Digital Inclusion Outcomes-Based Evaluation
In recent years, government agencies, private foundations, and community-based organizations have increasingly sought to understand how programs that promote digital inclusion lead to social and economic outcomes for individuals, programs, and communities. This push to measure outcomes has been driven, in part, by a larger trend to ensure that dollars are being used efficiently to improve lives rather than simply to deliver services. Digital Inclusion Outcomes-Based Evaluation, published by Benton Foundation, describes the challenges facing community-based organizations and other key stakeholders in using outcomes-based evaluation to measure the success of their digital inclusion programs and offers recommendations toward addressing these shared barriers.
The report is written by Dr. Colin Rhinesmith and Angela Siefer. Dr. Rhinesmith conducts original research for Benton and advises the foundation on new research opportunities. Rhinesmith is an assistant professor in the School of Library and Information Science at Simmons College and a faculty associate with the Berkman Klein Center for Internet & Society at Harvard University. Siefer is the Director of the National Digital Inclusion Alliance (NDIA). Angela envisions a world in which all members of society have the skills and resources to use the Internet for the betterment of themselves and their communities. Since 1997, Angela has worked on digital inclusion issues with local community organizations, the National Telecommunications and Information Administration, state governments, and the Schools, Health & Libraries Broadband (SHLB) Coalition.
In workshop discussions with practitioners, funders, policymakers, and other key digital inclusion stakeholders, the authors identified the core outcomes-based evaluation needs of the digital inclusion field:
- Shared vocabulary. One of the most pressing issues that emerged from the workshops is the lack of shared vocabulary across the field to define “digital inclusion” and “outcomes-based evaluation.” While most workshop participants articulated what these terms mean to their own work, it was more difficult for them to say how these terms are being implemented across the field.
- Outcomes-based evaluation tools and implementation support. Workshop participants expressed their concern with not knowing where to find and how to use the tools needed to conduct outcomes-based evaluation of digital inclusion programs. Many participants also indicated that staff within organizations would need training to take advantage of the most appropriate, and culturally-relevant, outcomes-based evaluation tools in ways that reflect the specific needs and aspirations of their communities.
- Need for common indicators. Participants explained they lacked access to common indicators, the measures programs select as markers for their success. Participants in our workshops often expressed concerns about what and how to measure these changes in ways that were also meaningful to the communities they serve.
- Time and resources. While most participants understood the value of outcomes-based evaluation, particularly how it could help them show the impact of their digital inclusion initiatives, few could dedicate the time and resources to support this work.
Outcomes-based evaluation can be helpful for funders of digital inclusion programs, policymakers, and for the organizations that offer digital inclusion programs. Outcomes-based evaluation can be useful in the program planning and implementation phases, as well as for measuring program outcomes over time. Therefore, this methodology is often iterative and requires organizations to engage in a continuous process of data gathering and analysis.
This new research builds off Dr. Colin Rhinesmith’s Digital Inclusion and Meaningful Broadband Adoption Initiatives, released in early 2016. That report identified the core offerings of digital inclusion organizations – from providing low-cost broadband, and the devices to connect to it, while helping new broadband adopters gain the skills they need to navigate the Internet and online services. In this national study of digital inclusion organizations, Dr. Rhinesmith also noted that most of the digital inclusion organizations that participated in this study did not have outcomes-based evaluation frameworks. However, all recognized the importance of having them.