Recommended Resources for Evaluating Program Improvement Efforts (including the SSIP)

This guide was developed by the IDEA Data Center (IDC) Evaluation Workgroup for use by IDC technical assistance providers when assisting state staff in planning State Systemic Improvement Plan (SSIP) evaluations. It identifies steps and considerations for developing a high-quality SSIP evaluation plan based on OSEP’s SSIP Phase II Guidance and Review Tools. It is not intended to provide standalone guidance to states, but some state staff and non-IDC TA providers may find it useful to consult. See the guide for more information on this and on accessing TA.

Source: Nimkoff, T., Fiore, T., and Edwards, J. (January 2016). A Guide to SSIP Evaluation Planning. IDEA Data Center. Rockville, MD: Westat. Retrieved from https://ideadata.org/sites/default/files/media/documents/2021-01/2-A-Guide_to_SSIP_Evaluation_Planning.pdf

This guide was developed by the Office of Special Education Programs (OSEP) for states’ use in reviewing and further developing SSIP evaluation plans for Phase III submissions. The elements included in this tool are derived from OSEP’s indicator measurement tables and Phase II review tool. The questions for consideration included for each element will assist States as they communicate the results of their SSIP implementation activities to stakeholders and organize the Phase III SSIP submission due to OSEP on April 3, 2017.

Source: Office of Special Education Programs (2016). SSIP Evaluation Plan Guidance Tool. US Department of Education. Washington, DC. Retrieved from https://ectacenter.org/~pdfs/grads360/12904.pdf

This manual is written for program managers and contains tips, worksheets, and samples to provide information to help you understand each step of the evaluation process.

Source: Administration for Children and Families, Office of Planning, Research and Evaluation (ACF OPRE) (2010). Retrieved from https://files.eric.ed.gov/fulltext/ED566135.pdf

This guidebook is written for the person who has never done an evaluation before. It provides step-by-step instructions on how to design and carry out an evaluation. It could also be used as a reference by people interested in certain phases of the evaluation process, such as writing performance indicators or designing a survey. The guidebook details how to:

1. develop outcome statements, indicators, and evaluation questions;
2. formulate an evaluation methodology and collect, assess, and summarize data, and
3. develop and disseminate evaluation findings and recommendations.

The guidebook also contains a glossary, sample outcome and indicator statements, evaluation resources, and real-life stories of how community organizations used evaluation tools. This resource is lengthy, but full of useful explanations of evaluation concepts and has some really wonderful worksheets and resources that can be used by states for the SSIP and also for TA providers to use with states during sessions and workshops.

Source: SRI International (supported by the Sierra Health Foundation) (2000). Retrieved from https://www.yumpu.com/en/document/view/50107005/we-did-it-ourselves-sierra-health-foundation

This manual is written for community-based organizations and provides a practical guide to program evaluation. It focuses on internal evaluation conducted by program staff, which will be useful for States planning on conducting their SSIP evaluation internally. The manual provides a nice overview of the evaluation process and includes the basic steps of planning for and conducting internal program evaluation, including practical strategies for identifying quantitative and qualitative data.

Source: Bond, S.L., Boyd, S.E., Rapp, K.A., Raphael, J.B. and Sizemore, B.A. (1997). Horizon Research. Retrieved from https://www.dcjs.virginia.gov/sites/dcjs.virginia.gov/files/publications/victims/takingstock.pdf

Directed at state staff engaged in building an early childhood system, it offers a framework for evaluating systems initiatives that connects the diverse elements involved in systems change. The executive summary includes general principles for evaluating systems initiatives and figures that support states in developing a theory of change and making decisions around evaluation planning.

Source: Coffman, Julia. (2007). BUILD Initiative. Framework for Evaluating Systems Initiatives (executive summary). Retrieved from https://buildinitiative.org/resource-library/a-framework-for-evaluating-system-initiatives/

This brief is written for early childhood researchers, program developers, and funders and seeks to introduce the importance of measuring implementation at multiple system levels and proposes tools for doing so, including a cascading logic model that makes connections between the outcomes and resources of different systems. The brief uses two illustrative examples:

1. a state's effort to improve the quality of infant-toddler child care and
2. a state's effort to improve child and family outcomes through the expansion of home visiting.

Source: Berghout, Anne M., Lokteff, Maegan, Paulsell, Diane. (2013). OPRE. Retrieved from https://www.acf.hhs.gov/sites/default/files/documents/opre/levels_brief_final_002.pdf

This workbook is written for public health program managers, administrators, and evaluators to support their construction of an effective evaluation report. It encourages a shared understanding of what constitutes a final evaluation report for people with all levels of evaluation experience. The workbook outlines six steps for report development and provides worksheets and examples for each step. It offers exercises on outlining a report, communicating results, and stakeholder inclusion. It is especially geared toward stakeholder participation in reporting which will help states as they consider how to include stakeholders in their SSIP.

Source: Centers for Disease Control and Prevention, National Center for Chronic Disease Prevention and Health Promotion, Office on Smoking and Health, Division of Nutrition, Physical Activity and Obesity. (2013). Developing an effective evaluation report: Setting the course for effective program evaluation. Retrieved from https://www.cdc.gov/eval/materials/developing-an-effective-evaluation-report_tag508.pdf

This guide is written for a broad audience to clarify the what, why, and how of logic models: what logic models are and the different types; why a program should develop a logic model; and how a logic model can be used to guide implementation and plan for evaluation. The guide includes templates and checklists that states can use to apply it to their specific SSIP. It provides useful explanations and definitions of evaluation terminology.

Source: W.K. Kellogg Foundation (Updated 2004). Retrieved from https://wkkf.issuelab.org/resource/logic-model-development-guide.html

This presentation is geared toward helping practitioners develop useful logic models. The slides address the hallmarks of well-constructed, useful logic models and how to use logic model for program evaluation planning, implementation, and using the findings. The slides do a nice job of breaking down and explaining the different components and terminology associated with a logic model. Additional resources and sites are also included. States can use this presentation as an introduction to logic models and as guidance for key components to create a meaningful logic model for their SSIP evaluation.

Source: Honeycutt, S. and Kegler, M. C. (2010). Honeycutt, S. and Kegler, M. C. AEA/CDC 2010 Summer Evaluation Institute. Retrieved from http://comm.eval.org/viewdocument/?DocumentKey=79c7da3d-6978-452c-8e8b-7056d3626966