• Logo: DaSy
  • Logo: ECTA Center
  • Logo: IDC
  • Logo: NCSI
 

Evaluating Implementation Process and Outcomes

During Phase III, states will collect, analyze, report, and use evaluation data based on the methods and timelines outlined in the Phase II plan. These data will be used to track implementation progress, track progress toward achieving the SIMR, and revise the improvement and evaluation plans. States will report progress and outcome data as well as revisions to the plans to OSEP in the Phase III SSIPs due in April 2017.

Data should be used regularly to monitor the improvement process and revise the improvement plan, as needed.

Considerations

  • Align the evaluation plan with the theory of action and logic model.
  • Review intended outcomes and ensure that they remain related to and logically follow the improvement strategies and related improvement activities.
  • Review performance criteria and indicators and make adjustments as needed based on implementation.
  • Make adjustments in data collection strategies/sources as needed to better measure intended outcomes.
  • Analyze data to address critical evaluation questions.
  • Review and adjust resources, as needed, to conduct all components of the evaluation plan.
  • Review and adjust, as needed, the individuals involved in each stage of the evaluation plan (data collection activities, data analyses, etc.), including stakeholders.
  • Continue to use data to support and guide improvement strategies and implementation processes.

Tools and Resources: Evaluating Process and Outcomes

  • This document provides a list of recommended resources to support evaluation planning for program improvement efforts including the SSIP. Resources relevant to early intervention and preschool special education are included in the list, which will be updated as new and relevant resources become available.

    Source: Winer, A., Nelson, R., Kahn, L., Derrington, T., Davies-Mercier, E., Cochenour, M., and Copa, N. (2015). Recommended resources for planning to evaluate improvement efforts. Retrieved from http://ectacenter.org/~pdfs/topics/ssip/plan_eval_program_improvement.pdf

  • This guide describes key steps for developing a well thought out plan for evaluating an SSIP. The guide provides considerations for how to incorporate each step into an evaluation plan, as well as a series of worksheets that correspond to each step and can be used to facilitate the planning process. Preferred use of the guide, along with its corresponding worksheets, is by TA providers in partnership with state staff.

    Source: IDEA Data Center. (2015). A guide to SSIP evaluation planning. Retrieved from https://ideadata.org/resource-library/5697cca3140ba0ca5c8b4599/

  • This sample action plan template was designed by DaSy, ECTA, IDC, and NCSI to provide states with a suggested format and examples of potential content for their Phase II SSIP improvement and evaluation plan. States should feel free to adapt the template or use one that best meets their needs and communicates how they will implement and evaluate their SSIP in Phase III. This template is based on a logic model approach. It links activities and steps needed to implement the improvement strategies with intended outcomes and uses the activities and outcomes as the basis for the evaluation plan.

    Source: Early Childhood Technical Assistance Center). (2015). Sample SSIP action plan template. Retrieved from http://ectacenter.org/~docs/topics/ssip/ssip_improvement_plan_template.doc

  • This resource was designed by the National Center for Systemic Improvement (NCSI) to provide states with a sample approach and tool to plan and track measures of State Systemic Improvement Plan (SSIP) implementation. This resource will assist states in addressing the SSIP requirements laid out in the State Performance Plan/Annual Performance Report (SPP/APR) Part B and Part C Indicator Measurement Tables and the SSIP Phase II OSEP Guidance and Review Tool, which call for the evaluation of implementation as well as outcomes.

    Source: National Center for Systemic Improvement. (2016). Implementation evaluation matrix. Retrieved from http://ncsi.wested.org/wp-content/uploads/2016/03/Implementation_Evaluation_Matrix-1.docx

  • This national webinar was hosted by NCSI, ECTA, and DaSy for state Part B and Part C staff and focused on strategies for assessing the impact of SSIP infrastructure improvements. Representatives from two state departments of education and two state Part C programs participated in a “virtual state panel” and shared their experiences with implementing infrastructure changes as well as their approaches to assessing the impact of those changes on their SSIP improvement strategies and ultimately, their SIMR.

    Source: National Center for Systemic Improvement. (2016). Assessing impact of infrastructure improvements. Retrieved from https://vimeo.com/169687158

  • The presentation focuses on how to use high-quality data to support effective implementation. Information is included on the use of data for decision-making and improvement and the conditions under which high-quality data can make the most difference.

    Source: Blasé, K. (2015). Building implementation capacity: Data to drive change. Retrieved from https://ideadata.org/resource-library/55c8c10b140ba0a8218b4574/

  • This white paper focuses on factors that could lead Part C or Part B state agencies to propose changes in their SIMR baselines or targets. The paper addresses questions that state agency personnel should propose when establishing baselines and targets and considerations that may need to be addressed when revising targets.

    Source: Ruggiero, T. and Kahn, L. (2015). Considerations for Making Changes to SIMR Baseline and Targets. Retrieved from https://ideadata.org/resource-library/5682b8ab140ba0fb0f8b45a7/

  • This research brief, which is available from the Office of Planning, Research, and Evaluation at the U.S. Department of Health and Human Services, addresses the importance of incorporating quality measures into the implementation evaluation process. Examples are provided on how quality and quantity constructs are assessed and examined in relation to early care and education program outcomes.

    Downer J. and Yazejian, N. (2013). Measuring the quality and quantity of implementation in early childhood interventions. Retrieved from http://www.acf.hhs.gov/programs/opre/resource/measuring-the-quality-and-quantity-of-implementation-in-early-childhood

  • This brief offers examples of how quality and quantity constructs are assessed and examined in relation to early care and education program outcomes.

    Source: Smith, B. J., Fox, L., Dunlap, G., Strain, P., Trivette, C. M., Perez Binder, D., Bovey, T., McCullough, K., & Blase, K. (2015). Planning guide to statewide implementation, scale-up, and sustainability of recommended practices. Retrieved from http://ectacenter.org/~pdfs/implement_ebp/ECTA_RP_StateGuide_2-2015.pdf

  • This assessment tool is for home visiting program leadership teams to use in assessing their status in the critical elements of program-wide implementation.

    Source: Trivette, C. and Jones, A. (2015). Reaching potential through recommended practices (RP2): Benchmarks of quality for home-visiting programs. Retrieved from http://ectacenter.org/~pdfs/calls/2015/decrp-2015-02-11/Benchmarks_Home%20Visiting.pdf

  • This assessment tool is for preschool special education programs’ leadership teams to use in assessing their status in the critical elements of program-wide implementation.

    Source: Trivette, C. and Jones, A. (2015). Reaching potential through recommended practices (RP2): Benchmarks of quality for classroom-based programs. Retrieved from http://ectacenter.org/~pdfs/calls/2015/decrp-2015-02-11/Benchmarks_Home%20Visiting.pdf

  • These activities will support evaluation teams in designing and developing fidelity assessments. The Designing a Fidelity Assessment activity allows teams to identify, categorize, and discuss challenges to implementing a fidelity assessment.

    Source: National Implementation Research Network and State Implementation and Scaling-up of Evidence-based Practices Center. (2016). Designing a fidelity assessment. Retrieved from http://implementation.fpg.unc.edu/resources/activity-7-1-designing-fidelity-assessment

  • These activities will support evaluation teams in designing and developing fidelity assessments. Once the essential components or functions of the EBPs have been identified, the Developing a Fidelity Assessment activity will support teams in brainstorming fidelity assessments.

    Source: National Implementation Research Network and State Implementation and Scaling-up of Evidence-based Practices Center. (2016). Developing a fidelity assessment. Retrieved from http://implementation.fpg.unc.edu/resources/activity-7-2-fidelity-module-7-capstone-developing-fidelity-assessment

  • This website can support teams in evaluating and planning for coaching and training systems and implementing and assessing best practices.

    Source: National Implementation Research Network and State Implementation and Scaling-up of Evidence-based Practices Center. (2016). Resource library: Evaluation and planning tools. Drivers. Retrieved from http://implementation.fpg.unc.edu/resources/results/taxonomy%3A23%2C40

  • This reference tool includes a series of steps that are required to collect high quality data needed to evaluate SSIP implementation and outcomes.  These steps can help states collect high quality data regardless of whether or not they are in the process of planning for or engaged in data collection.

    Retrieved from http://ectacenter.org/~pdfs/topics/ssip/Data_Pathway.pdf

  • This document is designed to assist states develop or refine their SSIP performance indicators. It includes a worksheet with a series of questions based on S.M.A.R.T. criteria (Specific, Measurable, Achievable, Relevant and Timely) that will help states write performance indicators, providing the information needed for the SSIP, and will articulate a rationale for making changes to existing performance indicators and their corresponding intended outcome statements.

    Retrieved from http://ectacenter.org/~pdfs/topics/ssip/Refining_SMART_Performance_Indicators.pdf

  • This document is designed for states who are currently working to refine and refocus their short and/or long term intended SSIP outcomes. It includes a worksheet to support states in identifying the intended outcomes that are most critical to the success of their SSIP and refining the language of those outcomes to best align to the theory of action.

    Retrieved from http://ectacenter.org/~pdfs/topics/ssip/Refining_Intended_SSIP_Outcomes.pdf

  • Developed by ECTA and DaSy, this document is designed to illustrate how a state might summarize and report data gathered through the System Framework self-assessment process to document infrastructure improvements in their Phase III SSIP or other program improvement efforts. A template for reporting progress is provided along with an example of hypothetical state data.

    Retrieved from http://ectacenter.org/~pdfs/sysframe/Reporting_Infrastructure_Improvements_2017-03-03.pdf

Links on this site are verified monthly. This page content was last updated on 2016-08-29 AML

Content hosted by The Early Childhood Technical Assistance Center

  • CB 8040
  • Chapel Hill, NC 27599-8040
  • phone: 919.962.2001
  • fax: 919.966.7463
  • email: ectacenter@unc.edu

The contents of this guide were developed under cooperative agreement numbers #H326R140006 (DaSy), #H326P120002 (ECTA Center), #H373Y130002 (IDC) and #H326R140006 (NCSI) from the Office of Special Education Programs, U.S. Department of Education. Opinions expressed herein do not necessarily represent the policy of the US Department of Education, and you should not assume endorsement by the Federal Government.

Project Officers: Meredith Miceli & Richelle Davis(DaSy), Julia Martin Eile (ECTA Center), Richelle Davis & Meredith Miceli (IDC), and Perry Williams & Shedeh Hajghassemali (NCSI)

  • OSEP's TA&D Network:
  • IDEAs that Work