Analysis Steps Used to Generate National Estimates for Progress Categories, Summary Statements, and Results FFY 2021

Methods for computing the national estimates

The national estimates were computed using three methods:

1. Computing weighted national estimates for all states
2. Computing the unweighted national estimates for all states
3. Computing the weighted national estimates for the reduced set of states with high-quality data

Process for computing weighted national estimates for all states

1. The first step was to compute a weighted numerator for each outcome and progress category for each state by multiplying the overall 2021–2022 child count from the The Part C Child Count and Settings and overall 2021–2022 child count from The Part B Child Count and Educational Environments for that state by the percentage of children reported in any given Office of Special Education Programs (OSEP) progress category in FFY 2021.
   Weighted numerator

   =

   percentage of children reported in an OSEP progress category

   ×

   child count
   Note that the weighted numerator is not the actual number of children in a progress category in a state but a calculated number that allows each state to carry a weight proportionate to its child count in the national percentages.
2. Once the weighted numerators for each state were computed, we summed across states to create the weighted national numerators for each outcome and progress category.
3. Next, the child counts across all states were summed to create the denominator for the national estimates.
4. Then we computed the national weighted percentage for each progress category by dividing the national weighted numerator (Step 2) by the denominator (Step 3). This produced the national percentage for each progress category.
5. Finally, we computed the national weighted summary statements based on the nationally weighted progress category percentages.

The results are presented in Table 3 and Table 4 for Part C and Table 9 and Table 10 for Part B Preschool.

Process for computing the unweighted national estimates for all states

1. The national unweighted estimates for progress categories were computed by taking the average across states within each progress category and outcome.
2. The national unweighted estimates for the summary statements were computed by taking the average across states within each summary statement and outcome.

These results are presented in Table 5 and Table 6 for Part C and Table 11 and Table 12 for Part B Preschool.

Process for computing the weighted national estimates for the reduced set of states with high-quality data

The process used to compute the weighted national estimates for all states was repeated with the reduced set. The results are presented in Table 7 and Table 8 for Part C and Table 13 and Table 14 for Part B Preschool.

Method used to identify states with high-quality data

Two quality criteria were used to identify which states, of all 50 states plus the District of Columbia, have high-quality data:

1. Data completeness, defined in Part C as a minimum of 28% of exiters included in the outcomes data and defined in Part B, Section 619 as a minimum of 12% of the count of children ages 3–5 included in the outcomes data.
2. Reasonable data patterning, defined in both Part C and Part B, Section 619 as 10% or less of children in progress category "a" and 65% or less of children in progress category "e."

Data completeness

The first quality criterion was that states measured a large enough proportion of their population. We eliminated states that were conducting sampling themselves because we had no metric for estimating the extent of missing data. Sampling means selecting a sample of children from the population instead of measuring the whole population. States were identified as conducting sampling based on state report in the FFY 2021 APR.

For Part C, we established a proxy for missing data by using the number of children the state reported in the 618 Exiting Data Table (2020–2021) as the denominator and the number of children included in the outcomes data as the numerator (as reported in the FFY 2021 State Performance Plan and Annual Performance Report). We removed states that reported outcomes data on fewer than 28% of children reported as exiting.

For Part B, we established a proxy for missing data by using the number of children ages 3–5 not in kindergarten the state reported in the Part B 618 Child Count Data Table (2021–2022) as the denominator and the number of children included in the outcomes data as the numerator. We removed states that reported outcomes data on less than 12% of their child count for Part B, Section 619.

There were Part B, Section 619 state programs that also reported outcomes data on more than 100% of their child count. This is an unexpected pattern, and these states were excluded from the analysis.

• 618 Exiting Data Table (2019–2020) retrieved from https://data.ed.gov/dataset/idea-section-618-data-products-static-tables-part-c-exiting-table-1/resources
• Part B 618 Child Count Data Table (2021–2022) retrieved from https://data.ed.gov/dataset/idea-section-618-data-products-static-tables-part-b-count-environ-table1/resources

Reasonable data patterning

The second data quality criterion was that states' data had "reasonable" patterns. Because outliers in data patterns are often indicators of questionable data quality, we established criteria for reasonable parameters of the "a" and "e" progress category percentages.

Progress category "a" includes children with the most significant delays and degenerative conditions who do not make any progress or who regress from entry to exit. We established a cutoff of 10% of children in progress category "a" as reasonable based on historical patterns. We removed states from the analysis that reported more than 10% in progress category "a" on one or more of the child outcomes.

Progress category "e" includes children who enter and exit at age expectations in the outcome area. This category is related to eligibility criteria; the percentage of children in category "e" will be lower for states with narrow eligibility categories and higher for states that serve a broader range of children. On the basis of historical data, we established a cutoff of 65% of children in progress category "e" as the reasonable limit. We removed states from the analysis that reported more than 65% of children in progress category "e" on one or more of the child outcomes.

Tables

• Table 1 and Table 2 indicate the numbers of states excluded at each of these steps.
• Table 3 and Table 4 present the weighted progress category and summary statement data for all Part C states.
• Table 5 and Table 6 progress category and summary statement data for all Part C states.
• Table 7 and Table 8 present weighted progress category and summary statement data for only the Part C states that met the quality criteria.
• Table 9 and Table 10 present the weighted progress category and summary statement data for all Part B preschool states.
• Table 11 and Table 12 present the unweighted progress category and summary statement data for all Part B preschool states.
• Table 13 and Table 14 present weighted progress category and summary statement data for only the Part B preschool states that met the quality criteria.

In Table 1 and Table 2, each state is only counted once. If the reason that a state's data were excluded from analysis is a combination of reasons, it is listed below with an "AND ALSO" between reasons. States that are sampling are only counted under "state is sampling", and never under another reason.

Table 1. Reasons for excluding Part C states

Table 2. Reasons for excluding Part B, Section 619 states

Table 3. Part C, all states included, weighted by child count: Percentages for OSEP progress categories (N=51)

Table 4. Part C, all states included, weighted by child count: Percentages for the OSEP summary statements (N=51)

Table 5. Part C, all states included, unweighted: Percentages for OSEP progress categories (N=51)

Table 6. Part C, all states included, unweighted: Percentages for the OSEP summary statements (N=51)

Table 7. Part C, states with high-quality data, weighted by child count: Percentages for OSEP progress categories (N=46)

Table 8. Part C, states with high-quality data, weighted by child count: Percentages for the OSEP summary statements (N=46)

Table 9. Part B preschool, all states included, weighted by child count: Percentages for OSEP progress categories (N=51)

Table 10. Part B preschool, all states included, weighted by child count: Percentages for the OSEP summary statements (N=51)

Table 11. Part B preschool, all states included, unweighted: Percentages for OSEP progress categories (N=51)

Table 12. Part B preschool, all states included, unweighted: Percentages for the OSEP summary statements (N=51)

Table 13. Part B preschool, states with high-quality data, weighted by child count: Percentages for OSEP progress categories (N=40)

Table 14. Part B preschool, states with high-quality data, weighted by child count: Percentages for the OSEP summary statements (N=40)

Please cite as: Early Childhood Technical Assistance Center, Center for IDEA Early Childhood Data Systems (2023). Analysis steps used to generate national estimates for child outcomes progress categories, summary statements, and results FFY 2021. Retrieved from https://ectacenter.org/eco/pages/analysissteps.asp

The content of this document was developed under a cooperative agreement, #H326P220002, and a grant, #H373Z190002, from the Office of Special Education Programs, U.S. Department of Education. However, the content does not necessarily represent the policy of the U.S. Department of Education, and you should not assume endorsement by the Federal Government. ECTA Center Project Officer, Julia Martin Eile, and DaSy Center Project Officers, Meredith Miceli and Richelle Davis.