Analysis Steps Used to Generate National Estimates for Progress Categories, Summary Statements, and Results FFY 2017

Methods for computing the national estimates

The national estimates were computed using three methods:

1. Computing the average of all states included, weighted by child count
2. Computing the average of all states included, unweighted
3. Computing the average of only states that met all data quality criteria, weighted by child count

Process for method 1: Computing weighted national estimates for all states

1. The first step was to compute a weighted numerator for each outcome and progress category for each state by multiplying the overall 2017–2018 child count for that state by the percentage of children reported in any given Office of Special Education Programs (OSEP) progress category in FFY 2017.
   Weighted numerator

   =

   child count

   ×

   percentage of children reported in an OSEP progress category
   Note that the weighted numerator is not the actual number of children in a progress category in a state but a calculated number that allows each state to carry a weight proportionate to its child count in the national percentages. For example:
   StateX_OC1a weighted numerator

   =

   StateX_OC1a

   ×

   StateX child count gives the number of children in State X who would be in progress category "a" based on its child count
2. Once the weighted numerators for each state were computed, we summed across states to create the weighted national numerators for each outcome and progress category.
3. Next, the child counts across all states were summed to create the denominator for the national estimates.
4. Then we computed the national weighted percentage for each progress category by dividing the national weighted numerator (Step 2) by the denominator (Step 3). This produced the national percentage for each progress category.
5. Finally, we computed the national weighted summary statements based on the nationally weighted progress category percentages.

The results of method 1 are presented in Tables 3 and 4 for Part C and Tables 9 and 10 for Part B Preschool.

Process for method 2: Computing the unweighted national estimates for all states

1. The national unweighted estimates for progress categories were computed by taking the average across states within each progress category and outcome.
2. The national unweighted estimates for the summary statements were computed by taking the average across states within each summary statement and outcome.

These results are presented in Tables 5 and 6 for Part C and Tables 11 and 12 for Part B Preschool.

Process for method 3: Computing the weighted national estimates for the reduced set of states with high-quality data

The process used to compute the weighted national estimates for all states was repeated with the reduced set. The results for method 3 are presented in Tables 7 and 8 for Part C and Tables 13 and 14 for Part B Preschool.

Method used to identify states with high-quality data

Two quality criteria were used to identify which states, of all 50 states plus the District of Columbia, have high-quality data:

1. Data completeness, defined in Part C as a minimum of 28% of exiters included in the outcomes data and defined in Part B 619 as a minimum of 12% of the count of children ages 3–5 included in the outcomes data.
2. Reasonable data patterning, defined in both Part C and Part B 619 as 10% or less of children in progress category "a" and 65% or less of children in progress category "e."

Missing data

The first quality criterion was that states measured a large enough proportion of their population— that they did not exceed a previously established maximum of missing data. We eliminated states that were conducting sampling themselves because we had no metric for estimating the extent of missing data. (Sampling means selecting a sample of children from the population instead of measuring the whole population.) States were identified as conducting sampling based on the FFY 2015 Results Matrix.

For Part C, we established a proxy for missing data by using the number of children the state reported in the 618 Exiting Data Table (2016-17) as the denominator and the number of children included in the outcomes data as the numerator (as reported in the FFY 2017 State Performance Plan and Annual Performance Report ). We narrowed the list of states to those that reported outcomes data on at least 28% of children reported as exiting.

For Part B, we established a proxy for missing data by using the number of 3- to 5-year-old children the state reported in the Part B 618 Child Count Data Table (2017–2018) as the denominator and the number of children included in the outcomes data as the numerator. We included states that reported outcomes data on at least 12% of their child count for Section 619.

• 618 Exiting Data Table (2017-18) retrieved from https://data.ed.gov/dataset/b57aaff7-4aab-4ec9-b565-055f0d291b4a/resource/1fd212b5-3d95-40a5-9e4e-9c2dc5d1ed93/download/1718-cchildcountandsettings-1.xlsx
• Part B 618 Child Count Data Table (2017–2018) retrieved from https://data.ed.gov/dataset/614c7388-64d7-444a-9f16-0180257f7c01/resource/872a4493-c2ef-40ce-9ceb-9348ddd2f9eb/download/1718-bchildcountandedenvironment-2.xlsx

Examination of data patterns

The second data quality criterion was that states' data had "reasonable" patterns. Because outliers in data patterns are often indicators of questionable data quality, we established criteria for reasonable parameters of the "a" and "e" progress category percentages.

Progress category "a" includes children with the most significant delays and degenerative conditions who do not make any progress or who regress from entry to exit. We established a cutoff of 10% of children in progress category "a" as reasonable based on historical patterns. We removed states from the pool that reported more than 10% in progress category "a" on one or more of the child outcomes.

Progress category "e" includes children who enter and exit at age expectations in the outcome area. This category is related to eligibility criteria; the percentage of children in category "e" will be lower for states with narrow eligibility categories and higher for states that serve a broader range of children. On the basis of historical data, we established a cutoff of 65% of children in progress category "e" as the reasonable limit. We removed states from the pool that reported more than 65% of children in progress category "e" on one or more of the child outcomes.

Tables

• Tables 1 and 2 indicate the numbers of states excluded at each of these steps.
• Tables 3 and 4 present the weighted progress category and summary statement data for all Part C states.
• Tables 5 and 6 present the unweighted data for all Part C states.
• Tables 7 and 8 present weighted data for only the Part C states that met the quality criteria.
• Tables 9 and 10 present the weighted progress category and summary statement data for all Part B preschool states.
• Tables 11 and 12 present the unweighted data for all Part B preschool states.
• Tables 13 and 14 present weighted data for only the Part B preschool states that met the quality criteria.

Table 1. Reasons for excluding Part C states

Table 2. Reasons for excluding Part B 619 states

Table 3. Part C, all states included, weighted by child count: Percentages for OSEP progress categories (N=51)

Table 4. Part C, all states included, weighted by child count: Percentages for the OSEP summary statements (N=51)

Table 5. Part C, all states included, unweighted: Percentages for OSEP progress categories (N=51)

Table 6. Part C, all states included, unweighted: Percentages for the OSEP summary statements (N=51)

Table 7. Part C, states with high-quality data, weighted by child count: Percentages for OSEP progress categories (N=48)

Table 8. Part C, states with high-quality data, weighted by child count: Percentages for the OSEP summary statements (N=49)

Table 9. Part B preschool, all states included, weighted by child count: Percentages for OSEP progress categories (N=51)

Table 10. Part B preschool, all states included, weighted by child count: Percentages for the OSEP summary statements (N=51)

Table 11. Part B preschool, all states included, unweighted: Percentages for OSEP progress categories (N=51)

Table 12. Part B preschool, all states included, unweighted: Percentages for the OSEP summary statements (N=51)

Table 13. Part B preschool, states with high-quality data, weighted by child count: Percentages for OSEP progress categories (N=45)

Table 14. Part B preschool, states with high-quality data, weighted by child count: Percentages for the OSEP summary statements (N=43)