# Analysis Steps Used to Generate National Estimates for Progress Categories, Summary Statements, and Results FFY2018

Multiple years of these data are available:

## Methods for computing the national estimates

The national estimates were computed using three methods:

- Computing the average of all states included, weighted by child count
- Computing the average of all states included, unweighted
- Computing the average of only states that met all data quality criteria, weighted by child count

### Process for method 1: Computing weighted national estimates for all states

- The first step was to compute a weighted numerator for each outcome and progress category for each state by multiplying the overall 2018-2019 child count for that state by the percentage of children reported in any given Office of Special Education Programs (OSEP) progress category in FFY2018.
Weighted numerator=child count×percentage of children reported in an OSEP progress categoryStateX_OC1a weighted numerator=StateX_OC1a×StateX child count gives the number of children in State X who would be in progress category "a" based on its child count
- Once the weighted numerators for each state were computed, we summed across states to create the weighted national numerators for each outcome and progress category.
- Next, the child counts across all states were summed to create the denominator for the national estimates.
- Then we computed the national weighted percentage for each progress category by dividing the national weighted numerator (Step 2) by the denominator (Step 3). This produced the national percentage for each progress category.
- Finally, we computed the national weighted summary statements based on the nationally weighted progress category percentages.

The results of method 1 are presented in Tables 3 and 4 for Part C and Tables 9 and 10 for Part B Preschool.

### Process for method 2: Computing the unweighted national estimates for all states

- The national unweighted estimates for progress categories were computed by taking the average across states within each progress category and outcome.
- The national unweighted estimates for the summary statements were computed by taking the average across states within each summary statement and outcome.

These results are presented in Tables 5 and 6 for Part C and Tables 11 and 12 for Part B Preschool.

### Process for method 3: Computing the weighted national estimates for the reduced set of states with high-quality data

The process used to compute the weighted national estimates for all states was repeated with the reduced set. The results for method 3 are presented in Tables 7 and 8 for Part C and Tables 13 and 14 for Part B Preschool.

## Method used to identify states with high-quality data

Two quality criteria were used to identify which states, of all 50 states plus the District of Columbia, have high-quality data:

- Data completeness, defined in Part C as a minimum of 28% of exiters included in the outcomes data and defined in Part B 619 as a minimum of 12% of the count of children ages 3–5 included in the outcomes data.
- Reasonable data patterning, defined in both Part C and Part B 619 as 10% or less of children in progress category "a" and 65% or less of children in progress category "e."

### Missing data

The first quality criterion was that states measured a large enough proportion of their population— that they did not exceed a previously established maximum of missing data. We eliminated states that were conducting sampling themselves because we had no metric for estimating the extent of missing data. (Sampling means selecting a sample of children from the population instead of measuring the whole population.) States were identified as conducting sampling based on the FFY 2015 Results Matrix.

For Part C, we established a proxy for missing data by using the number of children the state reported in the 618 Exiting Data Table (2017-18) as the denominator and the number of children included in the outcomes data as the numerator (as reported in the FFY2018 State Performance Plan and Annual Performance Report). We narrowed the list of states to those that reported outcomes data on at least 28% of children reported as exiting.

For Part B, we established a proxy for missing data by using the number of 3- to 5-year-old children the state reported in the Part B 618 Child Count Data Table (2018-2019) as the denominator and the number of children included in the outcomes data as the numerator. We included states that reported outcomes data on at least 12% of their child count for Section 619.

- 618 Exiting Data Table (2018-19) retrieved from https://www2.ed.gov/programs/osepidea/618-data/static-tables/2018-2019/part-c/child-count-and-settings/1819-cchildcountandsettings-1.xlsx
- Part B 618 Child Count Data Table (2018-2019) retrieved from https://www2.ed.gov/programs/osepidea/618-data/static-tables/2018-2019/part-b/child-count-and-educational-environment/1819-bchildcountandedenvironment-2.xlsx

### Examination of data patterns

The second data quality criterion was that states’ data had "reasonable" patterns. Because outliers in data patterns are often indicators of questionable data quality, we established criteria for reasonable parameters of the "a" and "e" progress category percentages.

Progress category "a" includes children with the most significant delays and degenerative conditions who do not make any progress or who regress from entry to exit. We established a cutoff of 10% of children in progress category "a" as reasonable based on historical patterns. We removed states from the pool that reported more than 10% in progress category "a" on one or more of the child outcomes.

Progress category "e" includes children who enter and exit at age expectations in the outcome area. This category is related to eligibility criteria; the percentage of children in category "e" will be lower for states with narrow eligibility categories and higher for states that serve a broader range of children. On the basis of historical data, we established a cutoff of 65% of children in progress category "e" as the reasonable limit. We removed states from the pool that reported more than 65% of children in progress category "e" on one or more of the child outcomes.

## Tables

- Tables 1 and 2 indicate the numbers of states excluded at each of these steps.
- Tables 3 and 4 present the weighted progress category and summary statement data for all Part C states.
- Tables 5 and 6 present the unweighted data for all Part C states.
- Tables 7 and 8 present weighted data for only the Part C states that met the quality criteria.
- Tables 9 and 10 present the weighted progress category and summary statement data for all Part B preschool states.
- Tables 11 and 12 present the unweighted data for all Part B preschool states.
- Tables 13 and 14 present weighted data for only the Part B preschool states that met the quality criteria.

### Table 1. Reasons for excluding Part C states

Reason | Number excluded |
---|---|

State is sampling | 2 |

Missing data (Reported outcomes data on less than 28% of exiters) | 2 |

"a" and "e" patterning (Had at least one outcome with category "a" greater than 10% or category "e" greater than 65%) | 0 |

States included in the analysis | 47 |

### Table 2. Reasons for excluding Part B 619 states

Reason | Number excluded |
---|---|

Sampling | 2 |

Missing data (Reported outcomes data on less than 12% of child count) | 1 |

"a" and "e" patterning (Had at least one outcome with category "a" greater than 10% or category "e" greater than 65%) | 5 |

Data determined not valid and reliable by OSEP | 1 |

No 3-5 Child Count data available for 2018-2019 | 1 |

State did not report outcomes data | 1 |

States included in the analysis | 40 |

### Table 3. Part C, all states included, weighted by child count: Percentages for OSEP progress categories (N=51)

Outcome 1 Social relationships | Percent |
---|---|

a: Children who did not improve functioning | 2.4 |

b: Children who improved functioning but not sufficiently to move nearer to functioning comparably to same-age peers | 22.1 |

c: Children who improved functioning to a level nearer to same-aged peers but did not reach it | 19.2 |

d: Children who improved functioning to reach a level comparable to same-aged peers | 26.9 |

e: Children who maintained functioning at a level comparable to same-aged peers | 29.4 |

Outcome 2 Knowledge and skills | Percent |

a: Children who did not improve functioning | 1.6 |

b: Children who improved functioning but not sufficiently to move nearer to functioning comparably to same-age peers | 20.5 |

c: Children who improved functioning to a level nearer to same-aged peers but did not reach it | 30.5 |

d: Children who improved functioning to reach a level comparable to same-aged peers | 32.8 |

e: Children who maintained functioning at a level comparable to same-aged peers | 14.5 |

Outcome 3 Action to meet needs | Percent |

a: Children who did not improve functioning | 2.0 |

b: Children who improved functioning but not sufficiently to move nearer to functioning comparably to same-age peers | 17.9 |

c: Children who improved functioning to a level nearer to same-aged peers but did not reach it | 23.2 |

d: Children who improved functioning to reach a level comparable to same-aged peers | 33.6 |

e: Children who maintained functioning at a level comparable to same-aged peers | 23.3 |

### Table 4. Part C, all states included, weighted by child count: Percentages for the OSEP summary statements (N=51)

Summary Statements | Outcome 1 | Outcome 2 | Outcome 3 |
---|---|---|---|

1. Of those children who entered the program below age expectations in [outcome], the percent that substantially increased their rate of growth in [outcome] by the time they exited. | 65 | 74 | 74 |

2. Percent of children who were functioning within age expectations in [outcome], by the time they exited. | 56 | 47 | 57 |

### Table 5. Part C, all states included, unweighted: Percentages for OSEP progress categories (N=51)

Outcome 1 Social relationships | Percent |
---|---|

a: Children who did not improve functioning | 1.5 |

23.5 | |

c: Children who improved functioning to a level nearer to same-aged peers but did not reach it | 29.1 |

d: Children who improved functioning to reach a level comparable to same-aged peers | 27.8 |

e: Children who maintained functioning at a level comparable to same-aged peers | 27.1 |

Outcome 2 Knowledge and skills | Percent |

a: Children who did not improve functioning | 1.2 |

24.2 | |

c: Children who improved functioning to a level nearer to same-aged peers but did not reach it | 28.6 |

d: Children who improved functioning to reach a level comparable to same-aged peers | 32.5 |

e: Children who maintained functioning at a level comparable to same-aged peers | 13.5 |

Outcome 3 Action to meet needs | Percent |

a: Children who did not improve functioning | 1.1 |

19.5 | |

c: Children who improved functioning to a level nearer to same-aged peers but did not reach it | 22.6 |

d: Children who improved functioning to reach a level comparable to same-aged peers | 35.1 |

e: Children who maintained functioning at a level comparable to same-aged peers | 21.7 |

### Table 6. Part C, all states included, unweighted: Percentages for the OSEP summary statements (N=51)

Summary Statements | Outcome 1 | Outcome 2 | Outcome 3 |
---|---|---|---|

1. Of those children who entered the program below age expectations in [outcome], the percent that substantially increased their rate of growth in [outcome] by the time they exited. | 64 | 71 | 74 |

2. Percent of children who were functioning within age expectations in [outcome], by the time they exited. | 55 | 46 | 57 |

### Table 7. Part C, states with high-quality data, weighted by child count: Percentages for OSEP progress categories (N=48)

Outcome 1 Social relationships | Percent |
---|---|

a: Children who did not improve functioning | 1.3 |

23.5 | |

c: Children who improved functioning to a level nearer to same-aged peers but did not reach it | 19.7 |

d: Children who improved functioning to reach a level comparable to same-aged peers | 26.8 |

e: Children who maintained functioning at a level comparable to same-aged peers | 28.6 |

Outcome 2 Knowledge and skills | Percent |

a: Children who did not improve functioning | 0.9 |

21.8 | |

c: Children who improved functioning to a level nearer to same-aged peers but did not reach it | 31.6 |

d: Children who improved functioning to reach a level comparable to same-aged peers | 33.4 |

e: Children who maintained functioning at a level comparable to same-aged peers | 12.4 |

Outcome 3 Action to meet needs | Percent |

a: Children who did not improve functioning | 0.8 |

17.9 | |

c: Children who improved functioning to a level nearer to same-aged peers but did not reach it | 24.0 |

d: Children who improved functioning to reach a level comparable to same-aged peers | 36.2 |

e: Children who maintained functioning at a level comparable to same-aged peers | 21.2 |

### Table 8. Part C, states with high-quality data, weighted by child count: Percentages for the OSEP summary statements (N=49)

Summary Statements | Outcome 1 | Outcome 2 | Outcome 3 |
---|---|---|---|

1. Of those children who entered the program below age expectations in [outcome], the percent that substantially increased their rate of growth in [outcome] by the time they exited. | 65 | 74 | 76 |

2. Percent of children who were functioning within age expectations in [outcome], by the time they exited. | 55 | 46 | 57 |

### Table 9. Part B preschool, all states included, weighted by child count: Percentages for OSEP progress categories (N=48)

Outcome 1 Social relationships | Percent |
---|---|

a: Children who did not improve functioning | 1.6 |

12.4 | |

c: Children who improved functioning to a level nearer to same-aged peers but did not reach it | 28.9 |

d: Children who improved functioning to reach a level comparable to same-aged peers | 31.4 |

e: Children who maintained functioning at a level comparable to same-aged peers | 25.7 |

Outcome 2 Knowledge and skills | Percent |

a: Children who did not improve functioning | 1.5 |

12.8 | |

c: Children who improved functioning to a level nearer to same-aged peers but did not reach it | 32.0 |

d: Children who improved functioning to reach a level comparable to same-aged peers | 34.0 |

e: Children who maintained functioning at a level comparable to same-aged peers | 19.7 |

Outcome 3 Action to meet needs | Percent |

a: Children who did not improve functioning | 1.7 |

11.9 | |

c: Children who improved functioning to a level nearer to same-aged peers but did not reach it | 24.6 |

d: Children who improved functioning to reach a level comparable to same-aged peers | 32.0 |

e: Children who maintained functioning at a level comparable to same-aged peers | 29.8 |

### Table 10. Part B preschool, all states included, weighted by child count: Percentages for the OSEP summary statements (N=48)

Summary Statements | Outcome 1 | Outcome 2 | Outcome 3 |
---|---|---|---|

81 | 82 | 81 | |

57 | 54 | 62 |

### Table 11. Part B preschool, all states included, unweighted: Percentages for OSEP progress categories (N=49)

Outcome 1 Social relationships | Percent |
---|---|

a: Children who did not improve functioning | 2.1 |

13.7 | |

c: Children who improved functioning to a level nearer to same-aged peers but did not reach it | 27.8 |

d: Children who improved functioning to reach a level comparable to same-aged peers | 31.9 |

e: Children who maintained functioning at a level comparable to same-aged peers | 24.5 |

Outcome 2 Knowledge and skills | Percent |

a: Children who did not improve functioning | 2.2 |

14.4 | |

c: Children who improved functioning to a level nearer to same-aged peers but did not reach it | 32.0 |

d: Children who improved functioning to reach a level comparable to same-aged peers | 35.1 |

e: Children who maintained functioning at a level comparable to same-aged peers | 16.3 |

Outcome 3 Action to meet needs | Percent |

a: Children who did not improve functioning | 2.3 |

13.8 | |

c: Children who improved functioning to a level nearer to same-aged peers but did not reach it | 22.8 |

d: Children who improved functioning to reach a level comparable to same-aged peers | 31.7 |

e: Children who maintained functioning at a level comparable to same-aged peers | 29.3 |

### Table 12. Part B preschool, all states included, unweighted: Percentages for the OSEP summary statements (N=49)

Summary Statements | Outcome 1 | Outcome 2 | Outcome 3 |
---|---|---|---|

78 | 80 | 76 | |

56 | 51 | 61 |

### Table 13. Part B preschool, states with high-quality data, weighted by child count: Percentages for OSEP progress categories (N=40)

Outcome 1 Social relationships | Percent |
---|---|

a: Children who did not improve functioning | 1.4 |

12.6 | |

c: Children who improved functioning to a level nearer to same-aged peers but did not reach it | 26.9 |

d: Children who improved functioning to reach a level comparable to same-aged peers | 31.1 |

e: Children who maintained functioning at a level comparable to same-aged peers | 28.0 |

Outcome 2 Knowledge and skills | Percent |

a: Children who did not improve functioning | 1.2 |

13.2 | |

c: Children who improved functioning to a level nearer to same-aged peers but did not reach it | 30.4 |

d: Children who improved functioning to reach a level comparable to same-aged peers | 33.9 |

e: Children who maintained functioning at a level comparable to same-aged peers | 21.2 |

Outcome 3 Action to meet needs | Percent |

a: Children who did not improve functioning | 1.3 |

11.9 | |

c: Children who improved functioning to a level nearer to same-aged peers but did not reach it | 22.6 |

d: Children who improved functioning to reach a level comparable to same-aged peers | 31.7 |

e: Children who maintained functioning at a level comparable to same-aged peers | 32.5 |

### Table 14. Part B preschool, states with high-quality data, weighted by child count: Percentages for the OSEP summary statements (N=40)

Summary Statements | Outcome 1 | Outcome 2 | Outcome 3 |
---|---|---|---|

82 | 82 | 81 | |

59 | 55 | 64 |

**Please cite as:**

Early Childhood Technical Assistance Center and Center for IDEA Early Childhood
Data Systems. (2020). *Analysis steps used to generate national estimates for child outcomes progress categories, summary statements, and results FFY2018*. Retrieved from https://ectacenter.org/eco/pages/analysissteps.asp

The content of this document was developed under a cooperative agreement, #H326P170001, and a grant, #H373Z120002, from the Office of Special Education Programs, U.S. Department of Education. However, the content does not necessarily represent the policy of the U.S. Department of Education, and you should not assume endorsement by the Federal Government. ECTA Center Project Officer, Julia Martin Eile, and DaSy Center Project Officers, Meredith Miceli and Richelle Davis.