Improving Systems, Practices and Outcomes

Analysis Steps Used to Generate National Estimates for Child Outcomes Progress Categories, Summary Statements, and Results FFY2016

Methods for computing the national estimates

The national estimates were computed using three methods:

  1. Computing the average of all states included, weighted by child count
  2. Computing the average of all states included, unweighted
  3. Computing the average of only states that met all data quality criteria, weighted by child count

Process for method 1: Computing weighted national estimates for all states

  1. The first step was to compute a weighted numerator for each outcome and progress category for each state by multiplying the overall 2016-2017 child count for that state by the percentage of children reported in any given Office of Special Education Programs (OSEP) progress category in FFY2016.
    Weighted numerator
    =
    child count
    *
    percentage of children reported in an OSEP progress category
    Note that the weighted numerator is not the actual number of children in a progress category in a state but a calculated number that allows each state to carry a weight proportionate to its child count in the national percentages. For example:
    StateX_OC1_a_weighted numerator
    =
    StateX_OC1_a
    *
    StateX_child count gives the number of children in State X who would be in progress category "a" based on its child count
  2. Once the weighted numerators for each state were computed, we summed across states to create the weighted national numerators for each outcome and progress category.
  3. Next, the child counts across all states were summed to create the denominator for the national estimates.
  4. Then we computed the national weighted percentage for each progress category by dividing the national weighted numerator (Step 2) by the denominator (Step 3). This produced the national percentage for each progress category.
  5. Finally, we computed the national weighted summary statements based on the nationally weighted progress category percentages.

The results of method 1 are presented in Tables 3 and 4 for Part C and Tables 9 and 10 for Part B Preschool.

Process for method 2: Computing the unweighted national estimates for all states

  1. The national unweighted estimates for progress categories were computed by taking the average across states within each progress category and outcome.
  2. The national unweighted estimates for the summary statements were computed by taking the average across states within each summary statement and outcome.

These results are presented in Tables 5 and 6 for Part C and Tables 11 and 12 for Part B Preschool.

Process for method 3: Computing the weighted national estimates for the reduced set of states with high-quality data

The process used to compute the weighted national estimates for all states was repeated with the reduced set. The results for method 3 are presented in Tables 7 and 8 for Part C and Tables 13 and 14 for Part B Preschool.

Method used to identify states with high-quality data

Two quality criteria were used to identify which states, of all 50 states plus the District of Columbia, have high-quality data:

  1. Data completeness, defined in Part C as a minimum of 28% of exiters included in the outcomes data and defined in Part B 619 as a minimum of 12% of the count of children ages 3–5 included in the outcomes data.
  2. Reasonable data patterning, defined in both Part C and Part B 619 as 10% or less of children in progress category "a" and 65% or less of children in progress category "e."

Missing data

The first quality criterion was that states measured a large enough proportion of their population— that they did not exceed a previously established maximum of missing data. We eliminated states that were conducting sampling themselves because we had no metric for estimating the extent of missing data. (Sampling means selecting a sample of children from the population instead of measuring the whole population.) States were identified as conducting sampling based on the FFY 2015 Results Matrix.

For Part C, we established a proxy for missing data by using the number of children the state reported in the XLS: 618 Exiting Data Table (2015-16) as the denominator and the number of children included in the outcomes data as the numerator (as reported in the FFY2016 State Performance Plan and Annual Performance Report ). We narrowed the list of states to those that reported outcomes data on at least 28% of children reported as exiting.

For Part B, we established a proxy for missing data by using the number of 3- to 5-year-old children the state reported in the XLS: Part B 618 Child Count Data Table (2016-2017) as the denominator and the number of children included in the outcomes data as the numerator. We included states that reported outcomes data on at least 12% of their child count for Section 619.

  • 618 Exiting Data Table (2015-16) retrieved from https://www2.ed.gov/programs/osepidea/618-data/static-tables/2015-2016/part-c/exiting/1516-cexiting-1.xlsx
  • Part B 618 Child Count Data Table (2016-2017) retrieved from https://www2.ed.gov/programs/osepidea/618-data/static-tables/2016-2017/part-b/child-count-and-educational-environment/1617-bchildcountandedenvironment-1.xlsx

Examination of data patterns

The second data quality criterion was that states’ data had "reasonable" patterns. Because outliers in data patterns are often indicators of questionable data quality, we established criteria for reasonable parameters of the "a" and "e" progress category percentages.

Progress category "a" includes children with the most significant delays and degenerative conditions who do not make any progress or who regress from entry to exit. We established a cutoff of 10% of children in progress category "a" as reasonable based on historical patterns. We removed states from the pool that reported more than 10% in progress category "a" on one or more of the child outcomes.

Progress category "e" includes children who enter and exit at age expectations in the outcome area. This category is related to eligibility criteria; the percentage of children in category "e" will be lower for states with narrow eligibility categories and higher for states that serve a broader range of children. On the basis of historical data, we established a cutoff of 65% of children in progress category "e" as the reasonable limit. We removed states from the pool that reported more than 65% of children in progress category "e" on one or more of the child outcomes.

Tables

  • Tables 1 and 2 indicate the numbers of states excluded at each of these steps.
  • Tables 3 and 4 present the weighted progress category and summary statement data for all Part C states.
  • Tables 5 and 6 present the unweighted data for all Part C states.
  • Tables 7 and 8 present weighted data for only the Part C states that met the quality criteria.
  • Tables 9 and 10 present the weighted progress category and summary statement data for all Part B preschool states.
  • Tables 11 and 12 present the unweighted data for all Part B preschool states.
  • Tables 13 and 14 present weighted data for only the Part B preschool states that met the quality criteria.
Table 1. Reasons for excluding Part C states
Reason Number excluded
State is sampling 2
Missing data (Reported outcomes data on less than 28% of exiters) 1
"a" and "e" patterning (Had at least one outcome with category "a" greater than 10% or category "e" greater than 65%) 2
States included in the analysis 46
Table 2. Reasons for excluding Part B 619 states
Reason Number excluded
Sampling 3
Missing data (Reported outcomes data on less than 12% of child count) 1
"a" and "e" patterning (Had at least one outcome with category "a" greater than 10% or category "e" greater than 65%) 2
No 3-5 Child Count data available for 2016-2017 2
States included in the analysis 43
Table 3. Part C, all states included, weighted by child count: Percentages for OSEP progress categories (N = 51)
Outcome 1 Social relationships Percent
a: Children who did not improve functioning 2.7
b: Children who improved functioning but not sufficiently to move nearer to functioning comparably to same-age peers 22.3
c: Children who improved functioning to a level nearer to same-aged peers but did not reach it 16.7
d: Children who improved functioning to reach a level comparable to same-aged peers 26.8
e: Children who maintained functioning at a level comparable to same-aged peers 31.5
Outcome 2 Knowledge and skills Percent
a: Children who did not improve functioning 1.8
b: Children who improved functioning but not sufficiently to move nearer to functioning comparably to same-age peers 22.9
c: Children who improved functioning to a level nearer to same-aged peers but did not reach it 26.7
d: Children who improved functioning to reach a level comparable to same-aged peers 33.8
e: Children who maintained functioning at a level comparable to same-aged peers 14.8
Outcome 3 Action to meet needs Percent
a: Children who did not improve functioning 2.4
b: Children who improved functioning but not sufficiently to move nearer to functioning comparably to same-age peers 19.2
c: Children who improved functioning to a level nearer to same-aged peers but did not reach it 21.8
d: Children who improved functioning to reach a level comparable to same-aged peers 33.5
e: Children who maintained functioning at a level comparable to same-aged peers 23.1
Table 4. Part C, all states included, weighted by child count: Percentages for the OSEP summary statements (N = 51)
Summary Statements Outcome 1 Outcome 2 Outcome 3
1. Of those children who entered the program below age expectations in [outcome], the percent that substantially increased their rate of growth in [outcome] by the time they exited. 64 71 72
2. Percent of children who were functioning within age expectations in [outcome], by the time they exited. 58 49 57
Table 5. Part C, all states included, unweighted: Percentages for OSEP progress categories (N = 51)
Outcome 1 Social relationships Percent
a: Children who did not improve functioning 1.9
b: Children who improved functioning but not sufficiently to move nearer to functioning comparably to same-age peers 21.8
c: Children who improved functioning to a level nearer to same-aged peers but did not reach it 19.5
d: Children who improved functioning to reach a level comparable to same-aged peers 28.3
e: Children who maintained functioning at a level comparable to same-aged peers 28.5
Outcome 2 Knowledge and skills Percent
a: Children who did not improve functioning 1.5
b: Children who improved functioning but not sufficiently to move nearer to functioning comparably to same-age peers 23.0
c: Children who improved functioning to a level nearer to same-aged peers but did not reach it 27.6
d: Children who improved functioning to reach a level comparable to same-aged peers 33.7
e: Children who maintained functioning at a level comparable to same-aged peers 14.2
Outcome 3 Action to meet needs Percent
a: Children who did not improve functioning 1.7
b: Children who improved functioning but not sufficiently to move nearer to functioning comparably to same-age peers 19.0
c: Children who improved functioning to a level nearer to same-aged peers but did not reach it 22.3
d: Children who improved functioning to reach a level comparable to same-aged peers 35.6
e: Children who maintained functioning at a level comparable to same-aged peers 21.5
Table 6. Part C, all states included, unweighted: Percentages for the OSEP summary statements (N = 51)
Summary Statements Outcome 1 Outcome 2 Outcome 3
1. Of those children who entered the program below age expectations in [outcome], the percent that substantially increased their rate of growth in [outcome] by the time they exited. 65 71 73
2. Percent of children who were functioning within age expectations in [outcome], by the time they exited. 57 48 57
Table 7. Part C, states with high-quality data, weighted by child count: Percentages for OSEP progress categories (N = 46)
Outcome 1 Social relationships Percent
a: Children who did not improve functioning 1.1
b: Children who improved functioning but not sufficiently to move nearer to functioning comparably to same-age peers 22.5
c: Children who improved functioning to a level nearer to same-aged peers but did not reach it 18.7
d: Children who improved functioning to reach a level comparable to same-aged peers 28.5
e: Children who maintained functioning at a level comparable to same-aged peers 29.1
Outcome 2 Knowledge and skills Percent
a: Children who did not improve functioning 0.9
b: Children who improved functioning but not sufficiently to move nearer to functioning comparably to same-age peers 21.9
c: Children who improved functioning to a level nearer to same-aged peers but did not reach it 28.7
d: Children who improved functioning to reach a level comparable to same-aged peers 34.0
e: Children who maintained functioning at a level comparable to same-aged peers 14.5
Outcome 3 Action to meet needs Percent
a: Children who did not improve functioning 1.0
b: Children who improved functioning but not sufficiently to move nearer to functioning comparably to same-age peers 19.1
c: Children who improved functioning to a level nearer to same-aged peers but did not reach it 23.3
d: Children who improved functioning to reach a level comparable to same-aged peers 36.4
e: Children who maintained functioning at a level comparable to same-aged peers 20.4
Table 8. Part C, states with high-quality data, weighted by child count: Percentages for the OSEP summary statements (N = 46)
Summary Statements Outcome 1 Outcome 2 Outcome 3
1. Of those children who entered the program below age expectations in [outcome], the percent that substantially increased their rate of growth in [outcome] by the time they exited. 67 73 75
2. Percent of children who were functioning within age expectations in [outcome], by the time they exited. 58 49 57
Table 9. Part B preschool, all states included, weighted by child count: Percentages for OSEP progress categories (N = 51)
Outcome 1 Social relationships Percent
a: Children who did not improve functioning 2.0
b: Children who improved functioning but not sufficiently to move nearer to functioning comparably to same-age peers 11.8
c: Children who improved functioning to a level nearer to same-aged peers but did not reach it 27.6
d: Children who improved functioning to reach a level comparable to same-aged peers 32.1
e: Children who maintained functioning at a level comparable to same-aged peers 26.6
Outcome 2 Knowledge and skills Percent
a: Children who did not improve functioning 1.9
b: Children who improved functioning but not sufficiently to move nearer to functioning comparably to same-age peers 12.7
c: Children who improved functioning to a level nearer to same-aged peers but did not reach it 31.2
d: Children who improved functioning to reach a level comparable to same-aged peers 34.5
e: Children who maintained functioning at a level comparable to same-aged peers 19.8
Outcome 3 Action to meet needs Percent
a: Children who did not improve functioning 2.1
b: Children who improved functioning but not sufficiently to move nearer to functioning comparably to same-age peers 11.2
c: Children who improved functioning to a level nearer to same-aged peers but did not reach it 22.2
d: Children who improved functioning to reach a level comparable to same-aged peers 31.3
e: Children who maintained functioning at a level comparable to same-aged peers 33.1
Table 10. Part B preschool, all states included, weighted by child count: Percentages for the OSEP summary statements (N = 51)
Summary Statements Outcome 1 Outcome 2 Outcome 3
1. Of those children who entered the program below age expectations in [outcome], the percent that substantially increased their rate of growth in [outcome] by the time they exited. 81 82 80
2. Percent of children who were functioning within age expectations in [outcome], by the time they exited. 59 54 65
Table 11. Part B preschool, all states included, unweighted: Percentages for OSEP progress categories (N = 51)
Outcome 1 Social relationships Percent
a: Children who did not improve functioning 1.9
b: Children who improved functioning but not sufficiently to move nearer to functioning comparably to same-age peers 12.8
c: Children who improved functioning to a level nearer to same-aged peers but did not reach it 26.7
d: Children who improved functioning to reach a level comparable to same-aged peers 32.9
e: Children who maintained functioning at a level comparable to same-aged peers 25.8
Outcome 2 Knowledge and skills Percent
a: Children who did not improve functioning 1.8
b: Children who improved functioning but not sufficiently to move nearer to functioning comparably to same-age peers 14.1
c: Children who improved functioning to a level nearer to same-aged peers but did not reach it 31.3
d: Children who improved functioning to reach a level comparable to same-aged peers 36.0
e: Children who maintained functioning at a level comparable to same-aged peers 16.8
Outcome 3 Action to meet needs Percent
a: Children who did not improve functioning 1.7
b: Children who improved functioning but not sufficiently to move nearer to functioning comparably to same-age peers 11.6
c: Children who improved functioning to a level nearer to same-aged peers but did not reach it 20.3
d: Children who improved functioning to reach a level comparable to same-aged peers 32.6
e: Children who maintained functioning at a level comparable to same-aged peers 33.8
Table 12. Part B preschool, all states included, unweighted: Percentages for the OSEP summary statements (N = 51)
Summary Statements Outcome 1 Outcome 2 Outcome 3
1. Of those children who entered the program below age expectations in [outcome], the percent that substantially increased their rate of growth in [outcome] by the time they exited. 80 81 79
2. Percent of children who were functioning within age expectations in [outcome], by the time they exited. 59 53 66
Table 13. Part B preschool, states with high-quality data, weighted by child count: Percentages for OSEP progress categories (N = 43)
Outcome 1 Social relationships Percent
a: Children who did not improve functioning 2.1
b: Children who improved functioning but not sufficiently to move nearer to functioning comparably to same-age peers 12.3
c: Children who improved functioning to a level nearer to same-aged peers but did not reach it 26.1
d: Children who improved functioning to reach a level comparable to same-aged peers 32.3
e: Children who maintained functioning at a level comparable to same-aged peers 27.3
Outcome 2 Knowledge and skills Percent
a: Children who did not improve functioning 2.0
b: Children who improved functioning but not sufficiently to move nearer to functioning comparably to same-age peers 12.9
c: Children who improved functioning to a level nearer to same-aged peers but did not reach it 29.5
d: Children who improved functioning to reach a level comparable to same-aged peers 34.5
e: Children who maintained functioning at a level comparable to same-aged peers 21.0
Outcome 3 Action to meet needs Percent
a: Children who did not improve functioning 2.3
b: Children who improved functioning but not sufficiently to move nearer to functioning comparably to same-age peers 11.3
c: Children who improved functioning to a level nearer to same-aged peers but did not reach it 21.1
d: Children who improved functioning to reach a level comparable to same-aged peers 31.6
e: Children who maintained functioning at a level comparable to same-aged peers 33.8
Table 14. Part B preschool, states with high-quality data, weighted by child count: Percentages for the OSEP summary statements (N = 43)
Summary Statements Outcome 1 Outcome 2 Outcome 3
1. Of those children who entered the program below age expectations in [outcome], the percent that substantially increased their rate of growth in [outcome] by the time they exited. 80 81 80
2. Percent of children who were functioning within age expectations in [outcome], by the time they exited. 60 56 65

Please cite as:

Early Childhood Technical Assistance Center & Center for IDEA Early Childhood Data Systems. (2018). Analysis steps used to generate national estimates for child outcomes progress categories, summary statements, and results FFY2016. Retrieved from http://ectacenter.org/eco/pages/analysissteps.asp

The contents of this document were developed under a grant from the U.S. Department of Education, #H373Z120002, and a cooperative agreement, #H326P170001, from the Office of Special Education Programs, U.S. Department of Education. However, those contents do not necessarily represent the policy of the U.S. Department of Education, and you should not assume endorsement by the Federal Government. DaSy Center Project Officers, Meredith Miceli and Richelle Davis and ECTA Center Project Officer, Julia Martin Eile.

  • IDEAs that Work: Office of Special Education Programs, U.S. Department of Education

The ECTA Center is a program of the FPG Child Development Institute of the University of North Carolina at Chapel Hill, funded through cooperative agreement number H326P170001 from the Office of Special Education Programs, U.S. Department of Education. Opinions expressed herein do not necessarily represent the Department of Education's position or policy.

Project Officer: Julia Martin Eile     © 2012-2018 ECTA Center

  • UNC Frank Porter Graham Child Development Institute