Unproven Assumptions

Foundation Built Upon Unproven Assumptions

Part Three NEPC Report on the Chetty Testimony in Vergara

I have always maintained that, while standardized tests may be able to reveal what a student knows or doesn’t know, standardized tests can’t tell us anything about why a student knows or doesn’t know something.  That becomes the basis of today’s critique of Dr. Chetty’s research results and testimony in the case of Vergara v. California. 

Moshe Adler, in his report published by the National Education Policy Center, addresses this very concern.   He writes as follows:

“The report first assumes that it is possible to measure the quality of a teacher by the standardized test results of his or her students and develops a statistical technique to conduct such a measurement.  The result of the measurement is called teacher ‘value-added,’ and the report concludes with the finding that a higher value-added score for the teachers causes higher incomes in adulthood for his or her students”  (page 2)

The most definitive research on ‘value-added models’, or VAM, was done for the Gates’ Foundation and reported in its “MET” Report.  Unfortunately, the statistical analysis of the MET Report presumes the presence of “random student assignment”, a concept which implies that each teacher’s class is a mirror image of each other teacher’s class.  Yet the MET Report itself acknowledges that “random student assignment” was not maintained throughout the study’s life and that some groups included in the study never did use “random student assignment”.  That factor alone destroys the credibility of the MET Report, and with it any reliance by Chetty on the ability to use standardized test scores to identify effective teachers. 

Yet, relying upon VAM, Dr Chetty then proceeds to extrapolate an “increased lifetime earnings” projection for students of “effective teachers”.   His projection is that each standard deviation above the mean teacher effective score produces 1.34% greater earnings per year.  You read that correctly, 1.34% @ year or about $286.  Dr. Chetty neglects to report the range of standard deviations found in the VAM analysis, so we get no sense of how big this deviation might be.  If we assume a standard bell curve, 33% of teachers will be within the range of the mean to +1 standard deviation (and another 33% will be in the range of –1 standard deviation).  In other words, 66% of teachers would be in the +/- 1 range, leaving very few in the +/- 2 or more standard deviations.

Adler goes on to say:

“Relying on these assumptions and findings, the (Chetty) report suggests that low value-added teachers should be fired or otherwise removed and replaced by high value-added teachers.”  (page 2)

If I read this correctly, California Superior Court Judge Treu is taking away the tenure and due process rights of every teacher in California on the basis of hotly contested allegations of research results and testimony, claiming a student of an “ineffective teacher” (a concept never defined) might lose 1.34% of his or her lifetime earnings.