This review of the "Leaders and Laggards" report was written by Bob Williams independently, not as part of the Think Tank Review Project. It is republished here with permission of the author. Williams focuses on the aspects of the report concerning alternative teacher certification and teacher tenure. He notes that the report's authors appear to begin with premise that "improving education requires weakening teacher tenure and union influence while supporting alternative certification and national programs to place inexperienced people ... into teaching positions with minimal training." Williams explains in this commentary how the report distorts the data in order to create state-by-state ratings that fit the authors' pre-determined agenda.

This review was written by Kevin Welner independently, not as part of the Think Tank Review Project. In addition to the free pdf download at the link above, it is available to subscribers of the journal "Teachers College Record", at http://www.tcrecord.org/Content.asp?ContentID=15872

Welner’s review describes how the Loveless report combines weak data with questionable analyses to manufacture an argument against detracking. Better treatment of these same data would, in fact, likely show that high-achieving Massachusetts middle school students in heterogeneous, untracked classrooms do as well or better than those in tracked classrooms – certainly in language arts (English) and maybe even in mathematics. He concludes that the report misleads in an attempt to convince policymakers to maintain tracking policies.

The report examines whether increasing competition from charter schools has a causal effect on the achievement of public school students in New York City, using a three-year longitudinal database of student test scores. As a measure of competition, it considers the percentage of students who left a public school for a charter school in the prior year. The statistical analysis suggests that increasing competition has no statistically significant impact on math test scores, but that it has small positive effects on language scores. The report does not conclusively demonstrate that the results are explained by increasing competitive pressure on public school administrators; they may also be explained by shifting peer quality or declining short-run class sizes in public schools.

Suggested Cite:

McEwan, P. J. (2009). Review of “Everyone Wins: How Charter Schools Benefit All New York City Public School Students.” Boulder and Tempe: Education and the Public Interest Center & Education Policy Research Unit. Retrieved [date] from http://epicpolicy.org/thinktank/review-Everyone-Wins.

"How New York City's Charter Schools Affect Achievement" estimates the effects on student achievement of attending a New York City charter school rather than a traditional public school and investigates the characteristics of charter schools associated with the most positive effects on achievement. Because the report relies on an inappropriate set of statistical models to analyze the data, however, the results presented appear to overstate the cumulative effect of attending a charter school. In addition, the report does not provide enough technical discussion and detailed description to enable a reader to assess the validity of some aspects of the report’s methodology and results. Policymakers, educators, and parents, therefore, should not rely on these estimates until the authors provide more technical detail and the analysis has undergone rigorous peer review.

Suggested Cite:

Reardon, S.F. (2009). Review of “How New York City’s Charter Schools Affect Achievement.” Boulder and Tempe: Education and the Public Interest Center & Education Policy Research Unit. Retrieved [date] from http://epicpolicy.org/thinktank/review-How-New-York-City-Charter

The Widget Effect: Our National Failure to Acknowledge and Act on Teacher Differences, published in June 2009 by the New Teacher Project, examines how 12 school districts across four states use teacher evaluation to make human resources decisions. It then proposes how to build teacher evaluation systems that are more credible and useful. Overall, the report portrays current practices in teacher evaluation as a broken system perpetuated by a culture that refuses to recognize and deal with incompetence and that fails to reward excellence. However, omissions in the report’s description of its methodology (e.g., sampling strategy, survey response rates) and its sample lead to questions about the generalizability of the report’s findings. In addition, while the rationale for the report’s policy recommendations is sound, the proposals are restricted to the findings from the study and fail to consider or to draw upon any promising teacher evaluation strategies in current use. Transforming the system rather than tinkering around the edges will require broader thinking and a commitment to provide much greater investment and support for innovation to build, test, and audit evaluation systems that can stand up to public scrutiny and be practically feasible.

Suggested Cite:

Pecheone, R. L. & Wei, R. C. (2009). Review of “The Widget Effect: Our National Failure to Acknowledge and Act on Teacher Differences.” Boulder and Tempe: Education and the Public Interest Center & Education Policy Research Unit. Retrieved [date] from http://epicpolicy.org/thinktank/review-Widget-Effect

The Center for Research on Education Outcomes (CREDO) at Stanford University conducted a large-scale analysis of the impact of charter schools on student performance. The center’s data covered 65-70% of the nation’s charter schools. Although results varied by state, 17% of the charter school students have significantly higher math results than their matched twins in comparable traditional public schools (TPS), while 37% had significantly worse results. The CREDO study strengthens the well-established, broader body of evidence showing average charter performance to be equal to, or perhaps lower than, the performance of traditional schools—-a body of evidence that is summarized in this review. The study also presents some state-level analyses concerning policy options; this review points out limitations with those analyses and also explores other policy implications of the report’s findings. The relative strength and comprehensiveness of the data set used for this study, as well as the solid analytic approaches of the researchers, makes this report a useful contribution to the charter school research base. Nevertheless, this review points out some weaknesses and areas for improvement, many of which represent commonplace limitations for this type of study that should be shared in the technical report.

Suggested Cite:

Miron, G. & Applegate, B. (2009). Review of “Multiple choice: Charter school performance in 16 states.” Boulder and Tempe: Education and the Public Interest Center & Education Policy Research Unit. Retrieved [date] from http://epicpolicy.org/thinktank/review-multiple-choice

The Friedman Foundation for Educational Choice has published multiple reports advocating for states to adopt policies that award tax credits to donors who fund private school vouchers. The three most recent reports focus on Indiana, Georgia, and Montana. For each state, the reports conclude that the policies would reduce government expenditures and make the educational finance system more efficient. This review looks at the Indiana, Georgia, and Montana reports and finds those conclusions highly suspect, pointing out that the reports present unsubstantiated claims and fail to adequately consider short- and long-term costs of such tax-credit schemes.

Suggested Cite:

Huerta, L. (2009). Review of three tuition-tax-credit voucher reports from the Friedman Foundation. Boulder and Tempe: Education and the Public Interest Center & Education Policy Research Unit. Retrieved [date] from http://epicpolicy.org/thinktank/reviewtuition-tax-credits

Reroute the Preschool Juggernaut is a new book published online by the Hoover Institution and authored by the Fordham Institution’s Chester Finn. It is an inaccurate and poorly reasoned attack on the movement to secure all children a good preschool education. The book cherry-picks a few weak studies to fit its preconceptions, and it builds the case for targeted programs based on errors, exaggeration, misrepresentation, and logical inconsistency.

Suggested Cite:

Barnett, W.S. (2009). Special Review of “Reroute the Preschool Juggernaut.” Boulder and Tempe: Education and the Public Interest Center & Education Policy Research Unit. Retrieved [date] from http://epicpolicy.org/thinktank/Special-Review-Reroute-Preschool-Juggern...

The study under review is the second-year evaluation report of the Milwaukee Parental Choice Program (MPCP), a publicly funded voucher program that allows low-income students in Milwaukee to attend secular and religious private schools in that city. Its primary finding is that there were no overall statistically significant differences in achievement growth in reading or math between MPCP and Milwaukee Public School (MPS) students over a one-year period. The study design and methods of analysis are sound overall. The relatively short duration of the treatment, however, raises questions about the usefulness of the findings.

Correction:
On June 24, 2009, the authors of the report under review brought to our attention an error in the original May 28, 2009 review of their work by Casey Cobb. Professor Cobb has revised his review accordingly. As revised, the review includes an important correction. In the initial review, Professor Cobb made an incorrect assumption about the sampling procedures used in the study. As a result, he drew the incorrect conclusion that the external validity of the findings was tempered by a non-random sample of MPCP students, and hence a non-representative sample of MPCP schools. As the study authors have pointed out, the MPCP sample was in fact drawn randomly.

Suggested Cite:

Cobb, C. D. (2009). Review of “The MPCP Longitudinal Educational Growth Study Second Year Report.” Boulder and Tempe: Education and the Public Interest Center & Education Policy Research Unit. Retrieved [date] from http://epicpolicy.org/thinktank/review-MPCP-longitudinal

Charter Schools in Eight States uses longitudinal data from eight states to evaluate the effects of charter schools on achievement, attainment, integration, and competition. The findings are mixed. Achievement: The study examines seven jurisdictions and finds insignificant effects on reading and math performance in five, and small negative effects in two others. Attainment: In the two jurisdictions with data, the study finds positive effects for charter high schools’ rates of graduation and college matriculation. Integration: The study finds no evidence that charter schools are skimming high-achieving students away from public schools, or that charter schools lead to increased racial/ethnic stratification, but these findings should be regarded as equivocal because the supporting analyses use highly aggregated data. Competition: The study finds no evidence that the average student achievement at public schools either increases or decreases in response to entry of charter schools to the educational “marketplace.” On the whole, the methods used in this report are exemplary. The authors describe their statistical analyses in a transparent manner that makes it possible for readers to form their own opinions about the strength of the argument being advanced. The review does raise questions about all four of the report's sections, particularly stressing some weaknesses in the data and analyses regarding the integration and competition findings. Overall, however, the report makes an important contribution to the empirical literature on charter school effectiveness.

Suggested Cite:

Briggs, D.C. (2009). Review of "Charter Schools in Eight States: Effects on Achievement, Attainment, Integration and Competition." Boulder and Tempe: Education and the Public Interest Center & Education Policy Research Unit. Retrieved [date] from http://epicpolicy.org/thinktank/review-Charter-Schools-Eight-States

The third-year evaluation of the federally funded Washington, D.C. voucher program shows that low-income students offered vouchers in the first two years of the program had modestly higher reading scores after three years but showed no significant difference in mathematics. The authors, however, interpret the results in ways that raise questions, given some of their own findings. The report downplays the implications of the subgroup results showing that higher reading scores for those offered vouchers were concentrated in among certain groups of students. Further, some of the most interesting results of the study were related to student choice behavior rather than increases in test scores. The report could have done far better in analyzing the results of the experiment by presenting them in a more nuanced fashion and by discussing the possibility of varied effects with different populations and in different contexts.

Suggested Cite: Carnoy, M. (2009). Review of “Evaluation of the DC Opportunity Scholarship Program: Impacts After Three Years” Boulder and Tempe: Education and the Public Interest Center & Education Policy Research Unit. Retrieved [date] from http://epicpolicy.org/thinktank/review-evaluation-dc-opportunity

According to a new study of Milwaukee public schools, student achievement has benefited from voucher-based school competition. A novel method, using geocoding, was proposed for measuring the degree of competition within the city of Milwaukee and, in turn, for determining whether such competition has increased or decreased the achievement of public school students. Though a more traditional measurement of competition was eventually used in lieu of geocoding, the authors of the study determined that the overall effect of competition on student outcomes was positive over the seven-year span for which data were available. Specifically, it was argued that increased school choice improves the academic performance of students in traditional public schools who are voucher eligible by means of system-wide competitive pressures. Based on a review of several key issues—-including statistical modeling and control, effect size interpretation, the role of explanation in causal inference, and the validity of reported conclusions—-the practical effect of competition through vouchers appears to be small, if not negligible. It is also suggested that a number of methodological issues would benefit from greater clarity.

Suggested Cite:

Camilli, G. (2009). Review of “The Effect of Milwaukee’s Parental Choice Program on Student Achievement in Milwaukee Public Schools” Boulder and Tempe: Education and the Public Interest Center & Education Policy Research Unit. Retrieved [date] from http://epicpolicy.org/thinktank/review-Effect-Milwaukee-Parental-Choice

The new Weighted Student Formula Yearbook 2009 from the Reason Foundation provides a simple framework for touting the successes of states and urban school districts that grant greater fiscal autonomy to schools. The report defines the Weighted Student Formula (WSF) reform extremely broadly, presenting a variety of reforms under the WSF umbrella. Accordingly, when the report concludes that WSF is successful and should be widely replicated, it is difficult to sort through the claims and recommendations. Moreover, the approach and recommendations lack critical inquiry, thought, or empirical analysis. Perhaps most disturbing is the fact that in a third of the specific districts presented in the report, the evidence of success provided predates the implementation of the reforms, and the Reason press release makes the outright claim that past improvements are somehow a function of yet-to-be-implemented reforms. While the report does provide some reasonable recommendations, they are overshadowed by others. Overall, the policy guidance provided by the Reason report is reckless and irresponsible.

Suggested Cite:

Baker, B. (2009). Review of “Weighted Student Formula Yearbook 2009.” Boulder and Tempe: Education and the Public Interest Center & Education Policy Research Unit. Retrieved [date] from http://epicpolicy.org/thinktank/review-Weighted-Student-Formula-Yearbook

A recent report from the Brookings Institution presents panel data analyses using student fixed effects models to estimate the impact of Milwaukee charter schools on student achievement. The report finds that attendance at a charter school is associated with small positive increases in math test scores and with no significant differences in reading scores. The positive effects estimates for math are largely limited to earlier years of the charter school program and to gains made during the first year in a charter, and the results might be dependent on the study’s approach to modeling transfers between schools. The review identifies several questions about the internal and external validity of the reported estimates. Nevertheless, the methods used in the study do have important strengths. When the results are considered together with the large body of research on charter schools, the conclusion that charter schools should not be expected to have large effects on achievement in urban schools is reasonable.

Suggested Cite:

Bifulco, R. (2009). Review of “The Impact of Milwaukee Charter Schools on Student Achievement.” Boulder and Tempe: Education and the Public Interest Center & Education Policy Research Unit. Retrieved [date] from http://epicpolicy.org/thinktank/review-impact-Milwaukee-charter

A report from the School Choice Demonstration Project examines issues concerning the funding formula used for the Milwaukee Parental Choice Program (MPCP). It finds that the program generates a net saving to taxpayers in Wisconsin but imposes a significant fiscal burden on taxpayers in Milwaukee. However, these findings depend significantly on how many students would have attended public school if the voucher option were not available, as well as on the actual resource requirements for those new voucher users. The report ignores the second assumption. It considers the first, although its key findings about the existence and magnitude of any savings and burdens are nonetheless dependent on that assumption. The issues raised by this report highlight some ways in which voucher policies are complex to design and implement and must be carefully evaluated on both efficiency and equity grounds.

Suggested Cite:

Belfield, C. R. (2009). Review of "The Fiscal Impact of the Milwaukee Parental Choice Program: 2009 Update." Boulder and Tempe: Education and the Public Interest Center & Education Policy Research Unit. Retrieved [date] from http://epicpolicy.org/thinktank/review-fiscal-impact-Milwaukee

Advocates of vouchers argue that nearby public schools will be forced to compete for students, leading to improvements for voucher users and non-users alike. Critics worry that the students who use vouchers to leave public schools will have parents with higher levels of education and be less expensive to educate, and that losing these students will cause those schools to enter spirals of decline. This new report purports to gather all available empirical evidence on the question of the competitive effects of vouchers, finding a strong consensus that vouchers help public schools. But the report, based on a review of 17 studies, selectively reads the evidence in some of those studies, the majority of which were produced by voucher advocacy organizations. Moreover, the report can’t decide whether or not to acknowledge the impact of factors other than vouchers on public schools. It attempts to show that public school gains were caused by the presence of vouchers alone, but then argues that the lack of overall gains for districts with vouchers should be ignored because too many other factors are at play. In truth, existing research provides little reliable information about the competitive effects of vouchers, and this report does little to help answer the question.

Suggested Cite:

Lubienski, C. (2009). Review of “A Win-Win Solution: The Empirical Evidence on How Vouchers Affect Public Schools.” Boulder and Tempe: Education and the Public Interest Center & Education Policy Research Unit. Retrieved [date] from http://epicpolicy.org/thinktank/review-win-win-solution

The Fordham Institute conducted a survey of experts who are advocates of private education, soliciting opinions about how private schools accepting public monies, such as from voucher programs, should be held accountable. The experts were in agreement that private schools should not accept regulation of their day-to-day operations, but disagreed among themselves whether to accept top-down standardized testing accountability. The Fordham researchers proposed a compromise position in which the more the private schools rely on public money, the more the schools should be subject to the same requirements as public schools. The Fordham study is a reasonable inquiry into the difficult area of private/public education ventures. Although the study does not arrive at a definitive position, it raises important issues about accountability in both private and public education.

Suggested Cite:

House, E. (2009). Review of “When Private Schools Take Public Dollars: What’s the place of accountability in school voucher programs?” Boulder and Tempe: Education and the Public Interest Center & Education Policy Research Unit. Retrieved [date] from http://epicpolicy.org/thinktank/review-when-private-schools-take-public-...

A new Mathematica Policy Research report suggests that alternatively certified teachers are as successful as traditionally certified teachers. However, few if any valid conclusions about certification policy can be drawn from the report because the study:
• Did not fully report and acknowledge in its conclusions the many analyses from the study finding that traditionally trained teachers outperformed alternative route teachers in both math and reading.
• Has a research design that favors finding few significant differences between groups, most notably its small sample size, sampling methods, and failure to distinguish the “treatments” that alternative certification and traditional certification teachers provided.
• Is relevant only to a very limited population of teachers in schools that hire many alternatively certified teachers, and is not generalizable to most states, districts, and schools that do not allow such programs and are more selective in their hiring.

Unfortunately, the Mathematica report is quick to draw broad and unqualified implications from the study, and it neglects to properly emphasize the study’s many limitations. Policy makers would do well to read it with caution.

Suggested Cite:

Corcoran, S. P., & Jennings, J. L. (2009). Review of “An Evaluation of Teachers Trained Through Different Routes to Certification: Final Report.” Boulder and Tempe: Education and the Public Interest Center & Education Policy Research Unit. Retrieved [date] from http://epicpolicy.org/thinktank/review-evaluation-of-teachers

Linda Darling-Hammond's separate review can be found at: http://edpolicy.stanford.edu/pages/pubs/pub_docs/mathematica_policy_brie...

A new report published by the Buckeye Institute for Public Policy Solutions is a minor variant on six similar reports published by the Friedman Foundation over the past three years. The new report repeats some of the errors in the previous reports, and it follows a parallel structure, arguing that the costs of dropping out are dramatic for the state of Ohio, and that last-chance charter schools for dropouts can increase graduation and address the dropout problem. However, the report’s claims about graduates for the 23 illustrative schools are inconsistent with the data reported by the state of Ohio for the year chosen, resulting in a dramatic overstatement of the graduation rates at the charters. The report also largely ignores the existing research literature on the personal and social benefits of educational attainment, the achievements of charter schools, and the factors associated with either completing or dropping out of high school. Further, the report fails to compare the alleged benefits of last-chance charter schools with plausible alternatives. State policymakers interested in increasing graduation would be better served by seeking out the available, well-researched scholarship on the topic.

Suggested Cite:

Dorn, S. (2009). Review of “The high cost of high school dropouts in Ohio.” Boulder and Tempe: Education
and the Public Interest Center & Education Policy Research Unit. Retrieved [date]
from http://epicpolicy.org/thinktank/review-the-high-cost

Two recent reports by Education Sector set out to examine the viability of proposals to revamp the No Child Left Behind choice provisions to allow students in failing schools to choose a school outside of their home school district. The findings of the reports are weakened by foundational assumptions about capacity and competition for space, as well as a failure to present alternatives. Further, the reports do not systematically consider the role of geographic variation, causing a potential underestimation of the impact of inter-district choice for those urban contexts where policy advocates believe it is most necessary. Instead of an analysis based on such definite assumptions, policy makers would be better served by an array of assumptions and analyses, presenting the full scope of potential outcomes.

Suggested cite:

Holme, J.J. & Richards, M.P. (2008). Review of “Plotting School Choice" and "In Need of Improvement." Boulder and Tempe: Education and the Public Interest Center & Education Policy Research Unit. Retrieved [date] from http://epicpolicy.org/thinktank/review-plottingschool-choice