New (Unvetted) Research about Washington DC’s Teacher Evaluation Reforms

VAMboozled!VAMboozled! A blog by Audrey Amrein-Beardsley

In November of 2013, I published a blog post about a “working paper” released by the National Bureau of Economic Research (NBER) and written by authors Thomas Dee – Economics and Educational Policy Professor at Stanford, and James Wyckoff – Economics and Educational Policy Professor at the University of Virginia.

In the study titled “Incentives, Selection, and Teacher Performance: Evidence from IMPACT,” Dee and Wyckoff (2013) analyzed the controversial IMPACT educator evaluation system that was put into place in Washington DC Public Schools (DCPS) under the then Chancellor, Michelle Rhee.

In this paper, Dee and Wyckoff (2013) presented what they termed to be “novel evidence” to suggest that the “uniquely high-powered incentives” linked to “teacher performance” via DC’s IMPACT initiative worked to improve the performance of high-performing teachers, and that dismissal threats worked to increase the voluntary attrition of low-performing teachers, as well as improve the performance of the students of the teachers who replaced them.

I critiqued this study in full (see both short and long versions of this critique here), and ultimately asserted that the study had “fatal flaws” which compromised the (exaggerated) claims Dee and Wyckoff (2013) advanced.

These flaws included but were not limited to that only 17% of the teachers included in this study (i.e., teachers of reading and mathematics in grades 4 through 8) were actually evaluated under the value-added component of the IMPACT system.

Put inversely, 83% of the teachers included in this study about teachers’ “value-added” did not have student test scores available to determine if they were indeed of “added value.”

That is, 83% of the teachers evaluated, rather, were assessed on their overall levels of effectiveness or subsequent increases/decreases in effectiveness as per only the subjective observational and other self-report data include within the IMPACT system.

Hence, while authors’ findings were presented as hard fact, given the 17% fact, their (exaggerated) conclusions did not at all generalize across teachers given the sample limitations, and despite what they claimed.

In short, the extent to which Dee and Wyckoff (2013) oversimplified very complex data to oversimplify a very complex context and policy situation, after which they exaggerated questionable findings, was of issue, that should have been reconciled or cleared out prior to the study’s release.

I should add that this study was published in 2015 in the (economics-oriented and not-educational-policy specific) Journal of Policy Analysis and Management (see here), although I have not since revisited the piece to analyze, comparatively (e.g., via a content analysis), the original 2013 to the final 2015 piece.

Anyhow, they are at it again. Just this past January (2017) they published another report, albeit alongside two additional authors: Melinda Adnot – a Visiting Assistant Professor at the University of Virginia, and Veronica Katz – an Educational Policy PhD student, also at the University of Virginia. This study titled “Teacher Turnover, Teacher Quality, and Student Achievement in DCPS,” was also (prematurely) released as a “working paper” by the same NBER, again, without any internal or external vetting but (irresponsibly) released “for discussion and comment.”

Hence, I provide below my “discussion and comments” below, all the while underscoring how this continues to be problematic, also given the fact that I was contacted by the media for comment.

Frankly, no media reports should be released about these (or for that matter any other) “working papers” until they are not only internally but also externally reviewed (e.g., in press or published, post vetting). Unfortunately, as they too commonly do, however, NBER released this report, clearly without such concern.

Now, we as the public are responsible for consuming this study with much critical caution, while also advocating that others (and helping others to) do the same. Hence, I write into this post my critiques of this particular study.

First, the primary assumption (i.e., the “conceptual model”) driving this Adnot, Dee, Katz, & Wyckoff (2016) piece is that low-performing teachers should be identified and replaced with more effective teachers. This is akin to the assumption noted in the first Dee and Wyckoff (2013) piece.

It should be noted here that in DCPS teachers rated as “Ineffective” or consecutively as “Minimally Effective” are “separated” from the district; hence, DCPS has adopted educational policies that align with this “conceptual model” as well. Interesting to note is how researchers, purportedly external to DCPS, entered into this study with the same a priori “conceptual model.” This, in and of itself, is an indicator of researcher bias (see also forthcoming).

Nonetheless, Adnot et al.’s (2016) highlighted finding was that “on average, DCPS replaced teachers who left with teachers who increased student achievement by 0.08 SD [standard deviations] in math.”

Buried further into the report they also found that DCPS replaced teachers who left with teachers who increased student achievement by 0.05 SD in reading (at not a 5% but a 10% statistical significance level).

These findings, in simpler but also more realistic terms, mean that (if actually precise and correct, also given all of the problems with how teacher classifications were determined at the DCPS level), “effective” mathematics teachers who replaced “ineffective” mathematics teachers increased student achievement by approximately 2.7%, and “effective” reading teachers who replaced “ineffective” reading teachers increased student achievement by approximately 1.7% (at not a 5% but a 10% statistical significance level).

These are hardly groundbreaking results as these proportional movements likely represented one or maybe two total test items on the large-scale standardized tests uses to assess DCPS’s policy impacts.

Interesting to also note is that not only… READ THE FULL BLOG POST HERE: VAMboozled! | VAMboozled! A blog by Audrey Amrein-Beardsley

Advertisements

Trump makes a deal with Iraq, and hundreds suddenly face deportation | Michigan Radio

Trump makes a deal with Iraq, and hundreds face deportation
The president of the Chaldean Community Foundation estimates about 300 ethnic Chaldean Catholics in Michigan alone are at risk of deportation to a country where their faith makes them a target. “Sending them back would be a death sentence for them,” he says.
Read more.

Source: Trump makes a deal with Iraq, and hundreds suddenly face deportation | Michigan Radio

FDA nominee Gottlieb’s many industry entanglements will be a focus of Wednesday hearing

On April 5, the Senate “HELP” Committee will hold a confirmation hearing for Scott Gottlieb, who’s in line to head the Food and Drug Administration.

This week, the attention’s on Judge Neil Gorsuch, President Trump‘s nominee to become an associate justice of the Supreme Court.

But there’s another position in play: the Senate Committee on Health, Education, Labor and Pensions held a confirmation hearing for Scott Gottlieb, in line to head the Food and Drug Administration.

read more. by Niv Sultan

Source: FDA nominee Gottlieb’s many industry entanglements will be a focus of Wednesday hearing

Those prized small donors? They may not be as small as you think

Small donor statistics may overstate their importance. That’s because many relatively “small donors” are actually one little slice of a much bigger donor.

Politicians rarely like to talk about the money that fuels their campaigns, and for good reason. The big players in campaign finance — billionaire megadonors, professional fundraisers, lobbyists — rarely get a warm reception from the public. Perhaps the only exception is the much-loved small donor.. read more by 

Source: Those prized small donors? They may not be as small as you think

Mischaracterization of Judicial Crisis Network’s ads reduces transparency, group says

The Campaign Legal Center has accused the Judicial Crisis of mischaracterizing ads supporting SCOTUS nominee Neil Gorsuch.

On the eve of the likely confirmation of Supreme Court nominee Judge Neil Gorsuch by a simple majority of the Senate, the Campaign Legal Center accused a conservative nonprofit of mischaracterizing ads supporting Gorsuch and thereby depriving the public of the ability to know who was behind the ads.

read more by Niv Sultan Source: Mischaracterization of Judicial Crisis Network’s ads reduces transparency, group says

Innovative crowdfunding platform CROWDPAC supporting Haley Stevens, potential opponent to MI-11’s GOP Rep. David Trott | Eclectablog

 In 2013, a new nonpartisan group called CROWDPAC launched a new fundraising model for potential political candidates. Similar to Kickstarter, CROWDPAC allows candidates to raise money through financial pledges.

The pledges are not collected unless the candidate actually decides to run, giving them a chance to see how much support they have before making their decision.

As an example, Kathryn Allen was considering a run against Utah Republican Congressman Jason Chaffetz so she started a CROWDPAC fundraising pledge drive.

She raised about $12,000. Not an insubstantial amount but not nearly enough to run against a sitting member of Congress who Chairs one of the most powerful House Committees.

Then Chaffetz had a townhall meeting were he suggested that people were deciding to buy iPhones rather than health insurance. The outrage that resulted from that tonedeaf and absurd comment turned out to be the catalyst for an outpouring of support for Allen.

As of today, she has raised over…

READ THE FULL BLOG POST HERE: Innovative crowdfunding platform CROWDPAC supporting Haley Stevens, potential opponent to MI-11’s GOP Rep. David Trott | Eclectablog

Foreclosure King Republican David Trott of MI-11 introduces bill to enrich his foreclosure business | Eclectablog

 Republican Congressman David Trott from Michigan’s 11th Congressional District has a long and sordid history profiting from the “human misery” caused when his foreclosure business threw victims of the Bush Recession out of their homes.

He didn’t just run a foreclosure-based law firm.

He created a vertically integrated machine to ensure that he got profit every step of the way:

CHRIS HAYES: What money is there to be made the process of ?

DAVID DAYEN: There’s not a whole lot of money to be made if you’re just doing the legal work. But Trott & Trott had a unique business strategy where they bought up practically every company down the line that gets a little bit of money out of the foreclosure process. There’s a requirement in Michigan that you have to put legal notices out in the newspaper. They bought the newspaper that did that. There’s a requirement that you have to do a title search. They bought the company that does the title searches and they get money off of that. They bought…

READ THE FULL BLOG POST HERE: Foreclosure King Republican David Trott of MI-11 introduces bill to enrich his foreclosure business | Eclectablog

Betsy DeVos Watch: When Role Models Visit a School for Girls

Betsy DeVos Watch: When Role Models Visit a School for Girls
by Jeff Smith (GRIID)
Last week, we kicked off the first installment in our series of articles under the heading of Betsy DeVos Watch.

In this week’s article we take a look at a visit from the Secretary of Education to a charter school in the nation’s capitol. DeVos was visiting Excel Academy in Washington, DC and began her brief comments by stating, “As we bring our education system into the 21st century, it is important we highlight the many types of different schools serving the varied and unique needs of individual students.”

Of course, there is nothing surprising about these comments or which school DeVos visited, since she has been saying for years that she thinks that schools other than public schools are the best direction the country can make.

However, what was more interesting… https://griid.org/2017/04/07/betsy-devos-watch-when-role-models-visit-a-school-for-girls/

Grand Rapids Institute for Information Democracy

Last week, we kicked off the first installment in our series of articles under the heading of Betsy DeVos Watch. 

In this week’s article we take a look at a visit from the Secretary of Education to a charter school in the nation’s capitol. DeVos was visiting Excel Academy in Washington, DC and began her brief comments by stating, “As we bring our education system into the 21st century, it is important we highlight the many types of different schools serving the varied and unique needs of individual students.”  

Of course, there is nothing surprising about these comments or which school DeVos visited, since she has been saying for years that she thinks that charter and private schools would be better for the country.

However, what was more interesting than the fact that Betsy DeVos visited Excel Academy, was who she brought with her. Accompanying the Secretary of…

View original post 419 more words