The Teachers’ Strike in Chicago…and VAMs

September 11th, 2012 at 8:10 am

Steve Greenhouse does a nice job of explaining what’s going on in this high visibility strike by Chicago’s public school teachers.  I need to learn more about the details before I develop my own view, but two things to keep in mind here.

First, public opinion turns out to be very important in these high visibility cases, and given today’s climate facing public sector unions, and teachers’ unions in particular, one should worry a lot about balance in how this action gets treated in the media.  (I was reminded of what may have been the most ridiculous line from the RNC a few weeks ago, from Gov Christie: “Democrats believe in teachers’ unions; we believe in teachers.”)

Second, note this line from the NYT article (my bold):

Eager to improve Chicago’s schools, [Mayor] Emanuel has taken several steps — among them pressing the school board to rescind a promised 4 percent raise — and made numerous demands that have infuriated the Chicago Teachers Union. He wants student test performance to count heavily in evaluating teachers for tenure, even though the union insists that is a highly unreliable way to assess teachers.

It’s not just unions.  Rigorous academic work, some of it associated with my friend and Berkeley Prof Jesse Rothstein, has shown that such testing—using “value-added models,” or VAMs–is often a highly incomplete and unfair way to evaluate teacher performance.

VAMs are designed to isolate individual teachers’ value-added—new skills and knowledge acquired by students—from year-to-year.  Often, as in the Chicago case, the results are then used for teacher promotion or demotion.

There’s nothing inherently wrong with that, assuming the tests are valid and reliable.  But Jesse’s careful work has convinced me that they’re not, at least not yet.

Here’s a review of the issue (and here’s the wonky backup work):

Using VAMs for individual teacher evaluation is based on the belief that measured achievement gains for a specific teacher’s students reflect that teacher’s “effectiveness.” This attribution, however, assumes that student learning is measured well by a given test, is influenced by the teacher alone, and is independent from the growth of classmates and other aspects of the classroom context. None of these assumptions is well supported by current evidence.

Most importantly, research reveals that gains in student achievement are influenced by much more than any individual teacher. Others factors include:

• School factors such as class sizes, curriculum materials, instructional time, availability of specialists and tutors, and resources for learning (books, computers, science labs, and more);

• Home and community supports or challenges;

• Individual student needs and abilities, health, and attendance;

• Peer culture and achievement;

• Prior teachers and schooling, as well as other current teachers;

• Differential summer learning loss, which especially affects low-income children; and

• The specific tests used, which emphasize some kinds of learning and not others and which rarely measure achievement that is well above or below grade level.

However, value-added models don’t actually measure most of these factors. VAMs rely on statistical controls for past achievement to parse out the small portion of student gains that is due to other factors, of which the teacher is only one. As a consequence, researchers have documented a number of problems with VAM models as accurate measures of teachers’ effectiveness.

So, problem one is the VAM is not yet able to reliably isolate teacher performance amidst all these other factors.

Researchers have different views on the VAM problem but problem two is widely agreed upon, and it’s something I, and I suspect every other parent with kids in public schools, have seen first hand.  People respond to incentives, and when their jobs are on the line, that amps up that incentive and response big time.

So you have teachers teaching to the test, even if the test is flawed and even if the test leaves out knowledge that the teacher (and parent) believes is essential to a child’s educational experience.

Again, my point is not that to endorse the Chicago teachers strike, though from what I can tell, they did not undertake it lightly.  My point is that it’s not just “the unions” who object to having their members’ job performance judged on the basis of VAMs.  In fact, their objections are founded on solid research.

Print Friendly

10 comments in reply to "The Teachers’ Strike in Chicago…and VAMs"

  1. Joe says:

    There is a lot of emotion around the whole teacher/teacher union issue. Skipping the whole public union issue, how should teachers be evaluated?

    A corporate marketing manager gets tied to revenue results yet those can be impacted by broader economic issues, poor sales reps and training and other issues outside of his/her responsibility or control. Yet that is how they get measured. Sometimes the reviews (and salary treatments) are subjective and not objective.

    Should teachers get evaluated on subjective measures by their bosses? If not then they have to accept some type of objective measurement. It can’t just be an endless list of why they can’t be measured. Ultimately, student performance is the “revenue” for them. It is what they are responsible for even though there may be other factors that impact performance.

    Two more points:

    - Without unions but with objective measurements/results teachers could be “free-agents” able to sell their skills and results on a free market.
    - If we decide to pay teachers more going forward and wanted to offer new grads a high starting pay (but with lower retirement costs), unions would prevent that from happening yet it might be what is needed to begin to bring in new blood from the top of their class


    • Daniel says:

      Joe, regardless of the complexities of the context in the business world, output is easy to measure – PROFIT or MONEY. Yes, it can be complicated to measure what influence one employee had over another in order to achieve (input), but the output is SIMPLE. Like I said, the bottom line is profit or money – as in how much an employee, individual branch office, or corporation makes.

      The teaching and learning process involves as many, if not more, variables in terms of the process (input), but the output is potentially the most complex product that any profession could produce – a person.

      There is no quantification on the process, no matter how much the corporate reformers desire. It just can’t be done. Now the question remains, how much damage will be caused to our kids and teachers before someone finally figures this out.

      And don’t be duped – expect that state mandated test scores will go up, as they have in TN. And also expect politicians to jump for joy and claim victory. But, in the end, all that means is that more teachers, administrators, and county level officials cheated or teachers concentrated on teaching to the test and rote memorization of learning, rather than learning how to think. The latter is what education is supposed to be about. However, the politicians have ruined that for us.


      • Joe says:

        Ultimately there are data points for evaluation of both districts and teachers. Graduation rates, college readiness, college admissions, college graduations, literacy – math and reading all can be quantified.

        At some point, teachers must have measurable objectives. If not, then how will we ever be able to evaluate and improve the output of our education system.

        I really don’t understand the counter point but I’m willing to listen. How would teachers be evaluated? What criteria?

        It simply is not good enough to say we need to spend (invest?) more money. We spend enough on education although how wisely we spend it is certainly open for debate.

        So just help me understand what we need to do to better understand and identify a successful teacher, reward them on a differentiated basis, and cull the bad ones out of the system.

        I’m also convinced that we should be recruiting new teachers by offering them higher starting salaries but with reduced long term compensation (pensions/benefits) thereby holding overall costs down.


        • Chris G says:

          >Graduation rates, college readiness, college admissions, college graduations, literacy – math and reading all can be quantified. At some point, teachers must have measurable objectives.

          Quoting John Tukey, probably my favorite mathematician: “The combination of some data and an aching desire for an answer does not ensure that a reasonable answer can be extracted from a given body of data.”

          As Jared notes, there’s no test which adequately controls for factors having nothing to do with teacher performance. Fine I suppose to aspire to developing a statistically-valid test for evaluating teacher performance but, at present, there’s no such thing. And I’m skeptical that such a test could ever be developed.

          >”If [there's no objective test], then how will we ever be able to evaluate and improve the output of our education system.”

          Then we’ll have to rely on qualitative and perhaps semi-quantitative measures. In other words, we’ll have to rely on the subjective judgment of people we believe to be reasonable.

          >”So just help me understand what we need to do to better understand and identify a successful teacher, reward them on a differentiated basis, and cull the bad ones out of the system.”

          See my comment above.

          >”I’m also convinced that we should be recruiting new teachers by offering them higher starting salaries but with reduced long term compensation (pensions/benefits) thereby holding overall costs down.”

          FWIW, I’m fine with idea of increasing salaries and reducing deferred compensation but what leads you to believe that will holds costs down? (I have no idea whether it would or wouldn’t. I’m just curious as to the basis for your assertion.) I see it more as an issue of dealing with the cost of education upfront rather than kicking the can down the road and hoping finances work out in the long run. If you pay as you go then you eliminate the risk of having a huge bill to pay off some time in the future, i.e., you don’t save at rate X figuring that’s enough to cover the cost of the bill only to discover when the bill comes due that you should have been saving at 2X.


  2. Jenny Robinson says:

    Hi there,
    I am very disturbed by Joe’s comments, and my gut response to them is, what world are you operating in? I am not a teacher. I have three young children and I totally support the Chicago teachers union. They are striking in the interests of the parents and the children they teach. What they want is what I as a parent want: a full curriculum including science, social studies, art, music, world languages, P.E., and libraries staffed with librarians…also, decent class sizes and well maintained buildings with enough outdoor space. Isn’t this what all children deserve, not just the sons and daughters of the Rahm Emanuels and Penny Pritzkers of the world? Please stop talking about revenue, or about maximizing value as if our kids are some widgets to come off the assembly line. Fund education, and let communities work with professional educators to educate our kids. Please don’t advocate tinkering with the tests to get finer readings. When standardized tests get the power to control the curriculum, we are all in trouble, and Pearson and the rest are siphoning off the funds that our schools and teachers need.

    Jenny


    • Rima Regas says:

      Our education system fails kids by treating them like buckets of sand, and the teachers like robots whose sole mission is to pour the sand in each bucket. That’s not how learning works. We’re not raising sheep or programming computers.

      Teachers know their students best. Instead of paying for tests that measure a very narrow slice of what our kids know, invest in senior teachers who can spot check the students’ work in each grade and help the teachers better supplement the education of children who may be behind. Invest in better textbooks and teaching materials. Add the arts back in. Add philosophy, ethics, and other subjects that promote critical thinking and stimulate a child’s natural curiosity.

      Standardized testing is an abomination. Vilifying teachers and punishing them in the way they are being punished will only lead to more of them leaving the profession and staying out.


    • Chris G says:

      >Please stop talking about revenue, or about maximizing value as if our kids are some widgets to come off the assembly line. Fund education, and let communities work with professional educators to educate our kids. Please don’t advocate tinkering with the tests to get finer readings.

      I’m with you 100%.


    • Chris G says:

      Christopher Lasch from his book The Revolt of the Elites:
      “… individuals cannot learn to speak for themselves at all, much less come to an intelligent understanding of their happiness and well-being, in a world in which there are no values except those of the market. . . . the market tends to universalize itself. It does not easily coexist with institutions that operate according to principles that are antithetical to itself: schools and universities, newspapers and magazines, charities, families. Sooner or later the market tends to absorb them all. It puts an almost irresistible pressure on every activity to justify itself in the only terms it recognizes: to become a business proposition, to pay its own way, to show black ink on the bottom line. It turns news into entertainment, scholarship into professional careerism, social work into the scientific management of poverty. Inexorably it remodels every institution in its own image.”


  3. Horst Lehrheuer says:

    I only had a chance to scan the blog and comments. And, I do not know very much about all the issues in this case. However, one thing caught my eye though: “…teaching to the test.”

    If that’s the case here, Ram Emanuel, or anyone else who keeps on trying this, does a disservice to the students and our society – to the best of my knowledge. Why? Here is how Heinz von Foerster put it: “Tests test tests!” – and not much more.

    This is not to say that I am against any tests. We just mustn’t forget what Heinz von Foerster said about this and realize that we non-trivial humans are not fully measurable or predictable “machines.”


  4. David C says:

    I think _in general_ employee evaluation systems are very difficult to get right. It is hard to make them consistent and fair, and there are always major side effects/unintended consequences. I gave and received performance appraisals for more than 30 years in large corporations, and my own observation is that they had a sizable negative impact.

    You can only really assess performance if you actually see the employee performing the task. And if you understand the task at least as well as the employee. Both of these factors are almost never present. Certainly in schools, the idea of a principal observing teachers in the classroom has never gone down well.

    I think that we should be focusing less on evaluation and more on providing support to help teachers improve. It is possible to assess whether teachers are open to improvement suggestions and whether they are making a sincere effort to adopt them. This would be a far more effective way to make tenure and retention decisions.


Leave a Reply

Your email address will not be published.

Current ye@r *