Comparing Apples to Oranges

“Our progress as a nation can be no swifter than our progress in education. The human mind is our fundamental resource.”
- John F. Kennedy

I've been closely following events in Wisconsin regarding Governor Walker vs the Public Sector Unions (notably Teachers) but have resisted blogging about it because it's a hot button topic for many people. What did catch my eye that I would like to share is the battle over statistics. Both sides have their talking points and "facts" to back them up. And when "facts" are challenged, neither side is eager to cede the point and admit error. So bad "facts" persist and keep being used in arguments.

A blog post by IowaHawk takes to task Paul Krugman, Nobel Winning Economist, who writes for The New York Times. Krugman wrote,

Only 5 states do not have collective bargaining for educators and have deemed it illegal. Those states and their ranking on ACT/SAT scores are as follows:
South Carolina – 50th
North Carolina – 49th
Georgia – 48th
Texas – 47th
Virginia – 44th

This is very damming evidence and a popular statistic used by pro-Union teachers. But IowaHawk dives deeper into the numbers. He notes that a student's race is a bigger factor on SAT scores than teacher performance.

A state's "average ACT/SAT" is, for all intents and purposes, a proxy for the percent of white people who live there.

Statistically, on average, white students get higher ACT/SAT scores.This has nothing to do with innate ability or genetics or aptitude of individuals. Instead economic differences (poverty), racism, family environment, and other society biases are to blame.

Wisconsin is 4% Black and 4% Hispanic. Texas is 12% black and 30% Hispanic. Since race influences average ACT/SAT scores, to make a fair comparison, students in the two states should be compared by race.

Here are numbers from the National Assessment of Educational Progress (NAEP), an annual standardized test given to 4th and 8th graders around the country to measure proficiency in math, science, and reading.

2009 8th Grade Math
White students: Texas 301, Wisconsin 294 (national 294)
Black students: Texas 272, Wisconsin 254 (national 260)
Hispanic students: Texas 277, Wisconsin 268 (national 260)

There are more stats at IowaHawk for 4th grade & 8th grade Math, Reading and Science but the numbers displayed here are typical. Within racial groups, all Texas students are above the national average and score better than Wisconsin. In Wisconsin the White students perform at or slightly above the national average, Hispanics above the average, and Black students below the average.

So instead of showing that Union Teachers are better, a nationalized 4th/8th grade test shows that Wisconsin produces average White students and poorly scoring Black students. Is this the final answer? Perhaps not. Perhaps a better Statistician will analyze the scores a different way to show something new and revealing.

For example other blogs point out another interesting difference in the SAT scores for Wisconsin vs Texas.
In 2010 Wisconsin placed 3rd in the nation on SAT scores. But only 4% of Wisconsin students took the SAT. This is known as selection bias. When only the best take a test, of course they perform better. 67% of Wisconsin students took the ACT and this dropped Wisconsin to 13th place. Still very, very good but it shows the difference that can occur in ranking depending on which students are tested.  In Texas 53% of all students took the SAT and 30% took the ACT.

Katherine Levin, a College Board spokeswoman for SAT calls comparisons between states "invalid" since the students taking the test are self-selected.

Levin, Feb. 25: Media and others often rank states, districts and schools on the basis of SAT scores despite repeated warnings that such rankings are invalid. The SAT is a strong indicator of trends in the college-bound population, but it should never be used alone for such comparisons because demographics and other nonschool factors can have a strong effect on scores. If ranked, schools and states that encourage students to apply to college may be penalized because scores tend to decline with a rise in percentage of test-takers.

Yet another factor to consider is poverty. Texas ranks 8th in most residents living below the poverty line. Wisconsin is 38th. So instead of saying Union teachers produce better college scores, we could use WI vs TX to prove that rich kids score better than poor kids.

Bottom Line

Beware of any argument based upon comparing "averages" without knowing the population mix behind the average. For example I could compare two colleges and show that students at A are on average shorter than B; so B must be biased against short people. What I don't reveal is that A is an all-girl school and B is not. Since girls are shorter on average than boys, this will pull down the total height average for school A.

Suppose the Ivy League schools argued that they had the most talented professors because their students (on average) get more of the most prestigious and well-paying jobs. The professors would be ignoring the fact that many of the brightest students, and students from the richest and best socially connected families, attend Harvard & Yale. So teaching ability could be just average and yet selection bias and social advantages give the students a preference in hiring. Some Wall Street firms hire ONLY from the Ivy League and look what resulted - a near collapse of the Financial System in 2008.

Update

A report showing it's the student, not the college, that determines success.
http://washingtonexaminer.com/blogs/beltway-confidential/2011/03/more-evidence-going-fancy-college-doesnt-pay