March 08, 2007

Lies, damned lies, and statistics

Not sure why I read this stuff, as I have no connection whatever to education in the US, but a couple of posts recently caught my eye, one on the Freakanomics blog, and the other on Pissed Off (Teacher).

Stephen Dubner of Freakanomics, dips into the mailbag and quotes this correspondent:
Hi guys. I know you guys have railed against confusing correlation with causality, and also discussed various statistical problems, but you do not seem to have addressed one that affects many real world decisions. I have noticed what appears to be an increase in decisions based on single variant analysis and averaging, often ignoring consequences of those decisions and/or the context of the analysis. It seems you ought to address these, as they are right up your alley. Some simple common examples...
He gives 3: Post Office boxes, schools and testing, and global warming. Here's an excerpt from the schools and testing example:
Another example happening in the SF Bay Area involves schools. The District looks at average standardized test scores of each school and closes the ones with the lowest average. They do not consider whether these ones which are lower have the poorest kids with least parent involvement and most transience and the like. But, by closing the school and moving them to a much larger school, these low performing kids continue to be low performers, they just do not pull down the average as much. Worse, these low performing kids often do worse (at least according to a recent study at Stanford) and/or drop out. Dropping out helps the averages, so you could argue it is a good policy (Texas allegedly uses that to improve their scores), but the consequences are likely dire for society (more crime or welfare or both). The problem is that the poor performing school may in fact have those particular kids doing much better than they would in another school (due to extra attention, etc), but decision makers are taught that “statistics do not lie”. This is the opposite of some of your situations (where statistics show “common sense” to be wrong).


Pissed Off steals a parody of NCLB:
My dentist is great! He sends me reminders so I don't forget checkups He uses the latest techniques based on research. He never hurts me, and I've got all my teeth.
When I ran into him the other day, I was eager to see if he'd heard about the new state program. I knew he'd think it was great.
"Did you hear about the new state program to measure effectiveness of dentists with their young patients?" I said.
"No," he said. He didn't seem too thrilled. "How will they do that?" "It's quite simple," I said. "They will just count the number of cavities each patient has at age 10, 14, and 18 and average that to determine a dentist's rating. Dentists will be rated as excellent, good, average, below average, and unsatisfactory. That way parents will know which are the best dentists. The plan will also encourage the less effective dentists to get better," I said. "Poor dentists who don't improve could lose their licenses to practice."
"That's terrible," he said.
"What? That's not a good attitude," I said. "Don't you think we should try to improve children's dental health in this state?"
"Sure I do," he said, "but that's not a fair way to determine who is practicing good dentistry."
"Why not?" I said. "It makes perfect sense to me."
"Well, it's so obvious," he said. "Don't you see that dentists don't all work with the same clientele, and that much depends on things we can't control? For example, I work in a rural area with a high percentage of patients from deprived homes, while some of my colleagues work in upper middle-class neighborhoods. Many of the parents I work with don't bring their children to see me until there is some kind of problem, and I don't get to do much preventive work. Also, many of the parents I serve let their kids eat way too much candy from an early age, unlike more educated parents who understand the relationship between sugar and decay. To top it all off, so many of my clients have well waterwhich is untreated and has no fluoride in it. Do you have any idea how much difference early use of fluoride can make?"
"It sounds like you're making excuses," I said. "I can't believe that you, my dentist, would be so defensive. After all, you do a great job, and you needn't fear a little accountability."

There's more. Go read the whole thing.

9 comments:

Steve N said...

Very interesting stuff indeed. This all ties into the offlist communicado you and I had about causal contexts and emergent properties. For a little background on one of the most education-relevant treatments of causal conditions, see J.L. Mackie's article entitled 'Causes and Conditions' which introduces his concept of insufficient but necessary parts of an unnecessary but sufficient (inus) conditions. You can find a clear summary here:
http://evans-experientialism.freewebspace.com/causation.htm

Steve N said...

...com/causation.htm

(link above was cut off) in previous comment.

for some reason, I can't post html in your comments, though I'm signed in to blogger.

Pissed Off said...

Nice to know someone reads the stuff I write.

Thanks.

Marco Polo said...

Likewise, POT.

Marco Polo said...

Steve N, I've looked at my blogger comments settings and can't see anything that would prevent you from adding HTML links. I'll try blogger help. If anyone else is having this problem, please email me

msheff said...

steve n, the link doesn't work, even with the cut off bit added on.

msheff said...

Let's see if I can do it. Here's the link.

brian said...

Similar piece I wrote several years ago:
Example: A person weighs 1200 pounds (The world record is considerably more than that). They lose 150 pounds a year for four years - that's 600 pounds they've lost. But are they going to be healthy at 600 pounds? Pass a simple health test? No, they still can't get out of bed without help. They are still going to score in the bottom 1% of people as far as health issues - just like they did when they weighed 1200 pounds - you have seemingly made zero progress in improving their health on the health test in 4 years - and if they lose 200 pounds in the fifth year they still weigh 400 pounds – they are still in the bottom 1% health wise and you’ve been working with them for 5 years!!! Why the disconnect? - because you are judging progress based on test results instead of reality. Will this person's family be thrilled that they have now lost 800 pounds? Will they think the health team has done a poor job? That is why parents often are happy with their child's progress when the tests seem to show otherwise - they know their kid has had 1200 pounds of issues as far as where they were educationally and they have made huge progress. That’s why teachers often get frustrated about some testing and teaching to the test.

Steve N said...

(see my comment at top)

OK, let's try this one:

http://tinyurl.com/2exlxj

I'll work on the html issue when I have a little more time.

Cheers, Steve