Well, statistician I am not, but I’m glad to see that some common sense still applies in the absence of statistical analysis skills. Tom Mountain is full of it and it makes me furious that he and FOX and the righty blogs bitching about the anti-racist math are too damned lazy to look anything up. It’s too easy to be a lousy journalist and make trouble for the school district.
Posted by Kate on 02/17/05 at 08:37 AM
On further thought, I might point out that Mountain willfully ignores the inherent variability in the test itself. A score of X is actually an indication that the kid’s (unknown, unknowable) real achievement level is within some range; X is an estimate. With sharply defined categories, it’s possible that a kid appearing in, say, the “Needs to improve” category is actually “Proficient”; the groupings are also estimates.
Was there anything in the MCAS reports about, say, mean or median scores? Or standard deviations? Did the category definitions remain fixed across years? What was done to determine that the 2002 and 2004 tests scored out identically, that a 75 in 2002 meant “exactly” the same thing as a 75 in 2004?
Then there’s the whole ludicrous idea of using just two data-points to define a “trend”.
Arrant politicizing of (anti)-science.
...it’s possible that a kid appearing in, say, the “Needs to improve” category is actually “Proficient”
Well, yes. The lad is a prime example. In 4th grade he took the MCAS and found them extremely boring and tedious. He even admitted that he guessed at half the answers just to get it over with. Consequently, he barely scored in the Needs Improvement in the Math section and barely a Proficient in the English section. The principal was all in favor of setting up an Ed Plan for him so he’d do better on the next exam. I said something to the effect of Don’t Be Ridiculous.
In 10th grade (2004) he was very serious (almost in a panic) about the exam and ended up with upper end Advanced scores in both English and Math, with Math slightly edging over English.
He’s a bright kid (3.89 High School GPA, so far). But these tests don’t really mean much to the kids until they hit about mid-year in 9th grade. Then they start to panic because they know they have to do pretty well on the 10th grade exams in order to graduate.
I think it’s a little early to try figuring out trends for the MCAS anyway. They’ve only been around for 7 years, but only 4 of those years were really dependably measurable. The tests were a little iffy during the first three years. And the scoring system changed between 2002 and 2003. It was reported as average raw/scaled scores. Now it’s something called a Proficiency Index. That was changed between 2002 and 2003.
Only one class (2006) has taken the MCAS from 4th grade through 10th grade and that class didn’t have a 6th grade math exam in 2000. It’s still too new to judge effectiveness (or not) of various curricula—since we are now very definitely teaching to the tests—so Tom Mountain is still full of it.
Posted by Kate on 02/17/05 at 04:20 PM
Next entry: Furnace fixing day
Previous entry: Decisions, decisions
Using chi-square analysis and reasonable rounding of the cell counts, I can confirm your argument.
Overall, the 2002 and 2004 distributions for 6th grade are somewhat different (p=0.03), but the difference arises solely from the groups with disabilities (p=0.0004) and limited English (p=0.02). In regular ed students, the p-value for the 2002 and 2004 cohorts is p=0.31, nowhere near significant.
In other words, there was no significant change in the regular ed scores between 2002 and 2004.