.comment-link {margin-left:.6em;}

Ed Knows Policy

EKP -- a local (Washington, DC) and national blog about education policy, politics, and research.

Who is Ed Researcher?

Wednesday, July 05, 2006

More bad news for National Board

[Update: The National Council on Teacher Quality says in their latest newsletter that: "Except for some alert reporter at the Hampton Roads Daily Press, nary a mention of this study can be found anywhere." Gee, thanks, NCTQ, for plagiarizing my title and missing the fact that I scooped the HRDP by 5 days. I guess blogs don't count.]


The National Board for Professional Teaching Standards (NBPTS) can't catch a break. They put out an RFP in 2002 to let a thousand dozen (research) flowers bloom and they got a pile of, um, a pile of studies from North Carolina, mostly, that were riddled with methodological weaknesses, but consistent in their story: National Board certified teachers are probably a hair better than non-certified teachers, but otherwise indistinguishable.

Now another study (pdf) by researchers at one of the U.S. Department of Education's regional labs (SERVE) delivers a disappointing message: National Board certification had "no clear pattern of effects on student achievement based on whether the teacher was Board certified."
(continued below)

As with the previous studies, I'm not sure that this one is on such solid ground. Like the ones before it, the SERVE study compares test scores for Board-certified with non-Board certified teachers and relies on a set of easily obtained control variables to remove selection bias. (Actually, the report is candid about how even basic control variables are not so easily obtained).

Studies like this have to worry about selection bias because: (a) those who seek National Board certification are not a random sample of the population, so we're not sure if we're getting the selection/certification effect or the training/value added effect of the certification process; and (b) teachers are not randomly sorted into their districts, schools, and communities, so we don't know the direction of causality. Principals may explicitly assign different types of students to NBCTs. Their Board certification may change where they teach, to be in harder or easier situations. Controlling for prior test scores (or end of grade exam scores), race, and free lunch participation seems like a a small bandaid on a gushing, oozing sore of a methodological problem.

The analysis actually has two parts even though I focused on just the first one. The first part is the comparison of test scores between NBCTs and non-NBCTs. The second part tries to pick off the most and least effective comparison teachers and do more in-depth study of how and why they differ from NBCTs. But the authors admit that they got a small, self-selected sample of volunteers for the second part, so it's hard to take much away from it. In fact, the recommendations at the end of the report boil down to: do more research and be more careful about interpreting existing research.

Amen.

8 Comments:

Anonymous Ben said...

What sort of methodology would you accept as credible to determine the relative value of National Board teaching certification? Or is it truly unknowable?

2:14 PM  
Blogger Ed Researcher said...

A good study would be one that meets the standards of the What Works Clearinghouse: a randomized experiment or a well-justified quasi-experiment.

It is possible to compare teachers within a school using random assignment of students to classrooms. See here for example. Natural experiments like student assignment lotteries also provide opportunties to net out selection bias.

Just sticking some variables in a regression, however, is not convincing, even if you throw around terms like "HLM" and "random effects".

Sure, if you don't have anything else, then regressions of post-test on pre-test with some student background measures is useful, but National Board has been around for a long time and they set out to sponsor new research that would settle the debate. So why didn't they invest their research dollars in more rigorous designs? WHy did they stress quantity of research over quality?

2:37 PM  
Blogger Ed Researcher said...

Oops. Bad link to the What Works Clearinghouse. Use this one instead.

3:06 PM  
Blogger Michele at AFT said...

I have only had time to skim this, but it looks like there were only 25 teachers in the NBCT group-- is that right? If so, I would be hesitant to draw ANY conclusions about the student achievement effects.

4:33 PM  
Blogger Ed Researcher said...

Michele,
Sample size was 307 teachers in 3 districts. 25 of the 307 were NBCTs.

That should probably be fine for what they were trying to do. The fact that the NBC and comparison sample were drawn from the same districts, schools, and even grade levels means you don't need as large a sample as if you were making comparisons across schools, districts, and grades.

3:23 PM  
Anonymous Anonymous said...

肿瘤 癌症 中国肿瘤网 肺癌 肺癌症状 肺癌治疗 胃癌 胃癌症状 胃癌治疗 肝癌 肝癌症状 肝癌治疗 食道癌 胰腺癌 贲门癌 甲状腺癌 直肠癌 结肠癌 宫颈癌 卵巢癌 乳腺癌 肾癌 前列腺癌 子宫肌瘤

3:33 AM  
Anonymous Anonymous said...

Who knows where to download XRumer 5.0 Palladium?
Help, please. All recommend this program to effectively advertise on the Internet, this is the best program!

5:14 AM  
Anonymous Anonymous said...

Some may feel squeamish about eating it, but rabbit has a fan base that grows as cooks discover how easy they are to raise — and how good the meat tastes.

11:36 AM  

Post a Comment

Links to this post:

Create a Link

<< Home