Treatments and payback curves

[Epidemiology examples used by marketing statistician -- proceed with caution.]

In one sense, education research is like much of the research done by epidemiologists: the outcomes researchers are most interested in are complex and take years, even decades to fully express themselves. In both cases, we are generally forced to rely on relatively simple, short-term indirect metrics.

Treatments (at least treatments with significant effects) are likely to have some effect on the relationship between the outcome and the indirect metrics. For certain questions, you're OK as long as the relationship maintains the same direction; for other questions, though, these changes have a way of unexpectedly causing serious problems.

One of the issues that comes up at this point is how short-term your metric can be. You might be able to get a read on the cardiovascular impact of a blood pressure drug in a couple of weeks.  The analogous impact of a diet and exercise program might take a year to show up. Health researchers are well aware of these concerns; education researchers, particularly those influential with the reform movement, tend to be more nonchalant about the robustness of short-term indirect metrics. Given the trend toward deferring pedagogical and policy decisions to these metrics, that's a big concern.

I pointed out a case earlier where learning a different approach to multiplication put a student at a disadvantage in elementary school but had a significant pay-off in more advanced grades. On the other end of the spectrum, there are methods of covering material that improve test scores for a short period but which don't necessarily lead to retention.

Ideally, the benefits of an education are spread out over decades. It may not be practical to measure those benefits directly, but we can keep in mind the limitations of our proxies. This point will be coming up again in this thread.