A randomised algebra trial found gains only in its second year

The classroom note

A school buying an algebra program in year one has almost no way to see whether it worked before deciding whether to keep paying for it. RAND's evaluation of Cognitive Tutor Algebra I, published in Educational Evaluation and Policy Analysis in June 2014 and posted as an external publication on RAND's own site, put that timing problem to a direct test. Across 147 school sites in seven states, schools were 'matched into similar pairs and randomly assigned to either continue with the current algebra curriculum for 2 years or to adopt' the software, which blends textbook-based instruction with computer-based, self-paced practice.

What the evidence says

The design was a matched-pair, school-level randomised trial, with two parallel experiments for middle and high schools because, as RAND's own research brief explains, students taking algebra early tend to be higher-achieving. The brief gives the sample precisely: 73 high schools and 74 middle schools across 51 districts, 'nearly 18,700 students' in the high-school arm and 'nearly 6,800 students' in the middle-school arm. The result was time-dependent: 'in the first year of implementation, posttest scores for the intervention group schools did not differ significantly from those for the control group schools,' but in year two high schools using the software showed a significant effect, 'equivalent to moving a student from the 50th percentile on the algebra posttest to the 58th percentile.' A later addendum confirmed 'a significant positive effect of .21 standard deviation units for high school students in the second year.' The middle-school effect was similar in size but not statistically significant.

The implementation question

RAND's brief also priced the comparison, estimating the tutoring curriculum 'to cost about $97 per student, compared with about $28 per student' for the existing textbook curricula, including infrastructure and training in that estimate. The researchers tested two explanations for the year-one gap: that returning teachers had gained experience with the software, which their own analysis of teacher cohorts did not confirm, and that all teachers by year two had learned to adapt the curriculum, for which their teacher-reported data gave 'some support.'

What holds and what fails

What holds, stated plainly by RAND itself, is a real effect for high schools in year two of ordinary, non-boutique implementation. What fails is any claim of quick results: the brief's own guidance to schools is that 'decisionmakers should be prepared for the possibility that CTAI will not produce positive effects immediately,' and middle-school results remain, in the researchers' own words, 'far less certain.'

• Is a procurement decision being judged on first-year data that this trial found uninformative?
• Does the quoted cost per student include infrastructure and training, or software alone?
• Is the age group in question the one the trial found a significant effect for?

Nearly a decade before generative AI tutors, RAND's own findings already establish that a real, randomised, at-scale effect can take a full year to appear and can cost more than three times an existing curriculum — a caution worth carrying into any pilot judged after a single term.