Proceedings of the National Academy of Sciences


Scaling Up Behavioral Science Interventions in Online Education.

Rene Kizilcec , Justin Reich, Mike Yeomans, Glenn. Lopez, Emma Brunskill, Christoph Dann, Selen Turkay, Joseph Jay Williams, Dustin Tingley


Online education is rapidly expanding in response to rising demand for higher and continuing education, but many online students struggle to achieve their educational goals. Several behavioral science interventions have shown promise in raising student persistence and completion rates in a handful of courses, but evidence of their effectiveness across diverse educational contexts is limited. In this study, we test a set of established interventions over two-and-a-half years, with ¼ million students, from nearly every country, across 247 online courses offered by Harvard, MIT, and Stanford. We hypothesized that the interventions would produce medium-to-large effects as in prior studies, but this is not supported by our results. Instead, using an iterative scientific process of cyclically pre-registering new hypotheses inbetween waves of data collection, we identify individual, contextual, and temporal conditions under which the interventions benefit students. Self-regulation interventions raised student engagement in the first few weeks but not final completion rates. Value-relevance interventions raised completion rates in developing countries to close the global achievement gap, but only in courses with a global gap. We find minimal evidence that state-of-the-art machine learning methods can forecast the occurrence of a global gap or learn effective individualized intervention policies. Our findings demonstrate how scaling behavioral science interventions across various online learning contexts can reduce their average effectiveness by an order of magnitude and the promise of iterative scientific investigations to uncover what works where for whom.


Kizilcec, R., Reich, J., Yeomans, M., Dann, C, Brunskill, M., Lopez, G., Turkay, S., Williams, J.J., Tingley, D. (2020) Scaling Up Behavioral Science Interventions in Online Education. Proceedings of the National Academy of Sciences.

Links to Research

More Research

Simulating more Equitable Discussions: Using Teacher Moments And Practice Based Teacher Education In Mathematical Professional Learning

Let’s hit the refresh button (a couple of times): Reimagining math curriculum and teacher learning to broaden participation in the math of the future

The power to change the equation: Mathematics teacher learning reimagined