Protocol for a Systematic Review with Meta-Analysis: Understanding Quality Characteristics of Edtech Interventions and Implementation for Disadvantaged Pupils

Abrami, P. C., Lysenko, L., & Borokhovski, E. (2020). The effects of ABRACADABRA on reading outcomes: An updated meta-analysis and landscape review of applied field research. Journal of Computer Assisted Learning, 36(3), 260–279. https://doi.org/10.1111/jcal.12417

Ahn, S., Ames, A. J., & Myers, N. D. (2012). A Review of Meta-Analyses in Education: Methodological Strengths and Weaknesses. Review of Educational Research, 82(4), 436–476. https://doi.org/10.3102/0034654312458162

Akçayır, G., & Akçayır, M. (2018). The flipped classroom: A review of its advantages and challenges. Computers & Education, 126, 334–345. https://doi.org/10.1016/j.compedu.2018.07.021

Alison, C. (2023). Guidance note on using implementation research in education. https://www.edu-links.org/sites/default/files/media/file/Guidance_Note_on_Using_Implementation_Research_in_Education.pdf

Archer, K., Savage, R., Sanghera-Sidhu, S., Wood, E., Gottardo, A., & Chen, V. (2014). Examining the effectiveness of technology use in classrooms: A tertiary meta-analysis. Computers & Education, 78, 140–149. https://doi.org/10.1016/j.compedu.2014.06.001

Bai̇Doo-Anu, D., & Owusu Ansah, L. (2023). Education in the Era of Generative Artificial Intelligence (AI): Understanding the Potential Benefits of ChatGPT in Promoting Teaching and Learning. Journal of AI, 7(1), 52–62. https://doi.org/10.61969/jai.1337500

Barzelay, M. (2007). Learning from Second-Hand Experience: Methodology for Extrapolation-Oriented Case Research. Governance, 20(3), 521–543. https://doi.org/10.1111/j.1468-0491.2007.00369.x

Bates, M. A., & Glennerster, R. (2017). The Generalizability Puzzle. Stanford Social Innovation Review. https://ssir.org/articles/entry/the_generalizability_puzzle

Belland, B. R. (2017). Instructional Scaffolding: Foundations and Evolving Definition. In B. R. Belland (Ed.), Instructional Scaffolding in STEM Education: Strategies and Efficacy Evidence (pp. 17–53). Springer International Publishing. https://doi.org/10.1007/978-3-319-02565-0_2

Belland, B. R., Walker, A. E., & Kim, N. J. (2017). A Bayesian Network Meta-Analysis to Synthesize the Influence of Contexts of Scaffolding Use on Cognitive Outcomes in STEM Education. Review of Educational Research, 87(6), 1042–1081. https://doi.org/10.3102/0034654317723009

Blei, D. M., & Lafferty, J. D. (2009). Topic models. In Text mining (pp. 101–124). Chapman and Hall/CRC. https://www.taylorfrancis.com/chapters/edit/10.1201/9781420059458-12/topic-models-david-blei-john-la%EF%AC%80erty

Blei, D. M., & Lafferty, J. D. (2009). Visualizing Topics with Multi-Word Expressions (arXiv:0907.1013). arXiv. http://arxiv.org/abs/0907.1013

Bold, T., Kimenyi, M., Mwabu, G., Ng’ang’a, A., & Sandefur, J. (2018). Experimental evidence on scaling up education reforms in Kenya. Journal of Public Economics, 168, 1–20. https://doi.org/10.1016/j.jpubeco.2018.08.007

Bond, M. (2020). Schools and Emergency Remote Education during the COVID-19 Pandemic: A Living Rapid Systematic Review. Asian Journal of Distance Education, 15(2), 191–247. https://eric.ed.gov/?id=EJ1285336

Bond, M., Buntins, K., Bedenlier, S., Zawacki-Richter, O., & Kerres, M. (2020). Mapping research in student engagement and educational technology in higher education: a systematic evidence map. International Journal of Educational Technology in Higher Education, 17(1), 2. https://doi.org/10.1186/s41239-019-0176-8

Cabı, E. (2018). The Impact of the Flipped Classroom Model on Students’ Academic Achievement. The International Review of Research in Open and Distributed Learning, 19(3). https://doi.org/10.19173/irrodl.v19i3.3482

Castro, F. G., Barrera, M., & Holleran Steiker, L. K. (2010). Issues and Challenges in the Design of Culturally Adapted Evidence-Based Interventions. Annual Review of Clinical Psychology, 6, 213–239. https://doi.org/10.1146/annurev-clinpsy-033109-132032

Chen, M.-H., Tseng, W.-T., & Hsiao, T.-Y. (2018). The Effectiveness of Digital Game-Based Vocabulary Learning: A Framework-Based View of Meta-Analysis. British Journal of Educational Technology, 49(1), 69–77. https://doi.org/10.1111/bjet.12526

Cheung, A. C. K., & Slavin, R. E. (2013). The effectiveness of educational technology applications for enhancing mathematics achievement in K-12 classrooms: A meta-analysis. Educational Research Review, 9, 88–113. https://doi.org/10.1016/j.edurev.2013.01.001

Clark, D. B., Tanner-Smith, E. E., & Killingsworth, S. S. (2016). Digital Games, Design, and Learning: A Systematic Review and Meta-Analysis. Review of Educational Research, 86(1), 79–122. https://doi.org/10.3102/0034654315582065

CooperGibson Research. (2022). Implementation of education technology in schools and colleges. Department for Education.

D’Angelo, C., Rutstein, D., Harris, C., Haertel, G., Bernard, R., & Borokhovski, E. (2014). Simulations for STEM Learning: Systematic Review and Meta-Analysis. SRI Education. https://www.sri.com/wp-content/uploads/2021/12/simulations-for-stem-learning-brief.pdf

Education Endowment Foundation, & Durham University. (2022). EEF Evidence Database Coding Guide – Main Data Extraction (Version 3). https://d2tic4wvo1iusb.cloudfront.net/production/documents/toolkit/MDE_CodingGuide_V3_March2022-1.pdf

EPPI Centre. (2003). Core Keywording Strategy. https://eppi.ioe.ac.uk/CMS/Portals/0/PDF%20reviews%20and%20summaries/EPPI_Keyword_strategy.pdf

Evans, D. K., & Popova, A. (2016). What Really Works to Improve Learning in Developing Countries? An Analysis of Divergent Findings in Systematic Reviews. The World Bank Research Observer, 31(2), 242–270. https://doi.org/10.1093/wbro/lkw004

Gerard, L., Matuk, C., McElhaney, K., & Linn, M. C. (2015). Automated, adaptive guidance for K–12 education. Educational Research Review, 15, 41–58. https://doi.org/10.1016/j.edurev.2015.04.001

Glewwe, P., & Muralidharan, K. (2016). Chapter 10 - Improving Education Outcomes in Developing Countries: Evidence, Knowledge Gaps, and Policy Implications. In E. A. Hanushek, S. Machin, & L. Woessmann (Eds.), Handbook of the Economics of Education (Vol. 5, pp. 653–743). Elsevier. https://doi.org/10.1016/B978-0-444-63459-7.00010-5

Hariton, E., & Locascio, J. J. (2018). Randomised controlled trials – the gold standard for effectiveness research. International Journal of Obstetrics & Gynaecology, 125(13), 1716–1716. https://doi.org/10.1111/1471-0528.15199

Harn, B., Parisi, D., & Stoolmiller, M. (2013). Balancing Fidelity with Flexibility and Fit: What Do We Really Know about Fidelity of Implementation in Schools? Exceptional Children, 79(3), 181–193. https://doi.org/10.1177/001440291307900204

Haßler, B. (2021). A mixed-methods evidence assessment for emergent fields of knowledge (Working Paper No. 24). EdTech Hub. https://doi.org/10.53832/edtechhub.0003

Haßler, B., Adam, T., Allier-Gagneur, Z., Blower, T., Brugha, M., Damani, K., Hennessy, S., Hollow, D., Jordan, K., Martin, K., Megha-Bongnkar, G., Murphy, M., & Walker, H. (2021). Methodology for literature reviews (Working Paper No. 10). EdTech Hub. https://doi.org/10.5281/zenodo.4557508

Haßler, B., El-Serafy, Y., Khalayleh, A., & McBurnie, C. (2021). Syṡtemic Mixed-Methods Research — a conceptual framework for scalable EdTech research. EdTech Hub. https://doi.org/10.53832/edtechhub.0001

Haßler, B., Haseloff, G., Adam, T., Akoojee, S., Allier-Gagneur, Z., Ayika, S., Bahloul, K., Kigwilu, P. C., Costa, D. D., Damani, K., Gordon, R., Idris, A., Iseje, F., Jjuuko, R., Kagambèga, A., Khalayleh, A., Konayuma, G., Kunwufine, D., Langat, K., … Winkler, E. (2020). Technical and Vocational Education and Training in Sub-Saharan Africa: A Systematic Review of the Research Landscape (1st ed.). VET Repository, Bundesinstitut für Berufsbildung, Bonn, Germany. https://doi.org/10.5281/zenodo.4288831

Haßler, B., McBurnie, C., Walker, H., Klune, C., & Huntington, B. (2024). Keyword Inventory for a systematic literature review with meta-analysis: Understanding Quality Characteristics of EdTech Interventions and Implementation for Disadvantaged Pupils [Other]. Open Development & Education. https://doi.org/10.53832/opendeved.1075

Higgins, S., Xiao, Z., & Katsipataki, M. (2012). The Impact of Digital Technology on Learning: A Summary for the Education Endowment Foundation. Full Report. Education Endowment Foundation. https://eric.ed.gov/?id=ED612174

Holmes, W., & Tuomi, I. (2022). State of the art and practice in AI in education. European Journal of Education, 57(4), 542–570. https://doi.org/10.1111/ejed.12533

Hong, Q. N., Fàbregues, S., Bartlett, G., Boardman, F. K., Cargo, M., Dagenais, P., Gagnon, M. P., Griffiths, F., Nicolau, B., O’Cathain, A., Rousseau, M., Vedel, I., & Pluye, P. (2018). The Mixed Methods Appraisal Tool (MMAT) version 2018 for information professionals and researchers. Education for Information, 34(4), 285–291. https://doi.org/10.3233/efi-180221

Illari, P. M., & Williamson, J. (2012). What is a mechanism? Thinking about mechanisms across the sciences. European Journal for Philosophy of Science, 2(1), 119–135. https://doi.org/10.1007/s13194-011-0038-2

Karboul, A. (2020, December 4). COVID-19 put 1.6 billion children out of school. Here’s how to upgrade education post-pandemic. World Economic Forum. https://www.weforum.org/agenda/2020/12/covid19-education-innovation-outcomes/

Kerwin, J. T., & Thornton, R. L. (2021). Making the Grade: The Sensitivity of Education Program Effectiveness to Input Choices and Outcome Measures. The Review of Economics and Statistics, 103(2), 251–264. https://doi.org/10.1162/rest_a_00911

Klune, C., Huntington, B., Touiyate, A., Kay, J., Zaman, M., Mansour, H., & Haßer, B. (2024). Scaling AI-driven evidence synthesis for targeting new education research: Accelerating progress towards SDG4 [Extended abstract submitted to the What Works Hub for Global Education Conference 2024]. Open Development & Education, Education Endowment Foundation. https://doi.org/10.53832/opendeved.1132

Kulik, J. A., & Fletcher, J. D. (2016). Effectiveness of Intelligent Tutoring Systems: A Meta-Analytic Review. Review of Educational Research, 86(1), 42–78. https://doi.org/10.3102/0034654315581420

Kunkel, A. (2015). The Effects of Computer-Assisted Instruction in Reading: A Meta-Analysis [University of Minnesota]. http://conservancy.umn.edu/handle/11299/175221

Leviton, L. C. (2017). Generalizing about Public Health Interventions: A Mixed-Methods Approach to External Validity. Annual Review of Public Health, 38(1), 371–391. https://doi.org/10.1146/annurev-publhealth-031816-044509

Lewin, C., Smith, A., Morris, S., & Craig, E. (2019). Using Digital Technology to Improve Learning: Evidence Review. Education Endowment Foundation. https://d2tic4wvo1iusb.cloudfront.net/production/documents/guidance/Using_Digital_Technology_to_Improve_learning_Evidence_Review.pdf?v=1698320282

Lin, H. (2014). Establishing an Empirical Link between Computer-Mediated Communication (CMC) and SLA: A Meta-Analysis of the Research. Language Learning & Technology, 18(3), 120–147.

Ludwig, J., Kling, J. R., & Mullainathan, S. (2011). Mechanism Experiments and Policy Evaluations. Journal of Economic Perspectives, 25(3), 17–38. https://doi.org/10.1257/jep.25.3.17

Major, L., Francis, G. A., & Tsapali, M. (2021). The effectiveness of technology-supported personalised learning in low- and middle-income countries: A meta-analysis. British Journal of Educational Technology, 52(5), 1935–1964. https://doi.org/10.1111/bjet.13116

Mbiti, I., Muralidharan, K., Romero, M., Schipper, Y., Manda, C., & Rajani, R. (2019). Inputs, Incentives, and Complementarities in Education: Experimental Evidence from Tanzania*. The Quarterly Journal of Economics, 134(3), 1627–1673. https://doi.org/10.1093/qje/qjz010

McNally, S., Ruiz-Valenzuela, J., & Rolfe, H. (2016). ABRA: Online Reading Support. Evaluation Report and Executive Summary. Education Endowment Foundation. https://eric.ed.gov/?id=ED581099

McNeish, D. (2017). Challenging Conventional Wisdom for Multivariate Statistical Models With Small Samples. Review of Educational Research, 87(6), 1117–1151. https://doi.org/10.3102/0034654317727727

Means, B., Toyama, Y., Murphy, R., & Baki, M. (2013). The Effectiveness of Online and Blended Learning: A Meta-Analysis of the Empirical Literature. Teachers College Record: The Voice of Scholarship in Education, 115(3), 1–47. https://doi.org/10.1177/016146811311500307

Moeyaert, M., Ugille, M., Natasha Beretvas, S., Ferron, J., Bunuan, R., & Van Den Noortgate, W. (2017). Methods for dealing with multiple outcomes in meta-analysis: a comparison between averaging effect sizes, robust variance estimation and multilevel meta-analysis. International Journal of Social Research Methodology, 20(6), 559–572. https://doi.org/10.1080/13645579.2016.1252189

Moore, G. F., Audrey, S., Barker, M., Bond, L., Bonell, C., Hardeman, W., Moore, L., O’Cathain, A., Tinati, T., Wight, D., & Baird, J. (2015). Process evaluation of complex interventions: Medical Research Council guidance. BMJ, 350, h1258. https://doi.org/10.1136/bmj.h1258

ONS. (2023). A level and other 16 to 18 results, Academic year 2022/23. https://explore-education-statistics.service.gov.uk/find-statistics/a-level-and-other-16-to-18-results/2022-23#dataBlock-5c6d58a3-56e3-4ffe-6c95-08dbd1502a35-tables

ONS. (2023). Key stage 1 and phonics screening check attainment, Academic year 2022/23. https://explore-education-statistics.service.gov.uk/find-statistics/key-stage-1-and-phonics-screening-check-attainment/2022-23

ONS. (2023). Key stage 2 attainment, Academic year 2022/23. https://explore-education-statistics.service.gov.uk/find-statistics/key-stage-2-attainment/2022-23

ONS. (2023). Key stage 4 performance, Academic year 2022/23. https://explore-education-statistics.service.gov.uk/find-statistics/key-stage-4-performance-revised

Outhwaite, L. A., Early, E., Herodotou, C., & Van Herwegen, J. (2023). Understanding how educational maths apps can enhance learning: A content analysis and qualitative comparative analysis. British Journal of Educational Technology, 54(5), 1292–1313. https://doi.org/10.1111/bjet.13339

Outhwaite, L. A., Faulder, M., Gulliford, A., & Pitchford, N. J. (2019). Raising early achievement in math with interactive apps: A randomized control trial. Journal of Educational Psychology, 111(2), 284. https://doi.org/10.1037/edu0000286

Outhwaite, L. A., Gulliford, A., & Pitchford, N. J. (2017). Closing the gap: efficacy of a tablet intervention to support the development of early mathematical skills in UK primary school children. Computers & Education, 108, 43–58. https://doi.org/10.1016/j.compedu.2017.01.011

Outhwaite, L., Ang, L., Herbert, E., Sumner, E., & Van Herwegen, J. (2023). Technology and Learning for Early Childhood and Primary Education [Report]. UNESCO. https://unesdoc.unesco.org/ark:/48223/pf0000386108.locale=en

Pawson, R., Greenhalgh, T., Harvey, G., & Walshe, K. (2005). Realist review – a new method of systematic review designed for complex policy interventions. Journal of Health Services Research & Policy, 10 Suppl 1, 21–34. https://doi.org/10.1258/1355819054308530

Piper, B., Jepkemei, E., Kwayumba, D., & Kibukho, K. (2015). Kenya’s ICT Policy in Practice: The Effectiveness of Tablets and E-readers in Improving Student Outcomes. FIRE: Forum for International Research in Education, 2(1). https://doi.org/10.18275/fire201502011025

Pokhrel, S., & Chhetri, R. (2021). A Literature Review on Impact of COVID-19 Pandemic on Teaching and Learning. Higher Education for the Future, 8(1), 133–141. https://doi.org/10.1177/2347631120983481

Pritchett, L., & Sandefur, J. (2015). Learning from Experiments When Context Matters. American Economic Review, 105(5), 471–475. https://doi.org/10.1257/aer.p20151016

Pustejovsky, J. E., & Tipton, E. (2021). Meta-analysis with robust variance estimation: Expanding the range of working models. Prevention Science, No Pagination Specified-No Pagination Specified. https://doi.org/10.1007/s11121-021-01246-3

Scammacca, N., Roberts, G., & Stuebing, K. K. (2014). Meta-Analysis With Complex Research Designs: Dealing With Dependence From Multiple Measures and Multiple Group Comparisons. Review of Educational Research, 84(3), 328–364. https://doi.org/10.3102/0034654313500826

Sharpe, D., & Poets, S. (2020). Meta-analysis as a response to the replication crisis. Canadian Psychology/Psychologie Canadienne, 61(4), 377. https://doi.org/10.1037/cap0000215

Sharples, J., Albers, B., Fraser, S., & Kime, S. (2019). Putting Evidence to Work: A School’s Guide to Implementation. https://d2tic4wvo1iusb.cloudfront.net/eef-guidance-reports/implementation/EEF-Active-Ingredients-Summary.pdf?v=1635355218

Sims, S., & Fletcher-Wood, H. (2018). Characteristics of effective teacher professional development: what we know, what we don’t, how we can find out.

Sims, S., Fletcher-Wood, H., O’Mara, A., Cottingham, S., Stansfield, C., Herwegen, J. V., & Anders, J. (2021). What are the Characteristics of Effective Teacher Professional Development? A Systematic Review and Meta-analysis.

Sokolowski, A., Li, Y., & Willson, V. (2015). The effects of using exploratory computerized environments in grades 1 to 8 mathematics: a meta-analysis of research. International Journal of STEM Education, 2(1), 8. https://doi.org/10.1186/s40594-015-0022-z

Steenbergen-Hu, S., & Cooper, H. (2013). A meta-analysis of the effectiveness of intelligent tutoring systems on K–12 students’ mathematical learning. Journal of Educational Psychology, 105(4), 970–987. https://doi.org/10.1037/a0032447

Sung, Y.-T., Chang, K.-E., & Liu, T.-C. (2016). The effects of integrating mobile devices with teaching and learning on students’ learning performance: A meta-analysis and research synthesis. Computers & Education, 94, 252–275. https://doi.org/10.1016/j.compedu.2015.11.008

Takacs, Z. K., Swart, E. K., & Bus, A. G. (2015). Benefits and Pitfalls of Multimedia and Interactive Features in Technology-Enhanced Storybooks: A Meta-Analysis. Review of Educational Research, 85(4), 698–739. https://doi.org/10.3102/0034654314566989

Takacs, Z. K., Swart, E. K., & Bus, A. G. (2015). Benefits and Pitfalls of Multimedia and Interactive Features in Technology-Enhanced Storybooks: A Meta-Analysis. Review of Educational Research, 85(4), 698–739. https://doi.org/10.3102/0034654314566989

Tipton, E., Pustejovsky, J. E., & Ahmadi, H. (2019). A history of meta‐regression: Technical, conceptual, and practical developments between 1974 and 2018. Research Synthesis Methods, 10(2), 161–179. https://doi.org/10.1002/jrsm.1338

Tipton, E., Pustejovsky, J. E., & Ahmadi, H. (2019). Current practices in meta‐regression in psychology, education, and medicine. Research Synthesis Methods, 10(2), 180–194. https://doi.org/10.1002/jrsm.1339

UK Government. (2023). Pupil premium: overview. GOV.UK. https://www.gov.uk/government/publications/pupil-premium/pupil-premium

UK Government. (2019). Realising the potential of technology in education: A strategy for education providers and the technology industry. https://assets.publishing.service.gov.uk/media/5ca360bee5274a77d479facc/DfE-Education_Technology_Strategy.pdf

UNCTAD. (2023). Technology and Innovation Report. https://unctad.org/system/files/official-document/tir2023_en.pdf

Van den Noortgate, W., López-López, J. A., Marín-Martínez, F., & Sánchez-Meca, J. (2015). Meta-analysis of multiple outcomes: A multilevel approach. Behavior Research Methods, 47, 1274–1294. https://doi.org/10.3758/s13428-014-0527-2

Verbruggen, S., Depaepe, F., & Torbeyns, J. (2021). Effectiveness of educational technology in early mathematics education: A systematic literature review. International Journal of Child-Computer Interaction, 27, 100220. https://doi.org/10.1016/j.ijcci.2020.100220

Vicentini, L., Day, L., Gill, V., Lillis, J., Komers, S., & Olausson, N. (2022). Future opportunities for education technology in England. Department for Education.

Vivalt, E. (2020). How Much Can We Generalize From Impact Evaluations? Journal of the European Economic Association, 18(6), 3045–3089. https://doi.org/10.1093/jeea/jvaa019

Walker, H., Moustafa, N., & Haßler, B. (2023). EdTech Impact in Education – An Analysis Prepared for the Education Endowment Foundation. OpenDevEd. https://doi.org/10.53832/opendeved.0294

West, M. (2023). An ed-tech tragedy? Educational technologies and school closures in the time of COVID-19 (K. Davidson & R. Yaghmour, Eds.). UNESCO. https://unesdoc.unesco.org/ark:/48223/pf0000386701

Williams, M. J. (2020). External Validity and Policy Adaptation: From Impact Evaluation to Policy Design. The World Bank Research Observer, 35(2), 158–191. https://doi.org/10.1093/wbro/lky010

Wouters, P., Van Nimwegen, C., Van Oostendorp, H., & Van Der Spek, E. D. (2013). A meta-analysis of the cognitive and motivational effects of serious games. Journal of Educational Psychology, 105(2), 249–265. https://doi.org/10.1037/a0031311

Xiao, Y., & Watson, M. (2017). Guidance on Conducting a Systematic Literature Review. Journal of Planning Education and Research, 39, 0739456X1772397. https://doi.org/10.1177/0739456x17723971

Zhai, X., Chu, X., Chai, C. S., Jong, M. S. Y., Istenic, A., Spector, M., Liu, J.-B., Yuan, J., & Li, Y. (2021). A Review of Artificial Intelligence (AI) in Education from 2010 to 2020. Complexity, 2021, 1–18. https://doi.org/10.1155/2021/8812542

Zheng, B., Warschauer, M., Lin, C.-H., & Chang, C. (2016). Learning in One-to-One Laptop Environments: A Meta-Analysis and Research Synthesis. Review of Educational Research, 86(4), 1052–1084. https://doi.org/10.3102/0034654316628645

Zheng, L. (2016). The effectiveness of self-regulated learning scaffolds on academic performance in computer-based learning environments: a meta-analysis. Asia Pacific Education Review, 17(2), 187–202. https://doi.org/10.1007/s12564-016-9426-9

Zheng, L., Li, X., Tian, L., & Cui, P. (2018). The effectiveness of integrating mobile devices with inquiry-based learning on students’ learning achievements: a meta-analysis. International Journal of Mobile Learning and Organisation, 12(1), 77–95. https://doi.org/10.1504/IJMLO.2018.089238