In Study of Human-AI Tutoring with U.S. Seventh Graders, Human Tutors Enhanced the Benefits of AI Tutors
Findings Can Help Spot Learning Progress, Identify Students Who Need More Support
Human tutoring and computer-based AI tutors are widely recognized for their effectiveness in supporting learning. However, human tutoring is expensive and difficult to scale, and AI tutors vary widely in their ability to adapt to students’ academic and motivational needs. In a new study, a team of researchers evaluated a year-long virtual human-AI tutoring program during classroom use of AI tutors. Students tutored in the program outperformed students tutored by AI alone.
The study was conducted by researchers at Carnegie Mellon University, the University of Hong Kong, and Stanford University. It is published as a chapter in Artificial Intelligence in Education: Proceedings of the 26th International Conference on Artificial Intelligence in Education, which took place July 22-26 in Palermo, Italy.
“Our findings suggest that human tutors enhance the benefits of AI tutors, with gains increasing with time on task,” explains Lee Branstetter, professor of economics at Carnegie Mellon’s Heinz College, who coauthored the study. “In addition to replicating other studies over a longer period, we showed that time on task data are a valuable indicator of learning progress and a way to identify students who need added support to get the most out of AI tutoring.”
High-impact human tutoring is a structured form of instruction delivered one-on-one or in small groups that supplements students’ classroom experiences and targets specific learning goals. This type of tutoring has been shown to be effective in boosting students’ learning outcomes. But the high costs of one-on-one tutoring have spurred researchers to explore alternatives that can approximate its benefits.
One approach has been the use of AI tutors, which aim to replicate the benefits of human tutoring in a more scalable and cost-effective way. In human-AI tutoring, human tutors provide personalized instructional and motivational support, while AI tutors ensure consistent, adaptive feedback and hints.
“This research shows the promise of collaboration between AI and human tutors to magnify the impact of tutoring on student outcomes. Seeing these meaningful improvements play out in real-world classrooms over a school year is an encouraging sign,” said Jordan Gutterman, a graduate student at Carnegie Mellon’s Heinz College, who coauthored this study as part of CMU’s commitment to engaging students in frontier research.
In this study, researchers evaluated human-AI tutoring over one school year, extending the amount of time studied in prior work. Using year-long log data and standardized state tests, they compared more than 350 U.S. seventh-grade students who received human-AI tutoring in 2023-2024 with nearly 320 students who received AI-only tutoring the previous year.
Half the students in the study were female and half were male; 65% were Hispanic, 17% were Asian, 10% were African American, 6% were White, and the rest were of other races and ethnicities. In addition, 14% received special education services, 88% were from families with socioeconomic disadvantage, and 18% were English language learners.
The human tutors were undergraduate students from a U.S. university who had been trained on tutoring strategies to identify and address students’ needs. While students could access AI tutoring at any time, they also had a designated period to use it; during this time, the human-AI group received virtual tutoring via Zoom.
The human-AI group demonstrated significantly higher growth and was 0.36 grade levels ahead by year’s end, the study found. Although there was no overall difference in state test scores, researchers found a significant interaction between human-AI tutoring and time on task (i.e., AI tutor use). This suggests that human tutors enhance the benefits of AI tutors, with gains increasing with time on task.
“We found that relatively small doses of human support can meaningfully improve learning outcomes over time,” says Ashish Gurung, a postdoctoral researcher at Carnegie Mellon’s Human-Computer Interaction Institute, who coauthored the study. “Our findings lay a strong foundation for rethinking how human-AI collaboration can enhance both the learning experience for students and the overall effectiveness of AI tutors.”
The research was funded by the Learning Engineering Virtual Institute.
Summarized from an article in Artificial Intelligence in Education: Proceedings of the 26th International Conference on Artificial Intelligence in Education, "Human Tutoring Improves the Impact of AI Tutor Use on Learning Outcomes," by Gurung, A (Carnegie Mellon University), Lin, J (University of Hong Kong, Carnegie Mellon University), Gutterman, J (Carnegie Mellon University), Thomas, DR (Carnegie Mellon University), Houk, A (Carnegie Mellon University), Gupta, S (Carnegie Mellon University), Brunskill, E (Stanford University), Branstetter, L (Carnegie Mellon University), Aleven, V (Carnegie Mellon University), and Koedinger, K (Carnegie Mellon University). Copyright 2025 The Authors. All rights reserved.
About Heinz College of Information Systems and Public Policy
The Heinz College of Information Systems and Public Policy is home to two internationally recognized graduate-level institutions at Carnegie Mellon University: the School of Information Systems and Management and the School of Public Policy and Management. This unique colocation combined with its expertise in analytics set Heinz College apart in the areas of cybersecurity, health care, the future of work, smart cities, and arts & entertainment. In 2016, INFORMS named Heinz College the #1 academic program for Analytics Education. For more information, please visit www.heinz.cmu.edu.