Heinz College student develops app to help evaluate medical residents
By Emma Folts
Last spring, Dr. Robert Sobehart, Dr. Aleta Mizner and Dr. Shiv Dua sought to improve the way physicians at Allegheny Health Network’s (AHN) Allegheny General Hospital (AGH) evaluate medical residents in the emergency medicine residency program. They knew the already busy physicians, who see up to 30 patients each shift, lacked a straightforward and easy-to-use system.
“We’ve used paper and QR-code evaluation systems in the past, both of which placed a significant burden on the attending physician,” Sobehart said. But the feedback is important for residents, who are training to become emergency medicine physicians.
They shared their evaluation problem with Rema Padman, Trustees Professor of Management Science and Healthcare Informatics at Heinz College. Padman sketched out a potential project and solution. Through a summer internship for one and a fall capstone for a team, Heinz College students could create a working application. The doctors were on board, so Padman shared news of the opportunity in one of her graduate classes.
Heinz College graduate student Siva Komaragiri (MSHCA ‘25) landed the summer internship last year and led the creation of the mobile and web app to solve the challenge.
The Emergency Medicine Milestone Application, known as EMMA, prompts a physician and resident to answer four brief questions together after each shift to assess the resident’s competency in a given area. The app ensures residents receive real-time feedback while standardizing the information physicians can use to complete required evaluations. The project was funded by a grant from the Academic Affairs Committee of US Acute Care Solutions.
The app could improve patient care, Komaragiri said. Before the app’s implementation, in July 2024, physicians in the program completed 19% of their post-shift assessments. The average completion rate jumped to 75% between September and December 2024, after EMMA’s implementation. Right now, about 25 attending physicians and 49 residents at AGHl are using the app.
In the fall 2024 semester, Komaragiri collaborated with six other graduate students on the project’s capstone component, with the team seeking to enhance the app’s features and robustness. Komaragiri worked alongside Gracie Siu (MSHCA ‘25), Katie Burgess (MISM ‘24), Aditya Deshmukh (MSHCA ‘25), Nirbhay Bagmar (MISM ‘24), XinQiao Luo (MSPPM-DA ‘24), and Michelle Wang (MSPPM-DA ‘25).
“Working to solve this problem seemed very interesting and impactful. It would help residents out, which would lead to helping patients out. It was a really good experience,” Komaragiri said.
How the app works and why it matters
Evaluations are important to the Accreditation Council for Graduate Medical Education (ACGME), the accrediting body for the emergency medicine program at AGH. The council requires that physicians evaluate a resident's performance at the end of each rotation. Rotations are made up of 18 to 21 shifts over the course of four weeks.
The capstone team surveyed 16 hospitals across the country and found that most are using paper-based systems or spreadsheets to assess residents, and all experienced the same challenges in doing so.
“We wanted to build a solution that could simplify this whole process,” Komaragiri said. “Oddly enough, it seems like a relatively simple problem to solve, but it isn’t. There are solutions out there, but they are still painstaking for physicians to complete.”
With EMMA, residents and physicians receive a notification after each shift instructing them to complete an evaluation. Together, they will answer two true-or-false questions, provide brief written responses and rank the resident’s performance, from novice to expert, in an assigned topic area. If the resident or physician does not complete the evaluation, the app will send a notification six hours following the shift.
The app uses a weighted system to randomly assign the topic area of a post-shift evaluation. For example, emergency stabilization — which includes how well a resident performs CPR trauma resuscitation — is more relevant to those in their second and third year of residency. The app gives a higher weight to that topic for second- and third-year residents than first-year residents.
EMMA also uses a natural language processing model to summarize the comments each physician has given residents. This not only allows residents to compare their performance to their cohort but also helps physicians generate reports that can be used for the ACGME’s mandatory evaluations.
Dua, who completed his residency training in 2018, said the app helps the hospital evolve to serve a new generation of learners. When he was a resident, he equated receiving periodic feedback from his physicians meant he was doing well. Today’s residents desire more frequent communication, he said.
The app also helps physicians more effectively assess residents, Dua said.
“Sometimes it's easier to have a more qualitative understanding of the individual, but there’s also a quantitative process that needs to be met. Now, we have specific prompts to catch people when they need to make improvements before they graduate,” Dua said.
By working together, the capstone team at Heinz College and AGH have been able to create something “pretty special,” Sobehart said.
“We're getting much better data on the residents, and we can help them very specifically improve in areas where they have true deficiencies. It makes our feedback and evaluation process much more specific and timely,” he continued.
Creating the app
During his internship, from May 2024 to August 2024, Komaragiri frequently visited AGH and met with physicians. The visits helped him identify the barriers physicians faced in completing post-shift assessments and the ways the app could help. Much of what Komaragiri learned built upon his coursework at Heinz College, he said.
By the end of his internship, Komaragiri had created a full-stack app, which has a frontend and a backend, that Sobehart, Mizner and Dua could test. In the fall 2024 semester, the capstone team implemented the app’s natural language processing model and created features that allow physicians to generate reports, add and remove users and create topics for evaluations. Their work on the app’s backend continued into the spring 2025 semester.
“This project is illustrative of what an integrative capstone experience is about — it’s about working in teams with a client organization on a complex real-world problem, with people who have different skill sets and perspectives on problem-solving,” Padman said.
She continued: “They all came together to work with clinicians, integrating all they learned throughout their respective graduate programs to create a technology product that worked. The application was thoughtfully designed and implemented to streamline the workflows of attending physicians and residents in a busy emergency department setting and also enhance the learning objectives of the residency training program.”
Throughout the process, the team learned several valuable lessons, Komaragiri said.
Because several physicians and residents were already using EMMA, the students had to ensure that updates to the app did not disrupt evaluations. They had to be realistic about the project’s timeline, as they had to build the user and application program interfaces from scratch. Komaragiri also learned how to manage a large team.
Despite the challenges, Komaragiri found the work fulfilling.
“The best part is that the clients came to Heinz every single week to get progress updates and provide any help that they possibly could. They were super helpful. They were equally excited about the project and wanted to see it grow,” Komaragiri said.
What comes next
The app has garnered interest within other departments at AGH and has received positive feedback at conferences, Dua said. Going forward, Dua, Sobehart and Mizner would like to conduct a validation study, determining whether the app helps other residencies within AHN.
In the meantime, Komaragiri aims to refine the app’s admin panel functionalities so that physicians can submit reports to ACGME without needing to extract the file and correct any typos. He also would like to incorporate predictive analytics that could identify and provide educational resources to residents struggling with a topic.
He’s interested in incorporating more artificial intelligence-driven features but is still determining which large language models to use.
“The possibilities are truly endless at this point. We have a bunch of ideas on how best to proceed and what else to incorporate, and that's amazing,” Komaragiri said.
Komaragiri chose Heinz College for his graduate studies because of the MSHCA program’s unique curriculum. Through his internship and capstone project, he was able to see how he could apply his coursework to solve real-world problems. After he graduates this spring, he plans to enter the healthcare field and would like to continue working for AHN.
“Helping patients brings me happiness. As long as my job fulfills this criteria, I am happy to do it,” Komaragiri said.