By Emma Folts
Upgrade your data analytics, cybersecurity, IT, and management skills from anywhere in the world.
An interdisciplinary curriculum highlighting data-driven management strategies and technological expertise prepares you to lead digital transformation in your organization.
The project quickly became a collaboration between Horvat and Rema Padman, Trustees Professor of Management Science and Healthcare Informatics at Heinz College. In the spring, the two began to develop evaluation standards for the visual language models. The researchers need to assess their usability, safety, value and outcomes, among other criteria, Padman said.
“There are so many components that one needs to evaluate these models on before they can be deployed at scale. The evaluations have to keep up with changing technologies, as well as the changing capabilities of end users as they get trained on AI-enabled tools,” Padman said.
As the project has progressed, more collaborators from across CMU and Pittsburgh have joined the fold. One collaborator is Fernando De La Torre, a research professor at the Robotics Institute and a leading expert in the field of computer vision and machine learning.
For years, he and his lab at CMU have developed algorithms that can understand and model subtle human behavior. They can track faces and bodily movements and understand which limbs are moving. Over the last few years, visual language models have transformed their work, as they can not only measure the body but also issue recommendations based on observations.
However, general-purpose visual language models do not yet understand the nuances of CPR delivery and, therefore, are unable to recognize incorrect hand placements or compression rhythms, De La Torre said. These subtleties are crucial to the correct delivery of CPR. To correct this problem, the team is training models on multi-camera footage of CPR trainings and simulated medical emergency drills at UPMC.
Next, the team will test whether the models can accurately assess new examples of CPR delivery. They will evaluate the models’ outputs through a software platform created this summer by Dheeraj Vislawath, Julian Ng-Thow-Hing and David Chung, all students in the School of Computer Science. Ideally, at least one of the models will consistently provide accurate feedback on the CPR demonstrations.
“I think there are a lot of opportunities for VLMs to explain a lot of actions in the medical field, and CPR is one that is kind of a no-brainer,” De La Torre said. “Now the big question is: Will a VLM be able to capture the fine granularity that you need to give feedback with footage from just one camera in an unstructured environment?”
Reyna Jones, a student in the Heinz College master’s program in healthcare analytics, helped develop a framework for collecting data and analyzing the models’ outputs. With that structure in place, the team is planning to collect broader swaths of training data and see how the model performs outside of a controlled, laboratory environment.
Jones said, “CPR is not, of course, a domain I was really familiar with, so getting the opportunity to not only see the analytics side of this work, but also read CPR guidelines and literature about the work that has been done so far, was really insightful.”
Vislawath, Ng-Thow-Hing and Chung presented their software platform in May at the Safar Symposium.
Going forward, the team intends to apply in October for a research grant from the National Institutes of Health, which would fund more expansive data collection and model testing. The researchers expect the project will take multiple years to complete.
“We still need to collect data at scale and fine-tune these algorithms. We are still in the process of understanding the limitations and the needs that we are going to have for this project,” De La Torre said. “At this point, we have a good hypothesis. But we need to validate it with real data.”
Horvat believes AI coaches will be the next wave of innovation to sweep through health care. Pittsburgh, he said, is well-positioned to develop the technology because of the strong relationships between its many hospitals and universities, including Carnegie Mellon. The collaboration between institutions creates opportunities for students, faculty and community partners to produce pioneering research.
Along with being a world-class institution for analytics and public policy, Heinz College showed Jones that “nothing is impossible,” she said
“It's a great learning environment, and you can always find a faculty member willing to support you,” Jones said. “I’ve found two really wonderful mentors in Dr. Padman and Dr. Horvat. Without Heinz College, I wouldn't have had them or known what was possible for me in the future.”