Why AI in healthcare needs human-centered design

I’m a firm believer in the promise of artificial intelligence (AI): a tool that, if used in a smart way, can give healthcare providers superpowers by putting the right information at their fingertips in the moments that truly matter to them and their patients. Yet I also believe that in order to deliver on that promise, we must focus on people – not technology – first.   

In conversations with healthcare leaders around the world, one challenge that typically comes up is the difficulty of finding enough specialized physicians to meet the growing need for care. At that point I like to ask: what if we could make your existing physicians twice as effective and efficient, enabling them to make better decisions while spending more time with patients?

That’s essentially the promise of AI – whether it’s through automating mundane chores that take away a physician’s focus from the patient, or by providing clinical decision support in the moments that ultimately have the power to lead to better health outcomes.

And yet, for all its technological advances, surprisingly few AI applications have successfully made their way from the research lab into clinical practice. What’s holding AI in healthcare back?

We already see the early contours of human-AI collaboration in radiology; a discipline that has always been at the forefront of digital transformation in healthcare, and which is now a fertile hotbed for AI innovation. With radiologists and imaging staff being in short supply in many parts of the world, facing ever-increasing workloads, they could certainly benefit from having a smart digital assistant.

But how do we ensure that AI complements these human experts such that together they achieve more than either could alone? Clearly, the answer isn’t going to present itself on a silver platter if we focus on technology in isolation.

I have long believed that in order to deliver meaningful innovation to people, you need to innovate with them rather than for them. As I argued at the Fortune Brainstorm Design conference some time ago, building algorithms in the confines of a research lab in itself will not bring us human-centered AI. Instead, we must first get out there and submerge ourselves into the daily reality of the people whose lives we are trying to improve.

That’s exactly what we are doing with our clinical and academic partners, such as the Catharina Hospital and the Eindhoven University of Technology (as part of our joint e/MTIC collaboration [i]) and Leiden University Medical Center in the Netherlands, conducting 360-degree on-site workflow analyses and co-creation sessions with medical experts to look beyond the technology itself and understand the full context of use.

Through this work, we are beginning to uncover the experience drivers that foster strong physician-AI collaboration in radiology. And what has become abundantly clear is that designing for human-centered AI involves much more than coming up with a superior user interface.

For example, early in the workflow, AI can help to optimize patient scheduling so that patients get seen as quickly as possible. Or, it can assist radiology technicians in the selection and preparation of the appropriate imaging exams for a specific patient to avoid unnecessary rescans. Once images have been acquired, AI can support radiologists in various ways, for example by pointing them to incidental findings that are easy to overlook, or by saving them time through automatic integration of findings into the report.

There are plenty more use cases one could think of, some of which we are actively exploring with our clinical partners. Identifying pain points and mapping opportunities along the entire workflow is a great way to kick off an AI innovation program, because it creates a shared focus on the value that AI can bring at every step.

But we must think even more holistically about workflow integration, looking at the diagnostic system at large. For example, AI could do a preliminary analysis of a patient’s CT scan to flag worrisome cases that require immediate follow-up with a PET-CT scan. In itself this would be a very useful application. It would only work, however, if space can be made available in the PET-CT schedule that same day, which presents a new workflow challenge (one that AI may be able to support with as well!). As this example shows, you need to look at the entire hospital’s way of working, not just individual workflow elements.

How do you help radiologists find the right balance between trust and vigilance?

First, it is essential they understand what an AI system can and cannot do, what data it was trained on, and what it has been optimized for. For example, an algorithm can be optimized to spot small tumors in the lungs, but it might therefore overlook bigger ones. This is not at all intuitive to the radiologist and requires active communication to prevent distrust. Radiologists will also want to know how well the AI does what it supposed to do, calling for transparency on its validation and performance.

In addition, we need to consider how we can make AI’s findings as understandable as possible and needed. While interpretability may not be as important when the stakes are low, in a high-stakes situation the black-box nature of an AI system can be an impediment to trust. Together with our clinical collaborators and academic partners such as the Eindhoven University of Technology, we are investigating various ways of addressing this problem – either by explaining the AI’s design philosophy in general, or by providing the rationale for individual recommendations.