Philips and Microsoft HoloLens 2: could augmented reality change the face of image guided therapy?
Estimated reading time: 5-7 minutes
Augmented reality has the potential to take image guided therapy, a modern descendent of open surgery, to the next level
At MWC Barcelona, one of the largest mobile events in the world, Microsoft took to the stage to unveil HoloLens 2, the new version of what it describes as ‘the ultimate mixed reality device’. The company closed out their event by showcasing a unique augmented reality concept for image-guided minimally invasive procedures developed together with Philips.
Involved in the partnership since its early stages, Atul Gupta, MD, Chief Medical Officer for Philips’ Image Guided Therapy business and a practicing physician, was together with the Microsoft team in Barcelona for the event, and joins us here to tell us more about augmented reality in image guided therapy and its potential for the future.
What is image guided therapy and how does it relate to surgery?
AG: Let’s start from the beginning. Surgery has a history going back thousands of years. For almost all of that time, surgery has been ‘open’: the physician carrying out the procedure had to open up and physically see and touch the part of the body they were trying to fix.
Today, imaging technologies like X-ray and ultrasound mean that we can see what’s going on inside the body – in real time – without actually opening up the patient. That’s the first part of the story. The second part is the miniature devices, such as balloons, catheters, and stents that can be navigated through the body to be used where they’re needed.
If you combine these two innovations, the result is image guided therapy. These procedures require just a small incision, often no larger than a pencil-point, through which the devices needed to fix the problem can be inserted and guided to the treatment area. This can include the heart, blood vessels, brain, liver and other major organs.
What are the benefits of image guided therapy compared to open surgery?
AG: For the patient, it means no big incisions, so no big scars and oftentimes a local rather than a general anesthetic. That results in a faster recovery and a better overall experience of their treatment and recovery. With open surgery, patients need to stay in hospital for days and sometimes weeks. With today’s image guided procedures they’re often walking out of the hospital the same day.
For hospitals, with image guided therapy they can treat more people, at a lower cost. Excitingly, there are many new procedures that we can perform with image guided therapy that simply aren’t possible with traditional open surgery.
How can Augmented Reality help improve image guided therapy?
AG: During a procedure, the team treating the patient need to keep track of a lot of different sources of information and make a lot of quick decisions based on that data. It’s the medical equivalent of an airplane cockpit.
When we launched our Philips Azurion image-guided therapy platform in 2017, we took a big step forward in terms of integration, with all of that information combined and intuitively controlled on a large LED screen suspended above the patient table.
With augmented reality, we can take it to the next level, fully immersing each member of the team carrying out the procedure in an environment tailored to their specific role.
When you’re driving a car, you want to keep both hands on the steering wheel and your eyes on the road, not get distracted by looking at screens and having to press buttons. Similarly, when I’m working on a patient I want to keep my hands on my instruments and my eyes on the patient
MD, Chief Medical Officer for Philips’ Image Guided Therapy business
Can you describe the concept that Philips has developed with Microsoft on HoloLens 2?
AG: When you’re driving a car, you want to keep both hands on the steering wheel and your eyes on the road, not get distracted by looking at screens and having to press buttons. Similarly, when I’m working on a patient I want to keep my hands on my instruments and my eyes on the patient.
The concept we’ve developed with Microsoft allows me to see the real world superimposed with live data and 3D medical imagery needed to guide our precision therapy, and importantly also lets me control the procedure with voice recognition, eye tracking and advanced gestures.
HoloLens 2 is far more immersive than its predecessor, and much more intuitive to control. Rather than having to shift my focus away from the patient to press buttons, I can issue verbal commands, and the system can also recognize where I’m looking and what I’m doing with my hands. Together with Microsoft, we’re inventing an even better way for doctors to focus on the patient during minimally invasive procedures.
To get a better understanding of what this means in practice, take a look at this video of the concept.
What comes next?
AG: We believe that augmented reality will play an important role in the future of image guided therapy, but there are many steps that still need to be taken before the technology becomes integrated into daily clinical practice.
We’re using the concept developed with Microsoft to gather further clinical insights to support the development of future commercially-available augmented reality solutions. And this is just one example of how we’re innovating in image guided therapy. For example, we’ve previously announced the development of another augmented reality solution for spine, cranial and trauma procedures.
We’ve already taken huge steps in creating an immersive, intuitive environment that enables physicians to focus on the patient. Augmented reality is a natural next step in our innovation journey.
Atul Gupta, MD, Chief Medical Officer for Philips’ Image Guided Therapy business. He’s also an actively practicing physician, with over 20 years’ experience in treating patients minimally invasive procedures. Follow Atul on Twitter.
The augmented reality concept for image guided therapy developed by Philips and Microsoft is being used to gather further clinical insights and is not yet available for sale.