Blog

Assessing the AR-VR opportunity

We can start by understanding the difference between them both and how they can most powerfully support patient and provider. AR overlays, or augments, what you see in real life with location-specific information and graphics[2]. One of the best known examples of the technology was the 2016 ‘Pokémon Go’ phenomenon, which populated users’ mobile screens with fictional characters over a normal camera view of the world around them. VR creates a fully rendered digital environment that replaces the user’s real-world environment. 

There are already many examples of AR-VR adoption in healthcare, altering everything from the way medical students learn before interventional procedures[3] to helping patients with PTSD[4] and reducing anxiety[5] in children undergoing blood tests or other painful procedures. A study by Duke University in 2016 claimed that virtual reality had helped patients suffering from spinal cord injuries[6] in partially restoring their mobility. 

In medical imaging VR can enable better planning to support patient safety by reducing their exposure to radiation dose. The process of building a radiation therapy plan for patients is a very time-consuming, complex exercise. One has to plan a very detailed radiation dose distribution in 3 dimensions, not only taking care of the complex 3D shape of a tumor. We have to build a minimal margin around the tumor and at the same time spare the healthy, critical structures from radiation doses, a root cause of many side effects of radiation therapies. VR has the potential to evaluate the dose distribution in 3D in an intuitive way, saving planning time with potentially greater precision and fewer side effects of the treatment.

Augmentation in the cath lab

Each new study and application points to the possibility of wider adoption of these immersive technologies, but a lot of consideration first needs to go into the context and the clinical setting for AR-VR, followed by clinical test cases, in order for that to occur. And getting this right requires extensive collaboration between a host of stakeholders: clinicians, data scientists, interaction designers, and other experts, and obviously a deep understanding of clinical care. 

Integrating AR into clinical workflows starts with the end user in mind, the clinician: studying how they move; how they stand, gesture or look; to shift attention from a 2D glass screen in their hands to a virtual screen projected in front of them. From a human interaction perspective, the goal is to combine clinical expertise and AR technology to be better than either of them could be on their own. 

We understand what those clinical needs are, and how departmental and hospital workflows operate. This level of understanding is the only way to successfully develop and implement AR-VR solutions that fit seamlessly into existing hospital workflows. For example, in partnership with Microsoft we have developed an AR innovation in image-guided therapies (Philips-Microsoft Hololens 2), which will transform the way physicians interact with patients and each other. 

AR as enabler of Digital Twin

During a TAVI procedure on the heart, the team treating the patient need to keep track of a lot of different data sources — real time information from all kinds of sensors — and make a lot of quick decisions based on that data. Through AR, each member of the team carrying out the procedure is doing so in an environment tailored to their specific role. Surgeons can take instant X-rays and 3D U/S scans and control the complete interventional solution space with voice recognition, eye tracking and advanced gestures. 

At a heart team meeting, multi-disciplinary teams can use AR to study and discuss a 3D model of the patient case, examining the case collaboratively. In the pre-operative planning phase, the physician can make a detailed plan using AR modelling, evaluate a virtual heart — what we call a digital twin[7] — test whether the valve is positioned correctly, and simulate the functional performance of the valve to predict the clinical outcome of the procedure. Through AR glasses you can see the catheter moving, adjust it and perform instant measurements. The digital twin uses AI and modelling techniques for precision diagnostics and treatment, using big data sets and complex calculation models for which we need simple tools that make the use and interaction simple and intuitive. AR/VR is a natural fit to provide advanced visualization and interaction tools to a digital twin environment. 

This can lead to better planning and first-time right interventions. Think of all of those valuable seconds saved through planning alone. It is immensely powerful and the value creation can be four fold: shorter procedure times; better clinical outcomes; better prediction and execution of the procedure; and, a more intuitive and comfortable experience for staff. 

Fail-fast, adapt and evolve

With AR-VR applications in healthcare likely to grow and evolve over the coming years, we have to move with the change by continuously adopting and learning new technologies, in an agile and fail-fast way. This is such a fast changing market, and it will only continue to change, which presents a big challenge to healthcare providers. 

Firstly, building the real time computing engine required for AR/VR capabilities requires a high performance computing infrastructure. We need to prepare our platforms for these real time, high speed (4D) data applications, a challenge not only for hospitals but also for other healthcare settings such as the home or the GP’s surgery. New applications, such as ‘over the shoulder support’ from remote experts will also require new communication protocols such as 5G and cloud computing. 

Healthcare is typically very slow in adopting to innovation. In the new era of IoMT and digital innovation we will see much faster waves of innovation which begs the question: how do we prepare healthcare systems on the high innovation speed of the gaming world on which AR-VR has evolved? How do we prepare IT architectures and platforms for the easy adoption of the latest and greatest AR/ VR has to offer? 

These are questions that we are busy answering, working in an ecosystem with our clinical, university and industry partners. Philips is much more a software company than most people realize. More and more, we augment our systems and devices with ‘smartness’ powered by software and unleashed by connectivity. The goal is to reach a level whereby 3D visualization and 3D interactions, enabled by AR-VR technology, become more feasible and affordable in healthcare not just as an sexy technology, but one that has proven impact and is seamlessly integrated in the delivery of quality patient-centric care. 

The mouse got us quite a long way in our relationship with technology, but the next great leap is well underway. Healthcare’s future is going to be more visual, more immersive and more interactive and while there are many steps to be taken before AR-VR technology becomes integrated into daily clinical practice, it will eventually be incremental in the standard care that we receive in the future.

[1] Centre for Computing History, ‘The history of the mouse’, http://www.computinghistory.org.uk/det/613/the-history-of-the-computer-mouse/#

[2] CNET, ‘AR and VR made simple’, https://www.cnet.com/how-to/ar-and-vr-made-simple/

[3] Elsevier, ‘Going virtual: How VR is guiding interventional radiology’, https://www.elsevier.com/connect/going-virtual-how-vr-is-guiding-interventional-radiology

[4] PWC, ‘What AR and VR technology means for healthcare,’ https://www.pwc.com/us/en/industries/health-industries/library/interview-with-ian-bonnet-ar-and-vr-in-healthcare.html

[5] West Australian, ‘WA first sees virtual reality goggles used at St John of God Health Care Hospitals to ease anxiety’ https://thewest.com.au/news/wa/wa-first-sees-virtual-reality-goggles-used-at-st-john-of-god-health-care-hospitals-to-ease-anxiety-ng-b88611201z

[6] IDG, ’Ways virtual reality could take off beyond gaming’, ‘https://www.techworld.com/picture-gallery/apps-wearables/ways-virtual-reality-could-take-off-beyond-gaming-3638229/

[7] Philips, ‘The rise of the digital twin in healthcare’, https://www.philips.com/a-w/about/news/archive/blogs/innovation-matters/20180830-the-rise-of-the-digital-twin-how-healthcare-can-benefit.html