Apr 10, 2019

The role of AR-VR technology in the future of care

By Wim Crooijmans
Research Program Manager in Philips for Ultrasound, Image Guided Intervention & Therapy and Cardiology

Estimated reading time: 8-10 minutes

Please fill in the subtitle of the blog (max 2 lines text - see Blog checklist for more information)

The way we interact with technology and data — how we consume information -- is evolving so quickly it is easy to forget the role played by the mouse. Douglas Engelbart’s invention in 1964, which enabled us to engage with 2D technology, consisted of a wooden shell circuit board and two metal wheels[1]. Before then there was no graphic user interface, so people couldn’t engage with technology.

The tech evolution from 2D to 3D and 4D has projected us forwards, with new ways to experience technology. As a consequence a new brand of healthcare is emerging which uses large amounts of medical data to build more immersive, visualization tools to support diagnosis and treatment. In image processing, for example, cardiovascular surgeons can now view individual segments of the heart using a 3D image, and even 3D print a patient’s heart, to better understand the anatomy prior to surgery. And just as we evolved from ‘mouse to swipe’ we are now going through a rapid evolution with augmented reality (AR) and virtual reality (VR), technologies which are natural and intuitive to us as 3D beings living in a 3D world. So, how should we assess the value and opportunity of this new wave of healthcare innovation?

Assessing the AR-VR opportunity

We can start by understanding the difference between them both and how they can most powerfully support patient and provider. AR overlays, or augments, what you see in real life with location-specific information and graphics[2]. One of the best known examples of the technology was the 2016 ‘Pokémon Go’ phenomenon, which populated users’ mobile screens with fictional characters over a normal camera view of the world around them. VR creates a fully rendered digital environment that replaces the user’s real-world environment.

VR has the potential to evaluate the dose distribution in 3D in an intuitive way, saving planning time with potentially greater precision and fewer side effects of the treatment.

There are already many examples of AR-VR adoption in healthcare, altering everything from the way medical students learn before interventional procedures[3] to helping patients with PTSD[4] and reducing anxiety[5] in children undergoing blood tests or other painful procedures. A study by Duke University in 2016 claimed that virtual reality had helped patients suffering from spinal cord injuries[6] in partially restoring their mobility.

In medical imaging VR can enable better planning to support patient safety by reducing their exposure to radiation dose. The process of building a radiation therapy plan for patients is a very time-consuming, complex exercise. One has to plan a very detailed radiation dose distribution in 3 dimensions, not only taking care of the complex 3D shape of a tumor. We have to build a minimal margin around the tumor and at the same time spare the healthy, critical structures from radiation doses, a root cause of many side effects of radiation therapies. VR has the potential to evaluate the dose distribution in 3D in an intuitive way, saving planning time with potentially greater precision and fewer side effects of the treatment.

Augmentation in the cath lab

Each new study and application points to the possibility of wider adoption of these immersive technologies, but a lot of consideration first needs to go into the context and the clinical setting for AR-VR, followed by clinical test cases, in order for that to occur. And getting this right requires extensive collaboration between a host of stakeholders: clinicians, data scientists, interaction designers, and other experts, and obviously a deep understanding of clinical care.

Integrating AR into clinical workflows starts with the end user in mind, the clinician: studying how they move; how they stand, gesture or look; to shift attention from a 2D glass screen in their hands to a virtual screen projected in front of them. From a human interaction perspective, the goal is to combine clinical expertise and AR technology to be better than either of them could be on their own.

We understand what those clinical needs are, and how departmental and hospital workflows operate. This level of understanding is the only way to successfully develop and implement AR-VR solutions that fit seamlessly into existing hospital workflows. For example, in partnership with Microsoft we have developed an AR innovation in image-guided therapies (Philips-Microsoft Hololens 2), which will transform the way physicians interact with patients and each other.

In the cath lab, there is a paradigm shift underway in how physicians interact with their environment. This technology delivers information through the virtual screen, in real-time. Rather than stopping and looking at a monitor, the information is right in front of the physician’s eyes, in 3D — allowing faster and more intuitive decision making.

AR as enabler of Digital Twin

During a TAVI procedure on the heart, the team treating the patient need to keep track of a lot of different data sources — real time information from all kinds of sensors — and make a lot of quick decisions based on that data. Through AR, each member of the team carrying out the procedure is doing so in an environment tailored to their specific role. Surgeons can take instant X-rays and 3D U/S scans and control the complete interventional solution space with voice recognition, eye tracking and advanced gestures.

At a heart team meeting, multi-disciplinary teams can use AR to study and discuss a 3D model of the patient case, examining the case collaboratively. In the pre-operative planning phase, the physician can make a detailed plan using AR modelling, evaluate a virtual heart — what we call a digital twin[7] — test whether the valve is positioned correctly, and simulate the functional performance of the valve to predict the clinical outcome of the procedure. Through AR glasses you can see the catheter moving, adjust it and perform instant measurements. The digital twin uses AI and modelling techniques for precision diagnostics and treatment, using big data sets and complex calculation models for which we need simple tools that make the use and interaction simple and intuitive. AR/VR is a natural fit to provide advanced visualization and interaction tools to a digital twin environment.

This can lead to better planning and first-time right interventions. Think of all of those valuable seconds saved through planning alone. It is immensely powerful and the value creation can be four fold: shorter procedure times; better clinical outcomes; better prediction and execution of the procedure; and, a more intuitive and comfortable experience for staff.

Fail-fast, adapt and evolve

With AR-VR applications in healthcare likely to grow and evolve over the coming years, we have to move with the change by continuously adopting and learning new technologies, in an agile and fail-fast way. This is such a fast changing market, and it will only continue to change, which presents a big challenge to healthcare providers.

Firstly, building the real time computing engine required for AR/VR capabilities requires a high performance computing infrastructure. We need to prepare our platforms for these real time, high speed (4D) data applications, a challenge not only for hospitals but also for other healthcare settings such as the home or the GP’s surgery. New applications, such as ‘over the shoulder support’ from remote experts will also require new communication protocols such as 5G and cloud computing.

Healthcare is typically very slow in adopting to innovation. In the new era of IoMT and digital innovation we will see much faster waves of innovation which begs the question: how do we prepare healthcare systems on the high innovation speed of the gaming world on which AR-VR has evolved? How do we prepare IT architectures and platforms for the easy adoption of the latest and greatest AR/ VR has to offer?

These are questions that we are busy answering, working in an ecosystem with our clinical, university and industry partners. Philips is much more a software company than most people realize. More and more, we augment our systems and devices with ‘smartness’ powered by software and unleashed by connectivity. The goal is to reach a level whereby 3D visualization and 3D interactions, enabled by AR-VR technology, become more feasible and affordable in healthcare not just as an sexy technology, but one that has proven impact and is seamlessly integrated in the delivery of quality patient-centric care.

The mouse got us quite a long way in our relationship with technology, but the next great leap is well underway. Healthcare’s future is going to be more visual, more immersive and more interactive and while there are many steps to be taken before AR-VR technology becomes integrated into daily clinical practice, it will eventually be incremental in the standard care that we receive in the future.