They have taken a fascinating approach to the challenge of Holoportation – the use of virtual, augmented and mixed reality to allow for users to virtually occupy the same space and interact in 3 dimensions while miles apart.
What makes James Edward Marks, Marius Matesan, Albert Kim and Chris Szkoda special is that they use affordable hardware – the Microsoft Kinect – and their bespoke software to allow anyone to create a holoportal. This means that the entire space it records can be viewed by a person with a Hololens. If you set up two holoportals, you can have an interactive experience.
The use cases are wide and mind-boggling. I focus on the healthcare aspects, so I immediately started to imagine what it might be like for a specialist to holoconsult with a patient, or advise a practitioner in the field. The holoportation isn’t limited to just people: you can also add 3d models, and perhaps even users from within VR allowing true mixed reality interactions. I visualise an Orthopaedic surgeon sharing a 3d model of a diseased joint and the propsed procedure ahead of an operation, or a GP speaking with a patient about their mental health while a 3rd person attends as support.
The current setup is a little lo-fi and the resolution isn’t perhaps good enough for detailed work, but even at this early stage I sense it is perfectly capable of allowing for clinical consultations. The question is then whether holoportation offers benefit over face-to-face, or teleconsultations?
This venture is also connected to a 5G testbed, so the next steps are to build a Holoportal at The Fusebox in Brighton, a co-working creative space for immersive tech that also happens to be a 5G site. With any luck I’ll be consulting by May.
Of all the remarkable clinical possibilities of Virtual Reality, it’s use as a novel therapy, or Digital Drug, excites me the most. What properties of the VR experience are ‘class’ effects, shared by all virtual worlds, and what are specific to the particular construct? Can we enhance the potency and minimise side-effects by going further and creating precision experiences, tailored to the individual? What about biofeedback? How do we create these experiences?
On Day 16, Dr Keith Grimes, The VR Doctor, has a closer look at Virtual Therapeutics, and shares the story of CALMO, the VR procedural distraction tool for blood sampling that was born at the Cardiff NHS Hack Day 2018. The story of how we went from pitch to patient illustrates one possible route to precision VR medicine.
When I was a child, aged 9 or 10, I used to love exploring. I’d pack my bag with important explorer’s tools, like a compass, binoculars, and some rope. I’d make sure I had food to survive if I got stuck somewhere. I’d also bring some games, some pens, and a notebook to draw comics. In short, I carried everything that I could possibly need to build something, draw it, or have fun waiting for the problem to go away.
Saturday morning, about 35 years later, and I’m packing my bag with important hacking tools. My laptop and iPad. My trusty Gear VR headset, stickers attached. Not one but two voice assistants, just in case. Everything is charged to the point of being uncomfortably warm. I have some experimental medical technology to baffle any medical folk I encounter. Oh, and I have my PS Vita and eBook reader for entertainment, and a nearly empty notebook. It’s almost as if I haven’t grown up.
NHS Hack day is a great place for the young at heart, the inquisitive, the restless, and the explorer of the art of the possible. This event was the third NHS Hack Day I’ve attended, the first being in Cardiff 3 years ago. As before, we had the expert organisational skills of Anne-Marie ‘@amcunningham‘ Cunningham and Helen ‘@DeckofPandas’ Jackson, and we settled into the fine facilities.
This event was number 19 in a series held around the country that bring together patients, developers, academics, clinicians, and interested hackers of all shapes and sizes, to collaborate on solving the problems we experience in the NHS. The tagline is ‘Geeks Who Love the NHS’, and its a perfect fit.
We started the day, as in all hacks, with the pitch. I’d scribbled down some half formed ideas about how VR could be used in my practice, and brought along some we’d discussed on the VR Doctors group – VR for wound care, VR for visual field testing, even VR to create a the sort of ‘suspect wall’ you see in police dramas where stern black and white photos are linked by red thread to heavily marked-up newspaper clippings (I was working on the ‘build it and see how it feels’ philosophy on that one). As it happened, James Dornan stood up and made the first pitch, and it was perfect – VR for reducing pain and distress during blood sampling.
Outside the pitchers met the unaffiliated, and the speed-dating began in earnest. Soon James & I had assembled a fabulous team of developers, medical students, entrepreneurs and even an Occupational Therapist. With that, and a quick decision on a name, we were off.
What I’ve learned from past hacks is that while everyone wants to change the world, 36 hours is not enough time to do it. A keen focus on the actual problem and a clear idea of what you’re going to manage in the time allotted appears to give you the greatest chance of success. For me, having something I could put in front of my patients on Monday morning was going to be my goal That, and having some fun.
We started a group by exploring the use case and user stories, settling on a simple VR app where the user would be transported to a calm, neutral space, to watch a glowing circle slowly expand and contract in time to a peaceful soundtrack. By using a ‘Box Breathing’ technique (in for 4, hold for 4, out for 4, hold for 4) we were building on an established relaxation technique and augmenting it with the immersion offered by VR.
With the 4 devs (James, Adam, Matthew and Paul, all from Wiggly Amps) forming a fearsome full-stack crew, Laura, Nat, Henry and I started to work on the background research. What size is the problem? What is currently being done? What works? What doesn’t work? What about VR?
An hour or so later and we broke for lunch, already able to test a rough demo. The circle was pulsing! You could watch it in VR! Things were looking good!
I can’t tell you how lucky we were to have the range of skills in our team, and the input of Vina Patel as our expert patient. Everything went back to the needs of the patient and the process. Design creep, ever present, was squashed mercilessly. Adam turned out a fantastic composition using breath sounds, ambient melodies, and even a space whale song inspired by my obsession with 80’s sandbox space simulation Starglider 2.
While Nat collated the technical information about the build for GitHub, the three medics got busy with the presentation. Laura Tan, a final year medic, pulled an incredible logo and animation from Powerpoint to breathe life into the product and shape the look and feel. By the time we closed the lids for the evening, we were all feeling pretty confident that we’d have something to show the next day.
What happens on a Hack Day night out stays on a Hack Day night out. Which is a shame, because whatever did happen led to an early hours coding blitz from James and the team and a working prototype, ready for everyone to test at 9am. I guess we know the secret of motivating a dev team after all.
Day 2, and with just over 5 hours left to present, the pressure was on. Thanks to James, that pressure was significantly less than usual though, so we were in the privileged position of being able to test and tweak the product. What colour scheme works? Where do we place the sound to get the right volume and spatial effects?
An interesting side task came about as we realised that the breath sounds supplied in Logic Pro didn’t quite cut it. We borrowed a smartphone microphone and set up a recording studio i the auditorium, taking it in turns to snort and wheeze therapeutically into the mic. I am proud to say that my respiratory efforts made the grade, and shall forever be captured in silico.
We set a deadline of 1pm to have Calmo finished, and we made that with some room to spare. I don’t mind saying that version 1, running on my Samsung Gear VR headset, was quite something. To have Adam’s music playing in perfect synchrony with the pulsing central sphere, as stars quietly drifted overhead was magical. Distracting. Relaxing. Everything we had aimed for. It was also immediately deployed to sooth the ragged brains of those that had worked tirelessly into the night.
All that was left was to polish up the presentation and give it a few test runs, and to try Calmo out on our test patient. I had hoped to get hold of the equipment to actually take blood, but in the end we had to resort to a quick pinch. Thankfully, our nervous subject felt sufficiently distracted as to not even feel my painful stimulus.
We were ready to share.
The final presentations to the massed teams and judges is always nerve-wracking, and it wasn’t helped by Laura’s laptop having a meltdown and a sudden urge to update as we got on stage.
We were one of 12 pitches. One of my favourites was Lymphie Selfie, an app to allow patients with Lymphoedema to collate a visual record of the ebb and flow of limb swelling they experience. To finish, the judges went to collaborate and the audience voted using a tally system. This being Hack Day, and collaboration being the name of the game, there were no outright winners, but I’m delighted to say that Calmo was voted by judges and audience alike to make the top three.
I couldn’t be happier with this – I’ve had a brilliant weekend, met some fabulous and talented people, and been part of the development of a product that will be available for my patients tomorrow. 28 and a half hours end-to-end. How about THAT for Just F****in’ Doing it!?
NHS Hack Day absolutely delivered on its promises, and having seen it evolve over the past 3 years, it clearly goes from strength to strength. I’d love to have more people attend, especially those that think they wouldn’t be needed. Diversity of thought and experience is crucial to build a varied and thriving hacking community in the NHS.
Innovation, Free at the point of delivery,
Based on need and not the ability to pay.
Nye Bevan would be proud.
It’s 8:30 on Monday morning, and I’ve just offered a patient Calmo to help with nerves ahead of a blood test. Pleasingly, it seemed to work exactly as we’d hoped. The patient said he felt as if he’d been on holiday ‘on the deck of the Voyager’. Somehow it was fitting to have a Trekker as the first beneficiary of the app.
January means Consumer Electronics Show, and Dr Keith Grimes, The VR Doctor, has donned his Rift and made a virtual visit to Vegas to check out all the announcements, and report back on his top 5 products to watch in 2018. From gaze tracking, mind reading headsets to cable cutting and iron pumping, the clinical possibilities just became a whole lot more interesting.
A new year brings hope and happiness for many, but for some people it’s a very different story. Depression is a condition that affects affects more than 300 million people around the world, and the number is rising. On Day 14 of my investigations I look at how Virtual Reality can provide an unusual way of helping people deal with symptoms and perhaps aid recovery.
Day 13, and it’s a quick trip into VR to look at the difference between Virtual and Augmented Reality as tools for patient education. Using the same model, but different platforms, Dr Keith compares and contrasts each and comes to the decision that…well, you’ll have to watch the video to find out more.