Mixed reality, artificial intelligence and augmented surgeons

Introduction

Computers offer the promise of being able to standardise surgical procedures and allow the surgeon, with the help of a machine, to become even more accurate. Thanks to computer-assisted surgery, the era of the “artist-surgeon”, where outcomes depended primarily on the operator’s skills, has given way to the era of the “digital surgeon”, also known as an “augmented surgeon” whose propensity for “human” error is eradicated thanks to the guidance offered by the machine. A particularly disturbing article in the British Medical Journal states that medical error is the third leading cause of death in the US.1 The stakes are therefore huge. Nevertheless, there is no denying the fact that the vast majority of surgeons have yet to incorporate digital technologies, borne from advances in 3D imaging and the rapid expansion of computer science, and which have been used in orthopaedic surgery for over 20 years, into their daily practice.

This article discusses the strengths and weaknesses of these technologies, including navigation, 3D printing and robotics. It offers a personal opinion of the future of these technologies and the future of orthopaedic surgery, following the arrival of a newcomer on the digital technology scene: Mixed Reality.

1. Impression 3D

3D printing is a new technology that has recently been applied to the realm of orthopaedics and trauma.

1.1. 3D printed implants

One of the ways in which implants are manufactured is to start with a block of composite material, and gradually machine it down to produce the desired shape. 3D printing works in the opposite way, whereby material is built up in layers to produce a shape designed using a computer. 3D printers use a laser beam to sinter a metal or plastic substrate, building up the complete model layer by layer. These printers can therefore print implants very quickly, recycle the unused powder substrate, and handle materials with high melting points such as titanium.

The two main advantages of 3D printing over traditional machining methods for making implants are the ability to produce very complex forms, even ones that cannot be machined, and the cheaper production of patient-specific implants (e.g. for reconstructive surgery). In addition, certain 3D printed components can be used in porous areas to optimise osseointegration and stress distribution, and ultimately maximise the long-term survivorship of the implant.

This technology is particularly useful in the treatment of tumour invasion into the pelvic and periacetabular structures, and for revision hip replacements with major cancerous defects, as well as more recently for trauma surgery. Digital patient data, whether from a scan or MRI, are used to create a 3D anatomical model of the bone that requires reconstruction. A fractured or invaded structure can be reconstructed in line with the patient’s native anatomy thanks to a 3D printed implant modelled on the opposing healthy side. Once assembled, these modelled implants are 100% patient-specific (Fig. 1).

This particular use is becoming quite popular in pelvic and spinal surgery, [2-5] although many other applications are also being discovered such as the production of osteosynthesis6 and osteotomy [7-8] plates with anatomic-matching designs and screw trajectories, and ACL femoral tunnel guides.[9] Most articles on this topic are from China. In fact, regulatory issues (such as CE marking) are the biggest barrier to the development of 3D implants in Europe and other countries.

Instruments can also be made using a 3D printer, offering benefits in terms of daily surgical practice and profitability.[10]

1.2. 3D printing for complex fractures

Initially reserved only for implants for complex defect repairs, this technology is now being more widely adopted for simpler and more routine cases, and is currently having a significant impact on our daily practice.

3D printing is being developed for periacetabular fractures and supports the surgeon throughout the process of visualising the various fragments and determining what reduction is required and how to achieve it. 3D printed models can be used preoperatively, then sterilised and reused during the procedure if required.[11-13]

For example, when treating pelvic fractures, 3D printing technology improves all stages of the patient care pathway: during diagnosis, the surgeon is better able to analyse the fracture, which is always a tricky process, and visualise the mechanism of injury; next, when planning the procedure and the surgery itself, the osteosynthesis plates can be produced in advance and positioned faster during the surgery, guaranteeing an anatomic match; finally, during the postoperative recovery the surgeon can use scan data to produce pre- and post-operative 3D models for comparison. The importance of 3D printing for educational and training purposes has been discussed already

1.3. Patient-specific 3D-printed cutting guides

Deep anatomical structures in close contact with the soft tissues can now be accessed thanks to patient specific instrumentation (PSI), which minimises portal size and tissue damage, results in more accurate implant positioning, and improves the chances of a good functional outcome. However, patient-specific cutting guides are a logistical burden (designed preoperatively using special software, then transferred to the manufacturer for production using 3D printing, before being delivered to the hospital), and PSI can even result in positional errors. In fact, they are often modelled using 3D scanning data which does not show cartilage. When placed in situ, the 3D-printed guide may not in fact fully match the patient’s anatomy because there may be cartilage or even embedded soft tissue in the way, resulting in implant malpositioning. However, these guides are still used to position the glenoid implant in shoulder replacement procedures, for want of a more reliable or easily accessible system currently available on the market. They are more reliable when used as osteotomy cutting guides (Fig. 2).[14]

An essential preliminary step when using 3D printing technology to make customisable cutting guides is implant position planning. Although there may justifiably be questions raised as to the benefits of these guides for prosthetic surgery, there is no doubt that the technology has triggered advances in planning software, which is not only particularly useful but represents a major step forward for surgery.[15,16]

1.4. 3D printing and immobilisation

Synthetic and plaster casts have also benefited from the advent of 3D printing technology. Splints can now be made to measure, modelled on either the injured or opposing limb. Offering a closer fit, they are perforated and do not require any batting (Fig. 3).

This means the skin can still be examined, which is not possible with traditional solid casts. They even allow the patient to clean the immobilised limb and reduce the risk of damp batting.[17]

2. Navigation and robotics

Here is a brief overview of the multitude of published articles on navigation. Manufacturers and orthopaedic surgeons have shown a growing interest in the ability to control certain variables during the actual procedure, such as optimal implant positioning and ligament tissue balance for knee, hip and, more recently, shoulder replacements, or pedicle screw positioning for spinal surgery.

Let us consider single and tricompartmental knee replacements (UKR and TKR). Many functional failures are attributed to poor implant position or ligament balancing, which is what has prompted the development of navigation systems since the turn of the century and more recently of robotic assistance. In order to understand the benefits of these two systems, let us first look at how they work. They comprise hardware, namely a computer and an infrared optical (or LED) system. Passive systems are also available to provide information for the surgeon, such as GPS. For a knee replacement, the infrared sensors (arrays) are fitted to the femur and tibia using pins, and to the various ancillary instruments including the cutting guide. A receiver (camera) mounted on the computer detects spatial movements of the various components. The information is then displayed on the screen for the surgeon to consult during the surgery. The software uses the data to reconstruct a 3D model of the knee, giving a dynamic view and allowing the surgeon to adjust the procedure based on both preoperative planning and intraoperative data. There are different categories of navigation system available, based on the data used for the navigation. CT Navigation requires a preoperative CT scan. The skeleton is modelled in 3D, then plans and reference landmarks are extracted and used as markers during the procedure, once the images have been recalibrated. The implants are therefore positioned based on a 3D model. Navigation is used to guide the implant position, in line with the preoperative plan. This system can be combined with a joint motion simulator in order to predict any potential conflicts between the implant and bone. Imageless navigation and hybrid systems do not use preoperative imaging but instead they use the bone morphing technique for data acquisition during the surgery to construct the reference landmarks (Fig. 4). The benefits of this technique are the lack of preoperative radiation (CT scan) and a faster planning stage. Nevertheless, some people do not think these systems are as accurate as preoperative CT navigation.

3D navigation systems can monitor several parameters, such as overall lower limb alignment, HKA, HKS and implant position. The latest navigation systems can also control ligament balance. The improvement in implant position in relation to the mechanical axis has been proven for navigation-assisted arthroplasties.[18-24] However, these same studies found no significant evidence of superiority for implant survival compared to traditional techniques, or for functional outcomes. This could be due to the length of follow-up for these various studies, which is currently fairly short (less than 5 years on average). Nonetheless, navigation is still clearly useful for more complex cases.

There is also an apparent improvement in ligament balance following arthroplasties performed with navigation assistance compared to traditional techniques. In fact, evaluating ligament balance is a highly complex process and involves numerous variables such as overall lower limb alignment, the position of the joint line, rotation, divergence, and implant position and size in flexion and extension.[25] Given the complexity and number of parameters to consider, computers are an obvious tool for helping the surgeon make the right decision concerning implant position and size (Fig. 5).[26]

Robots are a step up from simple navigation insofar as the navigation system is coupled with a robotic arm that can make the various cuts in preparation for positioning the implant. Robotic systems not only control the axis of the lower limb but also improve the accuracy of the surgeon's cut. Studies indicate that three factors, namely mechanical axis, implant position and ligament balance, are improved when using robotic assistance.[27,28] However, arthroplasty of the knee is not the only area in which computer-assisted surgery is gaining ground. All joints can now benefit from this technology (including the hip and spine), although not all are at the same stage of development.

Despite a mounting volume of data for knee surgery, with very interesting follow-up periods, this technology is still in its infancy for shoulder replacements. There are still no robotic systems for this joint. However, surgeons are using navigation to help position the glenoid implant. Stubig et al. conducted an anatomical study to compare non-navigated and navigated positioning of the glenoid wire to help guide the position of the metaglene and glenosphere during reverse shoulder arthroplasty.[29] Optimal positioning of these two components is a determining factor of implant survival. No significant difference was found for positioning parameters including inferior tilt (10° in theory, 7.5° in routine practice, radiologically confirmed), and inferior glenoid drilling distance (12mm in theory, and 14.5mm observed in routine practice). However, the glenoid version in the axial plane was more consistent when the navigated procedure was used. Theopold et al. found no significant difference, rather a trend towards improved accuracy of K-wire positioning and thus of the glenoid component and screws.[30] There is however a consensus as to the benefits for glenoid position planning during shoulder replacement surgery and on the need for navigation assistance during this stage of the procedure. Over the past decade, we have witnessed the emergence of patient specific instrumentation, and especially its planning software, which is the first stage of all these 3D printing techniques. We will discuss this further in the next section. As with knee surgery, the real benefits of navigation for shoulder replacement surgery can be seen in complex cases.

Another type of robotic-assisted surgery is used for the hands and peripheral nerves, involving the Da Vinci® robot for microsurgical procedures (vascular and nerve sutures). This robot uses several light sources and cameras for up to 20x optical magnification and remote surgery. The system has several arms offering a high level of accuracy. The surgeon sits behind the machine to control the robotic arms using joysticks, which increases the degrees of freedom and precision of the movements (fewer movements required; steady hands) (Fig. 6).[31]

Although computer-assistance would appear to be an irreversible trend in orthopaedic surgery, take-up of the technology has not been overwhelming. There are still limitations preventing its widespread use:

1. It is more expensive and uses arrays and pins, which are a potential source of complications;

2. The system takes longer to set up than conventional surgery, and offers no proven benefits for simpler cases;

3. Some techniques require a scan (i.e. additional radiation which, for some joints, is not needed for conventional surgery);

4. Above all, the recalibration stage can be time-consuming and has a risk of inaccuracy.

The barriers to the use of robots are even greater since not only is the equipment expensive, but the robot can only perform a limited number of steps during the surgery.

However, patient-perceived benefits are becoming a determining factor. Computer-assisted surgery, especially robotics, is currently a major commercial strategy for manufacturers of implantable medical devices. In 2013, the MAKO Surgical Corp©, a robotic arm designed in particular to assist during knee surgery, was purchased by Stryker© for $1.65 billion. More recently, in 2016, Montpellier-based Medtech© which developed the Rosa® robot was taken over by Zimmer-Biomet© in a deal worth €164 million. In 2017, Imascap© (a company offering 3D planning and printing solutions for glenoid implant positioning, based in Brest), was bought out by Wright Medical© for €75 million. These are just a few examples but are highly indicative of the current trend. France is a hotbed for innovation in computer-assisted surgery. This is thanks to the quality of the country's engineers, computer technicians and mathematicians, previously a sticking point for the digital surgery revolution.

These examples also illustrate how highly patients value the use of new technologies, as they gradually become more influential in the decision-making process for selecting orthopaedic equipment and implants, leaving behind the time when the surgeon was virtually the sole input when deciding which implant to use. The future therefore looks particularly bleak for companies who refuse to embrace the digital era. On the contrary, manufacturers of implantable devices who have made the switch to digital have a very bright future. The question now is whether 3D printing and robotics will become the go-to digital technology for daily surgical practice, or whether the winds of digital change will push orthopaedic surgery towards another technology altogether.

3. What is mixed reality and could it transform orthopaedic surgery?

Let us first explain the difference between mixed reality, virtual reality and augmented reality.

- Virtual reality (VR) means a system that can digitally simulate an entire environment, through a computer. Depending on which technology is used, numerous sensorial inputs plunge the user into a virtual world. VR has no direct application during the surgery itself, apart from as an adjunct to anaesthesia. However, the immersive nature of VR makes it extremely interesting as a training tool.

- Augmented reality (AR) is when computer-generated perceptual information (sounds, 2D/3D images, videos etc.) is superimposed on the actual world, in real time. The information is usually alphanumeric. For example, “head-up displays” can allow a combat helicopter pilot to view flight data displayed on a transparent screen directly inside the visor.

- Mixed reality (MR) is a type of augmented reality. The concept of mixed reality was developed by Microsoft with the first version of its HoloLens® glasses, unveiled in 2015 but not released until February 2016 in America and October 2016 in France. The difference with MR is that the HoloLens® glasses are in fact a minicomputer, with a laser and sensors that use artificial intelligence algorithms. The sensors allow the wearer to interact with the smartglasses via spoken commands, or by making gestures in front of the glasses which are fitted with two cameras. The glasses also have a laser and photogrammetry software that uses 3D mesh technology to model the surrounding environment (e.g. the ground, ceiling, a desk). A 3D virtual object (a hologram) can therefore be placed into the actual environment as scanned by the computer.

These three technologies can each be summed up in a single word: immersion for VR, integration for AR, and interaction for MR.

In December 2017, the first ever surgery was performed using mixed reality and the HoloLens® headset (https://www.youtube.com/watch?v=xUVMeib0qek&t=24s) (Fig. 7).[32]

The procedure was a reverse shoulder arthroplasty. During the surgery, the surgeon had access to three interfaces:

1. 3D images of the patient via the Holoportal® interface from Terarecon©, which was accessing the hospital’s PACS (Picture Archiving and Communication System), allowing the surgeon to manually superimpose the hologram of the scapula over the visible portion of the glenoid, and thus visualise the parts of the bone hidden under the soft tissues;

2. The surgical technique developed by Evolutis© (Interface HeadOp®) in holographic form on the 3D patient images taken from the preoperative planning stage. After every single stage, the surgeon can therefore compare what is happening against the planning, in order to achieve the correct implant position as determined prior to the surgery;

3. The Skype® interface allowing the surgeon to communicate with surgeons in the USA and England (who were watching the procedure on their computer, filmed by the two cameras on the HoloLens® headset, and who could also see the same holograms as the surgeon) (Fig. 8). The surgeon’s assistant also had a HoloLens® headset, so the holograms could be viewed simultaneously and in real time by both people.

This World First prompted further experiments with MR in other fields of surgery, such as maxillofacial surgery, plastic surgery, oncology and neurosurgery.[27-36] However, this was not a major surgical breakthrough, as the technology offered no real gains in terms of time or accuracy. For a surgeon working in his or her specialist field, being able to see the patient’s scan data on a screen in the operating theatre gives sufficient information to be able superimpose the images onto the surgical field with the imagination. However, this première was a good indication of the future potential for this technology in everyday surgery.

There are several possibilities. For example, in a not-so-distant future the surgeon may be able to use a computer during surgery and still remain sterile, by interacting not via a keyboard but using voice commands and/or gestures; the computer may be able to connect to the hospital’s IT system and allow observers outside the operating theatre to watch the surgery, with advances in digital technology lowering the price of the hardware and making it more widely accessible (the HoloLens® 1 costs around $3,500).

Looking further ahead, depending on the development of mixed reality devices (HoloLens® 2, 3 or 4, or a competitor system), we see potential for two very interesting developments:

1. The possibility of using MR laser technology (e.g. the HoloLens mesh) to automatically and continuously recalibrate the preoperative patient images in the operating theatre, in real time, without the need for sensors fitted to the patient. Navigation could therefore do away with bone arrays and instruments and thus overcome the current risk of inaccuracies during the recalibration stage. For example, scan data are currently used to model the contours of the patient’s skin and bones. The HoloLens laser would allow the HoloLens computer to model the patient’s skin. It would then automatically recalibrate the skin hologram against the patient’s skin contours, and automatically determine the position of the bones in real time.

2. Mass data acquisition by the cameras. Large volumes of surgical data captured during the routine use of MR glasses by multiple surgeons could be analysed by artificial intelligence algorithms and used to produce orthopaedic surgical software of the future. For example, an artificial intelligence (AI) algorithm could warn the surgeon of a nearby nerve before the surgeon has even seen it. Another algorithm could warn a head of department that a junior surgeon has encountered a problem whilst in the operating theatre, even before the surgeon has decided to call for assistance.

It is currently hard to imagine what information that these AI algorithms could discover in the future. In all likelihood, we do not even yet know what we should be looking for, but the huge potential is clear to anyone working in the field of AI and mass data. Faster 5G data transfer will help boost the mass acquisition of surgical data.

Conclusion

Mixed reality was developed for uses completely unrelated to medicine and orthopaedic surgery. However, the technology will result in the wider use of computer-assisted surgery within our field. The prerequisites for the large-scale adoption of a surgical innovation are discussed in an article by Barkun et al. [37]. We believe that MR meets these criteria:

1. The most important is ‘patient demand for the technology’. Patient demand for digital technologies to make surgery safer should not be underestimated. This is perfectly illustrated by the success of the Da Vinci robot in urology and the role played by patient associations in this success.

2. ‘Low cost to surgeons’. The HoloLens costs around $3,500.

3. Benefits ‘perceived by each stakeholder’. In simple terms, MR offers a computer that can be used inside the operating theatre whilst remaining perfectly sterile. The improved ergonomics and interactivity achieved by the new AI sensors on the recently released HoloLens® 2 suggest to us that this new system could be the trigger for widespread use of this technology in orthopaedic surgery.