Mixed Reality and AI for Safer Surgeries

Illustration: © IoT For All

Mixed reality and AI can help make surgeries safer by assisting surgeons during the process. From providing 3D imaging to handling instruments, AI is a vital part of the operating room. Here, we discuss what mixed reality means and how AI is taking surgeries to the next level.

Mixed reality and AI can help make surgeries safer by assisting surgeons during the process. From providing 3D imaging to handling instruments, AI is a vital part of the operating room.Click To Tweet

Artificial intelligence, machine learning, and computer vision are becoming an essential part of the healthcare industry. AI is helping doctors, nurses, and the hospital administration streamline patients’ records, accurately diagnose the medical condition, and provide better treatment.

The COVID-19 pandemic has increased the focus and adaptation of remote patient care and telemedicine to ensure that patients don’t suffer due to lockdowns. Doctors increasingly rely on AI-based technology and tools to provide patients with quality healthcare.

Technology like augmented reality (AR), virtual reality (VR), and mixed reality (MR) are being used in hospitals to assist doctors and train medical students. These are designed to assist doctors during surgeries to make the process safer and increase the success rate.

The market size value of AI in healthcare is $10.4 billion in 2021 and is estimated to reach $120.2 billion by 2028 at a CAGR (Compound Annual Growth Rate) of 48 percent. North America has shown the highest adoption rate of AI in healthcare at 58.9 percent in 2020. The same report suggests that the use of AI in the healthcare industry will grow rapidly in the Asia Pacific during the forecast period.

What is Mixed Reality?

Mixed reality is a combination of virtual reality and human vision. The AR filters used in mobile apps such as Instagram, Snapchat, etc., are a simple example of mixed reality solutions.

Surgeries require a lot of planning, practice, and precision. Doctors cannot afford to get distracted or lose focus when operating on a person. The use of AI in surgery aims to support doctors and supply them with the necessary information and surgical tools without disturbing them at any point.

Mixed reality makes it possible to use technology to assist doctors during surgeries and minimize risks.

Paul Milgram and Fumio Kishino first introduced the term mixed reality in 1994 in their paper titled A Taxonomy of Mixed Reality Visual Displays. MR combines computer vision, cloud computing, graphical processing, etc., to blend the physical and virtual worlds. Many companies have been developing MR applications that can be used in various industries.

MR and Healthcare

MR in healthcare can be explained using a head-mounted device that gives doctors a clear and better picture of the patient’s data chart and related images. The images of the affected area where the surgery has to be performed can be displayed directly on the patient. This increases accuracy and makes it easy for surgeons to be as precise as possible. MR devices are also changing the interaction method between doctors and patients. Data is more easily accessible and is used to understand the patient’s condition in-depth.

Microsoft’s HoloLens is a prime example of MR Headgear that creates realistic images and holograms to help medical students understand human anatomy. Medical schools have begun investing in MR devices to empower their students.

Mohammad Abdul Mukit, an MS student in electrical and computer engineering, a graduate fellow and research assistant at the University of Oklahoma, and Dr. Christian El Amm, a plastic and reconstructive surgeon, have collaborated to create an AI-based device (that accepts voice commands) to increase safety and efficiency during surgical reconstructions.

SentiAr, a digital health company, has developed a holographic AR platform to provide surgeons with 3D images of a specific anatomy and data visualizations of the patient’s medical chart.

The adoption of MR in the healthcare industry is still in its early stages. Technology can bring many more positive changes to improve healthcare facilities and surgical procedures.

Use of Mixed Reality in the Healthcare Industry

Mixed reality can help doctors, nurses, and healthcare professionals in several Mways. From providing real-time information to developing realistic 3D images, here’s what MR can do.

Resource Optimization

Mixed reality reduces the dependence on nurses and medical assistants. Data is automatically collected, processed, and projected as images and charts whenever the doctor requires it. This saves resources and reduces expenditure without compromising the quality of healthcare services.

Real-Time Data Access

The mixed reality headsets can provide data in real-time to surgeons. For example, if a surgeon wants crucial data about the patient during an emergency, the MR device can immediately pull it up from the database it is connected to and display it for the surgeon. AI works in the backend to make this possible.

Live Streaming Surgeries

The MR devices can record and live stream surgeries without interrupting the process. These videos are used to train medical students to become proficient at surgeries. It also helps doctors rewind and replay complex surgical steps and explain the processes clearly to the students.

Remote Patient Care

Patients can continue to be under home care and still get excellent treatment from their doctors. The latest data is updated to the systems to help doctors understand the patients’ current condition and prescribe medication accordingly. AI has made it possible to deliver quality patient care remotely during the pandemic.

Natural 3D Imaging

Mixed reality is known for creating realistic 3D images with depth and layers of detail. This feature is very useful when surgeons want to know how deep the infection is or when to stop the incision. Natural 3D imaging increases accuracy and precision in surgeries.

Virtual Surgery Intelligence

This is used to educate patients about what happens during the surgery. Being aware of the procedure will give patients more confidence and prevent them from being stressed or worried about the results.

Remote Assistance in Surgeries

MR headsets like Hololens come with microphones and sensors. Surgeons can directly interact with other experts and take their advice during a complicated, high-risk surgery. Experts can provide remote assistance to surgeons in the operation theater.

Simplified Data Sharing

Critical medical information is stored in a unified format and can be shared with the physicians without any delay. Doctors don’t have to wait for data to be collected, printed, and brought to them through the long route. AI has made data easily accessible while maintaining data security standards.

Use of Artificial Intelligence in Surgeries

The introduction of AI in surgery is relatively a new development. Integrating AI and ML in medicine hasn’t been an easy task. Though we have successfully initiated the adoption process at various levels.

From automating recurring tasks to streamlining patients’ records, and assisting technicians with 3D images of scans and X-rays, AI is now a part of the healthcare industry. But how does artificial intelligence help doctors in operation theaters? What is the use of AI in surgery?

AI in surgeries is used for imaging, navigation, and robotic assistance. Let’s learn more about the role of AI in surgery.

Pre-Operation Preparation

Pre-operation planning is where the surgeon uses the patient’s medical records and imaging to determine the most effective way to perform the surgery. Artificial intelligence helps surgeons at this stage by:

• Developing 3D images of the anatomy
• Providing machine learning algorithms to classify the images and information
• Identifying abnormalities in CT Scans
• Preparing for emergency care in advance
• Using deep learning neural networks predict failures and risks in real-time
• Improving critical care for patients with high risk, etc.

AI is used in preoperative medical imaging to classify, detect, segment, and register anatomical images through deep learning technology.

Surgical Guidance

Termed as intra-operative guidance, AI is used during the surgery in the form of robotic assistance. Computer-aided tools are used to perform Minimally Invasive Surgery (MIS) to reduce surgical trauma in patients.

By integrating learning strategies with computer-aided tools, surgeons can access a high level of visualization prior to any surgery. For example, a framework has been created online to learn soft-tissue tracking for MIS and to study the deformation of soft tissue using AI-based imaging and visualizations.

AI Robotics

Robotic assistance is also used as a part of the surgical process. The AI-powered robots help surgeons by arranging and positioning the instruments in a way that surgeons don’t need to ask or wait. The aim is to keep the surgeon fully focused on the operation by minimizing distractions.

Robots are trained using machine learning algorithms to understand the surgical process and provide the surgeon with the necessary information. The surgical robot developed by Asensus Surgical helps doctors with laparoscopic surgeries. The robot directly provides the size of tissue that needs to be operated on.

Reinforced learning (RL) is used to train the robots to perform subtasks during the surgery. The idea is to train robots to assist surgeons based on their gestures such as hand and head movement. Since each surgeon has a different style, training the robots needs more time. After all, precision is necessary for surgery, and we cannot be satisfied with 80 percent accuracy.

Examples of AI in Surgery

The following are some examples of AI-powered tools used in surgeries:

• Smart-Tissue Autonomous Robot (STAR) is an automated robot developed to perform surgeries if a surgeon is not available during an emergency. The robot will be trained on certain types of surgeries and techniques.
• Gestonurse is a robotic nurse that assists surgeons by handling the surgical instruments in the operation theater.
• Robotic Hair Restoration robots can harvest root follicles and graft them into the exact area on the scalp to perfection. It reduces the time taken by the surgeon to manually remove the hair follicles from the donor area.
• Da Vinci robotic cardiac surgery uses robot-manipulated tools and tiny instruments that allow doctors to perform surgeries with only a small incision on the chest. This surgery is used for various heart and chest-related surgeries.

AI’s entry into healthcare might have been tentative years ago. But it is clear that the future of surgeries and healthcare is heavily dependent on artificial intelligence.

Conclusion

Mixed Reality and AI in surgeries are the future of the healthcare industry. By using MR devices, hospitals can streamline communication between the staff, doctors, and patients. Mixed Reality devices are capable of recording and saving voice notes. Doctors can listen to these voice notes while examining patients or when free, and concentrate on each patient equally.

Artificial intelligence and robotics are likely to reduce the pressure on surgeons by performing certain surgeries without human assistance. Doctors will feel less stressed and deliver premium healthcare.