Artificial intelligence holds great promise in improving surgical risk assessment and surgical outcomes. Orsi Academy, Europe’s leading advocate for robotic and minimally invasive surgeries, has been upholding the promise by training healthcare professionals (surgeons) through proficiency-based progression. Today, the academy’s wide variety of training models are used to train participants as they progress along the learning curve. To build datasets for these training models and achieve precision in surgical robotics, Orsi is partnering with SuperAnnotate.
For this process, Orsi compared the speed and accuracy of SuperAnnotate’s annotation workforces against physician annotators. SuperAnnotate’s team was not only able to deliver 3x faster annotations, but they also provided better annotation quality compared to physician annotators.
Here’s why Orsi loves SuperAnnotate:
- 3x faster annotations: SuperAnnotate’s workflow automation and other advanced features allowed for dramatic acceleration and task completion.
- Pixel-perfect quality with surgical data: More precise annotation in terms of the level of detail when compared to physician annotators.
- Ease of use and nimble product development: An intuitive interface and opportunity to collaborate on new features.
- Exclusive DataOps services program: Hand-picked and vetted teams of professionals ready to take on projects of any volume and complexity.
Europe’s hub for surgical AI
Championing MedTech and surgical AI industries, Orsi academy is an independent institution where industry, clinical, and academic partners collaborate in training, R&D, and data analysis to improve the best practices in minimally invasive surgery. The open and inclusive scientific community at Orsi serves two major categories:
Healthcare professionals: Endorsed by the scientific society EAU and acknowledged by NASCE/UEMS, Orsi’s training program unites medical professionals from all over the world to teach them the newest minimally invasive healthcare practices.
Medtech companies: The Orsi Academy also provides MedTech companies with multifunctional (dry and wet) labs, facilities to host training or preclinical test, an auditorium for on-site or virtual lectures, 10 meeting rooms, and a fully equipped green key studio.
Problem
To create a safer environment for robotic surgeries and improve surgical assessment, Orsi Academy analyzed and weighed the results of annotated surgical data for medical images and videos. However, taking annotation projects of that sort often requires the intervention of doctors or medically-trained personnel. That has its downsides, as doctor annotators are massively expensive, difficult to scale, and rare to find. Facing these challenges, Orsi Academy proposed an experiment that challenged niche-expert-assisted annotation.
Finding a solution with SuperAnnotate
Orsi worked out a bottom-up approach (starting from a non-medical annotator to medical QA to ready-made datasets) in overseeing annotation for robotic instruments in robot-assisted partial nephrectomy (RAPN). Later it was validated in robot-assisted minimally invasive esophagectomy (RAMIE). Then, performance metrics were proposed to evaluate the efficacy of annotation methods for RAPN and RAMIE (including units such as time per frame per annotation modality).
Results
The two annotation approaches generated the following results for RAPN pixel annotation:
Approach 1: Annotations by medical students → In-house QA → Result: 12.6 min/image.
Approach 2: SuperAnnotate’s annotation service → In-house QA → Result: 4.49 min/image.
The demonstrated annotation time for RAMIE was similar for both approaches.
Approach #2 resulted in more accurate annotations when compared to annotations by medical students, as Pieter De Backer, the Artificial Intelligence lead at Orsi Academy, mentions.
“The fact that SuperAnnnotate had a local dedicated team of professionals sped up the annotation performance and improved the quality of instrument annotations. That, combined with our local QA team, made a combination of 1+1 makes 3.”
To wrap up the results, the second approach with SuperAnnotate can be replicated and applied to level up any surgical or potentially medical annotation project. While the study focuses on a specific part of an ML pipeline, i.e., dataset creation, it definitely sets a firm ground for promoting the use of medical AI, addressing the feasibility of successfully completing tasks such as instrument detection, segmentation, and pose estimation.
Key outcomes
SuperAnnotate transforms unstructured data into SuperData to train top-notch AI for any industry — even those that have historically required industry professional annotators — and Orsi was not an exception. By leveraging SuperAnnotate’s ever-evolving features and unique DataOps Services program, Orsi capitalized on two major aspects of its pipeline:
1. Annotation speed
2. Quality and accuracy of annotation
SuperAnnotate broke stereotypes about the need for niche-expert annotators, in this respect, medical professionals, to generate high-quality training datasets.
Orsi became a living example of how you could accelerate your annotation pipeline by at least 3x and save significant costs, all while getting more accurate annotations simply by investing in the right annotation platform and service.
“We’ve developed not a client relationship but rather a partnership in which both sides benefit,” Pieter De Backer recalls.
The majority of our customers experienced an increase in annotation accuracy by up to 85%. Join Orsi and other industry-leading companies to supercharge your datasets now!
