- Open Access
A patient-specific surgical simulator using preoperative imaging data: an interactive simulator using a three-dimensional tactile mouse
Journal of Computational Surgery volume 1, Article number: 10 (2014)
Preoperative simulation can greatly facilitate the safe and effective conduct of surgical procedures, especially for difficult and complex operations such as hepatectomy and pancreatectomy. A computer simulation system may greatly assist surgeons to preoperatively evaluate an operation and facilitate sharing of information among the operative staff. However, there are several problems with existing simulation systems. We have developed a new surgical simulation system using a patient’s own imaging data, which has some advantages over existing systems. Individual anatomical information obtained through an imaging study is used in this system. In addition it allows interactive control, similar to what a surgeon does in a real operation. Furthermore, a surgeon can control the system intuitively using a three dimensional tactile mouse. Changing the translucency of objects makes it easy to understand complex anatomical relationships. In conclusion, this new system is a patient-specific surgical simulator and can be applied to navigation surgery, medical education and patient communication.
Hepatectomy and pancreatectomy are among the most difficult abdominal surgery procedures to perform due in part to their anatomical complexity, vascular variation, and the wide variety of surgical techniques and tumor locations. Surgeons must maintain an adequate balance between function of residual tissue and curability, which poses an additional challenge.
Performing a preoperative simulation may be one of the best ways a surgeon can improve the safety of a surgical procedure. Currently, most surgeons simulate procedures mentally to perform them safely and effectively. However, this level of preoperational evaluation and the lack of realistic simulation allows for little information sharing among members of the team and objectivity. Computer simulation using three-dimensional (3D) visualization techniques helps to address these problems.
Many surgical simulators are available for preoperative evaluation and surgical education –, but several problems still exist. First, individual variations in anatomy and tumor location are not reflected in existing simulation systems. Simulators for preoperative evaluation, which are most commonly used for hepatectomy procedures, cannot simulate the steps of an operation in an interactive manner. Finally, nearly all simulators currently used in surgical education are programmed to present a limited number of prearranged surgical scenarios.
Significant advances in diagnostic imaging modalities and image processing software, which have enabled surgeons to obtain detailed anatomical information preoperatively for each patient, should be incorporated in surgical simulation. It would be especially useful for difficult procedures such as hepatectomy and pancreatectomy. The present study was undertaken to create a surgical simulator based on radiological images from an individual patient and demonstrate the feasibility of using this system in a real clinical environment.
This study was approved by the Ethics Committee of Jichi Medical University Hospital. The study consisted of four steps:
Obtaining imaging information. Multi-detector row CT (MDCT) scan (SOMATOM Definition Flash®, Siemens, Munich, Germany) images were obtained during preoperative evaluation of patients scheduled to undergo hepatectomy or pancreatectomy. A dynamic study using iodinated contrast medium was performed, following a routine imaging protocol for detailed imaging examinations. Arterial, portal venous, and equilibrium phases were imaged for planned hepatic resections, and the early arterial phase, late arterial phase, and portal/equilibrium phase were scanned for planned pancreatic resections. All CT scans were performed in the expiratory phase. All images were stored as DICOM (Digital Imaging and communication in Medicine) datasets (slice thickness, 1 mm) and sent to a workstation.
Segmentation. Image processing software (Synapse Vincent®, Fujifilm Medical, Tokyo, Japan) was used for segmentation of each component from the DICOM datasets. Automatic and manual segmentations were performed in combination. For hepatectomy procedures, segmentation of the portal vein, hepatic vein, and hepatic parenchyma was performed using both portal-venous and equilibrium phase images for each. Precontrast-enhanced and arterial phase images were used as references. For pancreatectomy procedures, arteries and pancreatic parenchyma were extracted from early and late arterial phase images, while the portal and surrounding veins were extracted from portal venous phase images. For all cases, tumors were extracted from the phase that allowed the easiest identification of the tumor. Other organs and objects, such as the biliary tract and biliary stents, were extracted as necessary. All segmented components were saved in separate STL (stereolithography) formats.
Creation of a virtual 3-Dimensional model. All segmented components were transferred to 3D modeling software (FreeForm®, Sensable, Woburn, MA, USA). Surface smoothing of each component was applied automatically, and manually corrected if necessary. Each component was colored for easy recognition. Finally, a virtual 3D model was created by integrating all segmented components in a virtual space. Positional information stored as DICOM data was used to automatically adjust three-dimensional relations among segments. Manual adjustment was performed if there were large positional differences between scan phases.
Preoperative simulation and intraoperative navigation. The surgical procedure was simulated using the virtual 3D model and manipulated within FreeForm® using a 3D tactile mouse that functioned as a variety of surgical instruments to cut, retract, and isolate the tissue (Figure 1a). FreeForm® was brought into the operating room and information was shared by displaying images on the operating room monitor (Figure 1b). As the actual operation progressed, the same techniques were carried out on the virtual 3D model.
Results and discussion
Process time required to create a virtual 3D model. It took approximately 2 h to segment all components for a hepatectomy simulation. It took approximately 2.5 h to segment all components for a pancreatectomy simulation. Creation of a virtual 3D model with the modeling software required approximately 30 min.
Simulation of surgical procedures. Surgical trainees under the direction of attending surgeons performed the simulations. During the simulations, the object opacity was changed to make otherwise invisible objects visible (Figure 2).
A) The following are the standard procedures with data from a healthy volunteer:
Lateral segmentectomy of the liver. First, the umbilical portion of the portal vein located within the hepatic parenchyma was identified and exposed. Next, the Glisson's capsule corresponding to the lateral segment was divided, and the hepatic parenchyma was resected. Finally, the root of the left hepatic vein was isolated and transected (Figure 3) (Additional file 1: Movie 1).
Right hepatectomy of the liver. First, the hepatic portal region was exposed, and the right side of Glisson’s capsule was identified and divided. Then, the resection line, which was calculated by image processing software, was drawn on the surface of the virtual 3D model. Following this, the hepatic parenchyma and smaller vessels were transected. Finally, the roots of the right hepatic vein and right inferior hepatic vein were transected. As a result of virtual resection, the middle hepatic vein was exposed on the resection plane of the residual hepatic parenchyma (Figure 4) (Additional file 2: Movie 2).
The following clinical cases are simulated in the study:
Hepatectomy. The tumor was located in segment 8 of the liver and a partial resection was performed. In this patient, the right hepatic vein was oversewn and the middle hepatic vein was partially exposed on the surface of the resection plane. Preoperative simulation results correlated well with the operative findings. Changing the translucency of the liver parenchyma allowed for easy anatomical evaluation of the relationship between the components (Figure 5).
Pancreatectomy. For this patient, we performed a pancreaticoduodenectomy for carcinoma of the head of the pancreas. The right hepatic artery originated from the superior mesenteric artery and ran behind the portal vein and the hepatic bile duct. The virtual image after resection effectively visualized the anatomical relationships between the artery, portal vein, bile duct, and pancreatic parenchyma. Image data were consistent with the surgical findings (Figure 6).
Preoperative simulation is a useful adjunct to ensure safety in the conduct of surgical procedures. Hepatectomy and pancreatectomy are particularly challenging abdominal surgical procedures, and require precise simulation in order to achieve a successful surgical procedure and a smooth post-operative course. Every surgeon mentally simulates the operation using imaging data preoperatively. However this practice is questionable in terms of its objectivity and reproducibility.
Computer simulation is a promising approach to address these problems. With recent advances in imaging technology and 3D visualization techniques, accurate images of each patient can be obtained preoperatively. We have published several reports on the use of 3D visualization techniques for the diagnosis of various diseases and therapeutic planning –. Other simulation systems have been recently described. The IRCAD software establishes a 3D virtual surgical planning system, and has been reported regarding the utility of patient-specific surgical simulation for laparoscopic liver surgery –. MeVis also supports planning a hepatectomy using patient specific data. These simulators are specialized software, and differ somewhat from this system, which allows easy application to various organs with a stepwise simulation of the operative procedure ,.
Arora et al. reported that mental practice with a virtual simulator can enhance surgical skill for the conduct of a laparoscopic cholecystectomy . However existing surgical simulators cannot simulate complex surgical procedures step-by-step, such as hepatectomy and pancreatectomy, and do not reflect individual anatomical information obtained from imaging studies.
The ideal surgical simulator should satisfy the following three conditions. First, it should reflect patient-specific anatomical information that is input into the simulator. Second, it should allow interactive control. Third, the user interface should be intuitive. Currently, some computer simulators are available for surgical training, but most have limited, pre-programmed scenarios ,,. In liver surgery, the use of an image processing workstation allows one to calculate resection volume and visualize completion images ,. However, this existing system is not interactive, so the steps of the operation cannot be simulated. This general-purpose software can be applied easily to various organs and simulate the procedure in a step-by-step fashion. Third, intuitive control is achieved with a 3D tactile mouse. In addition to the above, the opacity of each object is easily changed, allowing this simulator to make otherwise invisible objects visible.
Comprehensive anatomical understanding is intuitively established by confirming invisible internal structures through changes in transparency from every observational viewpoint on the monitor, including those that may be unrealistic when dealing with an actual patient. The understanding gained from simulation with this system gives a surgeon confidence when performing the actual surgical procedure.
This technique for surgical simulation does have limitations. First, these models are basically rigid and it is impossible to visualize a realistic operative field. Some of the physical characteristics of the tissue (e.g., firmness and elasticity) cannot be simulated with this method. Simulation of accurate organ deformation is also difficult. Further work is required to develop simulators capable of reflecting these characteristics of tissue ,. Second, creation of a virtual 3D model, especially one that involves segmentation of each component, demands significant time and effort. Each of the multiple steps required to create a virtual 3D model needs to be automated in order for this method to become more generally useful. Finally, the image processing software used in this simulation system is expensive. Synapse Vincent® and FreeForm® software cost US$100,000 and US$25,000, respectively.
We believe that in the future, application of this approach to modeling and simulation can be expanded to other facets of medical practice and education. For example, if this simulation system and the actual procedure could be performed simultaneously in the operating room, this system could serve as a surgical navigator. The understanding of complex anatomy and surgical procedures by surgical trainees at all levels could be greatly enhanced by using this system. Preoperative simulation not only allows attending surgeons and surgical trainees to achieve a common understanding but also helps medical students and other staff members to learn more about the operative procedure. Finally, this simulation system could be used as a way to visually communicate with patients, to better explain to them about an upcoming surgical procedure. The future directions of this study include objective and subjective evaluations of the system and will be evaluated using a questionnaire for both surgeons and patients.
We have developed a patient-specific surgical simulator for hepatectomy and pancreatectomy by creating a virtual 3D model using patient imaging data and have established the feasibility of using this system in a real clinical environment. In addition to its utility in surgical simulation, this method can be applied to navigation surgery, medical education and patient communication.
Digital Imaging and communication in Medicine
Sutton C, McCloy R, Middlebrook A, Chater P, Wilson M, Stone R: MIST VR. A laparoscopic surgery procedures trainer and evaluator. Stud Health Technol Inform 1997, 39: 598–607.
Marescaux J, Clement JM, Tassetti V, Koehl C, Cotin S, Russier Y, Mutter D, Delingette H, Ayache N: Virtual reality applied to hepatic surgery simulation: the next revolution. Ann Surg 1998,228(5):627–634. 10.1097/00000658-199811000-00001
O'Toole RV, Playter RR, Krummel TM, Blank WC, Cornelius NH, Roberts WR, Bell WJ, Raibert M: Measuring and developing suturing technique with a virtual reality surgical simulator. J Am Coll Surg 1999,189(1):114–127. 10.1016/S1072-7515(99)00076-9
Hyltander A, Liljegren E, Rhodin PH, Lonroth H: The transfer of basic skills learned in a laparoscopic simulator to the operating room. Surg Endosc 2002,16(9):1324–1328. 10.1007/s00464-001-9184-5
Saito S, Yamanaka J, Miura K, Nakao N, Nagao T, Sugimoto T, Hirano T, Kuroda N, Iimuro Y, Fujimoto J: A novel 3D hepatectomy simulation based on liver circulation: application to liver resection and transplantation. Hepatology 2005,41(6):1297–1304. 10.1002/hep.20684
Fairhurst K, Strickland A, Maddern G: The LapSim virtual reality simulator: promising but not yet proven. Surg Endosc 2011,25(2):343–355. 10.1007/s00464-010-1181-0
Sata N, Endo K, Shimura K, Koizumi M, Nagai H: A new 3-D diagnosis strategy for duodenal malignant lesions using multidetector row CT, CT virtual duodenoscopy, duodenography, and 3-D multicholangiography. Abdom Imaging 2007,32(1):66–72. 10.1007/s00261-006-9008-0
Shiozawa M, Sata N, Endo K, Koizumi M, Yasuda Y, Nagai H, Takakusaki H: Preoperative virtual simulation of adrenal tumors. Abdom Imaging 2009,34(1):113–120. 10.1007/s00261-008-9364-z
Endo K, Utano K, Togashi K, Yano T, Lefor AT, Yamamoto H, Yasuda Y, Sugimoto H: Virtual enteroscopy using air as the contrast material: a preliminary feasibility study. Dig Endosc 2010,22(3):205–210. 10.1111/j.1443-1661.2010.00986.x
Soler L, Delingette H, Malandain G, Ayache N, Koehl C, Clement JM, Dourthe O, Marescaux J: An automatic virtual patient reconstruction from CT-scans for hepatic surgical planning. Stud Health Technol Inform 2000, 70: 316–322.
Soler L, Marescaux J: Patient-specific surgical simulation. World J Surg 2008,32(2):208–212. 10.1007/s00268-007-9329-3
Mutter D, Dallemagne B, Bailey C, Soler L, Marescaux J: 3D virtual reality and selective vascular control for laparoscopic left hepatic lobectomy. Surg Endosc 2009,23(2):432–435. 10.1007/s00464-008-9931-y
MeVisLab: medical image processing and visualization., [http://www.mevislab.de]
Fraunhofer MEVIS: Computer Support for Image-Based Medicine. [http://www.mevisfraunhofer.de/en]
Arora S, Aggarwal R, Sirimanna P, Moran A, Grantcharov T, Kneebone R, Sevdalis N, Darzi A: Mental practice enhances surgical technical skills: a randomized controlled study. Ann Surg 2011,253(2):265–270. 10.1097/SLA.0b013e318207a789
Lamade W, Glombitza G, Fischer L, Chiu P, Cardenas CE Sr, Thorn M, Meinzer HP, Grenacher L, Bauer H, Lehnert T, et al.: The impact of 3-dimensional reconstructions on operation planning in liver surgery. Arch Surg 2000,135(11):1256–1261. 10.1001/archsurg.135.11.1256
Fujimoto J, Yamanaka J: Liver resection and transplantation using a novel 3D hepatectomy simulation system. Adv Med Sci 2006, 51: 7–14.
Radetzky A, Nurnberger A, Pretschner DP: Elastodynamic shape modeler: a tool for defining the deformation behavior of virtual tissues. Radiographics 2000,20(3):865–881. 10.1148/radiographics.20.3.g00mc13865
Zhu B, Gu L: A hybrid deformable model for real-time surgical simulation. Comput Med Imaging Graph 2012,36(5):356–365. 10.1016/j.compmedimag.2012.03.001
All authors have no disclosures and received no financial support.
KE and NS conceived of this study. KE carried out image processing and simulations. YI, AM, HS, YS, AS and MH participated in coordination and simulations in clinical cases. NS, AL and YY helped to draft the manuscript. All authors read and approved the final manuscript.
Electronic supplementary material
Authors’ original submitted files for images
About this article
Cite this article
Endo, K., Sata, N., Ishiguro, Y. et al. A patient-specific surgical simulator using preoperative imaging data: an interactive simulator using a three-dimensional tactile mouse. J Comput Surg 1, 10 (2014). https://doi.org/10.1186/s40244-014-0010-5
- Virtual reality
- Image processing