Skip to main content

Computing, Robotics, and Imaging for the Surgical Platform

A patient-specific surgical simulator using preoperative imaging data: an interactive simulator using a three-dimensional tactile mouse

Abstract

Preoperative simulation can greatly facilitate the safe and effective conduct of surgical procedures, especially for difficult and complex operations such as hepatectomy and pancreatectomy. A computer simulation system may greatly assist surgeons to preoperatively evaluate an operation and facilitate sharing of information among the operative staff. However, there are several problems with existing simulation systems. We have developed a new surgical simulation system using a patient’s own imaging data, which has some advantages over existing systems. Individual anatomical information obtained through an imaging study is used in this system. In addition it allows interactive control, similar to what a surgeon does in a real operation. Furthermore, a surgeon can control the system intuitively using a three dimensional tactile mouse. Changing the translucency of objects makes it easy to understand complex anatomical relationships. In conclusion, this new system is a patient-specific surgical simulator and can be applied to navigation surgery, medical education and patient communication.

Background

Hepatectomy and pancreatectomy are among the most difficult abdominal surgery procedures to perform due in part to their anatomical complexity, vascular variation, and the wide variety of surgical techniques and tumor locations. Surgeons must maintain an adequate balance between function of residual tissue and curability, which poses an additional challenge.

Performing a preoperative simulation may be one of the best ways a surgeon can improve the safety of a surgical procedure. Currently, most surgeons simulate procedures mentally to perform them safely and effectively. However, this level of preoperational evaluation and the lack of realistic simulation allows for little information sharing among members of the team and objectivity. Computer simulation using three-dimensional (3D) visualization techniques helps to address these problems.

Many surgical simulators are available for preoperative evaluation and surgical education [1]–[6], but several problems still exist. First, individual variations in anatomy and tumor location are not reflected in existing simulation systems. Simulators for preoperative evaluation, which are most commonly used for hepatectomy procedures, cannot simulate the steps of an operation in an interactive manner. Finally, nearly all simulators currently used in surgical education are programmed to present a limited number of prearranged surgical scenarios.

Significant advances in diagnostic imaging modalities and image processing software, which have enabled surgeons to obtain detailed anatomical information preoperatively for each patient, should be incorporated in surgical simulation. It would be especially useful for difficult procedures such as hepatectomy and pancreatectomy. The present study was undertaken to create a surgical simulator based on radiological images from an individual patient and demonstrate the feasibility of using this system in a real clinical environment.

Methods

This study was approved by the Ethics Committee of Jichi Medical University Hospital. The study consisted of four steps:

  1. 1.

    Obtaining imaging information. Multi-detector row CT (MDCT) scan (SOMATOM Definition Flash®, Siemens, Munich, Germany) images were obtained during preoperative evaluation of patients scheduled to undergo hepatectomy or pancreatectomy. A dynamic study using iodinated contrast medium was performed, following a routine imaging protocol for detailed imaging examinations. Arterial, portal venous, and equilibrium phases were imaged for planned hepatic resections, and the early arterial phase, late arterial phase, and portal/equilibrium phase were scanned for planned pancreatic resections. All CT scans were performed in the expiratory phase. All images were stored as DICOM (Digital Imaging and communication in Medicine) datasets (slice thickness, 1 mm) and sent to a workstation.

  2. 2.

    Segmentation. Image processing software (Synapse Vincent®, Fujifilm Medical, Tokyo, Japan) was used for segmentation of each component from the DICOM datasets. Automatic and manual segmentations were performed in combination. For hepatectomy procedures, segmentation of the portal vein, hepatic vein, and hepatic parenchyma was performed using both portal-venous and equilibrium phase images for each. Precontrast-enhanced and arterial phase images were used as references. For pancreatectomy procedures, arteries and pancreatic parenchyma were extracted from early and late arterial phase images, while the portal and surrounding veins were extracted from portal venous phase images. For all cases, tumors were extracted from the phase that allowed the easiest identification of the tumor. Other organs and objects, such as the biliary tract and biliary stents, were extracted as necessary. All segmented components were saved in separate STL (stereolithography) formats.

  3. 3.

    Creation of a virtual 3-Dimensional model. All segmented components were transferred to 3D modeling software (FreeForm®, Sensable, Woburn, MA, USA). Surface smoothing of each component was applied automatically, and manually corrected if necessary. Each component was colored for easy recognition. Finally, a virtual 3D model was created by integrating all segmented components in a virtual space. Positional information stored as DICOM data was used to automatically adjust three-dimensional relations among segments. Manual adjustment was performed if there were large positional differences between scan phases.

  4. 4.

    Preoperative simulation and intraoperative navigation. The surgical procedure was simulated using the virtual 3D model and manipulated within FreeForm® using a 3D tactile mouse that functioned as a variety of surgical instruments to cut, retract, and isolate the tissue (Figure 1a). FreeForm® was brought into the operating room and information was shared by displaying images on the operating room monitor (Figure 1b). As the actual operation progressed, the same techniques were carried out on the virtual 3D model.

Figure 1
figure 1

Simulation of the surgical procedure. a: Setup for pre-operative simulation. The virtual 3D model was controlled with a keyboard and a 3D tactile mouse. Rotation and opacity control of the virtual 3D model were controlled by a keyboard. A 3D tactile mouse was used to manipulate surgical instruments such as a scalpel, forceps, and scissors. b: Operating room setup including the simulation system. The same system was brought to the operating room and connected to a large external monitor located at the patient’s head to allow intraoperative visualization.

Results and discussion

Results

  1. 1.

    Process time required to create a virtual 3D model. It took approximately 2 h to segment all components for a hepatectomy simulation. It took approximately 2.5 h to segment all components for a pancreatectomy simulation. Creation of a virtual 3D model with the modeling software required approximately 30 min.

  2. 2.

    Simulation of surgical procedures. Surgical trainees under the direction of attending surgeons performed the simulations. During the simulations, the object opacity was changed to make otherwise invisible objects visible (Figure 2).

Figure 2
figure 2

Altering the opacity of the liver parenchyma. Images of the virtual 3D model were observed from the same point of view, and only the translucency of the liver parenchyma was changed. Left: No translucency of the parenchyma. Middle: Parenchyma is semi-translucent. Right: Parenchyma is completely translucent, facilitating visualization of the vascular structures within.

A) The following are the standard procedures with data from a healthy volunteer:

  1. a)

    Lateral segmentectomy of the liver. First, the umbilical portion of the portal vein located within the hepatic parenchyma was identified and exposed. Next, the Glisson's capsule corresponding to the lateral segment was divided, and the hepatic parenchyma was resected. Finally, the root of the left hepatic vein was isolated and transected (Figure 3) (Additional file 1: Movie 1).

    Additional file 1:Virtual left lateral segmentectomy. Lateral segmentectomy of the liver using a virtual 3D model. (1) Inferior view of the liver. (2) The umbilical portion was identified and exposed. Glisson's capsule corresponding to the lateral segment was divided. (3) Hepatic parenchyma was transected. The left hepatic vein is then clearly exposed. (4) Lateral segment was resected. (MOV 9 MB)

Figure 3
figure 3

Lateral segmentectomy of the liver using a virtual 3D model. (1) Inferior view of the liver. (2) The umbilical portion was identified and exposed. Glisson's capsule corresponding to the lateral segment was divided. (3) Hepatic parenchyma was transected. The left hepatic vein is then clearly exposed. (4) Lateral segment was resected.

  1. b)

    Right hepatectomy of the liver. First, the hepatic portal region was exposed, and the right side of Glisson’s capsule was identified and divided. Then, the resection line, which was calculated by image processing software, was drawn on the surface of the virtual 3D model. Following this, the hepatic parenchyma and smaller vessels were transected. Finally, the roots of the right hepatic vein and right inferior hepatic vein were transected. As a result of virtual resection, the middle hepatic vein was exposed on the resection plane of the residual hepatic parenchyma (Figure 4) (Additional file 2: Movie 2).

    Additional file 2:Virtual right hepatectomy. Right hepatectomy of the liver on a virtual 3D model. (1) Right side of Glisson’s capsule was exposed and divided. (2) Demarcation line in the liver parenchyma was identified. (3) Hepatic parenchyma and smaller vessels were transected. (4) Right hepatic vein was exposed in the half-translucency view and transected. (5) Virtual right lobectomy was completed. (6) The middle hepatic vein was exposed at the resection plane of the residual liver. (MOV 9 MB)

Figure 4
figure 4

Right hepatectomy of the liver on a virtual 3D model. (1) Right side of Glisson’s capsule was exposed and divided. (2) Demarcation line in the liver parenchyma was identified. (3) Hepatic parenchyma and smaller vessels were transected. (4) Right hepatic vein was exposed in the half-translucency view and transected. (5) Virtual right lobectomy was completed. (6) The middle hepatic vein was exposed at the resection plane of the residual liver.

  1. B)

    The following clinical cases are simulated in the study:

  2. a)

    Hepatectomy. The tumor was located in segment 8 of the liver and a partial resection was performed. In this patient, the right hepatic vein was oversewn and the middle hepatic vein was partially exposed on the surface of the resection plane. Preoperative simulation results correlated well with the operative findings. Changing the translucency of the liver parenchyma allowed for easy anatomical evaluation of the relationship between the components (Figure 5).

Figure 5
figure 5

Clinical case: partial resection of the liver. In this patient, the tumor was located in segment 8 of the liver. Cut ends of the right hepatic vein (RHV) and Glisson’s capsule at segment 8 (GS8) are shown. The middle hepatic vein (MHV) was exposed at the surface of the resection plane. The half-translucency of the hepatic parenchyma allowed for easy anatomical identification of each component.

  1. b)

    Pancreatectomy. For this patient, we performed a pancreaticoduodenectomy for carcinoma of the head of the pancreas. The right hepatic artery originated from the superior mesenteric artery and ran behind the portal vein and the hepatic bile duct. The virtual image after resection effectively visualized the anatomical relationships between the artery, portal vein, bile duct, and pancreatic parenchyma. Image data were consistent with the surgical findings (Figure 6).

Figure 6
figure 6

Clinical case: pancreaticoduodenectomy. The tumor was located in the head of the pancreas, and a pancreaticoduodenectomy was performed. In this patient, the right hepatic vein (RHV) originated from the superior mesenteric artery and ran along the posterior side of the portal vein (PV). The ligated gastroduodenal artery (GDA) and adjacent hepatic bile duct were visualized. Resection of the pancreatic parenchyma (P) was performed in the same manner as the actual operation.

Discussion

Preoperative simulation is a useful adjunct to ensure safety in the conduct of surgical procedures. Hepatectomy and pancreatectomy are particularly challenging abdominal surgical procedures, and require precise simulation in order to achieve a successful surgical procedure and a smooth post-operative course. Every surgeon mentally simulates the operation using imaging data preoperatively. However this practice is questionable in terms of its objectivity and reproducibility.

Computer simulation is a promising approach to address these problems. With recent advances in imaging technology and 3D visualization techniques, accurate images of each patient can be obtained preoperatively. We have published several reports on the use of 3D visualization techniques for the diagnosis of various diseases and therapeutic planning [7]–[9]. Other simulation systems have been recently described. The IRCAD software establishes a 3D virtual surgical planning system, and has been reported regarding the utility of patient-specific surgical simulation for laparoscopic liver surgery [10]–[12]. MeVis also supports planning a hepatectomy using patient specific data. These simulators are specialized software, and differ somewhat from this system, which allows easy application to various organs with a stepwise simulation of the operative procedure [13],[14].

Arora et al. reported that mental practice with a virtual simulator can enhance surgical skill for the conduct of a laparoscopic cholecystectomy [15]. However existing surgical simulators cannot simulate complex surgical procedures step-by-step, such as hepatectomy and pancreatectomy, and do not reflect individual anatomical information obtained from imaging studies.

The ideal surgical simulator should satisfy the following three conditions. First, it should reflect patient-specific anatomical information that is input into the simulator. Second, it should allow interactive control. Third, the user interface should be intuitive. Currently, some computer simulators are available for surgical training, but most have limited, pre-programmed scenarios [1],[4],[6]. In liver surgery, the use of an image processing workstation allows one to calculate resection volume and visualize completion images [16],[17]. However, this existing system is not interactive, so the steps of the operation cannot be simulated. This general-purpose software can be applied easily to various organs and simulate the procedure in a step-by-step fashion. Third, intuitive control is achieved with a 3D tactile mouse. In addition to the above, the opacity of each object is easily changed, allowing this simulator to make otherwise invisible objects visible.

Comprehensive anatomical understanding is intuitively established by confirming invisible internal structures through changes in transparency from every observational viewpoint on the monitor, including those that may be unrealistic when dealing with an actual patient. The understanding gained from simulation with this system gives a surgeon confidence when performing the actual surgical procedure.

This technique for surgical simulation does have limitations. First, these models are basically rigid and it is impossible to visualize a realistic operative field. Some of the physical characteristics of the tissue (e.g., firmness and elasticity) cannot be simulated with this method. Simulation of accurate organ deformation is also difficult. Further work is required to develop simulators capable of reflecting these characteristics of tissue [18],[19]. Second, creation of a virtual 3D model, especially one that involves segmentation of each component, demands significant time and effort. Each of the multiple steps required to create a virtual 3D model needs to be automated in order for this method to become more generally useful. Finally, the image processing software used in this simulation system is expensive. Synapse Vincent® and FreeForm® software cost US$100,000 and US$25,000, respectively.

We believe that in the future, application of this approach to modeling and simulation can be expanded to other facets of medical practice and education. For example, if this simulation system and the actual procedure could be performed simultaneously in the operating room, this system could serve as a surgical navigator. The understanding of complex anatomy and surgical procedures by surgical trainees at all levels could be greatly enhanced by using this system. Preoperative simulation not only allows attending surgeons and surgical trainees to achieve a common understanding but also helps medical students and other staff members to learn more about the operative procedure. Finally, this simulation system could be used as a way to visually communicate with patients, to better explain to them about an upcoming surgical procedure. The future directions of this study include objective and subjective evaluations of the system and will be evaluated using a questionnaire for both surgeons and patients.

Conclusions

We have developed a patient-specific surgical simulator for hepatectomy and pancreatectomy by creating a virtual 3D model using patient imaging data and have established the feasibility of using this system in a real clinical environment. In addition to its utility in surgical simulation, this method can be applied to navigation surgery, medical education and patient communication.

Additional files

Abbreviations

DICOM:

Digital Imaging and communication in Medicine

STL:

stereolithography

References

  1. Sutton C, McCloy R, Middlebrook A, Chater P, Wilson M, Stone R: MIST VR. A laparoscopic surgery procedures trainer and evaluator. Stud Health Technol Inform 1997, 39: 598–607.

    Google Scholar 

  2. Marescaux J, Clement JM, Tassetti V, Koehl C, Cotin S, Russier Y, Mutter D, Delingette H, Ayache N: Virtual reality applied to hepatic surgery simulation: the next revolution. Ann Surg 1998,228(5):627–634. 10.1097/00000658-199811000-00001

    Article  Google Scholar 

  3. O'Toole RV, Playter RR, Krummel TM, Blank WC, Cornelius NH, Roberts WR, Bell WJ, Raibert M: Measuring and developing suturing technique with a virtual reality surgical simulator. J Am Coll Surg 1999,189(1):114–127. 10.1016/S1072-7515(99)00076-9

    Article  Google Scholar 

  4. Hyltander A, Liljegren E, Rhodin PH, Lonroth H: The transfer of basic skills learned in a laparoscopic simulator to the operating room. Surg Endosc 2002,16(9):1324–1328. 10.1007/s00464-001-9184-5

    Article  Google Scholar 

  5. Saito S, Yamanaka J, Miura K, Nakao N, Nagao T, Sugimoto T, Hirano T, Kuroda N, Iimuro Y, Fujimoto J: A novel 3D hepatectomy simulation based on liver circulation: application to liver resection and transplantation. Hepatology 2005,41(6):1297–1304. 10.1002/hep.20684

    Article  Google Scholar 

  6. Fairhurst K, Strickland A, Maddern G: The LapSim virtual reality simulator: promising but not yet proven. Surg Endosc 2011,25(2):343–355. 10.1007/s00464-010-1181-0

    Article  Google Scholar 

  7. Sata N, Endo K, Shimura K, Koizumi M, Nagai H: A new 3-D diagnosis strategy for duodenal malignant lesions using multidetector row CT, CT virtual duodenoscopy, duodenography, and 3-D multicholangiography. Abdom Imaging 2007,32(1):66–72. 10.1007/s00261-006-9008-0

    Article  Google Scholar 

  8. Shiozawa M, Sata N, Endo K, Koizumi M, Yasuda Y, Nagai H, Takakusaki H: Preoperative virtual simulation of adrenal tumors. Abdom Imaging 2009,34(1):113–120. 10.1007/s00261-008-9364-z

    Article  Google Scholar 

  9. Endo K, Utano K, Togashi K, Yano T, Lefor AT, Yamamoto H, Yasuda Y, Sugimoto H: Virtual enteroscopy using air as the contrast material: a preliminary feasibility study. Dig Endosc 2010,22(3):205–210. 10.1111/j.1443-1661.2010.00986.x

    Article  Google Scholar 

  10. Soler L, Delingette H, Malandain G, Ayache N, Koehl C, Clement JM, Dourthe O, Marescaux J: An automatic virtual patient reconstruction from CT-scans for hepatic surgical planning. Stud Health Technol Inform 2000, 70: 316–322.

    Google Scholar 

  11. Soler L, Marescaux J: Patient-specific surgical simulation. World J Surg 2008,32(2):208–212. 10.1007/s00268-007-9329-3

    Article  Google Scholar 

  12. Mutter D, Dallemagne B, Bailey C, Soler L, Marescaux J: 3D virtual reality and selective vascular control for laparoscopic left hepatic lobectomy. Surg Endosc 2009,23(2):432–435. 10.1007/s00464-008-9931-y

    Article  Google Scholar 

  13. MeVisLab: medical image processing and visualization., [http://www.mevislab.de]

  14. Fraunhofer MEVIS: Computer Support for Image-Based Medicine. [http://www.mevisfraunhofer.de/en]

  15. Arora S, Aggarwal R, Sirimanna P, Moran A, Grantcharov T, Kneebone R, Sevdalis N, Darzi A: Mental practice enhances surgical technical skills: a randomized controlled study. Ann Surg 2011,253(2):265–270. 10.1097/SLA.0b013e318207a789

    Article  Google Scholar 

  16. Lamade W, Glombitza G, Fischer L, Chiu P, Cardenas CE Sr, Thorn M, Meinzer HP, Grenacher L, Bauer H, Lehnert T, et al.: The impact of 3-dimensional reconstructions on operation planning in liver surgery. Arch Surg 2000,135(11):1256–1261. 10.1001/archsurg.135.11.1256

    Article  Google Scholar 

  17. Fujimoto J, Yamanaka J: Liver resection and transplantation using a novel 3D hepatectomy simulation system. Adv Med Sci 2006, 51: 7–14.

    Google Scholar 

  18. Radetzky A, Nurnberger A, Pretschner DP: Elastodynamic shape modeler: a tool for defining the deformation behavior of virtual tissues. Radiographics 2000,20(3):865–881. 10.1148/radiographics.20.3.g00mc13865

    Article  Google Scholar 

  19. Zhu B, Gu L: A hybrid deformable model for real-time surgical simulation. Comput Med Imaging Graph 2012,36(5):356–365. 10.1016/j.compmedimag.2012.03.001

    Article  MathSciNet  Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Kazuhiro Endo.

Additional information

Competing interests

All authors have no disclosures and received no financial support.

Authors’ contributions

KE and NS conceived of this study. KE carried out image processing and simulations. YI, AM, HS, YS, AS and MH participated in coordination and simulations in clinical cases. NS, AL and YY helped to draft the manuscript. All authors read and approved the final manuscript.

Electronic supplementary material

Authors’ original submitted files for images

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (https://creativecommons.org/licenses/by/4.0), which permits use, duplication, adaptation, distribution, and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Endo, K., Sata, N., Ishiguro, Y. et al. A patient-specific surgical simulator using preoperative imaging data: an interactive simulator using a three-dimensional tactile mouse. J Comput Surg 1, 10 (2014). https://doi.org/10.1186/s40244-014-0010-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s40244-014-0010-5

Keywords