Department of Otolaryngology Head and Neck Surgery, Chinese PLA General Hospital, 28 Fuxing Road, Beijing 100853, ChinaChina
School of Optoelectronics, Beijing Institute of Technology, 5 Zhongguancun Road, Beijing 100081, ChinaChina
Also available as PDF (figures and tables appear in the PDF).
Recent year’s endoscopy is widely used in computer assisted surgeries. Three-Dimensional (3D) reconstruction has been presented due to the lack of depth information from endoscope images. One of the fundamental approaches in the domain of computer vision is Shape From Shading (SFS). This algorithm was proposed to obtain the shape of an object from a single intensity image. Because of the severe conditions are required in shape from shading to reconstruct 3D surface. The photometric calibration is proposed from the view of image processing. The calibration is important for illumination-based visualization techniques such as shape-from-shading. The result showed that the stability of surface reconstruction is improved when the photometric calibration is used before shape from shading. But the surface reconstruction from Shape From Shading (SFS) is the relative variation in the gray gradient field. So, the change from relative variation to absolute variation is necessary when the actual size of surroundings have to be known. Then the optical flow is introduced to solve this change in my paper. The optical tracker is also used in this system to capture the pose of endoscopy.
Keywords: Photometric calibration; 3D reconstruction; Shape from shading; Optical flow
3D reconstruction from endoscope image is a boomed technology in minimally invasive surgery, the lack of depth from endoscope image push the development of diverse technology in 3D reconstruction. The reconstruction of endoscopic sequence images1 is being frequently studied during the past years. The feature matching between sequence images is important for the subsequent reconstruction. Scale Invariant Feature Transform (SIFT) is wildly used in image matching but suffered from low matching pairs when employed in endoscope image. In this paper, the Shape From Shading (SFS) is implemented to 3D reconstruction due to its advantage. The surface is reconstructed from only a single image when the SFS is used.
SFS is one of the key technologies for three-dimensional reconstruction in computer vision. The principle is to use the change of single image gray to restore the relative height or 2 3 the lunar surface. He considered SFS problem as the inverse of the imaging process. The information of image gray is closely related with the intensity of surface reflection. Thus, the surface brightness can be changed to obtain the height information.
A few of recent works applied shape from shading to endoscopic images based on photometric calibration.4 However, in order to reconstruct an accurate shape from endoscopic images, the knowledge of light sources is necessary and important. The camera response function, light source intensity and light spatial distribution function are important when shape from shading method is used for the reconstruction. The gray card is produced by myself in this paper, which contains eighty percent to twenty percent gray. It can be regarded as the medium to calibrate the relationship between light source intensity and camera response function. When this relationship is illustrated with curve diagram, we can obtain the inverse of image intensity to compensate the original image.
In our work, an implemented photometric calibration is proposed which still performs well in synthetic image and real image. More detailed descriptions and experimental results of this method will be presented in Material and Methods. Finally, the conclusions are drawn in Discussion.
The image irradiance equation based on the Oren-Nayer model5 is widely used in shape from shading and can be expressed as follow formulas.
A B ∇z I(x, y) = R( p,q) = + (1) + ∇ 1+ ∇z 1 z
Generally speaking, the radiation source toward the radiation intensity is different in all directions, having directionality. The object with that character is called Lambert reflector. And the light intensity is defined in the formula above. If the camera coordinate system is set as reference system, and the height of object surface is set as z(x,y), so the object surface normal vector can be represented by a normal vector of the surface at various points n=(n1,n2,n3). p =∂z/∂x, q=∂z/∂y, p=-n1/n3, q=-n2/n3.
The SFS problem (1) can be formulated as the following Eikonal PDE:
∇z(x, y) = f 2 −1 ∀x ∈Ω
z x y x y x ( , ) ( , ) =ϕ ∀ ∈∂Ω
A numerical algorithm based on the high-order Go- 5,6 dunov fast sweeping scheme5,6 is proposed to solve the Eikonal
Partial Differential Equations (PDE) equation.
The gray card is produced with the same radiance, which contains eighty percent to twenty percent gray. It can be regarded as the medium to calibrate the relationship between light source intensity and camera response function Figure 1.
When we get the curved diagram in Figure 2, we can obtain the inverse of image intensity to compensate the original image and the result is showed in Figure 3. In order to demonstrate the validity of this photometric calibration, the reconstruction implemented in synthetic and real image and the result shown in Figures 4 and 5. The action of the experiment is inspired by the model in paper.
2 2
(2)
Figure 3. The upper set is indicating before calibra- tion and the under set is indicating after calibration.
(a)
(b)
Figure 5. The real image reconstruction. Left is the original im- age and right is the corresponding reconstruction result in set of (a) and (b).
Because 3D reconstruction help doctors diagnose, so choose suitable endoscope under the environment of 3D reconstruction method is particularly important. During the project investigation and experiment analysis shows that although the contrast method has its unique advantages, but also has disadvantages:1 To restore the depth of the surface is not absolute information, assuming assisted by endoscope can use external tracking equipment movement information, and then derive the depth of the surface information according to the relationship between image and object. But in the process of actual implementation need complex experimental system and equipment.2 Although a single image can be restored to form, but general surgery are taken in the process of video images. So that by extending the way of the contrast method combining reconstruction of different gray gradient information, we can get endoscopic coordinates of 3D tissues and organs.
According to the algorithm of defect group we put forward the solution. If we want to reconstruct the 3D surface shape; we should transit it to the world coordinate system and get endoscopic pose information. Although optical tracker can
locate endoscopic posture, but due to the lack of trace markers and the transformation of the relationship between endoscopic, so it is difficult to transfer 3D surface shape to world coordinates. Provide extended way for doctors under the endoscopic view of tissues and organs of the three-dimensional topography, can be a good guide for doctor’s surgery.
The small error in synthetic image demonstrates the validity of photometric calibration. The next mission is introducing the optical flow make the change from relative variation to absolute variation.
The authors declare that they have no conflicts of interest.
The study was supported by The National Youth Nature Foundation (30900385), The People’s Liberation Army General Hospital Clinical Scientific Research Foundation (2012FC- TSYS-3048).
The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.