3D video capturing for multiprojector realistic display
Today's 3D television receivers require viewers to wear special glasses or have a narrow viewing angle in which to experience the full 3D effect. We are researching a video system to realize super-realistic telecommunication free of these shortcomings. We previously proposed a large-screen, glasses-free, 3D-display method and developed several prototypes of such devices using high-definition projectors.1,2 However, the displayed 3D images are only computer-generated graphics or still images of real objects. It remains impossible to display 3D images of real moving objects like people that are crucial for realistic television. In the work described here, we employed a multicamera array to capture 3D images of real moving objects for a multiprojector-type display.
Figure 1 (top) and Figure 2 show our 3D camera system. We arrange multiple cameras in the same horizontal plane, with each apparatus aimed at the convergence point. The dashed lines in Figure 1 show the units' optical axes and their convergence at point O on plane P. The cameras are adjusted to reduce misalignment of their position, but we note that the distances from the cameras to the subject are not equal. The captured images, which are also electrically compensated by image signal processing for more accurate calibration, are corrected for color and brightness balance in each image.3,4
In our experiments, we used 30 high-definition video cameras to capture moving subjects: see Figure 2 (left). The cameras operate through synchronization of control signals. Distance D from the camera array to convergence plane P is 3m, and pc (the pitch of the camera units) is 65mm. The range of the capturing angles is about ±17°.
We determined the camera arrangement by considering the eventual arrangement of liquid-crystal-on-silicon projectors for the 3D receiver/display system, especially the positions of the projector array and the screen. In the glasses-free-3D display system shown at the bottom of Figure 1, reconstruction angle θn of nth projector units is expressed by θn= arctan (npd/L), where pd is the pitch of the projector units in the horizontal direction and L is the distance between the projector array and the screen. On the other hand, in our 3D camera system, when we use the same number of camera units as display units, the capturing angle of the nth camera is φn= arctan (npc/D), where D is the distance from the camera array to convergence plane P. To represent the correct multiviewpoint image of the 3D image, since φn must equal θn, we determined the camera pitch to be pc= Dpd/L.
The display uses an array of projectors: see Figure 2 (right).2 Images are projected, superimposed, on a special screen that combines anisotropic diffuser film with a condenser lens. The film has a small angle in the horizontal direction relative to the incident light and a wide diffusion angle in the vertical direction. These characteristics enable the system to produce different images at various horizontal angles, allowing users to observe parallax images based on the horizontal position. Conversely, in the vertical direction, the incident light becomes widely diffused, eliminating the effects of the projection angle in that direction.
Unlike the camera array, the projector units in the prototype 70-inch 3D display are arranged in parallel. They project their individual images onto the screen through off-axis or ‘shifted’ lenses. To represent correct 3D images, the captured multiviewpoint images are geometrically compensated by image processing to optimize the projector arrangement.
We successfully captured and displayed 3D video of real moving objects in our 3D video system for the first time. Figure 3(a–c) shows the reconstructed 3D images as observed from the left, center, and right, respectively. Parallax images exist based on the observation location.
In summary, we can capture and display 3D images of moving objects using our prototype 3D video system. In the future, we plan to investigate a fast image-processing method for multiviewpoint images for high-speed capture and display of moving 3D objects.
The 3D display system was developed with JVC KENWOOD Holdings Inc.
Masahiro Kawakita received his BS and MS in physics from Kyushu University (1988 and 1990), respectively, and his PhD in electronic engineering from Tokyo University, Japan (2005). In 1990, he joined NHK, Tokyo. Since 2010, he has worked for NICT. His current research interests include 3D video systems.
Sabri Gurbuz received a PhD from Clemson University (2002). He worked for Philips Research Labs (2001) and Advanced Telecommunications Research Institute International (ATR, 2003–2009). Since 2009, he has been a researcher at NICT. His research interests include signal and image processing, 3D imaging, and computer vision. He is a member of IEEE and SPIE.
Shoichiro Iwasawa is an expert researcher at NICT and a member of the Association for Computing Machinery. He earned a PhD in electrical engineering and electronics from Seikei University, Japan. His fields of interest include 3D display, pervasive computing, computer graphics, and computer vision.
Roberto Lopez-Gulliver received an MS from Ritsumeikan University (1993) and a PhD from Kobe University (2006), both in computer science. He is an expert researcher at NICT. His research interests include multiuser human-computer interaction, computer graphics, and autostereoscopic 3D displays.
Hiroshi Ando is the group leader of the Multimodal Communication Group at the Universal Media Research Center, NICT, and head of the Department of Perceptual and Cognitive Dynamics, Media Information Science Laboratories, ATR, Japan. He received his PhD in computational neuroscience from the Massachusetts Institute of Technology (Cambridge, MA) in 1993.
Naomi Inoue, a director of the Universal Media Research Center, has been involved in ultrarealistic communication research since 2006. He was promoted to a director in 2010. He has served on the program committee of two international conferences on universal communication and 3D systems and applications.
Sumio Yano received his BA and PhD in engineering from the University of Electro-Communications (1977 and 1993, respectively). He joined NHK in 1977. He was a loan employee at ATR and NICT. He currently works at NHK Science and Technology Research Laboratories.
