The occurrence of missing regions in output images is a critical issue when rendering a scene from novel vantage points using depth-based view synthesis. These regions typically have to be filled using inpainting algorithms, which are slow and might yield unconvincing results. Understanding the likelihood of the occurrence of these missing regions can help us design better, application-specific data representations and camera systems by knowing which vantage points should be captured and stored to minimize disocclusion holes in the synthesized novel views. In this paper, we propose a statistical model that predicts the likelihood of missing data in synthesized images as a function of the viewpoint translation. Scene-dependent model parameters are efficiently estimated using simple shift and scaling transformations on the source depth images without needing view synthesis.

The perceptions of comfort and presence in virtual reality (VR) are tightly linked to the richness of the visual cues that are made available to the viewer and the fidelity with which they are rendered. In this work, we present two subjective studies, the first of which is aimed at understanding how the quality of experience in VR is affected by the presence and absence of motion parallax and binocular stereopsis. The second one reports an illusion where, in the absence of stereo disparity, objects are perceived to be significantly larger compared to when the stereo cue is available, with the average magnification being as high as 2x for objects up-close. We conclude that while rendering the correct motion parallax is critical for mitigating visual discomfort, the viewers’ ability of accurate size perception is largely dependent on binocular stereopsis.

Allowing viewers to explore virtual reality in a head-mounted display with six degrees of freedom (6-DoF) greatly enhances the associated immersion and comfort. It makes the experience more compelling compared to a fixed-viewpoint 2-DoF rendering produced by conventional algorithms using data from a stationary camera rig. In this work, we use subjective testing to study the relative importance of, and the interaction between, motion parallax and binocular disparity as depth cues that shape the perception of 3D environments by human viewers. Additionally, we use the recorded head trajectories to estimate the distribution of the head movements of a sedentary viewer exploring a virtual environment with 6-DoF. Finally, we demonstrate a real-time virtual reality rendering system that uses a Stacked OmniStereo intermediary representation to provide a 6-DoF viewing experience by utilizing data from a stationary camera rig. We outline the challenges involved in developing such a system and discuss the limitations of our approach.

Motion parallax is an important cue for depth perception. Rendering it accurately can lead to a more natural and immersive virtual reality (VR) experience. We introduce Stacked Omnistereo, a novel data representation that can render immersive video with six degrees of freedom (DoF). We compare the proposed representation against other depth-based and image-based motion parallax techniques using natural as well as synthetic scenes. We show that the proposed representation can synthesize plausible, view-dependent specular highlights, is compact compared to light fields, and outperforms state-of-the-art VR representations by up to 3 dB when evaluated with 6 DoF head motion.

Cinematic virtual reality (VR) aims to provide immersive visual experiences of real-world scenes on head-mounted displays. Current cinematic VR systems support stereo cues, but not focus cues that are important for depth perception and comfortable viewing. We propose a new content representation, depth augmented stereo panorama (DASP), which permits generating light fields (LFs) across the observer’s pupils, achieving an order of magnitude reduction in data requirements compared to the existing techniques. The LF generation and refocusing capabilities of DASP are evaluated for both, computer-generated and real scenes. Results indicate that DASP can successfully create stereo as well as support focus cues.

Cinematic virtual reality (VR) aims to provide immersive visual experiences of real-world scenes on head-mounted displays. Current cinematic VR systems employ omnidirectional stereo videos from a fixed position, and therefore do not address head-motion parallax, which is an important cue for depth perception. We propose a new 3D video representation, referred to as depth augmented stereo panorama (DASP), to address this issue. DASP is developed considering data capture, postproduction, streaming, and rendering stages of the VR pipeline. The capabilities of this representation are evaluated by comparing the generated viewports with those from known 3D models. Results indicate that DASP can successfully create stereo and induce head-motion parallax in a predefined operating range.