Light field imaging based on a single-tier camera equipped with a microlens array – also known as integral, holoscopic, and plenoptic imaging – has currently risen up as a practical and prospective approach for future visual applications and services. However, successfully deploying actual light field imaging applications and services will require developing adequate coding solutions to efficiently handle the massive amount of data involved in these systems. In this context, self-similarity compensated prediction is a non-local spatial prediction scheme based on block matching that has been shown to achieve high efficiency for light field image coding based on the High Efficiency Video Coding (HEVC) standard. As previously shown by the authors, this is possible by simply averaging two predictor blocks that are jointly estimated from a causal search window in the current frame itself, referred to as self-similarity bi-prediction. However, theoretical analyses for motion compensated bi-prediction have suggested that it is still possible to achieve further rate-distortion performance improvements by adaptively estimating the weighting coefficients of the two predictor blocks.
Therefore, this paper presents a comprehensive study of the rate-distortion performance for HEVC-based light field image coding when using different sets of weighting coefficients for self-similarity bi-prediction. Experimental results demonstrate that it is possible to extend the previous theoretical conclusions to light field image coding and show that the proposed adaptive weighting coefficient selection leads to up to 5 % of bit savings compared to the previous self-similarity bi-prediction scheme.
Light field imaging based on microlens arrays – also known as plenoptic, holoscopic and integral imaging – has recently
risen up as feasible and prospective technology due to its ability to support functionalities not straightforwardly available
in conventional imaging systems, such as: post-production refocusing and depth of field changing. However, to
gradually reach the consumer market and to provide interoperability with current 2D and 3D representations, a display
scalable coding solution is essential.
In this context, this paper proposes an improved display scalable light field codec comprising a three-layer hierarchical
coding architecture (previously proposed by the authors) that provides interoperability with 2D (Base Layer) and 3D
stereo and multiview (First Layer) representations, while the Second Layer supports the complete light field content. For
further improving the compression performance, novel exemplar-based inter-layer coding tools are proposed here for the
Second Layer, namely: (i) an inter-layer reference picture construction relying on an exemplar-based optimization
algorithm for texture synthesis, and (ii) a direct prediction mode based on exemplar texture samples from lower layers.
Experimental results show that the proposed solution performs better than the tested benchmark solutions, including the
authors’ previous scalable codec.
Holoscopic imaging became a prospective glassless 3D technology to provide more natural 3D viewing experiences to the end user. Additionally, holoscopic systems also allow new post-production degrees of freedom, such as controlling the plane of focus or the viewing angle presented to the user. However, to successfully introduce this technology into the consumer market, a display scalable coding approach is essential to achieve backward compatibility with legacy 2D and 3D displays. Moreover, to effectively transmit 3D holoscopic content over error-prone networks, e.g., wireless networks or the Internet, error resilience techniques are required to mitigate the impact of data impairments in the user quality perception. Therefore, it is essential to deeply understand the impact of packet losses in terms of decoding video quality for the specific case of 3D holoscopic content, notably when a scalable approach is used. In this context, this paper studies the impact of packet losses when using a three-layer display scalable 3D holoscopic video coding architecture previously proposed, where each layer represents a different level of display scalability (i.e., L0 - 2D, L1 - stereo or multiview, and L2 - full 3D holoscopic). For this, a simple error concealment algorithm is used, which makes use of inter-layer redundancy between multiview and 3D holoscopic content and the inherent correlation of the 3D holoscopic content to estimate lost data. Furthermore, a study of the influence of 2D views generation parameters used in lower layers on the performance of the used error concealment algorithm is also presented.
Holoscopic imaging, also known as integral imaging, has been recently attracting the attention of the research
community, as a promising glassless 3D technology due to its ability to create a more realistic depth illusion than the
current stereoscopic or multiview solutions. However, in order to gradually introduce this technology into the consumer
market and to efficiently deliver 3D holoscopic content to end-users, backward compatibility with legacy displays is
essential. Consequently, to enable 3D holoscopic content to be delivered and presented on legacy displays, a display
scalable 3D holoscopic coding approach is required.
Hence, this paper presents a display scalable architecture for 3D holoscopic video coding with a three-layer approach,
where each layer represents a different level of display scalability: Layer 0 - a single 2D view; Layer 1 - 3D stereo or
multiview; and Layer 2 - the full 3D holoscopic content. In this context, a prediction method is proposed, which
combines inter-layer prediction, aiming to exploit the existing redundancy between the multiview and the 3D holoscopic
layers, with self-similarity compensated prediction (previously proposed by the authors for non-scalable 3D holoscopic
video coding), aiming to exploit the spatial redundancy inherent to the 3D holoscopic enhancement layer.
Experimental results show that the proposed combined prediction can improve significantly the rate-distortion
performance of scalable 3D holoscopic video coding with respect to the authors’ previously proposed solutions, where
only inter-layer or only self-similarity prediction is used.
This paper proposes a network-aware macroblock (MB) coding mode decision method, which is both error resilient and
coding efficient. This method differs from traditional mode decision methods since MB mode decisions are made by
simultaneously taking into account: i) their rate-distortion (RD) cost and also ii) their impact on error resilience by
considering feedback information from the underlying network regarding current error characteristics. By doing so, the
amount of Intra coded MBs can be varied to better suit, in a cost efficient way, the current state of the network and,
therefore, further improve the decoded video quality for a given packet loss rate. The proposed approach outperforms a
network-aware version of the H.264/AVC reference software with cyclic MB Intra refresh, for typical test sequences
encoded at various bit rates and for several error conditions in terms of packet loss rate.