The plenoptic function was originally defined as a complete record of the 3D structure of radiance in a scene and its dependence on a number of different parameters including position, angle, wavelength, polarization, etc. Recently-developed plenoptic cameras typically capture only the geometric aspects of the plenoptic function. Using this information, computational photography can render images with an infinite variety of features such as focus, depth of field, and parallax. Less attention has been paid to other, nonspatial, parameters of the plenoptic function that could also be captured. In this paper, we develop the microlens-based image sensor (aka the Lippmann sensor) as a generalized plenoptic capture device, able to capture additional information based on filters/modifiers placed on different microlenses. Multimodal capture can comprise many different parameters such as high-dynamic range, multispectral, and so on. For this paper we explore two particular examples in detail: polarization capture based on interleaved polarization filters, and capture with extended depth of field based on microlenses with different focal lengths.
Recent realizations of hand-held plenoptic cameras have given rise to previously unexplored effects in photography.
Designing a mobile phone plenoptic camera is becoming feasible with the significant increase of computing
power of mobile devices and the introduction of System on a Chip. However, capturing high numbers of views is
still impractical due to special requirements such as ultra-thin camera and low costs. In this paper, we analyze a
mobile plenoptic camera solution with a small number of views. Such a camera can produce a refocusable high
resolution final image if a depth map is generated for every pixel in the sparse set of views. With the captured
multi-view images, the obstacle to recovering a high-resolution depth is occlusions. To robustly resolve these, we
first analyze the behavior of pixels in such situations. We show that even under severe occlusion, one can still
distinguish different depth layers based on statistics. We estimate the depth of each pixel by discretizing the
space in the scene and conducting plane sweeping. Specifically, for each given depth, we gather all corresponding
pixels from other views and model the in-focus pixels as a Gaussian distribution. We show how it is possible to
distinguish occlusion pixels, and in-focus pixels in order to find the depths. Final depth maps are computed in
real scenes captured by a mobile plenoptic camera.
The Lytro camera is the first implementation of a plenoptic camera for the consumer market. We consider it a successful
example of the miniaturization aided by the increase in computational power characterizing mobile computational
photography. The plenoptic camera approach to radiance capture uses a microlens array as an imaging system focused
on the focal plane of the main camera lens. This paper analyzes the performance of Lytro camera from a system level
perspective, considering the Lytro camera as a black box, and uses our interpretation of Lytro image data saved by the
camera. We present our findings based on our interpretation of Lytro camera file structure, image calibration and image
rendering; in this context, artifacts and final image resolution are discussed.
A special section of the Journal of Electronic Imaging (JEI) will replace the conference proceedings for papers presented at the SPIE conference on Mobile Computational Photography (SPIE Conference 8667D). The papers will be published early in 2013 and can be viewed at http://electronicimaging.spiedigitallibrary.org, Vol. 22 (2013).
For many photographers today, mobile photography is photography. According to statistics available on the photo-sharing site CrossRef[[XSLOpenURL/]], over the last year the most popular camera used to take photos on the site has been an Apple iPhone. In fact, during the summer of 2012, the iPhone 4 and the iPhone 4S occupied the number one and number two spots. The success of mobile photography perhaps goes hand in glove with the rise of social networking sites. Mobile photography has enabled any individual with a cell phone (the ubiquitous mobile camera) to quickly and easily share visual images of their lives. Mobile cameras are cheap, small, lightweight, and connected—pictures can be taken anytime and anywhere—and shared immediately.
Processing and rendering of plenoptic camera data requires significant computational power and memory bandwidth. At
the same time, real-time rendering performance is highly desirable so that users can interactively explore the infinite
variety of images that can be rendered from a single plenoptic image. In this paper we describe a GPU-based approach
for lightfield processing and rendering, with which we are able to achieve interactive performance for focused plenoptic
rendering tasks such as refocusing and novel-view generation. We present a progression of rendering approaches for
focused plenoptic camera data and analyze their performance on popular GPU-based systems. Our analyses are validated
with experimental results on commercially available GPU hardware. Even for complicated rendering algorithms, we are
able to render 39Mpixel plenoptic data to 2Mpixel images with frame rates in excess of 500 frames per second.
The focused plenoptic camera is based on the Lippmann sensor: an
array of microlenses focused on the pixels of a conventional image sensor. This device samples the radiance, or plenoptic
function, as an array of cameras with large depth of field, focused at a certain plane in front of the microlenses. For the
purpose of digital refocusing (which is one of the important applications) the depth of field needs to be large, but there are
fundamental optical limitations to this. The solution of the above problem is to use and array of interleaved microlenses
of different focal lengths, focused at two or more different planes. In this way a focused image can be constructed at any
depth of focus, and a really wide range of digital refocusing can be achieved. This paper presents our theory and results of
implementing such camera. Real world images are demonstrating the extended capabilities, and limitations are discussed.
Plenoptic cameras are intended to fully capture the light rays in a scene. Using this information, optical elements can
be applied to a scene computationally rather than physically-allowing an infinite variety of pictures to be rendered after
the fact from the same plenoptic data. Practical plenoptic cameras necessarily capture discrete samples of the plenoptic
function, which together with the overall camera design, can constrain the variety and quality of rendered images. In this
paper we specifically analyze the nature of the discrete data that plenoptic cameras capture, in a manner that unifies the
traditional and focused plenoptic camera designs. We use the optical properties of plenoptic cameras to derive the geometry
of discrete plenoptic function capture. Based on this geometry, we derive expressions for expected resolution from a
captured plenoptic function. Our analysis allows us to define the "focused plenoptic condition," a necessary condition in
the optical design that distinguishes the traditional plenoptic camera from the focused plenoptic camera.
Digital images from a CCD or CMOS sensor with a color filter array must undergo a demosaicing process to
combine the separate color samples into a single color image. This interpolation process can interfere with the
subsequent superresolution process. Plenoptic superresolution, which relies on precise sub-pixel sampling across
captured microimages, is particularly sensitive to such resampling of the raw data. In this paper we present an
approach for superresolving plenoptic images that takes place at the time of demosaicing the raw color image
data. Our approach exploits the interleaving provided by typical color filter arrays (e.g., Bayer filter) to further
refine plenoptic sub-pixel sampling. Our rendering algorithm treats the color channels in a plenoptic image
separately, which improves final superresolution by a factor of two. With appropriate plenoptic capture we show
the theoretical possibility for rendering final images at full sensor resolution.
Depth estimation in focused plenoptic camera is a critical step for most applications of this technology and poses
interesting challenges, as this estimation is content based. We present an iterative algorithm, content adaptive,
that exploits the redundancy found in focused plenoptic camera captured images. Our algorithm determines for
each point its depth along with a measure of reliability allowing subsequent enhancements of spatial resolution
of the depth map. We remark that the spatial resolution of the recovered depth corresponds to discrete values
of depth in the captured scene to which we refer as slices. Moreover, each slice has a different depth and will
allow extraction of different spatial resolutions of depth, depending on the scene content being present in that
slice along with occluding areas. Interestingly, as focused plenoptic camera is not theoretically limited in spatial
resolution, we show that the recovered spatial resolution is depth related, and as such, rendering of a focused
plenoptic image is content dependent.
Plenoptic cameras, constructed with internal microlens arrays, capture both spatial and angular information, i.e., the full 4-D radiance, of a scene. The design of traditional plenoptic cameras assumes that each microlens image is completely defocused with respect to the image created by the main camera lens. As a result, only a single pixel in the final image is rendered from each microlens image, resulting in disappointingly low resolution. A recently developed alternative approach based on the focused plenoptic camera uses the microlens array as an imaging system focused on the image plane of the main camera lens. The flexible spatioangular trade-off that becomes available with this design enables rendering of final images with significantly higher resolution than those from traditional plenoptic cameras. We analyze the focused plenoptic camera in optical phase space and present basic, blended, and depth-based rendering algorithms for producing high-quality, high-resolution images. We also present our graphics-processing-unit-based implementations of these algorithms, which are able to render full screen refocused images in real time.
Proc. SPIE. 5666, Human Vision and Electronic Imaging X
KEYWORDS: Mathematical modeling, Image fusion, Visual process modeling, Image processing, Transform theory, Light sources and illumination, Visual system, High dynamic range imaging, Human vision and color perception, RGB color model
The Healing Brush is a tool introduced for the first time in Adobe Photoshop (2002) that removes defects in images by seamless cloning (gradient domain fusion). The Healing Brush algorithms are built on a new mathematical approach that uses Fibre Bundles and Connections to model the representation of images in the visual system. Our mathematical results are derived from first principles of human vision, related to adaptation transforms of von Kries type and Retinex theory. In this paper we present the new result of Healing in arbitrary color space. In addition to supporting image repair and seamless cloning, our approach also produces the exact solution to the problem of high dynamic range compression of17 and can be applied to other image processing algorithms.
3 February 2014 | San Francisco, California, United States
Mobile Computational Photography
4 February 2013 | Burlingame, California, United States
SC980: Theory and Methods of Lightfield Photography
Lightfield photography is based on capturing discrete representations of all light rays in a volume of 3D space. Since light rays are characterized with 2D position and 2D direction (relative to a plane of intersection), lightfield photography captures 4D data. In comparison, conventional photography captures 2D images. Multiplexing this 4D radiance data onto conventional 2D sensors demands sophisticated optics and imaging technology. Rending an image from the 4D lightfield is accomplished computationally based on creating 2D integral projections of the 4D radiance. Optical transformations can also be applied computationally, enabling effects such as computational focusing anywhere in space.
This course presents a comprehensive development of lightfield photography, beginning with theoretical ray optics fundamentals and progressing through real-time GPU-based computational techniques. Although the material is mathematically rigorous, our goal is simplicity. Emphasizing fundamental underlying ideas leads to the development of surprisingly elegant analytical techniques. These techniques are in turn used to develop and characterize computational techniques, model lightfield cameras, and analyze resolution.
The course also demonstrates practical approaches and engineering solutions. The course includes a hands-on demonstration of several working plenoptic cameras that implement different methods for radiance capture, including the micro-lens approach of Lippmann, the mask-enhanced "heterodyning" camera, the lens-prism camera, multispectral and polarization capture, and the plenoptic 2.0 camera. One section of the course is devoted specifically to the commercially available Lytro camera. Various computational techniques for processing captured data are demonstrated, including basic rendering, Ng's Fourier slice algorithm, the heterodyned light-field approach for computational refocusing, glare reduction, super-resolution, artifact reduction, and others.