This PDF file contains the editorial “Retinex at 40” for JEI Vol. 13 Issue 01

In the mid-1950s Land, while developing instant color film, repeated Maxwell's 1861 three-color projection experiments. By accident, a two-color red and white projection appeared on the screen. Fascinated by the multicolored images that he saw, Land studied two-color phenomena extensively, published a series of papers, and developed a prototype red and white television system with Texas Instruments. This work describes Land's original red and white projections using equipment now at the Rowland Institute. In the late 1960s and early 1970s, McKee, Benton, and McCann investigated color images from stimuli that excited only rods and long-wave (L) cones. They used dark adaptation curves, flicker-fusion rates, the Stiles-Crawford effect, and apparent sharpness to differentiate rod and middle- (M) and short-wave (S) cone responses. They showed that color from rods and L cones under the right stimulus conditions was nearly identical to cone-cone color.

We have extended the experiments of McCann et al. by incorporating their Mondrian stimulus into a dynamic color matching (DCM) technique that allows the subject to match accurately the color of any test patch under sequential changes of illuminant. Changes in the perceived color of a central test stimulus as a result of surround illuminant changes are investigated in a number of successful binocular and dichoptic experiments. The contribution made by distant patches to instantaneous color constancy (ICC) is found to be small, with the immediate surround (i.e., <1.5-deg separation) contributing most of the constancy effect. A modified DCM technique was developed to study ICC in patients with unilateral damage to the primary visual cortex. The results of this part of the study suggest that retinal mechanisms do not contribute to ICC. Experiments carried out in subjects with cerebral achromatopsia reveal the existence of hidden constancy, even when the subjects are unable to see the result of functioning ICC mechanisms. Dichoptic implementation of the DCM technique yields reduced but significant ICC index values with more equal contribution from distant surrounds. Our findings suggest that early stage processing of chromatic information in the primary visual cortex contributes significantly to ICC, with only small contributions from extrastriate areas of the cortex.

We review the conditions that are necessary for the perception of transparency, and describe the spatiochromatic constraints for achromatic and chromatic transparent displays. These constraints can be represented by the generalized convergence model and are supported by psychophysical data. We present an alternative representation of the constraints necessary for transparency perception, which is based on an analogy with a model of color constancy and the invariance of cone-excitation ratios. We show that the invariant-ratios model is a special case of the generalized convergence model. We argue that the spatial relations in an image are preserved when a Mondrian-like surface is partially covered by a transparent filter, and therefore show an intriguing link between transparency perception and color constancy. Finally, we describe experiments to relate the strength of the transparency percept with the number of unique patches in the image display. We find that the greater the number of surfaces in the display that are partially covered by a transparent filter, the stronger the impression of transparency.

This work recounts the research on capturing real-life scenes, calculating appearances, and rendering sensations on film and other limited dynamic-range media. It describes the first patents, a hardware display used in Land's Ives Medal Address in 1968, the first computer simulations using 20×24 pixel arrays, psychophysical experiments and computational models of color constancy and dynamic range compression, and the Frankle-McCann computationally efficient retinex image processing of 512×512 images. It includes several modifications of the original approach, including recent models of human vision and gamut-mapping applications. This work emphasizes the need for parallel studies of psychophysical measurements of human vision and computational algorithms used in commercial imaging systems.

Many different descriptions of Retinex methods of lightness computation exist. We provide concise MATLAB^TM implementations of two of the spatial techniques of making pixel comparisons. The code is presented, along with test results on several images and a discussion of the results. We also discuss the calibration of input images and the postRetinex processing required to display the output images.

Our goal is to understand how the Retinex parameters affect the predictions of the model. A simplified Retinex computation is specified in the recent MATLAB^TM implementation; however, there remain several free parameters that introduce significant variability into the model's predictions. We extend previous work on specifying these parameters. In particular, instead of looking for fixed values for the parameters, we establish methods that automatically determine values for them based on the input image. These methods are tested on the McCann-McKee-Taylor asymmetric matching data, along with some previously unpublished data that include simultaneous contrast targets.

Digital photography systems often render an image from a scene-referred description with very wide dynamic range to an output-referred description of much lesser dynamic range. Global tone maps are often used for this purpose, but can fail when called on to perform a large amount of range compression. A modified luminance formulation of the Retinex ratio-reset-product-average algorithm produces a smoothly changing contrast mask of great benefit, but it too can fail where high-contrast edges are encountered. A small but critical modification to the Retinex equation—the introduction of a ratio modification operator—changes the nature of the generated contrast mask, so that it is simultaneously smooth in regions of small contrast ratios, but extremely sharp at high-contrast edges. A mask produced in this way compresses large and undesirable contrast ratios while preserving, or optionally enhancing, small ratios critical to the sensation of image contrast. Processed images may appear to have a greater contrast despite having a shorter global contrast range. Adjusting the new operator prior to processing gives control over the degree of compression at high-contrast edges. Changing the operator during processing gives control over spatial frequency response.

We present a comparison between two color equalization algorithms: Retinex, the famous model due to Land and McCann, and Automatic Color Equalization (ACE), a new algorithm recently presented by the authors. These two algorithms share a common approach to color equalization, but different computational models. We introduce the two models focusing on differences and common points. An analysis of their computational characteristics illustrates the way the Retinex approach has influenced ACE structure, and which aspects of the first algorithm have been modified in the second one and how. Their interesting equalization properties, like lightness and color constancy, image dynamic stretching, global and local filtering, and data driven dequantization, are qualitatively and quantitatively presented and compared, together with their ability to mimic the human visual system.

We present our analysis of the Frankle-McCann Retinex algorithm and propose extensions that result in a better rendition of the image. These extensions suppress artifacts and reduce the number of iterations required to get to the final image. In addition, our investigation suggests that the Retinex algorithm can be viewed as a process of estimating the per-pixel gains necessary to achieve dynamic range compression in the original image. Experimental results are provided that demonstrate the efficacy of this approach as compared to using Retinex to generate the final image.

The Retinex algorithm presents an interesting mix of a locally connected iterative algorithm and a multiresolution analysis of the image. By recasting the algorithm, using wavelets, the behavior of the algorithm comes to light. This allows generalizations to be proposed, by changes in both the multiresolution structure and the iterative update structure.

There has been a revivification of interest in the Retinex computation in the last six or seven years, especially in its use for image enhancement. In his last published concept (1986) for a Retinex computation, Land introduced a center/surround spatial form, which was inspired by the receptive field structures of neurophysiology. With this as our starting point, we develop the Retinex concept into a full scale automatic image enhancement algorithm—the multiscale Retinex with color restoration (MSRCR)—which combines color constancy with local contrast/lightness enhancement to transform digital images into renditions that approach the realism of direct scene observation. Recently, we have been exploring the fundamental scientific questions raised by this form of image processing. 1. Is the linear representation of digital images adequate in visual terms in capturing the wide scene dynamic range? 2. Can visual quality measures using the MSRCR be developed? 3. Is there a canonical, i.e., statistically ideal, visual image? The answers to these questions can serve as the basis for automating visual assessment schemes, which, in turn, are a primitive first step in bringing visual intelligence to computers.

In the photorealistic image synthesis process, an accurate approximation of the spectral light radiance field of a synthetic scene is carefully reproduced, with the goal of generating a synthetic image that is indistinguishable from the actual one. The paradigm of photorealism requires a comparison of the real scene and its synthetic reproduction, and the two visual representations should be studied under the same conditions to make a correct comparison and evaluate the degree of accuracy. To reproduce the same observing conditions, we need to define a sort of synthetic observer (given the impossibility to enter into a synthetic world) to compensate the deep differences in the viewing conditions, between the real and synthetic images. Various solutions have been proposed to this end; most of them are based more on perceptive measures of the human visual system (HVS) under controlled conditions, rather than on the HVS behavior under real conditions, e.g., observing a natural image and not a controlled black and white or colored pattern. Besides the comparison problem, difficulties can arise from the visualization phase, whose purpose is to display the final results of the simulation model on a monitor screen or printed paper. This is known as the tone reproduction problem, and in most cases, one has to find the best solution to compress an extended dynamic range of the computed light field into the limited range of displayable colors. Several solutions have been proposed to solve this problem. On the contrary, no mapping is usually made in case of low luminance and extremely limited dynamic range images, and consequently photorealism and visual appearance are lost. We propose a working hypothesis to solve the appearance and the tone reproduction problems in the synthetic image generation, integrating the Retinex model into the photorealistic image synthesis context, including in this way a model of the HVS in the image synthesis process.

In image processing, color is generally treated as an autonomous entity that is separate from spatial dimensions. Consequently, color processing and color mapping are predominantly done as point operations, linking one color to another, without considering spatially neighboring colors. It is well known that this point-wise approach does not capture the spatial dependencies of color that we humans experience in everyday life. More precise mathematical models of color vision that take into account spatial dependencies are generally computationally expensive. This leads to a continuing predominance of point-wise color processing. We introduce some simple spatially dependent processing techniques and demonstrate the potential advantages of even such simple schemes. The underlying assumption is that simple spatial operations allow the "hiding" of artifacts and errors in a way that is less objectionable to a human observer.

Traditional color appearance modeling has recently matured to the point that available, internationally recommended models such as CIECAM02 are capable of making a wide range of predictions, to within the observer variability in color matching and color scaling of stimuli, in somewhat simplified viewing conditions. It is proposed that the next significant advances in the field of color appearance modeling and image quality metrics will not come from evolutionary revisions of colorimetric color appearance models alone. Instead, a more revolutionary approach will be required to make appearance and difference predictions for more complex stimuli in a wider array of viewing conditions. Such an approach can be considered image appearance modeling, since it extends the concepts of color appearance modeling to stimuli and viewing environments that are spatially and temporally at the level of complexity of real natural and man-made scenes, and extends traditional image quality metrics into the color appearance domain. Thus, two previously parallel and evolving research areas are combined in a new way as an attempt to instigate a significant advance. We review the concepts of image appearance modeling, present iCAM as one example of such a model, and provide a number of examples of the use of iCAM in image reproduction and image quality evaluation.

We begin with a brief review of the basic issues and a range of algorithms related to scene rendering. Then the results of an initial comparison of the McCann99 Retinex and iCAM algorithms for use in rendering three high dynamic range gray-scale images are reported. The images are also rendered using an image-specific manual tone curve. Ten observers performed a rank ordering of the printed renderings, and then mean rankings were computed. The results varied by image, with iCAM being most preferred for one of the images, McCann99 Retinex being most preferred for one of the images, and a tie for all three renderings for the third image. Five relative qualitative trends in the results are also described.

We conduct an exhaustive survey of image thresholding methods, categorize them, express their formulas under a uniform notation, and finally carry their performance comparison. The thresholding methods are categorized according to the information they are exploiting, such as histogram shape, measurement space clustering, entropy, object attributes, spatial correlation, and local gray-level surface. 40 selected thresholding methods from various categories are compared in the context of nondestructive testing applications as well as for document images. The comparison is based on the combined performance measures. We identify the thresholding algorithms that perform uniformly better over nondestructive testing and document image applications.

The morphological binary hit-miss operator has been used extensively to locate features within a binary image. We propose a grayscale hit-miss operator that detects signal shapes and is applicable to scalar-valued functions on one, two, or more dimensions. The hit and miss structuring elements define the lower and upper bounds of the signal: If a signal lies between the hit and miss templates, then the hit-miss operator will produce a one output; otherwise, it will respond with zero. We incorporate a fuzzy logic element to the hit-miss operator to indicate how strongly the signal matches the hit-miss templates.

We present a comparative analysis of features such as compression rate, image loss, sensitivity to resolution variation, etc., for still image file formats in the most widespread use today for image storage and transmission on computer networks.

The application of images and video has increased significantly in recent years. It is important to develop indexing techniques for searching images and video in the compressed domain. Among the compression techniques, discrete-wavelet-transform based techniques have become popular because of their excellent energy compaction and multiresolution capability. As a result, the newly established JPEG2000 image compression standard is based on discrete wavelet transform. We propose two progressive image indexing techniques in the JPEG2000 framework. In the first technique, a significant bitmap and a 2-D histogram of significant bits are used as the image index. In the second technique, a vector comprising variance and mean of the numbers of bitplanes used to encode individual code blocks is employed as the image index. The proposed index can be generated easily from the packet header of a JPEG2000 compressed image, resulting in a very fast search and retrieval. Experimental results show that the proposed techniques provide good indexing performance.

The usefulness of electronic document delivery and archives rests in large part on advances in compression technology. Documents can contain complex layouts with different data types, such as text and images, having different statistical characteristics. To achieve better image quality, it is important to make use of such characteristics in compression. We exploit the transform coefficient distributions for text and images. We show that the scheme in baseline JPEG does not lead to minimum mean-square error if we have models of these coefficients. Instead, we discuss an algorithm designed for this performance that involves first classifying the blocks, and then estimating the parameters to enable a biased reconstruction in the decompression value. Simulation results are shown to validate the advantages of this method.

Video streaming, or the real-time delivery of video over a data network, is the underlying technology behind many applications including video conferencing, video on demand, and the delivery of educational and entertainment content. In many applications, particularly ones involving entertainment content, security issues, such as conditional access and copy protection, must be addressed. To resolve these security issues, techniques that include encryption and watermarking need to be developed. Since video sequences will often be compressed using a scalable compression technique and transported over a lossy packet network using the Internet Protocol (IP), security techniques must be compatible with the compression method and data transport and be robust to errors. We address the issues involved in the watermarking of rate-scalable video streams delivered using a practical network. Watermarking is the embedding of a signal (the watermark) into a video stream that is imperceptible when the stream is viewed, but can be detected by a watermark detector. Many watermarking techniques have been proposed for digital images and video, but the issues of streaming have not been fully investigated. A review of streaming video is presented, including scalable video compression and network transport, followed by a brief review of video watermarking and the discussion of watermarking streaming video.

Semifragile watermarking techniques aim to prevent tampering and fraudulent use of modified images. A semifragile watermark monitors the integrity of the content of the image but not its numerical representation. Therefore, the watermark is designed so that the integrity is proven if the content of the image has not been tampered with, despite some mild processing on the image. However, if parts of the image are replaced with the wrong key or are heavily processed, the watermark information should indicate evidence of forgery. We compare the performance of eight semifragile watermarking algorithms in terms of their miss probability under forgery attack, and in terms of false alarm probability under nonmalicious signal processing operations that preserve the content and quality of the image. We propose desiderata for semifragile watermarking algorithms and indicate the promising algorithms among existing ones.

Earth observation missions have recently attracted a growing interest from the scientific and industrial communities, mainly due to the large number of possible applications capable to exploit remotely sensed data and images. Along with most remote sensing imagery now available in digital form, which can be easily copied and distributed through the Internet, the need of copyright protection arises as well. Such a problem has been largely addressed by resorting to image watermarking technology. We present a novel watermarking method based on a chaotic spread spectrum and apply it to the image products of remote sensing. The correlation property of chaotic sequences is analyzed. It is shown that the chaotic signal is more suitable to provide remote sensing image watermarks. In the retrieval process, the information signal is extracted by using a standard correlator. Experimental results show that the proposed chaotic watermarking system for remote sensing images is robust to various image manipulations, including resizing, cropping, filtering, and compression. Compared to the conventional spread spectrum watermarking approach, the proposed chaotic watermark is superior.

We review 20 years of development in the field of 3-D laser imaging. An overview of 3-D digitizing techniques is presented with an emphasis on commercial techniques and systems currently available. It covers some of the most important methods that have been developed, both at the National Research Council of Canada (NRC) and elsewhere, with a focus on commercial systems that are considered good representations of the key technologies that have survived the test of years.

We present a novel method for color interpolation that is specially designed for color-filter-array images acquired with charge-coupled device (CCD) detectors. The interpolation technique combines edge direction weighting and the local gain approach to determine missing sensor information with high accuracy but low computational costs. Simulations demonstrate that the proposed algorithm can achieve high spatial resolution and color fidelity for camera imaging systems. The averaged peak signal-to-noise ratio (PSNR) is improved about 2.3, 1.1, and 0.4 dB compared to bilinear interpolation, cubic B-spline, and discriminated color correlation, respectively. Moreover, the new color interpolation uses low computational complexity in terms of the number of additions and multiplications to be performed, and it requires less memory than the cubic B-spline and discriminated color correlation methods.

This PDF file contains the editorial “Computer Vision and Fuzzy-Neural Systems” for JEI Vol. 13 Issue 01