The well-established Modulation Transfer Function (MTF) is an imaging performance parameter that is well suited to describing certain sources of detail loss, such as optical focus and motion blur. As performance standards have developed for digital imaging systems, the MTF concept has been adapted and applied as the spatial frequency response (SFR). The international standard for measuring digital camera resolution, ISO 12233, was adopted over a decade ago. Since then the slanted edge-gradient analysis method on which it was based has been improved and applied beyond digital camera evaluation. Practitioners have modified minor elements of the standard method to suit specific system characteristics, unique measurement needs, or computational shortcomings in the original method. Some of these adaptations have been documented and benchmarked, but a number have not. In this paper we describe several of these modifications, and how they have improved the reliability of the resulting system evaluations. We also review several ways the method has been adapted and applied beyond camera resolution.
In this paper we address the problem of Image Quality Assessment of no reference metrics,
focusing on JPEG
corrupted images. In general no reference metrics are not able to measure with the same
performance the distortions within their possible range and with respect to different image
contents. The crosstalk between content and distortion signals influences the human perception.
We here propose two strategies to improve the correlation between subjective and objective
quality data. The first strategy is based on grouping the images according to their spatial
complexity. The second one is based on a frequency analysis. Both the strategies are tested on
two databases available in the literature. The results show an improvement in the correlations
no reference metrics and psycho-visual data, evaluated in terms of the Pearson Correlation
The capture and retention of image detail are important characteristics for system design and subsystem selection. An
established imaging performance measure that is well suited to certain sources of detail loss, such as optical focus and
motion blur, is the Modulation Transfer Function (MTF). Recently we have seen the development of image quality
methods aimed at more adaptive operations, such as noise cleaning and adaptive digital filtering. An example of this is
the measure of texture (image detail) loss using sets of overlapping small objects, known as dead leaves targets. In this
paper we investigate the application of the above method to image compression. We apply several levels of JPEG and
JPEG 2000 compression to digital images that include scene content that is amenable to the texture loss measure. A
modified form of the method was used. This allowed direct target compensation without data smoothing. Following a
camera simulation, the texture MTF and acutance were computed. The standard deviation of the acutance measure was
0.014 (relative error of 1.63%), found by replicate measurements. Structured similarity index (SSIM) values, used for
still and video image quality evaluation, were also computed for the image sets. The acutance and SSI results were
similar; however the relationship between the two showed an offset between the JPEG and JPEG 2000 images sets.
A significant challenge in the adoption of today's digital imaging standards is a clear connection to how they relate to
today's vernacular digital imaging vocabulary. Commonly used terms like resolution, dynamic range, delta E, white
balance, exposure, or depth of focus are mistakenly considered measurements in their own right and are frequently
depicted as a disconnected shopping list of individual metrics with little common foundation. In fact many of these are
simple summary measures derived from more fundamental imaging science/engineering metrics, adopted in existing
Four important underlying imaging performance metrics are; Spatial Frequency Response (SFR), Opto-Electronic
Conversion Function (OECF), Noise Power Spectrum (NPS), and Spatial Distortion. We propose an imaging
performance taxonomy. With a primary focus on image capture performance, our objective is to indicate connections
between related imaging characteristics, and provides context for the array of commonly used terms. Starting with the
concepts of Signal and Noise, the above imaging performance metrics are related to several simple measures that are
compatible with testing for design verification, manufacturing quality assurance, and technology selection evaluation.
In September 2000, INCITS W1 (the U.S. representative of ISO/IEC JTC1/SC28, the standardization committee for office equipment) was chartered to develop an appearance-based image quality standard.<sup>(1),(2)</sup> The resulting W1.1 project is based on a proposal<sup>(3)</sup> that perceived image quality can be described by a small set of broad-based attributes. There are currently six <i>ad hoc</i> teams, each working towards the development of standards for evaluation of perceptual image quality of color printers for one or more of these image quality attributes. This paper summarizes the work in progress of the teams addressing the attributes of Macro-Uniformity, Colour Rendition, Gloss & Gloss Uniformity, Text & Line Quality and Effective Resolution.
One of the first ISO digital camera standards to address image microstructure was ISO 12233, which introduced the SFR, spatial frequency response, based on the analysis of edge features in digital images. The SFR, whether derived from edges or periodic signals, describes the capture of image detail as a function of spatial frequency. Often during camera testing, however, there is an interest in distilling SFR results down to a single value that can be compared with acceptable tolerances. As a measure of limiting resolution, it has been suggested that the frequency at which the SFR falls to, e.g., 10%, can be used. We use this limiting resolution to introduce a sampling efficiency measure, being considered under the current ISO 12233 standard revision effort. The measure is the ratio of limiting resolution frequency to that implied by the image (sensor) sampling alone. The rationale and details of this measure are described,
as are example measurements. One-dimensional sampling efficiency calculations for multiple directions are included in a two-dimensional analysis.
A flatbed reflection scanner is a tempting device to use as a surrogate for a microdensitometer in the evaluation of print
image quality. Since reflection scanners were never designed with this purpose in mind, many concerns exist regarding their usefulness as a microdensitometer surrogate. This paper addresses the concerns regarding scan uniformity that must be addressed in order to qualify a reflection scanner for use in print image quality evaluation.
Edition 2 of ISO 12233, Resolution and Spatial Frequency Response (SFR) for Electronic Still Picture Imaging, is likely
to offer a choice of techniques for determining spatial resolution for digital cameras different from the initial standard.
These choices include 1) the existing slanted-edge gradient SFR protocols but with low contrast features, 2) polar coordinate sine wave SFR technique using a Siemens star element, and 3) visual resolution threshold criteria using a continuous linear spatial frequency bar pattern features. A comparison of these methods will be provided. To establish the level of consistency between the results of these methods, theoretical and laboratory experiments were performed by members of ISO TC42/WG18 committee. Test captures were performed on several consumer and SLR digital cameras using the on-board image processing pipelines. All captures were done in a single session using the same lighting conditions and camera operator. Generally, there was good conformance between methods albeit with some notable differences. Speculation on the reason for these differences and how this can be diagnostic in digital camera evaluation will be offered.
Inexpensive and easy-to-use linear and area-array scanners have frequently substituted as colorimeters and densitometers
for low-frequency (i.e., large area) hard copy image measurement. Increasingly, scanners are also being used for high
spatial frequency, image microstructure measurements, which were previously reserved for high performance
microdensitometers. In this paper we address characteristics of flatbed reflection scanners in the evaluation of print
uniformity, geometric distortion, geometric repeatability and the influence of scanner MTF and noise on analytic
measurements. Suggestions are made for the specification and evaluation of scanners to be used in print image quality
standards that are being developed.
There are no fundamental differences between today's mobile telephone cameras and consumer digital still cameras that
suggest many existing ISO imaging performance standards do not apply. To the extent that they have lenses, color filter
arrays, detectors, apertures, image processing, and are hand held, there really are no operational or architectural
differences. Despite this, there are currently differences in the levels of imaging performance. These are driven by
physical and economic constraints, and image-capture conditions. Several ISO standards for resolution, well established
for digital consumer digital cameras, require care when applied to the current generation of cell phone cameras. In
particular, accommodation of optical flare, shading non-uniformity and distortion are recommended. We offer proposals
for the application of existing ISO imaging resolution performance standards to mobile imaging products, and
suggestions for extending performance standards to the characteristic behavior of camera phones.
Recently, two ISO electronic imaging standards aimed at digital capture device dynamic range metrology have been issued. Both ISO 15739 (digital still camera noise) and ISO 21550 (film scanner dynamic range) adopt a signal-to-noise ratio (SNR) criterion for specifying dynamic range. To resiliently compare systems with differing mean-signal transfer, or Electro-Optical Conversion Functions (OECF), an incremental SNR (SNR<i>i</i>) is used. The exposure levels that correspond to threshold-SNR values are used as endpoints to determine measured dynamic range. While these thresholds were developed through committee consensus with generic device applications in mind, the methodology of these standards is flexible enough to accommodate different application requirements. This can be done by setting the SNR thresholds according to particular signal-detection requirements. We will show how dynamic range metrology, as defined in the above standards, can be interpreted in terms of statistical hypothesis testing and confidence interval methods for mean signal values. We provide an interpretation of dynamic range that can be related to particular applications based on contributing influences of variance, confidence intervals, and sample size variables. In particular, we introduce the role of the spatial-correlation statistics for both signal and noise sources, not covered in previous discussions of these ISO standards. This can be interpreted in terms of a signal's spatial frequency spectrum and noise power spectrum (NPS) respectively. It is this frequency aspect to dynamic range evaluation that may well influence future standards. We maintain that this is important when comparing systems with different sampling settings, since the above noise statistics are currently computed on a per-pixel basis.
Inexpensive and easy-to-use linear and area-array scanners have frequently substituted as densitometers for low-frequency (i.e., large-area) hard copy image metrology. Increasingly, they are also being tasked for high spatial frequency, image microstructure metrology, which is reserved for high-performance microdensitometers that use microscope optics, photomultiplier tubes (PMT), and log amps. It is hard to resist their adoption for such use though, given the convenience level. Their high speed, large scan areas, auto-focus, discomfiting low cost, and low operator skill requirements makes one question if their use for such purpose is somehow too good to be true. To confidently judge their limitations requires a comprehensive signal and noise spatial frequency performance evaluation with respect to available driver options. This paper will outline and demonstrate evaluation techniques that use existing ISO metrology standards for modulation transfer function (MTF), noise, and dynamic range with a comparison to a Photometric Data Systems (PDS) microdensitometer
For digital image acquisition systems, analysis of image noise often focuses on random sources, such as those associated with quantum signal detection and signal-independent fluctuations. Other important noise sources result in pixel-to-pixel sensitivity variations that introduce repeatable patterns into the image data. In addition, since most analyses use a nominally uniform target area to estimate image noise statistics, target noise can often masquerade as noise introduced by the device under test. We described a method for distilling various fixed-pattern and temporal noise sources. The method uses several replicate digital images, acquired in register. In some cases, however, evaluation of digital scanners reveals, scan-to-scan variation in the image registration to the input test target. To overcome this limitation, a modified noise estimation method is described. This includes a step to correct this scan-to-scan misregistration. We also show how the measurement of temporal and fixed pattern noise sources can be achieved via the noise color covariance from a single test image.
Modulation transfer function (MTF) metrology and interpretation for digital image capture devices has usually concentrated on mid- to high-frequency information, relative to the half-sampling frequency. These regions typically quantify characteristics and operations such as sharpening, limiting resolution, and aliasing. However, a potential wealth of low-frequency, visually significant information is often masked in existing measurement results because of spatial data truncation. For print or document scanners, this influences measurements in the spatial frequency range of 0 to 2.0 cycles/mm, where the effects of veiling flare, micro flare, and integrating cavity effect (ICE) often manifest themselves. Using a form of edge-gradient analysis based on slanted edges, we present a method for measurement of these characteristics. By carefully adapting this well-established technique, these phenomena can be quantified. We also show how, in many cases, these effects can be treated as other spread-function or device-MTF components. The theory and field metrology of several devices using the adapted technique are also presented.
This paper describes a digital image capture simulator that incorporates a lens model based on point-spread function (PSF) data from a commercial lens design package. This lens model has significant advantages over a simple MTF based model, because it accounts for all image degrading aberrations commonly encountered in image capture system. Lens design data in the form of a set of highly resolved PSFs are generated off-line using the lens design package. The data are compared for a range of wavelengths, image plane locations and field positions. To simulate a specific imaging system, these PSFs are re-sampled according to the sensor pixel pitch, system spectral sensitivities, sensor location, etc. The input to the simulator can be either a digital test target or a digital image of a real scene, which is highly over-sampled with respect to the final simulated image and show spatial and spectral characteristics are well known. The simulator output, in the form of the raw data generated by an actual digital capture device, will be highly useful in the parametric study of the design parameters of image capture systems. The performance and limitations of the lens component of the simulator are described.
SC807: Digital Camera and Scanner Performance Evaluation: Standards and Measurement
This is an updated course on imaging performance measurement methods for digital image capture devices and systems. We introduce several ISO measurement protocols for camera resolution, tone-transfer, noise, etc. We focus on the underlying sources of variability in system performance, measurement error, and how to manage this variability in working environments. The propagation of measurement variability will be described for several emerging standard methods for; image texture, distortion, color shading, flare and chromatic aberration. Using actual measurements we demonstrate how standards can be adapted to evaluate capture devices ranging from cell phone cameras to scientific detectors. New this year, we will be discussing the use of raw files to investigate intrinsic signal and noise characteristics of the image-capture path.
SC825: Imaging Performance Evaluation for Digital Cameras, Cell-phone Cameras and Scanners
This is an updated theory-to-practice course on imaging performance measurement methods for digital image capture devices and systems. We focus on science-based standard ISO measurement protocols* for tone-transfer, speed, resolution, noise, dynamic range, and color. Using actual measurements we demonstrate how standard methods can be adapted to measure capture devices and evaluate vendor compliance for various capture systems. Because practical metrology and field application can limit measurement precision and accuracy, we will identify ways to maintain measurement utility in the presence of error sources. ISO-compliant executable software will be provided and demonstrated. In addition, several available alternative methods and analysis software will be explained and compared.
*(ISO 12233, 16067-1, 16067-2, 15529, 15739, 21550, and 17321),
SC513: Digital Camera/Scanner Evaluation Workshop: A Critical Look at ISO Standards, Tools and Limitations
This theory-to-practice course on speed, resolution, dynamic range, and noise metrology for digital image capture is based on our previous highly rated courses offered at this Symposium. Improved this year, it takes the form of a workshop. We focus on standardized measurement protocols for resolution (ISO 12233, 16067-1, 16067–2, & 15529), speed (ISO 12232), and image dynamic range/noise (ISO 15739, 21550). We will also describe and compare alternative methods currently under consideration like CIPA DC-003 and polar sine-wave techniques. About half of our day will be spent addressing various challenges to reliable evaluation and system comparison using actual test data and software tools. These examples will expose error sources (rarely addressed in the standards), and reveal ways to maintain measurement utility and workflow.
SC745: Basic Testing and Calibration of Digital Imagers
The course is aimed at providing an understanding of basic performance testing of image acquisition for digital cameras and scanners, in both consumer and technical applications. The assembly of basic elements, optics, detector, supporting electronics and image processing, are described. We then identify several key imaging performance metrics for such systems, and testing methods consistent with ISO standards. How the standard methods are applied to several types of image acquisition systems and field conditions will be explained using results from actual measurements. It is common to apply compensation for practical performance limitations either in the device or supporting driver software. We describe and demonstrate several approaches to this for; spatial distortion, white-point balance, resolution and sharpness, scene uniformity, and detector fixed-pattern noise, and pixel defects. Attendees should then be prepared for other courses on, e.g., Color Management, understanding sources of performance variation.