The processes and procedures used to develop both ANSI and International Standards can appear daunting and unwieldy to newcomers to the world of standards. However, once understood, these processes and procedures can become valuable resources in the development of the co-operative agreements that we call standards. This is particularly true in the area of image quality for electronic system specification, measurement and design - the subject of this conference. This is a subject area where the interests of several different application technologies converge as do the responsibilities of several different standards communities. A brief overview of the structure of ANSI, ISO, and IEC will be given, along with a roadmap of the key groups (and their inter-relationships) applicable to image quality. A summary will be given of the key image-quality standardization activities currently underway in these groups. This overview will draw on practical examples of the issues, challenges and solutions experienced by the author in many years of involvement in both ANSI and ISO standards activities, particularly in ISO/TC130 (Graphic technology) and ISO/TC42 (Photography), which are two of the key committees that have a vested interest in image quality and color reproduction in digital systems.

The eight Technical Committees in CIE Division 8 all address areas related to image quality. These range from color difference equations for images, to color appearance modeling, to communication of color information. The Division is oriented more toward solving well-known image quality problems than toward identifying or quantifying new ones.

To address the standarization issues of perceptually based image quality for printing systems, ISO/IEC JTC1/SC28, the standarization committee for office equipment charactered the W1.1 project with the responsibiltiy of drafting a proposal for an international standard for the evaluation of printed image quality. One of the W1.1 task teams is charactered to address the issue of 'Gloss and Gloss Uniformity". This paper summarizes the current status and technical progress of this ad hoc team in 2003.

This paper describes the status and technical progress of the INCITS W1.1 Text and Line Quality ad hoc team. The team has defined the scope of the work and developed several perceptual sub-attributes which comprise the Text Quality and Line Quality attributes, together with preliminary drafts of test patterns suitable for their quantification. Data is provided on initial attempts at quantifying some Text Quality sub-attributes.

A common need of the INCITS W1.1 Macro Uniformity, Color Rendition and Micro Uniformity ad hoc efforts is to digitize image quality test targets and derive parameters that correlate with image quality assessments. The digitized data should be in a colorimetric color space such as CIELAB and the process of digitizing will introduce no spatial artifacts that reduce the accuracy of image quality parameters. Input digitizers come in many forms including inexpensive scanners used in the home, a range of sophisticated scanners used for graphic arts and scanners used for scientific and industrial measurements (e.g., microdensitometers). Some of these are capable of digitizing hard copy output for image quality objective metrices, and this report focuses on assessment of high quality flatbed scanners for that role. Digitization using flatbed scanners is attractive because they are relatively inexpensive, easy to use, and most are available with document feeders permitting analysis of a stack of documents with little user interaction. Other authors have addressed using scanners for image quality measurements. This paper focuses (1) on color transformations from RGB to CIELAB and (2) sampling issues and demonstrates that flatbed scanners can have a high level of accuracy for generating accurate, stable images in the CIELAB metric. Previous discussion and experimental results focusing on color conversions had been presented at PICS 2003. This paper reviews the past discussion with some refinement based on recent experiments and extends the analysis into color accuracy verification and sampling issues.

This paper describes the status of the INCITS W1.1 macro-uniformity ad hoc team, towards development of standards for perceptual image quality for color printers. The team has defined the macro-uniformity attribute, has developed several test patterns to be used for subjective and objective evaluations and has defined test patterns and methods to address color conversions of digitizing devices. A set of print samples, originating from diverse printing systems, has been established, and digitization of these samples is in progress. These activities and next steps are reviewed in this paper.

INCITS W1 is the U.S. representative of ISO/IEC JTC1/SC28, the standardization committee for office equipment. In September 2000, INCITS W1 was chartered to develop an appearance-based image quality standard. The resulting W1.1 project is based on a proposal that perceived image quality could be described by a small set of broad-based attributes. There are currently five ad hoc W1.1 teams, each working on one or more of these image quality attributes. This paper summarizes the work of the W1.1 Microuniformity ad hoc team. The agreed-upon process for developing the W1.1 Image Quality of Printers standards is described in a statement located on the INCITS W1.1 web site (ncits.org/tc_home/w11htm/incits_w11.htm), and the process schematic is reproduced here as Figure 1, (in which a final, independent confirmation step has been excluded for brevity).

The color rendition ad hoc team of INCITS W1.1 is working to address issues related to color and tone reproduction for printed output and its perceptual impact on color image quality. The scope of the work includes accuracy of specified colors with emphasis on memory colors, color gamut, and the effective use of tone levels, including issues related to contouring. The team has identified three sub-attributes of color rendition: (1) color quantization -- defined as the ability to merge colors where needed, (2) color scale -- defined as the ability to distinguish color where needed, and (3) color fidelity -- defined as the ability to match colors. Visual definitions and descriptions of how these sub-attributes are perceived have been developed. The team is presently defining measurement methods for these, with the first of the sub-attributes considered being color quantization. More recently, the problem of measuring color fidelity has been undertaken. This presentation will briefly review the definitions and appearance of the proposed sub-attributes. The remainder of the discussion will focus on the progress to date of developing test targets and associated measurement methods to quantify the color quantization and color fidelity sub-attributes.

JPEG compression is highly non-linear and non-stationary. This renders the system scene dependent and thus calculation of its MTF is challenging. Evaluation using differing methodologies yields vastly contradictory results. Further, local MTF can vary significantly with respect to position within sub-image blocks for a given measurement technique. Estimation of the first order Wiener kernel transform has been shown to provide a consistent and intuitive alternative to calculation of the MTF. The lack of reliable MTF curves for such a non-linear system may lead to poor performance of image quality metrics calculated using them. This work examines the use of first order Wiener kernel transform curves in the calculation of SQRI_n and PIC metrics for JPEG compression compared to those previously calculated using a traditional edge and sine wave technique. New analysis of previous psychometric experimentation is presented. Results are discussed and conclusions drawn.

A new method for testing the resolution of digital cameras has been developed. The new method is an extension of the ISO 12233 Slanted-edge Spatial Frequency Response test. The new method computes the spatial frequency response along the edge of a circle. It is especially well adapted to inexpensive imaging systems with rotationally symmetric blur and lens distortion. In addition to presenting the new method, a set of practical improvements, which can be applied to both the slanted-edge and circular-edge methods, is described.

The important criteria used in subjective evaluation of distorted images include the amount of distortion, the type of distortion, and the distribution of error. An ideal image quality measure should therefore be able to mimic the human observer. We present a new image quality measure that can be used as a multidimensional or a scalar measure to predict the distortion introduced by a wide range of noise sources. Based on the Singular Value Decomposition, it reliably measures the distortion not only within a distortion type at different distortion levels but also across different distortion types. The measure was applied to Lena using six types of distortion (JPEG, JPEG 2000, Gaussian blur, Gaussian noise, sharpening and DC-shifting), each with five distortion levels.

Modulation transfer function (MTF) metrology and interpretation for digital image capture devices has usually concentrated on mid- to high-frequency information, relative to the half-sampling frequency. These regions typically quantify characteristics and operations such as sharpening, limiting resolution, and aliasing. However, a potential wealth of low-frequency, visually significant information is often masked in existing measurement results because of spatial data truncation. For print or document scanners, this influences measurements in the spatial frequency range of 0 to 2.0 cycles/mm, where the effects of veiling flare, micro flare, and integrating cavity effect (ICE) often manifest themselves. Using a form of edge-gradient analysis based on slanted edges, we present a method for measurement of these characteristics. By carefully adapting this well-established technique, these phenomena can be quantified. We also show how, in many cases, these effects can be treated as other spread-function or device-MTF components. The theory and field metrology of several devices using the adapted technique are also presented.

Inspection of electrophotographic print cartridge quality and compatibility requires analysis of hundreds of pages on a wide population of printers and copiers. Although print quality inspection is often achieved through the use of anchor prints and densitometry, more comprehensive analysis and quantitative data is desired for performance tracking, benchmarking and failure mode analysis. Image quality measurement systems range in price and performance, image capture paths and levels of automation. In order to address the requirements of a specific application, careful consideration was made to print volume, budgetary limits, and the scope of the desired image quality measurements. A flatbed scanner-based image quality measurement system was selected to support high throughput, maximal automation, and sufficient flexibility for both measurement methods and image sampling rates. Using an automatic document feeder (ADF) for sample management, a half ream of prints can be measured automatically without operator intervention. The system includes optical character recognition (OCR) for automatic determination of target type for measurement suite selection. This capability also enables measurement of mixed stacks of targets since each sample is identified prior to measurement. In addition, OCR is used to read toner ID, machine ID, print count, and other pertinent information regarding the printing conditions and environment. This data is saved to a data file along with the measurement results for complete test documentation. Measurement methods were developed to replace current methods of visual inspection and densitometry. The features that were being analyzed visually could be addressed via standard measurement algorithms. Measurement of density proved to be less simple since the scanner is not a densitometer and anything short of an excellent estimation would be meaningless. In order to address the measurement of density, a transfer curve was built to translate the gray scale values that are the fundamental scanner output into an estimation of density. The correlation coefficient was excellent and was deemed to be adequate for this application. This paper addresses the automation process, image analysis and data management required by this application.

For digital image acquisition systems, analysis of image noise often focuses on random sources, such as those associated with quantum signal detection and signal-independent fluctuations. Other important noise sources result in pixel-to-pixel sensitivity variations that introduce repeatable patterns into the image data. In addition, since most analyses use a nominally uniform target area to estimate image noise statistics, target noise can often masquerade as noise introduced by the device under test. We described a method for distilling various fixed-pattern and temporal noise sources. The method uses several replicate digital images, acquired in register. In some cases, however, evaluation of digital scanners reveals, scan-to-scan variation in the image registration to the input test target. To overcome this limitation, a modified noise estimation method is described. This includes a step to correct this scan-to-scan misregistration. We also show how the measurement of temporal and fixed pattern noise sources can be achieved via the noise color covariance from a single test image.

The Image Systems Evaluation Toolkit (ISET) is an integrated suite of software routines that simulate the capture and processing of visual scenes. ISET includes a graphical user interface (GUI) for users to control the physical characteristics of the scene and many parameters of the optics, sensor electronics and image processing-pipeline. ISET also includes color tools and metrics based on international standards (chromaticity coordinates, CIELAB and others) that assist the engineer in evaluating the color accuracy and quality of the rendered image.

We theorize that there may be four types of differences (I, II, III, IV) between the expert observers and the naive observers when judging printing quality. Using data from a printing quality competitiveness study, we show that the expert observers may judge printing quality differently from naive observers depending on the test image and the specific printing defects the samples may have. We use correlation coefficients of the rank scores of samples to reveal various degrees of disagreement between the two groups for all the samples of an image. Significance test of each individual pair of samples were used to identify those pairs that have the largest discrepancy for each image when judged by the two groups. The visual differences between the samples were also analyzed. We further used scaled values distribution plots generated by bootstrap to explicitly show the differences both in printing quality ranking and attributes ranking. For the samples in this study, there was good agreement between the two groups for color images when engine artifacts were least obvious, making the type I and type III difference unlikely. For the mono print, the two groups ranked the samples differently in many pairings and ranked them in opposite directions. The data show that the expert group tended to weigh on engine printing artifacts more than the naive group, indicating the type IV difference.

The brighter, more portable projectors that have emerged in recent years have driven an increase in opportunities to use projectors in well-lit rooms and to project images on objects other than special screens. However, the color and tone of color images projected under such diverse environments are subject to degradation due to the effects of ambient light and of the projection surface. Therefore, using color sensors to detect environmental effects, we developed WallShot, a technology for correcting the color and tone of a projected image. We validated the effect of the corrections through psychophysical experiments.

This paper describes a regression model for predicting customer preference from objective image quality metrics for black and white printers, copiers and multifunction systems. In order to quantify customer preference for monochrome images the quantitative preference system was previously developed. Using this system, a preference survey with five different customer-type documents was used to obtain the preference data. Objective image quality metrics were obtained from a scanner-based measurement system. Using this regression model, typically 80% or more of the variation of the overall preference can be explained by six objective image quality metrics: Relative TRC Error; Mottle; Visual Noise; Visual Structure; Streaks and Bands; and Relative Dynamic Range Reduction. The results also provide the relative significance of these attributes for the different kinds of customer images.

Two psychophysical experiments were performed scaling overall image quality of black-and-white electrophotographic (EP) images. Six different printers were used to generate the images. There were six different scenes included in the experiment, representing photographs, business graphics, and test-targets. The two experiments were split into a paired-comparison experiment examining overall image quality, and a triad experiment judging overall similarity and dissimilarity of the printed images. The paired-comparison experiment was analyzed using Thurstone's Law, to generate an interval scale of quality, and with dual scaling, to determine the independent dimensions used for categorical scaling. The triad experiment was analyzed using multidimensional scaling to generate a psychological stimulus space. The psychophysical results indicated that the image quality was judged mainly along one dimension and that the relationships among the images can be described with a single dimension in most cases. Regression of various physical measurements of the images to the paired comparison results showed that a small number of physical attributes of the images could be correlated with the psychophysical scale of image quality. However, global image difference metrics did not correlate well with image quality.

Many images are received for distribution and reproduction without the possibility of a comparison to the source image. Image quality is difficult to measure and quantify without such a source image. However, test photographs consisting of memory colors can be compared to a remembered image. Test forms consisting of photographs of the real world are the only source of memory colors. They are used to test a graphic-arts color workflow. Such tests: (1) Display the color quality of various output devices. (2) Describe color differences by comparing the difference between reproduction of a test photograph and the remembered colors. (3) Help achieve a preferred state of color reproduction. (4) Characterize reproduction gamut problems of an output device. (5) Remotely trouble-shoot a color workflow. The human visual system cannot provide a precise measurement of quality but does provide direction and goals for a correction process. The criteria for the selection of image content and the usefulness of test photographs are described.

ISO 20462, a three-part standard entitled “Psychophysical experimental methods to estimate image quality,” is being developed by WG18 (Electronic Still Picture Imaging) of TC42 (Photography). As of late 2003, all three parts were in the Draft International Standard (DIS) ballot stage, with publication likely during 2004. This standard describes two novel perceptual methods, the triplet comparison technique and the quality ruler, that yield results calibrated in just noticeable differences (JNDs). Part 1, “Overview of psychophysical elements,” discusses specifications regarding observers, test stimuli, instructions, viewing conditions, data analysis, and reporting of results. Part 2, “Triplet comparison method,” describes a technique involving simultaneous five-point scaling of sets of three stimuli at a time, arranged so that all possible pairs of stimuli are compared exactly once. Part 3, “Quality ruler method,” describes a real-time technique optimized for obtaining assessments over a wider range of image quality. A single ruler is a series of ordered reference stimuli depicting a common scene but differing in a single perceptual attribute. Methods for generating quality ruler stimuli of known JND separation through modulation transfer function (MTF) variation are provided. Part 3 also defines a unique absolute Standard Quality Scale (SQS) of quality with one unit equal to one JND. Standard Reference Stimuli (SRS) prints calibrated against this new scale will be made available through the International Imaging Industry Association.

In this paper, a new no reference metric for video quality assessment is presented. The proposed metric provides a measure of the quality of a video based on a feature that we believe is relevant for the human observers: the motion. The metric is based on an unconventional use of a data hiding system. The mark is inserted in specific areas of the video using a fragile embedding algorithm. The exact embedding location is determined by the amount of motion between pairs of consecutive frames. The mark is embedded exclusively into 'moving areas' of the video. At the receiver, the mark is extracted from the decoded video and a quality measure of the video is estimated by evaluating the degradation of the extracted mark. Simulation results indicate that the proposed quality metric is able to assess the quality of videos degraded by compression.

The perceived quality of video sequences is generally measured using standard subjective methods. It has been argued that these methods, which typically consist of scaling judgment tasks, are affected by context effects. Context effects are observed when the perceived quality of a video sequence is influenced by the perceived quality of the other video sequences included in the test. Several studies have confirmed the presence of context effects. However, the same studies are ambiguous with respect to the issue of which methods are affected the most. In addition, context effects have been investigated mainly with non-expert viewers. In this study, we investigated context effects in both expert and non-expert viewers. Two experiments were conducted to investigate the relationships between context effects, level of expertise, and type of subjective method. In Experiment 1, we measured range and frequency context effects for two different subjective assessment methods, a double stimulus method (i.e., DSCQS) and a comparison scaling method, using non-expert viewers. We found no frequency context effect with both methods, and a marginal range context effect with the DSCQS method. In Experiment 2, we obtained the same measurements with expert viewers. We found no context frequency effect for both subjective methods, and a very small range context effect for the comparison method only.

Image evaluation tasks are often conducted using paired comparisons or ranking. To elicit interval scales, both methods rely on Thurstone's Law of Comparative Judgment in which objects closer in psychological space are more often confused in preference comparisons by a putative discriminal random process. It is often debated whether paired comparisons and ranking yield the same interval scales. An experiment was conducted to assess scale production using paired comparisons and ranking. For this experiment a Pioneer Plasma Display and Apple Cinema Display were used for stimulus presentation. Observers performed rank order and paired comparisons tasks on both displays. For each of five scenes, six images were created by manipulating attributes such as lightness, chroma, and hue using six different settings. The intention was to simulate the variability from a set of digital cameras or scanners. Nineteen subjects, (5 females, 14 males) ranging from 19-51 years of age participated in this experiment. Using a paired comparison model and a ranking model, scales were estimated for each display and image combination yielding ten scale pairs, ostensibly measuring the same psychological scale. The Bradley-Terry model was used for the paired comparisons data and the Bradley-Terry-Mallows model was used for the ranking data. Each model was fit using maximum likelihood estimation and assessed using likelihood ratio tests. Approximate 95% confidence intervals were also constructed using likelihood ratios. Model fits for paired comparisons were satisfactory for all scales except those from two image/display pairs; the ranking model fit uniformly well on all data sets. Arguing from overlapping confidence intervals, we conclude that paired comparisons and ranking produce no conflicting decisions regarding ultimate ordering of treatment preferences, but paired comparisons yield greater precision at the expense of lack-of-fit.

The method of paired comparison is often used in experiments where perceptual scale values for a collection of stimuli are desired, such as in experiments analyzing image quality. Thurstone's Case V of his Law of Comparative Judgments is often used as the basis for analyzing data produced in paired comparison experiments. However, methods for determining confidence intervals and critical distances for significant differences based on Thurstone's Law have been elusive leading some to abandon the simple analysis provided by Thurstone's formulation. In order to provide insight into this problem of determining error, Monte Carlo simulations of paired comparison experiments were performed based on the assumptions of uniformly normal, independent, and uncorrelated responses from stimulus pair presentations. The results from these multiple simulations show that the variation in the distribution of experimental results of paired comparison experiments can be well predicted as a function of stimulus number and the number of observations. Using these results, confidence intervals and critical values for comparisons can be made using traditional statistical methods. In addition the results from simulations can be used to analyze goodness-of-fit techniques.

For design criteria of displayed images and for the judgment of image quality, it is very important to dispose of a trustful formula for the contrast sensitivity of the human eye. The contrast sensitivity function or CSF depends on a number of conditions. Most important are the luminance and the viewing angle of the object, but surround illumination can also play a role. In the paper a practical formula is given for a standard observer. This formula is derived from a more general physical formula for the contrast sensitivity. In this paper also the effects of orientation angle and surround luminance will be treated. The orientation angle will be incorporated in the formula and a correction factor will be given for the dependence on surround luminance.

Sharpness enhancement is widely used technique for improving the perceptual quality of an image by emphasizing its high-frequency component. In this paper, a psychophysical experiment is conducted by the 20 observers with simple linear unsharp masking for sharpness enhancement. The experimental result is extracted using z-score analysis and linear regression. Finally using this result we suggest observer preferable sharpness enhancement method for digital television.

The colorimetric model for predicting the luminance and chromaticity coordinates produced by Digital Light Processing (DLP) Projection (PJ) TV has been proposed. A series of test colors were measured. A polynomial function fitted the tone reproduction curves with five coefficients per RGB primary channel. The characterization of neutral color due to the use of spoke light was discussed. Experimental results showed that the proposed method is very effective in the colorimetry in DLP PJ TV.

The ISO TC42/WG18-20-22-23 and ANSI/I3A IT10 Technical Committees have now been developing digital photography standards for over a decade. This work has led to the publication of standards on digital imaging terminology, digital camera ISO speed measurements, resolution measurements, OECF (linearity) measurements, image formats and metadata, and picture transfer protocol (PTP). More recently, standards on color encoding specifications and color architectures, a JPEG 2000 profile for digital cameras, camera noise and dynamic range measurements, digital camera specification reporting, and scanner resolution have been finalized. Work in progress includes image quality subjective testing methods, digital camera color characterization, and scanner dynamic range measurements. This paper will review past and current technical challenges, and the state of the solutions provided. In most cases, development includes a significant and innovative research component, which is discussed in relation to fundamental imaging issues. These standards are viewed from a broad digital photography perspective, and placed in context with other work in this area. In addition to providing a forum for the development of standards, technical committees are an important avenue for interaction between companies, user groups, and the government. Such avenues can have a great impact on emerging technologies.