Spatial aliasing quantification and analysis of existing imaging sensors: NASA’s Hubble Space Telescope, OrbView-3 OHRIS, and commercial-off-the-shelf camera for autonomous robots

Abstract. Spatial aliasing in its most pronounced form is seen as a Moiré pattern in (sampled) images. Less dramatically, aliasing is a form of image quality (IQ) degradation and exists at some level within any (sampled) image. More sampling tends to improve IQ due to less aliasing; however, there are drawbacks. Spatial aliasing has been recently quantified by Mudge [Appl. Opt.62(13), 3260–3264 (2023)] for imaging sensors (optics plus detector). This quantification allows a trade to be made between the acceptable aliasing errors imbedded within the (sampled) image and the penalty, or cost, associated with additional sampling, e.g., increased complexity, data throughput and storage, and reduced signal-to-noise ratio for fixed arrays or increased scan time. In this work, we examine several existing and useful imaging sensors along with their imagery and aliasing errors to appreciate how well these existing systems are designed with respect to sampling to better inform how future systems could potentially be improved. Finally, from these analyses, a 2% aliasing error rule is extracted initiating a universal aliasing boundary.


Introduction
With digitization, sampling systems are unavoidable in the 21st century, and errors associated with sampling or the insufficient sampling producing aliasing is an issue in general and in remote sensing imaging in particular.Aliasing, a result of insufficient sampling, is a form of image quality (IQ) degradation.4][5] Unfortunately, this theorem is without a quantifiable penalty for not meeting this minimum sampling frequency, and it is common for imaging sensors (optics plus detector) to spatially sample well below the Nyquist-Shannon sampling theorem's recommendation. 1,2(An aliasing graphical depiction is provided in figure 10.6 of Ref. 6.) For imaging sensors, the input is the object or, more specifically, the object spectrum and is typically a broad-band spatial spectrum-common for sensor systems.If the system has a broad-band input and an anti-aliasing filter [for imaging sensors, this is the optical transfer function (OTF)], which attenuates the high frequencies and (possibly) has a high frequency cut-off, then the Nyquist-Shannon sampling theorem should be considered more of a maximum sampling frequency and not a minimum, 7 and the differences in the sampled image with respect to the unsampled or truth image, an aliasing error, has been quantified for imaging sensors. 7This computed aliasing error not only shows up in a visual inspection of the sampled image but also when the sampled image is sent to a neural network, which can represent a convolution operation amongst others, [8][9][10] and recognizing this fact makes aliasing even more critical for many end-to-end imaging systems.The need to balance the aliasing error imbedded within the sampled image versus sensor complexity and data throughput (neural network) and storage is critical in today's real-time or near real-time imaging systems.In addition, the concepts provided here and in Ref. 7 apply to any sensor or sampling system whether it be an imaging sensor or a sensor that collects any data in any of the three spatial directions or the temporal.
Applying the aliasing error developed in Ref. 7 to currently existing imaging sensors is the focus of this work.The aim is to appreciate how well existing systems are sampled to better inform how future systems could potentially be improved by trading aliasing error and sampling frequency given existing imagery.Note that increasing the sampling frequency leads to increased sensor complexity, reduced data throughput and storage capability, reduced signal-to-noise ratio for fixed arrays, increased scan time for mechanical scanning systems, or reduced field of view for fixed arrays and fixed detector (pixel) size.
In determining the aliasing error, an estimate of the OTF (H), anticipated object (often referred to as a scene) spectrum (O), and detector pitch (δ) or its reciprocal sampling frequency ( 1 δ ) must be known.We have chosen three differing imaging sensors to capture a wide audience within optical sciences (and the sciences in general): (1) NASA's space viewing Hubble Space Telescope (HST), ( 2) the high-resolution Earth viewing commercial satellite OrbView-3 OHRIS instrument, and (3) a commercial-off-the-shelf (COTS) black/white camera relevant to autonomous robots.The relevant sensors parameters are presented in Table 1.
The objects employed are the same two somewhat bounding object spectra used in Ref. 7 and are provided in Fig. 1: (1) 1ðσÞ unity over all frequencies (white) representing a point source object in the spatial domain with a significant amount of high frequency content is similar to viewing a distant star common in astronomy and used in sensor alignment, 11 and (2) an extended object given as an inverse or reciprocal frequency spectrum of E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 0 1 ; 1 1 4 ; 3 5 2 where C is an arbitrary constant value set to unity with units of the spectrum, σ ang is the angular spatial frequency in deg −1 , σ 0 ¼ 1 deg −1 , β and γ are tuning parameters set to 0.9 and 0.01, respectively, with γ preventing the spectrum from becoming undefined at zero frequency.This extended object could represent an indoor environment or an urban area, and β could differ given the particular environment 12,13 where we have chosen β ¼ 0.9 in this work which is more on the side of a natural object rather than indoor. 12The unit deg −1 is typically used in object spectrum plotting and is related to the spatial frequency in linear space (σ) at the image plane by the focal length.Both of the proposed objects are considered real even functions.For the first object, 1ðσÞ, the high spatial frequencies of the point source are higher than an actual star, would ; t e m p : i n t r a l i n k -; e 0 0 2 ; 1 1 7 ; 3 1 1 The aliasing quantity is a comparison between summed sampled image spectra, Ĩðσ s Þ, and unsampled (or truth) image spectra, Iðσ s Þ ¼ IðσÞj σ¼σ s , over the sampled spatial frequency (σ s ) and is where σ is the truth spatial frequency.In this work, the full band aliasing quantity will be analyzed and not broken into sub-bands per Ref. 7. The unsampled image is tied to the object by object-image relationship in the frequency domain which is ; t e m p : i n t r a l i n k -; e 0 0 4 ; 1 1 7 ; 1 6 5 where HðσÞ is the sensor-level OTF.For our analysis, we have limited the full sensor transfer function to simply the OTF and acknowledge that an imaging sensor is two dimensional but only consider one here for simplicity. 7f the modulus of the OTF or modulation transfer function (MTF) is used in place of the OTF, which is often all that is readily available, this implies that a higher than or equal to sampling frequency is required for the same aliasing error.The sampling frequency selection can be thought of as an absolute maximum when using the MTF in place of the OTF.

Spatial Aliasing Error and Analysis
A summary aliasing error (ϵ) is given in Table 2 with the full analysis details being provided in Secs.3.1-3.3.The tabulated values are the aliasing error and its corresponding sampling frequency for each of the three imaging sensors and two objects discussed prior.For comparison purposes and in the last two columns, the sampling frequency for an aliasing error ϵ × 100% of 2% and Q are provided 1,2 where the value Q is where λ 0 is the reference wavelength and f∕# is the f-number, which is focal length divided by the effective entrance aperture diameter, and the optical cut-off frequency is 1 λ 0 f∕# .One take-away is that Q is not well correlated with aliasing error across differing imaging sensors and objects.This is because Q contains the optical cut-off frequency and does not contain the full sensor transfer function or even the OTF nor does it consider the anticipated object spectrum, but both are contained within the aliasing error.

NASA's Space Viewing HST
HST was launched from the Space Shuttle Discovery in 1990.The Wide Field Planetary Camera (WFPC) is the imaging work-horse for the HST observatory.The original WFPC was replaced by WFPC2 that included corrective optics for the large aberrations present in the main HST telescope primary mirror.There are readily available measured OTF values for the WFPC2 sensor that led us to utilize imagery from WFPC2 to illustrate aliasing errors.The OTFs are provided in Fig. 2 and show the optics OTF (real and tiny imaginary part) as well as the detector OTF.The optics OTF is calculated using measured, on-orbit, phase, and amplitude as presented by Krist and Burrows. 14On the detector side, we consider that the detector OTF consists only of the blurring due to the detector (pixel) aperture. 1,2Only the optics and detector OTF components are considered as dominate relevant contributions to obtain an approximate sensor-level OTF (H).
A plot of the aliasing quantity is shown in Fig. 3 for each of the two anticipated objects with HST's as-built sampling frequency is marked with a black square (67 mm −1 ).This frequency for the fully sampled image has an aliasing error of 9.3% for the 1ðσÞ object used in phase retrieval and 0.08% for the R β;γ ðσÞ object (see Table 2) using Eq. ( 2).In addition, there is a colored square marking the ½, ¼, and ⅛ reduced sampling frequencies for comparison purposes.The OTF is fixed since the detector size is maintained, but the sampling frequency is reduced (pitch increased) by removing detectors (pixels) from the image, e.g., for ½ reduced, every other detector is removed forcing a 50% fill factor.Providing a sense of how the aliasing quantity compares with actual images is an important aspect of this analysis.Specifically, we will consider two different classes of imagery from HST WFPC2/PC instrument where PC is planetary camera instrument: (1) a starfield and (2) an extended object.We use an image of the Fornax constellation as a starfield to represent a set of point source objects 15 and an image of Jupiter as the extended object. 16Both the Fornax and Jupiter images are acquired using the F555W filter in WFPC2.The Fornax images in Fig. 4 show the full sampling image followed by three reduced samplings as given above, and Fig. 5 shows the same but for science imagery of Jupiter.Each of the image's sampling frequency and aliasing quantity is shown in Fig. 3 and denoted by a colored square for the respective sampling frequency.
For the Fornax constellation image in Fig. 4, when going from the full sampling to ½ sampling (black to red boxed images), there is a visually significant IQ degradation with some stars hardly visible whereas this is not true for the image of Jupiter in Fig. 5. Based on the aliasing values in Fig. 3, this difference is due to the fact that Fornax constellation image aliasing error increases significantly from 9.3% (arguably too high) to 47% given by the blue curve, and on the other hand, the increase is only from 0.08% to 0.4% with the Jupiter image (red curve) which is relatively minor.For the Jupiter image in Fig. 5, this noticeable degradation occurs between the ¼ sampling and ⅛ sampling (green to blue boxed images) where the estimated aliasing error (red curve in Fig. 3) goes from 1.0% to 1.9%.The reason for the difference in the above two image sets is a result of vastly differing slopes in the two curves after the knee in the curve moving in right-to-left direction in Fig. 3.However, what is consistent in both of the image sets is, once the aliasing error gets somewhat near 2% or above the IQ degradation starts to become visibly evident.For this reason, a black dashed reference line is placed in Fig. 3 where α ¼ 1.02 and ϵ × 100% ¼ 2%.

Earth Viewing OrbView-3 OHRIS Space Telescope
The OrbView-3 OHRIS spacecraft (similar to IKONOS) was launched in June 2003 and carried the High Resolution Imaging System (OHRIS).The OHRIS mission provided commercially available, high resolution [1-m ground sample distance (GSD)] global imagery from an approximate altitude 470 km.The OHRIS sensor suite included a panchromatic imaging sensor as well as a four channel, multispectral sensor with 4-m spatial GSD.
The OHRIS imagery is available from the United States Geological Survey in Basic Enhanced (Level 1B) radiometrically corrected format.The enhanced panchromatic imagery has been sharpened with a high pass Laplacian filter in preparation for pan-sharpening of the multispectral imagery. 17This enhanced sharpening provides a modest boost to the mid-spatial frequency MTF.
In this work, we utilize imagery from the OHRIS panchromatic sensor to analyze aliasing errors in an Earth viewing sensor.The on-orbit optics OTF amplitude and phase are not as readily available as for HST.However, several sets of published values for the OHRIS sensorlevel MTF are available (see Sec. 2) and were estimated using on-orbit edge features. 18,19n Fig. 6, there are two measured sensor-level MTF curves plotted: Ross et al. 18 and Kohm and Tira. 19This aliasing analysis uses only the Ross et al. 18 MTF and not the Kohm and Tira, 19 and the main reason for this is the Ross et al. 18 data extend out to 167 mm −1 whereas the Kohm  and Tira 19 data extend only to 84 mm −1 .Nonetheless, it should be noted that over the first 84 mm −1 the curves are nearly identical.The Ross et al. 18 data in Fig. 6 have been zero padded after 167 mm −1 to increase the truth image to near continuous in the spatial domain.
The fully sampled image has an aliasing error of 13% for the 1ðσÞ object and for the R β;γ ðσÞ object is 0.6% (see Table 2) using Eq. ( 2).As done prior, the square markings in Fig. 7 are for the ½, ¼, and ⅛ reduced sampling, and a black dashed reference line denotes α ¼ 1.02 and ϵ × 100% ¼ 2%.Again, the OTF is fixed since the detector size is maintained, but the sampling frequency is reduced (pitch increased) by removing detectors from the image, e.g., for ½ reduced, every other detector is removed forcing a 50% fill factor.
In Fig. 8, a full sampling full sized Earth image of the Naval Air Weapons Station China Lake acquired with OrbView-3 OHRIS is provided as a context image.In Fig. 9, a full sampling sub-image of the image in Fig. 8 is provided where a dashed box denotes the sub-image position.The full sampling sub-image is black boxed and followed by three reduced sampling frequencies of the same sub-image in red, green, and blue boxes as done in Sec.3.1.Each of the sub-image's sampling frequency and aliasing quantity is shown in Fig. 7 and denoted by a colored square for the respective sampling frequency.
Visually there is a noticeable difference between the full sampling frequency black boxed image and the ½ reduced sampling frequency red boxed image (∼3% error) as seen by the Fig. 6 OHRIS space telescope sensor-level MTFs: Ross et al. 18 and Kohm and Tira. 19The aliasing analysis uses the Ross et al. 18 MTF.runway lines becoming dashed in the upper right corner.In the HST data, aliasing errors near 2% or above tend to show a noticeable degradation visually in the image, and this is similar for the OHRIS imagery.The ½ reduced sampling frequency red boxed image in Fig. 9 begins a pattern where the runway diagonal solid line becomes dashed, and as the sampling is further reduced, the dashes become dots and grow in size and are further spaced-a demonstration of aliasing being created.
In general, the R β;γ ðσÞ object is suited for this application, but this may not always be the case.For example, a Sun glint can be considered a point-like object, and the more appropriate curve for this is the 1ðσÞ object spectrum or blue curve in Fig. 7.A dramatic increase in the aliasing quantity occurs when jumping from the red curve to the blue leading to the aliasing error increasing from 0.6% to 13% error.The sub-images used in Fig. 9 is indicated with a dashed box.

COTS Imaging Sensor for Autonomous Robots
To commercialize robots, imaging sensors must be physically compact and cost effective, and since robots operate in real time, the throughput of the sampled image must be relatively high indicating a low sampling frequency is desirable.However, this is in direct conflict with aliasing error particularly when using a real-time neural network as previously discussed.In this section, we examine a standard COTS imaging sensor to provide a sense of the aliasing error to the autonomous robot community.
The imaging sensor or camera used in this analysis is a ThorLabs DCU224M charged-couple device (CCD) with a size of 29 mm by 29 mm by 29 mm, weights 43 g, and 82 mW to power for the Sony ICX205AL CCD.The imaging sensor's optics used for the data collection is a COTS Schnieder Kreuznach Cinegen 1.4/8-0902 lens assembly having an adjustable f-number set to 5.6, and the size is 37 mm long and 34 mm in diameter with a weight of 90 g and no power consumption.
Unfortunately, the optics OTF is not readily available from the manufacture.In lieu of this, the sensor-level MTF has been determined via a slanted-edge test as done with the OHRIS imaging sensor from Sec. 3.2.The on-axis sensor-level MTF testing results are shown in Fig. 10 where the MTF has been zero padded.
The fully sampled image has an aliasing error of 33% for the 1ðσÞ object and for the R β;γ ðσÞ object is 2.5% (see Table 2) using Eq. ( 2).Again, the square markings in Fig. 11 are for the ½, ¼, and ⅛ reduced sampling frequencies for red, green, and blue, respectively, with a black dashed 10 COTS imaging sensor-level MTF.line denotes α ¼ 1.02 and ϵ × 100% ¼ 2%.As before, the sampling frequency is reduced (pitch increased) by removing detectors from the image, e.g., for ½ reduced, every other detector is removed forcing a 50% fill factor.
In Fig. 12, a full sampling full sized unsharpened image of street intersection is provided as a context image.In Fig. 13, a full sampling sub-image of the image in Fig. 12 is provided where a dashed box denotes the sub-image position.The full sampling sub-image is black boxed on the far left and is followed by three reduced sampling frequencies of the same sub-image in red, green, and blue boxes as done in Secs.3.1 and 3.2.Each of the sub-image's sampling frequency and aliasing quantity is shown in Fig. 11 and denoted by a colored square for the respective sampling frequency.
There is a visual difference between the full black boxed image and the ½ reduced sampling frequency red boxed image as noted by the bike lane solid lines on the right side of the subimage, and a significant difference across the whole image from the ½ reduced sampling frequency to the ¼ reduced sampling frequency green boxed image.The black boxed image is estimated to have a 2.5% aliasing error (arguably too high), and the ½ reduced sampling frequency green boxed image has a 7.7% which is well above the 2% number using the appropriate  red curve in Fig. 11.In the HST data and the OHRIS data from Secs.3.1 and 3.2, aliasing errors near 2% or above tend to show a noticeable degradation in the image, and this is also consistent for the COTS imagery.As in the OHRIS images, the full reduced sampling frequency red boxed image begins a pattern where the diagonal solid line becomes dashed, and as the sampling is further reduced, the dashes become dots and grow in size and are further spaced-demonstrates aliasing being created.Similarly, there would be a significant jump in the aliasing error increasing from 2.5% to 33% should a point source exist within the image as discussed in Sec.3.2.
In this analysis, focal plane detector array is black/white but often a focal plane detector array has a Bayer color filter array applied used to provide color images.Given an RGGB pattern, the color band detector pitch has twice the black/white detector pitch, in this case 9.3 μm, causing a drop in the sampling frequency ( 1 δ ) to 108 mm −1 .Had this type of camera been used in place of the black/white camera, the aliasing error for each color band image would be 7.7% (the red square marking in Fig. 11) which is significantly worse than 2.5% for the black/white camera by more than a factor of three (and from 33% to 89% for the 1ðσÞ blue curve with again nearly a factor of three).

Discussion
The aliasing quantity given in Eq. ( 3) could be adjusted to better handle inversions in the OTF, which is a rare occurrence.One thought is to pull in the absolute value inside the summation as given in the below equation for an adjusted aliasing quantity E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 0 6 ; 1 1 7 ; 4 8 6 This assures that no subtraction of values occurs during the summation due to an OTF inversion, and it gives the same value as the original aliasing quantity when the OTF real part is always positive and zero imaginary part.The drawbacks to this adjusted aliasing quantity are that the physics of the problem becomes diluted, and when there is a zero crossing or an inversion, the adjusted aliasing quantity is discontinuous much like the MTF with an OTF inversion.However, in general, the sensor would be designed such that there is little to no inversion issues in the sensor-level OTF, and therefore, the original aliasing quantity given in Eq. (3) will generally suffice.
As a thought experiment, consider an object (input) spectrum that is not broadband but is a single tone of σ 0 , e.g., a Dirac delta function δðσ − σ 0 Þ, and is at a frequency that is greater than half the sampling frequency or σ 0 > 1 2δ .Since the maximum sampled frequency is 1 2δ by virtue of the Nyquist-Shannon sampling theorem, 5 then the truth image spectrum is zero within the sampled frequencies or P ∀ σ s ≥0 Iðσ s Þ ¼ 0, but this is not the case for the sampled image spectrum.This spectrum is entirely aliased where the high spatial frequency is masquerading as a low spatial frequency or solely disinformation (Aliasing can be thought of as disinformation since it is incorrect information imbedded in the image.)giving P ∀ σ s ≥0 Ĩðσ s Þ ≠ 0. Therefore, Eq. ( 2) gives ϵ ¼ ∞.This makes good mathematical sense because all of the signal is aliased indicating there is no part of the sampled image spectrum that is correct and is entirely disinformation, and the error should be infinite as calculated by Eq. ( 2).This single tone input is where the Nyquist-Shannon sampling theorem applies strictly and uncompromisingly.

Conclusions
The theory developed in Ref. 7 has been applied to a wide variety of existing and useful imaging sensors having vastly differing focal lengths (see Table 1).The COTS camera has the worst aliasing error, and the HST has the least aliasing error with the OHRIS landing in the middle.With a near 2% or greater aliasing error, a visually noticeable IQ degradation appears giving rise to a "2%" aliasing error rule-of-thumb as a potential universal aliasing boundary.For the HST starfield calibration, there is significantly more aliasing error and a case is made after taking an image to reposition by one half a sample and take another image-a process called drizzling, which is done on HST. 21Combining the two images would increase the sampling frequency by a factor of 2 to 134 mm −1 , and the HST starfield calibration aliasing error would come down to 0.002% which is well below 2% per Fig. 3 and Table 2. Drizzling is not necessary for the R β;γ ðσÞ object, i.e., Jupiter since the aliasing error is already well below the "2%" number at 0.08% per Table 2.The downside of drizzling is that it takes time, but using the aliasing error, the minimum amount of needed drizzling can be established (in this case, three images in total for the two dimensions) to minimizing calibration down time.

Fig. 3
Fig. 3 HST aliasing quantity (α) as a function of sampling frequency: 1ðσÞ (blue curve) to closely represents a point source or starfield object and the R β;γ ðσÞ (red curve) represents a natural extended object.

Fig. 7
Fig. 7 OHRIS aliasing quantity (α) as a function of sampling frequency: 1ðσÞ (blue curve) represents a point source object and the R β;γ ðσÞ (red curve) represents a natural extended object.

Fig. 8
Fig.8OHRIS Earth image of Naval Air Weapons Station China Lake in southern California is shown with full sampling.20The sub-images used in Fig.9is indicated with a dashed box.

Fig. 11
Fig. 11 COTS aliasing quantity (α) as a function of sampling frequency: 1ðσÞ (blue curve) represents a point source object and the R β;γ ðσÞ (red curve) represents a natural extended object.

Fig. 12
Fig.12COTS image of a typical San Francisco Bay Area suburb.The sub-images used in Fig.13are indicated with a dashed box.

Table 1
Top-level imaging sensors parameters.estimatedaliasingerror to that of an actual star, and can be thought of as an upper bound.The extreme lower bound would be a Dirac delta frequency distribution which corresponds to a flat image.However, a more representative object lower bound is the second object given by R β;γ ðσÞ.This spectrum appears to be established by examining images not objects which implies that the data have passed through a low pass filter, e.g., OTF.If true, high spatial frequencies have been attenuated giving a slightly lower estimated aliasing error and can thus be further considered a lower bound.Arguably, the object is more difficult to anticipate or estimate as opposed to OTF (anti-aliasing filter) when choosing a sampling frequency, and neither object has a standout single tone which would tend to generate a Moiré pattern within the image [figure6.26(rooftop) of Ref. 1 and figure 15 (striped shirt) of Ref. 2].

Table 2
Imaging sensor's aliasing error and its corresponding sampling frequency as well as the sampling frequency for an ϵ × 100% of 2% and Q.