Compound eye image scanner optimally arranged for image combining and suppressing axial chromatic aberration by wavefront coding

Abstract. A compound eye optical system design allows us to construct a compact image scanner with a large field of view and a large depth of field, both of which are beneficial for use in a copier’s flatbed scanner. Our prototypes are constructed of 32 or 28 imaging units, which are ommatidia, and the representations from the imaging units are electrically combined. Since the arrangement of the imaging units is a key factor in combining the images correctly without any loss at the boundaries, we discuss two types of arrangements along with results of their prototypes: a one-line arrangement of varying magnification imaging units and a two-line staggered arrangement of telecentric imaging units. We also discuss the effectiveness of wavefront coding (WFC) in suppressing chromatic aberration, which is caused using cheap refractive imaging units, along with experimental results of applying both cubic- and quartic-phase-modulated WFC.


Introduction
A compound eye imaging system, which greatly reduces the size of optics, provides an abundant variety of new options in structures and functions. 1,2 The document scanner used in a copier must be compact but with a wide field of view, have a high 600-dpi resolution, and work with a short object distance. Therefore, a system implementing compound eye optics has been applied in this field. Two types of optics have been used to create a whole image from plural images by ommatidia: directly creating an optically superimposed image and combining images in electrical processing.
In the first method, which creates an optically superimposed image, each unit image is erected by relaying an intermediate image. Anderson used two arrays of lenses to form an erect composite image 3 for a close-up imaging system. Meyer et al. 4 developed a 2D compact camera called oCley. For line image scanners, some patents adopt a refractive lens array 5 or a reflective concave mirror lens array, 6 thus making an erect optical image with identical magnification. The most commercially successful contact image scanner is a gradient index lens array, 7 which is widely used in automated teller machines or automatic document feeders embedded in the paperguide components of copiers. As documents flow in close contact with sensors, such applications need only a short depth of field (DOF). One merit of optically superimposing systems is that they do not need an image-combining process. On the other hand, such systems cannot afford a large DOF because the overlap of the images between adjacent imaging units becomes misaligned when the object distance changes. A copier's flatbed scanner needs a large DOF because when a thick book is opened and placed down, a gap inevitably appears at the gutter between pages. Consequently, optically superimposing systems are unsuitable for flatbed scanners.
In the second method, which creates a whole image by combining images through electrical processing, plural imaging units reduce the images. Since it does not involve optically overlapped images between adjacent imaging units, a large DOF is possible if appropriate image processing combines the images. Brückner et al. 8 developed a thin 2D camera called eClay using such an electrical combining process.
Here we develop a compact imaging scanner for use as a copier's flatbed scanner. We previously developed two types of image scanners to achieve compactness: a lensless image scanner 9 and a compound eye image scanner implementing the above electrical combining processing. 10,11 Although the former type is very thin, its 300-dpi resolution is insufficient for a copier. The latter consists of plural reflective mirror imaging units, resulting in no chromatic aberration; however, high accuracy in the mirror shape and alignment, especially the tilt adjustment, is required. Therefore, in this report, we use refractive optics as imaging units in a compound eye scanner to lessen the sensitivity of assembly. Although refractive optics cause axial chromatic aberration, which appears as color blurring at the image's edge, we suppress this effect by applying a wavefront coding (WFC) technique. WFC, first reported by Dowski and Cathey, 12 is effective for extending the DOF by distorting the wavefront with cubic-phase modulation and a successive deblurring process. Wach et al. 13 showed that WFC also effectively suppresses axial chromatic aberration. Although cubic-phase WFC has a large effect on extending the DOF, the artifact in the deblurred image might be obtrusive when the restoration kernel mismatches the point spread function (PSF). 14 The PSF of the cubic-phase WFC has direction dependence, and the modulation transfer function (MTF) along the diagonal direction tends to be null within the spatial-frequency pass band, which degrades the contrast of the restored image in the diagonal direction pattern. 15 Therefore, various patterns of axially symmetric modulated WFC have been proposed, including a logarithmic function, 16 a quartic function, 17,18 and an annular phase mask. 19 In this report, we present three key contributions: (a) a compound eye optical system can be applied to an image scanner product that requires compactness and high image quality, shown by specific design examples and the evaluation results of prototypes. (b) By applying a WFC technique to the compound eye optics, the imaging units can have good image quality despite their simple structure of just two plastic lenses. (c) Prototypes leveraging cubic and quartic WFC are compared in real images, for the first time to the best of our knowledge.
First, we discuss the optimal arrangement of our compound eye image scanner. We designed and fabricated two different layouts: a one-line arrangement of varying magnification imaging units and a two-line staggered arrangement of telecentric imaging units (Fig. 1). Second, we discuss the effects and side-effects of cubic-and quartic-phase-modulated WFC by comparing  20 That study showed that cubic WFC extends the DOF farther, whereas quartic WFC does not generate angular-dependent image quality as cubic WFC does. Accordingly, we demonstrate a compound eye optical system with practical use of WFC that is suitable for a flatbed scanner with a large DOF.
2 Two Arrangements of Compound Eye Image Scanner Figure 1 shows two different arrangements of our compound eye image scanner. Figure 1(a) shows a one-line arrangement of varying magnification imaging units, where all of the imaging units are aligned on a single line, creating a 310-mm field of view. Each imaging unit must cover a larger field of view than the diameter of the first lens so that no missing area is seen at the boundaries between imaging units. Therefore, it cannot be telecentric in the object space since the field of view grows with the object distance. Then the overlapping areas between the adjacent fields of view change with the object distance, as shown in Fig. 2, which illustrates three overlapped images by three imaging units in two different object distances. Image processing to combine adjacent unit images is performed in the following series. (1) Pattern matching is performed between the overlapped images. (2) The size of the overlapped width in the X direction is determined. (3) Unit images are resized in such a way that the size of those overlapped areas is identical. (4) The unit images are combined into an output image. Since this process is slightly complicated, it may cause some mistakes in particular cases.
On the other hand, Fig. 1(b) shows a two-line staggered arrangement of telecentric imaging units, which have telecentricity in the object space so that the field of view of each unit is stable regardless of the object distance. We previously reported its reflective mirror version. 10,12 Since the diameter of the first lens is larger than the field of view due to telecentricity, the imaging units are aligned in a two-line staggered arrangement. As the object is scanned along the Y direction relative to the image scanner, the unit images shift one after the other with a gap in the two lines in the Y direction [ Fig. 1(b)]. The image-combining processing for this arrangement is easier than that for the one-line arrangement because the image size does not change with the object distance. However, the optical design of the illumination is more severe because illuminance uniformity is required along the Y direction. Pattern-matching processing for the overlapping area remains necessary because the shift value between the images on the two lines may fluctuate slightly due to the scan speed fluctuation in the Y direction, which is discussed in Sec. 4.

Experiment on One-Line Arrangement of Varying Magnification
Imaging Units

One-Line Arrangement without WFC: Prototype A
We first designed prototype A, which is an image scanner with a one-line arrangement but without WFC. The cross-sectional optical layout of the four imaging units in the X − Z plane is shown in Fig. 3. Each unit is constructed of two lenses and an aperture stop that is placed directly above the second lens. Prototype A has the following target specifications: 600-dpi resolution (42.3-μm pitch), 1.2-mm DOF, and 310-mm field of view. Since the magnification ratio of each imaging unit is designed to be 1/2, the sensor pixel's pitch is 21.15 μm. An image sensor uses three lines to acquire color, on which red, green, and blue color filters are formed. The field of view of each unit is 10.8 mm, and the imaging units are aligned in a 9.8-mm pitch, resulting in 1.0-mm overlapping areas at the boundaries of the imaging units. The number of imaging units needed to cover the 310-mm field of view is 32, and the numerical aperture is NA ¼ 0.1.
One key design guide in each imaging unit is that it should be nearly object-space telecentric. The angle between the principal ray launched from the edge of the field of view and the optical axis should be as small as possible because the pattern-matching process (discussed in Sec. 2) must be correctly performed. According to the change in the object distance, the smaller the change in the field of view is, the easier the pattern-matching process is. Suppose, for example, the object is a chart of a black-and-white grating pattern with a spatial frequency of 5.9 line-pairs/mm (lp/mm), which is equivalent to 300-dpi resolution. Then the image-matching processing may be inaccurate when the change in the field of view is larger than a 5.9-lp∕mm, 169-μm period. The imaging unit must be as close to telecentric in the object space as possible so that the change in the overlapping area does not exceed 169 μm within the 1.2-mm DOF specification.
The uniformity of the optical characteristics in the field of view is another critical point in optical design for a compound eye image scanner. If such optical characteristics as MTF or a distortion degrade uniformity in the X direction, the unevenness is repeated with the period of the imaging unit, attracting the viewer's notice. Therefore, we designed the first lens with a free-form surface to suppress unevenness. This design's prototype is shown in Fig. 4. Figure 4(a) shows the entire appearance of the imaging part, Fig. 4(b) shows a side view with an illumination module, and Fig. 4(c) shows the top view of the imaging part with its cover removed, where the first lenses are aligned in a oneline pattern. An example image taken by prototype A is shown in Fig. 5(a). Figure 6(a) shows images of a black-and-white grating pattern with 4.9 lp∕mm at Z f ¼ 0, 0.6, and 1.2 mm, where Z f is the difference from the focal position shown in Fig. 3. Intensity profiles of the images along the X direction for blue (B), green (G), and red (R) are also shown in Fig. 6. Although their intensity profiles are almost the same at Z f ¼ 0, the contrast of B decreases faster than the other two colors with an increase in Z f , and the white part of the picture at Z f ¼ 1.2 mm is yellow.
The coloring of the small pitch grating is attributed to axial chromatic aberration. Figure 7(a) shows the calculated through-focus MTF in the B, G, and R colors at 4.9 lp∕mm in the object space. The difference between the values at the maximum of the three graphs shows axial chromatic aberration, which was caused by the chromatic dispersion of the lens material. The differences in the MTF values among the three colors at Z f ¼ 0, 0.6, and 1.2 mm closely reflect the contrasts of the profiles in Fig. 6. Since the lenses are mass-produced by plastic injection    molding, it is difficult to use achromatic lenses that are composed of two individual lenses made from glass with different amounts of dispersion. Therefore, we applied WFC, discussed below.

One-Line Arrangement with Cubic-Phase-Modulated WFC: Prototype B
Prototype B resolves the problem of chromatic aberration in prototype A by applying WFC with cubic-phase modulation, which should be added at the stop aperture to identically modulate an image everywhere in the field of view. Because the front surface of the second lens lies just under the stop aperture, the phase modulation of WFC can be superimposed on the surface. Since the lens is shaped by mold injection, replicating it is easy with phase modulation. Prototype B is identical to prototype A except for the function of the superimposed surface. A surface sag function for cubic-phase-modulated WFC (cubic WFC) is expressed as E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 0 1 ; 1 1 6 ; 4 3 5 where X and Y are the horizontal coordinates of the surface in mm units and α is the coefficient of the cubic-phase modulation in mm −2 units. The diameter of the stop aperture is 2.2 mm and α is set to 0.008 mm −2 . Figures 3(b) and 3(c) show the surface sag profile along the X direction in prototypes A and B. The broken line in Fig. 3(c) is the superimposed component of the cubicphase modulation expressed by Eq. (1), which implies that its addition to the solid graph of Fig. 3(b) is the solid graph of Fig. 3(c). The maximum sag amount inside the stop aperture is 10.65 μm, which equals 10.3 wavelengths, using refractive index n ¼ 1.53 and wavelength λ ¼ 0.55 μm. The amount of phase difference is almost identical to that in an earlier work. 13 Figure 8 shows spot diagrams in (a) prototype A without WFC and (b) prototype B with cubic WFC. Spot diagrams are distorted into triangle shapes by cubic WFC, and the differences in the spot size among colors or object distances are reduced in prototype B. Figure 7(b) shows the calculated through-focus MTF in B, G, and R colors at 4.9 lp∕mm in the object space for  prototype B. Compared to Fig. 7(a), the graphs are flat against the object distance, and the differences among colors are small in return for small peak values.
Prototype B was composed by exchanging the second lenses for the cubic-phase-modulated ones. Figure 9 shows the images of a point source, made from a pinhole placed before an LED light source. The image shapes resemble the spot diagrams in Fig. 8(b).
The raw images of the "ABC" characters taken by prototype B are shown in Fig. 5(b), and the images restored by a deblurring process (noted in Appendix A) are shown in Fig. 5(c). Although the raw images by the cubic WFC [ Fig. 5(b)] are more blurred than those without WFC [ Fig. 5(a)], the restored images of Fig. 5(c) are clearer than those of Fig. 5(a). Figure 6(b) displays images of a grating pattern and the profiles by prototype B, in which a deblurring process is performed to increase contrast. The images show no pseudo-color, and their profile shows no contrast difference among colors. Figure 10(a) shows deblurred images of "289 dpi" at three different levels of Z f . An artifact is found near the oblique line of character "2" [Fig. 10(a-2)]. Figure 11(a) shows the corresponding MTF graphs at Z f ¼ −0.8 mm along the X and diagonal directions. Some of the diagonal MTFs have nearly zero values at a spatial frequency lower than the Nyquist frequency, which comes from the cubic-phase-modulated PSF. This situation accounts for the artifact. 15 3.3 One-Line Arrangement with Quartic-Phase-Modulated WFC: Prototype C Prototype C suppresses the artifact found in prototype B. The function of the quartic-phasemodulated WFC (quartic WFC) based on spherical aberration is axial symmetric. 17 The following sag function describes the lens surface superimposed to produce quartic WFC: Fig. 9 Measured PSF in prototype B from Z f ¼ −1.2 to 1.2 mm for blue, green, and red. Fig. 10 Comparison of deblurred images: (a) prototype B: (a-1) "289 dpi" taken at Z f ¼ −0.8, 0, and 0.8 mm, where image size is 203 × 93 px (8.6 mm × 3.9 mm in object space) and (a-2) magnification of "2" at Z f ¼ −0.8 mm; (b) prototype C: (b-1) "247 dpi" taken at Z f ¼ 0, 1, and 2 mm, where image size is 185 × 85 px (7.8 mm × 3.6 mm in object space) and (b-2) magnification of "2" at Z f ¼ 0. (2) where r is the radial distance in mm units, and a and b are coefficients. The fourth power term produces spherical aberration with a side effect of the focal-point shift. The squared term is used to cancel the focal-point shift.
The basic lens structure before adding WFC modulation in prototype C was redesigned to extend the DOF specification to 2 mm. Figure 12(a) shows the ray trace of the lens design before adding WFC, resulting in the spot diagrams shown in Fig. 12(c). The front surface of the second lens is flat, located at the stop aperture. The added lens sag function that gives quartic WFC is expressed by Eq. (2) with coefficients of a ¼ 0.042 mm −3 and b ¼ −0.015 mm −1 [ Fig. 12(b)]. The modulated spot diagrams by the WFC [Fig. 12(d)] yield only slight differences among the colors and object distances. Figure 11(b) shows the MTF in prototype C and shows no zero value of the MTF under the Nyquist frequency. Since the phase modulation in quartic WFC is axial symmetric, the MTF has no directional dependency. Figure 13 shows the through-focus MTF  before and after adding the WFC modulation. The large difference in MTF among the colors shown in Fig. 13(a) is decreased by adding WFC, as shown by the solid and broken lines in Fig. 13(b). Since the ability to extend DOF is weaker in quartic WFC than in cubic WFC, the graphs in Fig. 13(b) are not flatter than those in Fig. 7(b). The broken lines of Fig. 13(b) show the expected MTF by performing the deblurring process for the WFC, showing that the MTF values are over 0.4 even at Z f ¼ 2 mm.
Prototype C was then fabricated and the resulting pictures are shown in Figs. 10(b) and 14. The magnified picture in Fig. 10(b-2) shows that it no longer has an artifact around the edge of the character "2." Figure 14 shows that the deblurring process produces clear images even for the fine structure of Chinese characters.
The measured MTF values at 4.9 lp∕mm fit well with the simulation curve shown in Figs. 13(b) and 13(c). The striped black and white patterns with 4.9 and 7.0 lp∕mm are read by prototype C, and the images are restored by the deblurring process. Then the contrasts of the wave patterns taken before and after the deblurring process are calculated [Eq. (3) where max and min are the maximum and minimum values of the waves of the striped pattern profile.
Although prototype C has good image quality within the specified DOF, it suffers from the image-combining process due to the compound eye system when the object distance is over the DOF range. Image-combining error may occur in rare cases such as when a repeated fine pattern Fig. 13 Through-focus MTF in prototype C: (a) calculated MTF before adding WFC at 4.9 lp∕mm and (b), (c) calculated and measured MTF at 4.9 lp∕mm and 7.0 lp∕mm, respectively. Solid and broken lines are calculated MTF before and after deblurring process, respectively. Solid and blank markers are measured MTF before and after deblurring process, respectively. Graph colors correspond to B, G, and R in image. exists on the overlapped area and the object distance exceeds the DOF. If the amount of the image's shift on the overlapped area exceeds the period of the repeated pattern, it is difficult to determine the correct positions to combine. One solution is to get information of the correct-combining positions by adding range sensors. If the object distance near the overlapped area is known, the shift amount of the overlapped image can be determined. Such range sensors can be easily installed by adding point-source LEDs very close to the image sensor chips on the sensor substrate. 21 However, we discuss another arrangement system for a compound eye image scanner in the next section because the prototype C system is complicated.
4 Two-Line Staggered Arrangement with Quartic-Phase-Modulated WFC: Prototype D A two-line staggered arrangement allows us to make a telecentric system in the object space because it produces space between adjacent objective lenses on one line. Since the image size is identical regardless of the object distance in object-telecentric systems, the image-combining process is easier than in a one-line arrangement system. Figure 15(a) shows the ray trace layout of the object-telecentric system for prototype D. The numerical aperture is NA ¼ 0.047, which is smaller than prototype C to increase its DOF. Figures 15(b) and 15(c) show spot diagrams before and after adding the quartic-phase modulation, showing the WFC effect. Figure 15(d) shows through-focus MTF graphs at 4.9 and 7.5 lp∕mm before adding the WFC, revealing a small DOF and large differences among the three colors. After adding the WFC [ Fig. 15(e)], the graph widths increase in exchange for smaller peak values. The peak position of the graphs for the lower spatial frequency of 4.9 lp∕mm is smaller than that of 7.5 lp∕mm. The decrease is a feature of spherical aberration. Figure 16(a) shows prototype D from the top with its white illumination unit. Rectangular apertures of the first lenses are aligned in a two-line staggered arrangement, and the boundaries between the apertures are overlapped in the X direction. Figure 16(b) depicts the flip side of prototype D with its sensor substrate removed, showing a two-line staggered array of the second lenses.
Here we discuss the required image processing package for our compound eye image scanner. An example of the raw image by prototype D is shown in Fig. 17  the offset in the Y direction between adjacent images. The following image processing (1) to (5) is performed in sequence.
(1) Correction of chromatic aberration of magnification in the X direction. Figure 17 A custom chart for this calibration is read by the prototype scanner to calculate the misalignment parameters. (4) Combining position correction for scan speed fluctuation. The scan speed fluctuation in the Y direction causes the combining position to misalign between adjacent unit images. The misalignment amount is at most a few pixels in the Y direction. This processing compensates for the misalignment. Two parts of the adjacent unit pictures in the  overlapped area are extracted. The degree of similarity is calculated at each shift step within a few pixels in the Y direction using a method of the sum of absolute differences. The matching position is determined as the point of the largest degree of similarity. The boundaries of each unit image are stitched using the matching position parameters. (5) Image intensity matching between adjacent unit images. The differences of the 2-mm reading positions between adjacent imaging units in the Y direction may cause a slight image intensity difference. The entire image intensity distribution is fine-tuned to match the image intensities at the overlapped areas.
The final image after the above image processing from Fig. 17(a) is shown in Fig. 18(a). Figure 18(b) displays the magnification of the rectangular area in Fig. 18(a), and includes two boundaries, showing flawless image combining.

Summary
We studied the most suitable arrangement of plural imaging units and WFC to suppress the effect of axial chromatic aberration for developing an image scanner with a large field of view and a large DOF. We fabricated four prototypes and compared the results.
Prototypes A, B, and C with one-line arrangement had different WFC types. Prototype A had no WFC, producing color blurring at the edges of an image when it was defocused. Prototype B with cubic WFC successfully suppressed the color blurring, although it produced a side effect of an artifact at the edge along the oblique directions. Prototype C with quartic WFC suppressed both the color blurring and the artifact in the oblique directions. However, the image-combining process made mistakes in some particular charts at large defocused positions since the imaging units in the one-line arrangement are forced to have varying magnification with a change in the object distance.
Prototype D had a two-line staggered arrangement of telecentric imaging units and quartic WFC. The telecentricity in the object space makes no errors in image combining, even at a large defocused position.
The above experimental results lead us to the following conclusions. We confirmed the suppression effect of axial color aberration by WFC. Although the DOF's extension effect is larger in the cubic WFC than in the quartic WFC, the quartic WFC produces no artifact in the deblurred image, making it suitable for such human-viewing usages as a document copier. A one-line arrangement may be applied when the expected object range is small; a two-line staggered arrangement is suitable for a larger DOF.
Consequently, we developed a color image scanner with 600-dpi resolution and a large field of view of 310 mm for a flatbed scanner in a copier without any axial color aberration or artifact in the image with quartic WFC.

Appendix A: Deblurring Process
A blurred image through WFC optics is restored by a deblurring process using Fourier transform. Blurred image B is expressed by the expression: E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 0 4 ; 1 1 6 ; 5 8 5 where O is the original image, P is the PSF, and the asterisk (*) denotes convolution. The original image can be restored by the following expression by adding a Wiener filter to suppress the noise effect: where F denotes the Fourier transform and γ is the signal-to-noise ratio parameter in the Wiener filter. Since the Fourier transform process requires computational resources, the process we performed was replaced by convolution: E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 0 6 ; 1 1 6 ; 4 3 6 The term of the inverse Fourier transform is constant for blurred image B, making it possible to calculate it in advance and store it in memory. The smaller the matrix size of the kernel term is, the lighter the calculation power is. The matrix could be trimmed to become as small as possible without degrading the image quality.