Real-time three-dimensional fingerprint acquisition via a new photometric stereo means

Abstract. A real-time means for three-dimensional (3-D) fingerprint acquisition is presented. The system is configured with only one camera and some white light-emitting diode lamps. The reconstruction is performed based on the principle of photometric stereo. In the algorithm, a two-layer Hanrahan–Krueger model is proposed to represent the finger surface reflectance property instead of the traditional Lambert model. By the proposed lighting direction calibration and the nonuniform lighting correction methods, surface normal at each image point can be accurately estimated by solving a nonlinear optimization problem. Finally, a linear normal transformation is implemented for the enhancement of 3-D models. The experiments are implemented with real finger and palm prints, and the results are also compared with traditional means to show its feasibility and improvement in the reconstruction accuracy.


Introduction
Fingerprinting is a classical topic in biometric and computer vision domains and has gained wide application in our daily life.With any fingerprinting technique, how to capture the high-quality fingerprint image is always the principle concern.Current fingerprint acquisition instruments usually consist of an image sensor and a touch panel.With such a touchable operation manner, the captured fingerprint images are usually degraded by improper finger placement, skin deformation or slippage, smearing of fingers, sensor noise, etc. 1 To overcome these disadvantages, a technique named touch-less fingerprinting is emerging recently. 2By employing a three-dimensional (3-D) scanning procedure, 3-D models of fingerprints can be obtained with a contactless operation.In comparison with traditional two-dimensional (2-D) fingerprint images, plentiful fingerprint information can be retrieved from the 3-D fingerprint models, thus making the subsequent fingerprint recognition more reliable. 3ompared with the traditional 2-D image-based fingerprinting techniques, 3-D fingerprinting is still a new research domain that appears in recent years.The major technical challenge involved in 3-D fingerprinting is how to capture the 3-D model of live fingerprint precisely and efficiently.Various computer vision techniques have been applied for this purpose, such as shape from silhouette, structured light systems (SLS), stereo vision, etc. [4][5][6][7][8][9][10][11] In Ref. 6, a multiple-view system that consists of five cameras and a set of 16 green light-emitting diode (LED) arrays is proposed for the 3-D fingerprint scanning.Multiple cameras are used to capture the fingerprint images under different viewpoints and LED lightings.And the corresponding silhouettes are extracted for the 3-D modeling of the fingerprints via shape from silhouette method.Based on the 3-D fingerprint acquisition system proposed in Ref. 6, to make the 3-D fingerprint images compatible with current 2-D fingerprint systems, an unwrapping algorithm is presented in Ref. 7. The equidistance unwrapping approach is utilized to minimize the distortion while preserving the ground-truth of the fingerprint.By unfolding the 3-D fingerprint in such a way, it resembles the effect of virtually rolling the 3-D finger on a 2-D plane.In Ref. 8, an SLS is developed for the 3-D fingerprint acquisition.While the fringe projector operates at a blue wavelength, a green illumination is used to image the papillary lines.The fringe pattern analysis technique is applied for the 3-D depth recovery from phase information. 9he nonparametric unwrap approach is applied to preserve the distances between surface points.A 3-D fingertip scanning system that consists of one projector and two industrial cameras is presented in Ref. 10. 3-D fingerprint models are achieved via SLS and stereo-vision means, and the 3D models are unwrapped to compare with 2-D counterparts.In Ref. 11, a high-speed SLS is established for the 3-D scanning of fingerprints.The system is configured with a digital light processing projector and a camera.A shifting sine wave pattern is projected onto the finger surface and the images with pattern illuminations are captured synchronously.Via the proposed decoding algorithm, high-density 3-D points of the finger surface can be obtained.However, such 3-D scanning-based approaches usually suffer from the translucent finger skin and thus degrade the 3-D reconstruction accuracy.Moreover, the complicated structure and high cost of the hardware also make these techniques unaffordable to end users and prevent the touchless 3-D fingerprinting technology for wide applications.
In this paper, an efficient 3-D fingerprint acquisition method based on a simple hardware setup is presented.The system only consists of one camera and some LED lights.A shiny ball with known radius is used to calibrate the lighting directions of different LEDs.Considering nonuniform lighting conditions by the LEDs, a correction procedure is introduced to calibrate each LED light.Compared with traditional photometric stereo (PS) methods, which are usually implemented with Lambert reflectance assumption, a two-layer Hanrahan-Krueger (HK) model 12 is introduced to model the finger surface's reflectance property more accurately.An objective function is derived from HK irradiance equation based on PS principle, and the Levenberg-Marquardt method 13 is used to solve the equation for accurate estimation of surface normal.Finally, a linear surface normal transformation is adopted to enhance the reconstructed 3-D fingerprint model.Experiments on real fingerprints and palm prints are experimented to demonstrate its feasibility and 3-D reconstruction accuracy.
The rest of the paper is organized as follows.Reflectance modeling algorithm of the finger surface and how to solve the surface normal from the model are introduced in Sec. 2. Section 3 describes the adopted methods for lighting direction calibration, nonuniform lighting correction, and 3-D fingerprint model enhancement.Section 4 presents the experimental results on real fingerprints and the comparisons with traditional method.Conclusions and future work are offered in Sec. 5.

Reflectance Property Modeling of Finger Surface
PS is an important approach in computer vision that is usually used to estimate the surface normal by observing the target surface under various illuminations.This technique was first introduced by Woodham. 14Various PS-based methods have been proposed in the past decades, and their major concerns are usually focused on the following issues: calibration of the lighting directions, modeling of the surface reflectance property, 3-D reconstruction under nonuniform lighting conditions, etc. [15][16][17][18][19][20] The modeling of surface reflectance property is a crucial issue in the PS algorithms, since it directly determines how the incident lights be modulated by the target surface as well as the final surface normal estimation.The most popular surface reflectance description is the Lambert model, which assumes the target surfaces with ideal diffuse reflection property.Given three or more known illumination conditions, surface normal can be efficiently calculated by solving some linear irradiance equations. 21kin surface like fingerprint is a kind of translucent material that contains certain multiple scattering and specular reflections.It cannot be well represented by a traditional linear reflection model like Lambert, especially in the case where high reconstructing precision is demanded.To model the human skin more precisely, a nonlinear reflection descriptor named Torrance and Sparrow (TS) model is introduced in Ref. 22.As a physical-based model, the TS model assumes the skin reflectance consists of Lambertian and purely surface scattering components.Incorporating an uncalibrated PS method, reflectance parameters of skin surface can be well estimated and the negative effects from generalized bas-relief ambiguity can be reduced. 23In comparison, the HK model considers the skin as a layered surface based on one-dimensional linear transport theory. 12The underlying principle is that the amount of light reflected by a material that exhibits subsurface scattering can be calculated by summing the amount of lights reflected by each layer times the percentage of light that actually reaches that layer.Hence, it is a more reasonable description for translucent surfaces like human skin and fingers.
With reference to the HK model, in this paper, the finger surface is modeled as a two-layer material that consists of the epidermis and the dermis.The layers have different reflectance parameters that determine how the incident lights are reflected as illustrated by Fig. 1.
According to Ref. 12, the irradiance equation of a twolayer HK model can be written as where L r is the irradiance, which is theoretically equal to pixel intensity, L i is the intensity of light source, σ a and σ s are the absorption section and scattering cross-section, respectively, T 12 and T 21 are the Fresnel transmittance terms for the lights entering and leaving the surface, d is the thickness of epidermis layer, ρ is the albedo, θ i is the incidence angle between the normal vector and the light direction, θ r is the outgoing angle between the normal vector and the reflection light direction, ϕ is the angle between the light and the view directions, g is the mean cosine value of the phase function, p is the function of ϕ, and g is as given in Eq. ( 2) and is used to determine in which direction the light is likely to scatter. (2) Define surface normal vector as n ¼ ðn x ; n y ; n z Þ, light direction vector as l ¼ ðl x ; l y ; l z Þ, reflection light direction as r ¼ ðr x ; r y ; r z Þ, and view direction as z ¼ ðz x ; z y ; z z Þ.All these vectors are normalized.Then, cos θ i , cos ϕ, and cos θ r in Eq. ( 1) can be rewritten in the form of inner product between two of these vectors as Suppose all illumination conditions are known, and g, T 12 and T 21 are constant and known over the whole surface.Then there are total seven unknown parameters involved in Eq. ( 1), i.e., n x , n y , n z , ρ, d, σ s , and σ a .Define x for the unknowns as x ¼ ðn j x ; n j y ; n j z ; ρ j ; d j ; σ j s ; σ j a Þ, where j indicates the j'th surface point.Given K images with each one taken under different illumination conditions, the objective function can be formulated as argmin x EðxÞ; where EðxÞ ¼ where L j;k r and I j;k indicate the irradiance value and pixel intensity on the k'th image of the j'th surface point.The Levenberg-Marquardt algorithm 13 can be used to solve x from the above equation.
3 System Calibration

Calibration of Lighting Directions
In Ref. 24, two cameras and a shiny ball with unknown position and radius are used for the lighting direction calibration.Since only one camera is adopted in our system, a shiny ball with known radius r is used instead.With Zhang's camera calibration method, 25 focal length f and the camera center C can be estimated.With reference to the camera coordinate frame, lighting direction of the i'th light source can be represented as l i ¼ ðlx i ; ly i ; lz i Þ as shown in Fig. 2(a).
According to the following two observations in the perspective image of a sphere-(1) the line passing through the camera center and any boundary point of the sphere in the image plane is a tangent line to the sphere and (2) the perpendicular distance from the center of the sphere to a tangent line of the sphere is the radius of the sphere-we can get the following equation: where B i is a boundary point on the image plane with a checked position ði i ; j i ; fÞ, jCSj is the distance between point C and point S, CS is the vector that connects the two points.Note that Eq. ( 5) contains only three unknowns, S ¼ ðs x ; s y ; s z Þ, i.e., three components of the sphere center.Suppose m boundary points B i , i ∈ ð1; : : : ; mÞ can be obtained via edge detection algorithm, 26 then an error function can be defined as By minimizing this error function EOF, we have three equations about S, and the optimal values of the sphere location can be solved.The specular point Pði 1 ; j 1 Þ on the image plane can be detected by finding the brightest image point.Then we can get its corresponding surface point S 1 as well as surface normal vector N as illustrated by Fig. 2(b).With N and R, the lighting direction l can be calculated as

Correction for Nonuniform Lighting Conditions
Notice that the distribution of the illumination (DOI) for each LED lamp is usually different and nonuniform.Figure 3(a) shows a homogeneous white paper surface under the illumination of one LED.
Suppose l k is the k'th LED light whose DOI needs to be calibrated; a white planar board with Lambertian surface is used for its correction.The procedure can be implemented by the following steps: a.Take an image of the planar board under the illumination of l k , and define it as I k ði; jÞ, i ∈ image Height, j ∈ image Width; b.Search the biggest values in I k , i.e., I k max ¼ max i;j fI k g; c.Substitute I k max into Eq.( 8) to calculate p k ði; jÞ for all pixels.
With the above correction, image intensity to each light can be generally corrected to a uniform distribution.Figure 4 gives a comparison between original and corrected fingerprint images.

Enhancement of Surface Normal
The corrected images will be used as input to calculate surface normal at each surface point via the algorithms described in Sec. 2. As a postprocedure, a linear surface normal transformation is adopted consequently.In the algorithm, the average of normal vectors n over a local patch w is calculated as a local reference.And the difference between original surface normal and this reference vector can be amplified as This procedure aims to amplify the angle between two neighboring normal vectors and thus improves the visual effect and contrast of the reconstructed 3-D fingerprint models.

Experiments and Discussion
The experimental setup consists of one camera (Prosilica GC650, 90FPS) with a resolution of 659 × 493 pixels and seven LED lamps (white color, 0.2 W, emitting angle 120 deg) as shown in Fig. 5   Real fingerprints and palm prints are used for the experiments, and the results are compared with traditional PS methods where the Lambert reflectance model is adopted.The experiments are implemented with the following steps: a. Calibration of lighting directions as described in Sec.3; b.Take seven pictures of the object; then for each surface point, rank seven pixel intensities in decent order.The highest and two lowest values are removed, so as to avoid the saturate and shadow points that do not obey the HK model; c.Use the remaining four images as input of the algorithm described in Sec.2; d.Apply normal transformation to enhance the reconstructed 3-D model.

Comparison with Traditional PS Method
In our experiment, the parameters of g, T 12 and T 21 in Eq. ( 1) are set to {0.8, 1.25, 0.8}.The unknown parameters of d j , σ j s and σ j a in Eq. ( 1) are initialized to f0.085 mm; 50 mm −1 ; 3.8 mm −1 g for the fingerprint experiments and f0.12 mm; 30 mm −1 ; 4.5 mm −1 g for the palm print experiment.To enhance the recovered 3-D model, w ¼ 5, k ¼ 1.5 is set to Eq. ( 9).
There are still technical challenges for current instruments to get the 3-D ground truth of elastic surface like skin.That means it is difficult to evaluate the precision of reconstructed 3-D fingerprint models directly.As an alternative approach, a homogeneous white paper surface is used for experiment as shown by Fig. 6.Surface normal at each image pixel is calculated, and a standard deviation of only 1.65 deg is obtained.It shows high reconstruction accuracy can be guaranteed by the proposed calibration and modeling methods, except for some distinct errors that appeared in the marginal image regions affected by inhomogeneous lighting conditions.Similar accuracy evaluations are also performed on some synthetic data with known depth map as shown by Figs. 7 and 8.The data can be downloaded from Ref.    experiment is also conducted on the palm print with same parameter setting as shown by Fig. 10.From the results we can see that skin features like the finger and palm ridges can be clearly retrieved in 3-D space.To solve the nonlinear model for each image pixel, our algorithm takes more time than the traditional linear Lambert model.The whole computation time from images input to final 3-D model is given in Table 1.The efficiency can be further improved by more efficient developing tools like Visual C++ and parallel processing devices like graphics processing units in the future.

Enhancement of 3-D Models Under Different Parameters
The parameters of w and k in Eq. ( 9) play an important role in the rendering of 3-D fingerprint model.In this experiment, different w and k are tested as shown in Fig. 11.From the results we can see that with the increase of enhancing parameters, the fingerprint ridges can be clearly emphasized.However, the ridges are also usually broken subject to the overenhancement as shown in Fig. 11(f).

Reconstruction Results with Different Image Numbers
In this experiment, three, five, and seven images are used for the fingerprint 3-D reconstruction, and the results by the proposed method and traditional Lambert method are also   compared respectively.The captured seven fingerprint images under different LED illuminations are as shown in Fig. 12.To make the comparison unbiased, all the 3-D models are presented without enhancement processing.Figure 13 shows the 3-D reconstruction result with seven images.Figures 13(a) and 13(b) show the results by our method and traditional Lambert model, respectively.Figures 13(c) and 13(d) give close-up views.In comparison, without the enhancement process, Fig. 13(c) demonstrates more finger ridge details with sharp contrast than Fig. 13(d).It means higher-quality 3-D model can be achieved with the proposed method.Experimental results with five and three images are given in Figs. 14 and 15, respectively.From these results, we can see that once the number of images declined to three, as illustrated in Fig. 15, there is no big difference between our method and the traditional method.It is because that there are seven unknowns in Eq. ( 1) and it needs more images to achieve a stable solution.

Conclusions and Future Work
This article presents a novel method for 3-D fingerprint acquisition based on the principle of PS.Compared with previous 3-D scanning-based methods, the proposed system is rather simple and only contains a camera and some LED lamps.To calibrate the lighting direction, a shiny sphere with known radius is used.To model the reflectance property of finger surface, a two-layer HK reflection model is proposed instead of traditional Lambert reflectance model.Considering the nonuniform lighting property of LEDs, a simple correction procedure is also introduced.Finally, the reconstructed 3-D model is enhanced via a linear transformation process.To verify feasibility of the proposed algorithm and system, the experiments are implemented with real finger and palm prints.The results by traditional Lambert model are also provided for comparison to demonstrate the improvements in 3-D reconstruction accuracy.Future work can address how to improve the correction method for nonuniform lighting conditions, especially to the marginal image regions.Moreover, how to apply the obtained 3-D fingerprint models to the recognition phase and to establish an extensive 3-D fingerprint database will be an urgent task in the future.

Fig. 1
Fig.1The finger skin is represented by a two-layer Hanrahan-Krueger model.The incident light L i is reflected twice by the epidermis and dermis layers; the reflected lights L r are captured by the camera.

Figure 3 (
Fig. 2 (a) Illustration of lighting direction calibration.(b) Relation of normal vector N, reflection direction R, and the incident light direction l on the specular point S i .

Fig. 3
Fig. 3 (a) A white paper under the illumination of one light-emitting diode (LED) light; the red circle is the brightest spot, which indicates the illumination distribution not uniform.(b) The diagram of values of p calculated from (a).

Fig. 4
Fig. 4 (a) Original fingerprint image.(b) Intensity of the corrected image looks more homogenous.

Fig. 6
Fig. 6 Accuracy evaluation of three-dimensional (3-D) reconstruction results.(a) A flat paper surface is used for the experiment.(b) The reconstructed planar surface.(c) 3-D plane under various viewpoints.(d) Distribution of the surface normal deviations.

Fig. 7
Fig. 7 Accuracy evaluation with synthetic data: saddle-shape.(a) Seven saddle images with virtual illuminations.(b) to (d) The reconstructed 3-D model under various viewpoints.(e) Distribution of the reconstruction error in pixel unit.
(a).Focal length of the lens (Navitar NMV-25M23) is 25 mm, and the operating distance (from the finger place to the front of lens) is ∼90 mm.The LED lamps are evenly mounted on a lampshade with radius of 75 mm.The lampshade has a distance ∼65 mm to the finger place.The LED positions are finely adjusted to point to the finger place region.An external I/O board is developed to synchronize the camera and LEDs to control the whole capturing time within 0.1 s.The algorithm is implemented under MATLAB2010a on a PC with 2.7 GHz CPU and 2 G RAM.

Fig. 8
Fig. 8 Accuracy evaluation with synthetic data: cone.(a) Seven cone images with virtual illuminations.(b) to (d) The reconstructed 3-D model under various viewpoints.(e) Distribution of the reconstruction error in pixel unit.
27. Seven synthetic images with virtual illuminations are generated as shown by Figs.7(a) and 8(a).The reconstructed 3-D models under various viewpoints are displayed by Figs.7(b) to 7(d) and 8(b) to 8(d).The reconstruction errors (in the unit of pixel) with respect to the ground truth models are presented by Figs.7(e) and 8(e).It shows an average reconstruction deviation far less than one pixel and demonstrates the accuracy of the proposed algorithm.In the experiment with fingerprints, the original four images with different lighting conditions are as shown in Fig.9(a).The reconstructed 3-D model by traditional PS method, which adopts Lambert reflectance model, is as shown in Fig. 9(b).The reconstructed 3-D model by the proposed method is given by Fig. 9(c).

Figure 9 (
d) shows the obtained albedo map by the proposed method.Another
Figure 11(a) shows the original fingerprint image.The reconstructed 3-D fingerprint without surface normal enhancement is as shown in Fig. 11(b).Figures 11(c) to 11(e) show the 3-D models with surface normal enhancing parameters of w ¼ 5, k ¼ 1.5, w ¼ 7, k ¼ 2.0 and w ¼ 11, k ¼ 4.0, respectively.

Fig. 11 3
Fig. 11 3-D fingerprint models under different surface enhancing parameters w and k.(a) Original fingerprint image under one LED illumination.(b) 3-D fingerprint model without surface normal transformation.(c) Result with parameters of w ¼ 5, k ¼ 1.5.(d) Result with parameters of w ¼ 7, k ¼ 2.0.(e) Result with parameters of w ¼ 11, k ¼ 4.0.(f) Cropped area to show overenhancement.

Fig. 12
Fig. 12 Seven fingerprint images captured under different LED illuminations.

Fig. 13 3
Fig. 13 3-D reconstruction results of finger print with seven images.(a) Result by the proposed method.(b) Result by the traditional Lambert model.(c) and (d) The cropped images for close observation.

Fig. 14 3
Fig. 14 3-D reconstruction results of finger print with five images.(a) Result by the proposed method.(b) Result by the traditional Lambert model.(c) and (d) The cropped images for close observation.

Fig. 15 3
Fig. 15 3-D reconstruction results of finger print with three images.(a) Result by the proposed method.(b) Result by the traditional Lambert model.(c) and (d) The cropped images for close observation.

Table 1
Calculation time by the nonlinear model and linear Lambert model (in seconds).