1 November 2011 Assessment approach for image mosaicing algorithms
Author Affiliations +
Optical Engineering, 50(11), 110501 (2011). doi:10.1117/1.3646746
In this paper, a novel quantitative assessment approach for image mosaicing algorithms is proposed by evaluating the quality of an output mosaic image and comparing with the input images. Five evaluation indices-entropy, clarity, registration error, peak signal-to-noise ratio, and structural similarity-are calculated and combined with the analytic hierarchy process for comprehensive assessment. This approach can overcome the lack of panoramic image and is easy to implement. Experimental results show the objectivity and the validity of the proposed approach.
Zou, Chen, and Zhang: Assessment approach for image mosaicing algorithms



Image mosaicing has impacted a wide variety of application areas.1 It is to combine multiple images with overlapping areas into a wide field or high-resolution image.2 The process is divided into two steps. The first step is image registration whose purpose is to find the coordinate transformation between a pair of images and align them. The second is image blending which aims at smoothing the transition areas and minimizing visual flaws, such as ghost, exposure difference, or misalignments. The performance of the algorithm is, therefore, both related to these two steps.

To evaluate the performance of the image mosaicing algorithm quantitatively is indispensable for algorithm comparisons and improvements. However, this aspect of research is still in an early stage of development due to the difficulties in obtaining the reference panoramas. The general solutions usually require manual interventions. In Ref. 3, it rates the mosaicing algorithm based on the comparison between the output yielded on standard data sets and the ground truth stitched manually. In Ref. 4, the input data sets are collected by virtual camera, a software module simulating the imaging process of an actual device. The shooting parameters are fully customizable so that it is possible to compose the ground truth. In Ref. 5, a reference image is specified as the real world scenery in a test environment. Similarly, the inputs are simulated by applying a group of common imaging distortions on narrow views of the reference image.

In this paper, an assessment approach for image mosaicing algorithms without reference panoramas is proposed. Five evaluation indices are calculated and combined with the analytic hierarchy process (AHP) for comprehensive assessment. It can reduce manual interventions and make the evaluation much more convenient.


Assessment Scheme for Image Mosaicing

A novel assessment scheme for image mosaicing is established as seen in Fig. 1. The most notable advantage of the approach is that it does not need a panoramic image but the original input narrow-angle image instead for comparison. It is deemed that the gathered image for stitching is the most realistic view from that angle in that projection. It is fair enough for partial comparison. The indices extracted in this aspect belonging to full-reference assessment are calculated by comparing the similarity between the original input images and their neighboring images, or their inverse warped partial mosaics. As for the whole mosaic image, it is up to its own performance in perception. The related indices belonging to no-reference assessment are calculated by measuring the output mosaics globally.

Fig. 1

The framework of the proposed approach.



Evaluation Criteria and Comprehensive Assessment


No-Reference Criteria

The no-reference criteria include two indices. The first one is entropy for characterizing the texture and information amount of the output mosaics Imos, denoted as C1. The higher the value of C1, the richer the image information. The second one is clarity and defined as follows.


[TeX:] \documentclass[12pt]{minimal}\begin{document}\begin{eqnarray} &&\hspace*{-5pt}C_2 = {\rm Clarity}_{{\rm Sobel8}} (I_{{\rm mos}}) =\! \sum\limits_{x = 1}^M \!{\sum\limits_{y = 1}^N {\left| {H(x,y)} \right|^2 } },{\rm }H(x,y)\! >\! T_N,\nonumber\\ &&\hspace*{-5pt}H(x,y) = \sqrt {\sum( {I_{{\rm mos}} (x,y)*S_j } )^2}\, , \end{eqnarray}\end{document} C2= Clarity Sobel 8(I mos )=x=1My=1NH(x,y)2,H(x,y)>TN,H(x,y)=(I mos (x,y)*Sj)2,
where Si are Sobel masks in eight directions (0 deg, 45 deg, …, 315 deg),6 TN is the standard deviation of Imos, and M×N is the size of Imos. The higher the value of C2, the less the blur of the mosaic image.


Full-Reference Criteria

The full-reference criteria include registration error, peak signal-to-noise ratio (PSNR) and structural similarity (SSIM), which are defined as C3, C4, and C5.


[TeX:] \documentclass[12pt]{minimal}\begin{document}\begin{eqnarray} C_3 &=& \frac{1}{{N - 1}}\sum\limits_{n = 1}^{N - 1} {\varepsilon _n }\nonumber\\ & =& \frac{1}{{N - 1}}\sum\limits_{n = 1}^{N - 1} {\left( {\frac{1}{P}\sum\limits_{p = 1}^P {\big\| {x_{jp} - H_j^{ - 1} H_i x_{ip} } \big\|} } \right)} \end{eqnarray}\end{document} C3=1N1n=1N1ɛn=1N1n=1N11Pp=1PxjpHj1Hixip


[TeX:] \documentclass[12pt]{minimal}\begin{document}\begin{eqnarray} C_4 = \frac{1}{N}\sum\limits_{n = 1}^N {{\rm PSNR}(\hat r_n,s_n)} = \frac{1}{N}\sum\limits_{n = 1}^N {20\lg \frac{{255}}{{\sqrt {{\rm MSE}(\hat r_n,s_n)} }}},\nonumber\\ \end{eqnarray}\end{document} C4=1Nn=1N PSNR (r̂n,sn)=1Nn=1N20lg255 MSE (r̂n,sn),


[TeX:] \documentclass[12pt]{minimal}\begin{document}\begin{eqnarray} C_5 &=& \frac{1}{N}\sum\limits_{n = 1}^N {{\rm SSIM}(\hat r_n,s_n)}\nonumber\\ \!&=&\! \frac{1}{N}\!\sum\limits_{n = 1}^N \!\frac{1}{K}\! {\sum\limits_{k = 1}^K {[l_n (x_k,y_k)]^\alpha [c_n (x_k,y_k)]^\beta [s_n (x_k,y_k)]^\gamma } }\!, \end{eqnarray}\end{document} C5=1Nn=1N SSIM (r̂n,sn)=1Nn=1N1Kk=1K[ln(xk,yk)]α[cn(xk,yk)]β[sn(xk,yk)]γ,
where P is the number of feature pairs in neighboring images, and the transform relations are shown in Fig. 2, N is the number of input images or interested regions, [TeX:] $\hat{r}_n$ r̂n is the inverse warped partial mosaic image, and sn is the input image. Refer to Ref. 8 for more parameter details of SSIM. The lower the value of C3, the smaller the registration error. The higher the value of C4, the more continuous in blending intensity. The higher the value of C5, the less structure difference between the input sequence and the synthetic mosaics.

Fig. 2

The computation process of registration error.



Comprehensive Assessment Based on AHP

For different applications, the demands for image mosaicing are usually different. The above criteria play roles of varying importance. To determine the weight coefficients, the problem is modeled as a hierarchy, shown in Fig. 3, and pairwise comparisons of the AHP are performed. The comprehensive assessment result is obtained by three steps.

Fig. 3

The hierarchy tree of evaluation indices for visual quality.


The first step is constructing a set of pairwise comparison matrices U using a scale of values ranging from 1 (equal importance) to 9 (absolute importance).


[TeX:] \documentclass[12pt]{minimal}\begin{document}\begin{eqnarray} U &=& \left[ {\begin{array}{c@{\quad}c@{\quad}c@{\quad}c} {u_{11} } & {u_{12} } & {\ldots} & {u_{1n} } \\ {u_{21} } & {u_{22} } & {} & {u_{2n} } \\ {\ldots} & {} & {} & {} \\ {u_{n1} } & {u_{n2} } & {} & {u_{nn} } \\ \end{array}} \right], \end{eqnarray}\end{document} U=u11u12...u1nu21u22u2n...un1un2unn,
where uij is the relative importance value of the i’th factor compared to the j’th, therefore, uij = 1/uji, uii = 1. The elements of U can be determined by statistical data, intensity of requirement on issues, or experts’ knowledge. Taking the application in photograph editing for instance, they can be defined as Table 1. For the criterion layer, the local full-reference criteria are deemed to be slightly important over the global no-reference criteria, since photograph editing requires high accuracy and the evaluation usually pays close attention to local details of the mosaics. Other matrices can be also determined in this way, seen in Table 1.

Table 1

An example on comparison matrices and weights in the hierarchy.

Criterion layerIndex layer B1Index layer B2

The second step is computing a vector of priorities in each layer. It equals the normalized eigenvector [w1, w2, …, wn]T corresponding to the maximum eigenvalue of U, where wi is calculated by


[TeX:] \documentclass[12pt]{minimal}\begin{document}\begin{eqnarray} w_i = {{\sqrt[n]{{\prod\limits_{j = 1}^n {u_{ij} } }}} / {\sum\limits_{i = 1}^n {\sqrt[n]{{\prod\limits_{j = 1}^n {u_{ij} } }}} }}\, . \end{eqnarray}\end{document} wi=j=1nuijn/i=1nj=1nuijn.
In addition, the judgments must pass through a consistency check.9 The computation results of the above case are also listed in Table 1.

The third step is synthesizing priority by aggregating the relative weights up the hierarchy. The comprehensive assessment result is expressed as


[TeX:] \documentclass[12pt]{minimal}\begin{document}\begin{eqnarray} A = \sum\limits_{i = 1}^m {\left( {w_i \sum\limits_{j = 1}^n {w_j^i C_i } } \right)}, \end{eqnarray}\end{document} A=i=1mwij=1nwjiCi,
where wi is weight coefficient of evaluation factor in criterion layer, and [TeX:] $w_j^i $ wji is weight coefficient of evaluation index of the i’th criterion factor in the j'th index layer, Ci is the normalized index data. In the example, the computed overall weight vector becomes W = [0.083, 0.250, 0.160, 0.140, 0.367].


Experiments and Conclusions

Two algorithms for photograph editing are compared here, denoted by #1 and #2. They are both based on a projective model, but extracting different features for registration. The blending methods are also different, #1 using center-weighting but #2 using optimal seam stitching. In order to test the validity of the proposed method, a series of mosaics stitched by these algorithms were evaluated by 15 volunteers. They observed each pair of mosaics under the same viewing conditions and voted for the algorithm. The algorithm with more votes was deemed as the better one. For all the test scenes, #2 was superior to #1 in our subjective experiments. Due to space limitations, we present five typical scenes with different complexities and textures for illustration. We also compared with another evaluation method on the basis of rms intensity error between the final mosaics and the reference panoramas stitched by hand. The mosaicing results and the reference panoramas are shown in Fig. 4. The computed index data in the proposed scheme are shown in Table 2. The relative parameters are set as recommendations in references. Their assessment results are shown in Table 3. They are all consistent with subjective judgments. Although some local moving objects might be lost by #2 as seen in Fig. 4, which results in lower full-reference evaluation, its global impression significantly defeats the local details by avoiding the blending ghost. However, some evaluation results of the compared method are contrary to subjective judgments as seen in Table 3, marked by “×,” mainly caused by moving objects which increase the rms intensity error if taking a large portion in the input image and lost in the final mosaics.

Fig. 4

Assessment experiments: the first row is the mosaicing results of #1; the second row is the mosaicing results of #2; the third row is manual stitched panoramas.


Table 2

Values of the evaluation indices.

sc. 1#16.830822187.40.327824.00250.9991
sc. 2#17.247516138.50.209827.84490.9998
sc. 3#17.261812553.50.381626.95030.9998
sc. 4#16.795616256.30.202526.42370.9997
sc. 5#17.070915771.40.267725.16560.9996

Table 3

The comparison of assessment results.

Proposed methodCompared method
No-ref.Full-ref.Overallrms error
sc. 1#10.50610.62690.58670.2556×
sc. 2#10.55010.61320.59220.1258
sc. 3#10.53070.60680.58140.1031
sc. 4#10.53370.63550.60150.1151
sc. 5#10.50600.64220.59680.1041×

The above experiments show the objectivity and the stability of the proposed method. Owing to multicriteria and AHP weights selection, the assessment is robust even if there are variances and disturbances in different scenes. Indices extension and parameter analysis will be the focus in future research.


This work is supported by Chinese National Science Fund for Distinguished Young Scholars, Grant No. 60925011.


1.  L. H. Zou, J. Chen, J. Zhang, and J. Lu, “Real-time wide-field monitoring system based on image mosaicing technique and its application,” Proceedings of the 18th IFAC World Congress, Milano, Italy, pp. 14928–14933 (2011). Google Scholar

2.  H. Lin, J. Si, and G. P. Abousleman, “Dynamic point selection in image mosaicking,” Opt. Eng. 45, 030501 (2006). 10.1117/1.2180794 Google Scholar

3.  M. Brown, “Multi-image matching using invariant features,” Ph.D. thesis, University of British Columbia, Vancouver, BC, pp. 74–87 (2005). Google Scholar

4.  P. Azzari, L. D. Stefano, and S. Mattoccia, “An evaluation methodology for image mosaicing algorithms,” Proceedings ACIVS 2008, Juan-les-Pins, France, pp. 89–100 (2008). Google Scholar

5.  J. Boutellier, O. Silvén, M. Tico, and L. Korhonen, “Objective evaluation of image mosaics,” Commun. Comp. Info. Sci. 21, 107–117 (2009). 10.1007/978-3-540-89682-1 Google Scholar

6.  T. Jiang, Y. Tan, and Q. Liu, “Research of a clarity-evaluation function of image based on Sobel,” Comp. Dig. Eng. 36(8), 129–131 (2008). Google Scholar

7.  W. Xu and J. Mulligan, “Performance evaluation of color correction approaches for automatic multi-view image and video stitching,” IEEE Conference on CVPR, San Francisco, USA, pp. 263–270 (2010). Google Scholar

8.  Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image quality assessment: from error visibility to structural similarity,” IEEE Trans. Image Process. 13(4), 1–14 (2004). 10.1109/TIP.2003.819432 Google Scholar

9.  E. Kwon and W. I. Ko, “Evaluation method of nuclear nonproliferation credibility,” Ann. Nucl. Energy 36, 910–916 (2009). 10.1016/j.anucene.2009.03.012 Google Scholar

Li-hui Zou, Jie Chen, Juan Zhang, "Assessment approach for image mosaicing algorithms," Optical Engineering 50(11), 110501 (1 November 2011). https://doi.org/10.1117/1.3646746


Error image aware content restoration
Proceedings of SPIE (December 09 2015)
A novel blind image restoration algorithm of UWB system
Proceedings of SPIE (October 30 2009)
Evaluation of image appeal in consumer photography
Proceedings of SPIE (June 02 2000)

Back to Top