↑ Return to Tutorials

Feature-based Vs direct alignment

Both feature-based and area-based alignment have their own pros and cons. In principle, feature-based matching is invariant to the strength of the geometric deformation, while it may fail when the image content is weakly-textured, periodic etc. On the other hand, area-based (direct) alignment methods are able to align such images, while they need a good initialisation when the deformation is strong, in order to avoid being trapped in local extrema. Therefore, combining the methods might be a better choice in some cases. In what follows, we verify the pros and cons of each method alone, and show the potential of their combination.

Where the feature matching fails

Let us consider the two following pairs of images. A homography relates the Chess images, while a Euclidean transform relates the Triangle images. (Strictly speaking, the real transformation for Chess pair should be lens-undistortion+homography).

Chess Image (to be warped) Chess Template

 

   
Triangle Image (to be warped) Triangle Template

While the extraction of features in the Triangle pair makes no sense, we are able to extract SURF features [4] from the Chess pair. Even so, the periodic content of images may make the matching unreliable. Let us load the images, extract features and match the descriptors.

>> img = imread('chessImage.png'); 
>> tmp = imread('chessTemplate.png');
>> [d1, l1] = iat_surf(img,128);
>> [d2, l2] = iat_surf(tmp,128);
>> [map, match, imgInd, tmpInd] = iat_match_features(d1,d2,.95);
>> iat_plot_correspondences(img,tmp,l1(imgInd,1:2)',l2(tmpInd,1:2)');

 

Correspondences

 

As shown, only arbitrary correspondences are obtained owing to the periodic content. We intentionally use a high angle-ratio (in the function iat_match_features) to get more correspondences. Apparently, even a RANSAC-based scheme [5] would not be able to filter out the mismatches. Note that we here ignore the fact that a chessboard appears, as opposed to the calibration scenario.

Area-based alignment can easily deal with such images. For example, the alignment of Chess pair can be done as follows:

>> par.transform = 'homography'; par.levels = 1; par.iterations = 30;
>> ECCWarp = iat_ecc(img, tmp, par);

Then, we can warp the image img and compare it with the template tmp :

>> [M,N] = size(tmp);
>> [wimECC, supECC] = iat_inverse_warping(img, ECCWarp, par.transform, 1:N, 1:M);
>> [~, errorECC] = iat_error2gray(tmp,wimECC,supECC);
>> figure; imshow(tmp); figure; imshow(uint8(wimECC)); figure;imshow(errorECC);

 

Template Warped image Alignment error
(lower error=more white)

As shown above, the warped image is very close to the template. Gray variations in the error image (mostly in black squares) are due to the intensity variation in images. It is worth pointing out here that, unlike ECC, Lucas-Kanade may fail here because the image, whose gradient information is used towards the alignment, is noisy and has not very similar appearance compared to the template. However, if you change the role of image and template, Lucas-Kanade should be able to register the images.

It makes sense here to show the mosaic we obtain with the final warp:

>> mosaicECC = iat_mosaic(tmp,img,ECCWarp);
>> figure; imshow(uint8(mosaicECC));

 

Mosaic

Triangle image pair is a simpler case and both Lucas-Kanade [1] and ECC [2] are able to register them. For example, we use the Lucas-Kanade algorithm here:

>> img = imread('triangleImage.png'); 
>> tmp = imread('triangleTemplate.png');
>> par.transform = 'euclidean'; par.iterations = 15;
>> LKWarp = iat_LucasKanade(img, tmp, par);
>> [M,N] = size(tmp);
>> [wimLK, supLK] = iat_inverse_warping(img, LKWarp, par.transform, 1:N, 1:M);
>> [~, errorLK] = iat_error2gray(tmp,wimLK,supLK);
>> figure; imshow(tmp); figure; imshow(uint8(wimLK)); figure;imshow(errorLK);

 

Template Warped image Alignment error
(lower error=more white)

Where the direct method fails

Recall the pair of the text images:

Template Image (to be warped)

The rotation angle between these images is about 10 degrees. Direct methods cannot resolve this case because they are trapped in local extrema without being properly initialized. It is important to note here that such a rotation with another image content can be resolved with a direct method, i.e., it is the image content which makes this case more difficult. Running ECC, we get the following warped image, even if we choose a multi-resolution alignment:

>> img = imread('textImage.png'); 
>> tmp = imread('textTemplate.png');
>> par.transform = 'affine'; par.iterations = 25; par.levels = 3;
>> ECCWarp = iat_ecc(img, tmp, par);
>> [M,N] = size(tmp);
>> [wimECC, supECC] = iat_inverse_warping(img, ECCWarp, par.transform, 1:N, 1:M);
>> figure; imshow(uint8(wimECC));
Warped text Image

Apparently, the above warped image cannot be registered with the template one. As with any non-linear optimization algorithm, global optimum can be reached after a proper initialization. Let us consider here an initial transform, with 8 degrees rotation and some reasonable translation.

 >> par.levels = 1; par.iterations = 25; theta = -pi/180*8;
>> par.initwarp = [cos(theta) -sin(theta) -25; sin(theta) cos(theta) 20];

If we call again the algorithm as above (we do not repeat all the steps here), we get the following warped image, thanks to par.initwarp. Notice that a single-level alignment is sufficient here.

>> ECCInitWarp = iat_ecc(img, tmp, par);
>> [wimECCInit,supECCInit]=iat_inverse_warping(img,ECCInitWarp,par.transform,1:N,1:M);
>> figure;imshow(uint8(wimECCInit));

 

Warped text Image after ECC’s initialization

In order to show the contribution of the initialization, we plot the error image after the manual initialization and after further running the alignment.

>> [wimInit, supInit]=iat_inverse_warping(img, par.initwarp, par.transform, 1:N, 1:M);
>> [~, errorInit] = iat_error2gray(tmp,wimInit,supInit);
>> [~, errorECCInit] = iat_error2gray(tmp,wimECCInit,supECCInit);
>> figure; imshow(errorInit); figure; imshow(errorECCInit);

Click on images to see the real-size.

Error of initialization Error after alignment

In summary, an obvious question arises: how one can obtain a proper initialization? Feature-based alignment seems to be a good choice, as shown in the next section.

Combine the methods

Let us consider the following pair of images that are related by homography (real captures of a wall from different viewpoints, see here).

 
Image (to be warped) Template

Apparently, there is a partial overlap that makes the direct method non applicable. On the other hand, the partial overlap does not guarantee an accurate alignment via feature-matching, since the repeatability of any detector decreases with the change in viewpoint.

Let us first do the feature matching using directly the function iat_surf_correspondences. Note that the function iat_surf_correspondences does not call the MEX implementation of the function iat_matching_descriptors

>> img = imread('wallImage.png'); 
>> tmp = imread('wallTemplate.png');
>> [X1,X2] = iat_surf_correspondences(img,tmp,.9);
>> iat_plot_correspondences(img,tmp,X1',X2');

 

Correspondences from SURF-matching

and let us use RANSAC to filter out outliers and get the optimum transform:

>> X1h = iat_homogeneous_coords(X1); 
>> X2h = iat_homogeneous_coords(X2);
>> [inliers,ransacWarp]=iat_ransac(X2h,X1h,'homography','maxInvalidCount',10,'tol',.1);
>> iat_plot_correspondences(img,tmp,X1(:,inliers),X2(:,inliers);

 

 
Ransac-filtered correspondences

To visually see the quality of the alignment, we plot the mosaic owing to the alignment as well as the grayscale error in the overlapped area.

>> mosaicFeat = iat_mosaic(tmp,img,ransacWarp); 
>> [M,N] = size(tmp);
>> [wimFeat, supFeat] = iat_inverse_warping(img, ransacWarp, transform, 1:N, 1:M); 
>> [~, errorFeat] = iat_error2gray(tmp,wimFeat,supFeat);
>> figure; imshow(errorFeat); figure; imshow(uint8(mosaicFeat));

Click on images for full resolution

 
Error of the overlapped area (left) and mosaic (right) obtained from feature-based alignment (SURF matching)

The location error of the detector leads to minor misalignment as shown by both error and mosaic images (see the images at full resolution).

Let us now consider the area-based alignment. Starting from the identity transformation, the alignment would completely fail owing to the strong deformation and the partial overlap. However, a direct method being initialised by the above result can lead to an accurate alignment. To verify this, we use ECC algorithm initialised by the warp ransacWarp which was obtained by feature matching, and we accordingly plot the error and the mosaic.

>> par.transform = 'homography'; par.iterations = 15; par.initwarp = ransacWarp;
>> [ECCWarp]=iat_ecc(img,tmp,par);
>> [wimECC, supECC] = iat_inverse_warping(img, ECCWarp, transform, 1:N, 1:M); 
>> [~, errorECC] = iat_error2gray(tmp,wimECC,supECC);
>> mosaicECC = iat_mosaic(tmp,img,ECCWarp);
>> figure; imshow(errorECC); figure; imshow(uint8(mosaicECC));

 

Click on images for full resolution

Error (left) and mosaic (right) from combinatorial alignment (SURF-matching+ECC)

It becomes clear that, in non-degenerate cases, this kind of alignment can be more effective than each method alone, as also pointed out in [3].

Demos – Matlab scripts

IAT provides Matlab scripts (demos) that automatically run alignment, display results and compare the methods.

Relevant demos: areaVsFeatureAlignmentDemo

References

[1] S. Baker, I. Matthews, “Lucas-Kanade 20 years on: A unifying framework Part I”, IJCV, vol.56, no. 3, 2004

[2] G.D.Evangelidis, E.Z.Psarakis, “Parametric Image Alignment using Enhanced Correlation Coefficient”, IEEE Trans. on PAMI, vol.30, no.10, 2008

[3] Richard Szeliski, “Computer Vision, Algorithms and Applications”, Springer, 2010

[4] H.Bay, A.Ess, T.Tuytelaars, L.V.Gool, “SURF: Speeded Up Robust Features”, Computer Vision and Image Understanding (CVIU), Vol. 110, No. 3, pp. 346–359, 2008

[5] Martin A. Fischler and Robert C. Bolles (June 1981). “Random Sample Consensus: A Paradigm for Model Fitting with Applications to Image Analysis and Automated Cartography”. Comm. of the ACM 24 (6): 381–395