↑ Return to Tutorials

Non-rigid alignment with SIFT-flow

SIFTflow was primarily designed for matching similar images of different scenes [1]. However, it can be easily used when images of the same scene, that undergo non-rigid deformation, must be registered. The algorithm estimates the optimum flows that minimize an energy criterion, while the energy minimisation is done by a multi-layer belief propagation scheme. In what follows, we explain the steps in order to align the images using SIFTflow algorithm. This tutorial was built based on the SIFTflow demo, kindly provided by Ce Liu.

Let us load the following pair of Mars images that undergo a non-rigid geometric deformation.

Image 1 Image 2 (to be warped)

 

>> im1 = imread('Mars1HalfSize.png');
>> im2 = imread('Mars2HalfSize.png');

While global rigid alignment is not sufficient here, dense local alignment (per pixel) is required. SIFTflow works on SIFT-images whose pixels are SIFT descriptors [2]. A function iat_dense_sift geratates such images as follows:

>> patchszie = 8; % half of the window size for computing SIFT
>> gridpsacing = 1; % sampling step
>> SiftIm1=iat_dense_sift(im2double(im1),patchsize,gridspacing);
>> SiftIm2=iat_dense_sift(im2double(im2),patchsize,gridspacing); 

Note that SIFT-images have patchsize rows and columns less compared to original images owing to the window needed to produce SIFT descriptors (in other words, a border of size patchsize/2 is removed).

To visualise such images, one could use the function iat_sift2rgb which projects each SIFT-pixel to the three most principal components of a pre-trained PCA model and a linear model maps in turn the three coefficients to RGB space.

>> figure;imshow(iat_sift2rgb(SiftIm1));title('SIFT image 1');
>> figure;imshow(iat_sift2rgb(SiftIm2));title('SIFT image 2');

SIFTflow algorithm needs some parameterization that is explained in the help file of the function iat_SIFTflow. Once the parameters are set, the algorithm can be invoked and compute the flows from SiftIm1 to SiftIm2. Default values are used for non-defined parameters.

>> SIFTflowpara.alpha=2;
>> SIFTflowpara.d=40;
>> SIFTflowpara.gamma=0.005;
>> SIFTflowpara.nlevels=4;
>> SIFTflowpara.wsize=5;
>> SIFTflowpara.topwsize=20;
>> SIFTflowpara.nIterations=60;
>> [vx, vy, energylist] = iat_SIFTflow(Sift1, Sift2, SIFTflowpara);

The function iat_SIFTflow returns the flows in X and Y dimensions as well as a structure energylist that contains the minimum energy per iteration per level.

Unlike rigid alignment, one should do the warping based on flows in a pixel basis using the function iat_pixel_warping. However, since vx, vy have the size of SIFT-images, im2 should be first resized by ignoring a border of size patchsize/2.

>> newIm2 = im2(patchsize/2+1:end-patchsize/2, pathcsize/2+1:end-patchsize);
>> [warpedIm2, support] = iat_pixel_warping(newIm2, vx, vy);

Next, cropped im1 can be compared with the warped image in order to visualise the alignment error:

>> newIm1 = im1(patchsize/2+1:end-patchsize/2, pathcsize/2+1:end-patchsize);
>> [~, grayerror] = iat_error2gray(newIm1,warpedIm2,support);
>> figure;imshow(newIM1);title('Image1');
>> figure;imshow(uint8(warpedIm2)); title('Warped Image2');
>> figure;imshow(grayerror); title('Registration error'); 

 

     
Image1 Warped Image2 Registration error
(lower error=more white)

The misalignment can be also visualised in color using the function iat_error2rgb. As shown below, this function makes misalignments appear with hot pink and lawn green colors.

>> figure;imshow(iat_error2rgb(newIm1,warpedIm2));title('Colorized alignment error');

 

Colorised alignment error

IAT also provides the function iat_flow2rgb to visualise dense pixel displacements or flows:

>> figure;imshow(iat_flow2rgb(vx,vy));title('SIFT flow field');

 

SIFT flow field

Find more about SIFTflow HERE

Demos – Matlab scripts

Relevant IAT demos: SIFTflowDemo

References

[1] C. Liu, J. Yuen, A. Torralba: SIFT Flow: Dense Correspondence across Scenes and its Applications, IEEE Trans. on PAMI, vol. 33, no. 5, 2011

[2] D.G. Lowe, Distinctive image features from scale-invariant keypoints, IJCV, vol. 60, no. 2, 2004