Epipolar geometry and stereo vision
By Kendall Merriman
Project:
My original goal was much more ambitious, but in the end, numerous setbacks effectively made the project scope much smaller then the initial goal. The following tasks were accomplished:
- Extraction of the Fundamental matrix from matched points
- Search along epipolar lines for match points
- Determine relative depth of items in the scene
The fundamental matrix is generated by using the basic 8-point algorithm, which requires only 8 pairs of matched points. However, I found that this method, though correct, produced matrices that were not entirely useful. As such, I decided to try and get the matrix that minimized the error over as many matches as the user provides, allowing for more accurate epipolar lines.
For more information on the fundamental matrix and epipolar lines, see the references.
This project uses DirectShow to connect to the cameras and grab frames, which are processed to produce the results shown. Once the images are pulled, they are displayed to the user, who, after selecting enough matches between the images, can generate the matrix and start exploring the scene, mainly by cricking on things. The following options are available:
- While any window is active:
- While the "Ball display" is active:
- left and right arrows - grow and shrink the squares, respectively
- While a camera is active:
- Left click - add point (before matrix calculation) or draw epipolar line for a point (after matrix calculation)
- right click - find a match for a point
- middle click - find a match and add the object to the depth map
- E - detect and show edges for the current image (please pause first, behavior is undefined otherwise). Originally inteded for use in auto-feature matcher, but was never fully finished
- C - calculate Fundamental matrix
- left and right - show next/previous calibration point
- 1-9 - show the nth calibration point, where n = the number pressed (can only access the first 9 points this way
The application will store the matrix it calculates automatically, so if the setup has not changed, it will not need to be recalculated. The user will be prompted about this at startup.
System Requirements:
- Windows (XP tested, vista may work)
- Two webcams which can run simultaneously (as a general rule, this means two non-identical webcams. Mine are a Logitech quickcam deluxe for notebooks and a Creative optia AF)
- DirectShow (should be installed by default)
Results:
An example of an epipolar line. The line shown in the right window is for the point in the left window. The point in the right is the match chosen for that point in the left.
An example of selecting objects. The red and green squares indicate the red and green balls, which were selected to add to the depth display. Notice that the red is in front of the green, as they are in the image.
Download here
References
Hartley, R.~I. and Zisserman, A., Multiple View Geometry in Computer Vision, Cambridge University Press, 2004
Wikipedia on Epipolar Geometry
Wikipedia on The Fundamental Matrix