Multiple View Geometry

Multiple View Geometry is a fundamental area in computer vision that studies the relationships between multiple 2D images of a 3D scene. This field provides the mathematical foundations for understanding how cameras capture scenes and how we can reconstruct 3D information from multiple 2D observations. It plays a crucial role in various applications such as 3D reconstruction, robot navigation, augmented reality, and object recognition.

Introduction

At its core, Multiple View Geometry deals with the geometric constraints that exist between different views of the same scene. When a 3D scene is captured from multiple viewpoints, there are inherent geometric relationships between the resulting images. These relationships are governed by the projective geometry that underlies the imaging process.

The essence of Multiple View Geometry lies in understanding and exploiting these geometric constraints to solve various problems in computer vision, such as camera calibration, 3D reconstruction, and motion estimation.

Camera Models

The basis of Multiple View Geometry is the camera model, which describes how 3D points in the world are projected onto the 2D image plane. The most commonly used models are:

• Pinhole camera model: The simplest model that assumes an ideal camera with no lens distortions. This model forms the foundation for most formulations in Multiple View Geometry.
• Extended camera models: More realistic models that account for lens distortions, finite apertures, and other physical properties of real cameras.

In the pinhole camera model, a 3D point X in world coordinates is projected to a 2D point x in image coordinates according to the projection equation:

where K is the camera calibration matrix, R is the rotation matrix, t is the translation vector, and [I | 0] is the matrix that selects the first three coordinates for projection.

Epipolar Geometry

Epipolar geometry is the intrinsic projective geometry between two views. It represents the geometric relationship between two images and does not depend on the scene structure. The key concepts of epipolar geometry include:

• Epipole: The point in an image where all projection rays from the other camera intersect.
• Epipolar line: The projection of the ray from the camera center through a point in the first image onto the second image.
• Epipolar plane: The plane formed by the two camera centers and a 3D point.

Fundamental Matrix

The fundamental matrix F encapsulates the epipolar geometry between two views. For a pair of corresponding points x in the first image and x' in the second image, the fundamental matrix satisfies:

The fundamental matrix is a 33 matrix of rank 2. It can be estimated from point correspondences between two images and is central to many applications such as stereo matching and 3D reconstruction.

The fundamental matrix relates to the camera matrices and the essential matrix E as follows:

where K and K' are the calibration matrices of the two cameras.

Essential Matrix

The essential matrix E relates corresponding points in two images in normalized image coordinates (i.e., coordinates after removing the effects of camera calibration). For normalized points x and x', we have:

The essential matrix can be factorized to obtain the relative rotation and translation between the two cameras:

where [t] is the skew-symmetric matrix of the translation vector t, and R is the rotation matrix. This decomposition is fundamental for structure-from-motion applications.

Homography

While the fundamental matrix and essential matrix describe the general relationship between two views of a 3D scene, a homography H represents a special case where the scene points lie on a plane or the cameras undergo pure rotation.

A homography is a 33 matrix that maps points x in one image to corresponding points x' in another image:

Homographies are useful for many applications such as image stitching, planar object recognition, and augmented reality.

Triangulation

Triangulation is the process of determining the 3D position of a point given its projections in two or more images and the camera parameters. Given two camera matrices P and P' and corresponding image points x and x', the 3D point X can be found by solving:

where P^+ and P'^+ denote the pseudo-inverse of the camera matrices. Due to noise in image measurements, this problem is typically solved in a least squares sense.

Structure from Motion

Structure from Motion (SfM) is a technique that uses multiple view geometry to recover both 3D structure and camera motion from a sequence of images. The general SfM pipeline consists of the following steps:

1. Feature detection and matching across images
2. Initial reconstruction using a minimal set of views
3. Incremental addition of more views and points
4. Bundling adjustment to optimize structure and motion parameters

Multiple view geometry provides the mathematical foundation for each of these steps, from filtering correct matches using the fundamental matrix to triangulating 3D points and optimizing the reconstruction via bundle adjustment.

Applications

Multiple view geometry has numerous applications in computer vision and related fields:

• 3D Reconstruction: Creating 3D models of scenes from multiple 2D images
• Robot Navigation: Enabling robots to understand their environment and navigate autonomously
• Augmented Reality: Placing virtual objects in real-world imagery with correct pose
• Object Recognition: Identifying objects across different viewpoints
• Image Stitching: Creating panoramic images from multiple photos
• Motion Tracking: Following objects across multiple video frames

Conclusion

Multiple View Geometry provides the fundamental mathematical tools for understanding and manipulating the relationship between multiple 2D images of a 3D scene. By exploiting the geometric constraints inherent to multi-view imaging, we can recover the structure of the 3D world and the motion of cameras, enabling a wide range of applications in computer vision and robotics.

As imaging technology advances and computational power increases, the principles of Multiple View Geometry continue to find new applications and form the basis for more sophisticated vision algorithms, pushing the boundaries of what machines can perceive and understand about the three-dimensional world we inhabit.