In this project, we achieve augmented reality by drawing cubes on top of a box.
We start by producing a video of a box that has been marked with twenty-five "keypoints". We move the camera during the video to capture the box from different camera positions and angles.
We manually label each keypoint in the first frame of the video.
We also assign three-dimensional "world coordinates" to each keypoint. We can do this easily because the keypoints are evenly-spaced on the box.
We need to know the two-dimensional "image coordinates" of every keypoint in every frame. We already know the image coordinates of keypoints in the first frame.
For each keypoint, we track the keypoint's changing image coordinates with a median flow tracker. We use twenty-five trackers to determine the image coordinates of every keypoint in every frame.
We now have enough information the compute the "camera projection matrix" of each frame. A camera projection matrix maps world coordinates to their corresponding image coordinates.
To draw an object in a particular frame, we specify the object's world coordinates. Then we compute its image coordinates and connect them with lines. In the video below, every drawn cube has a base whose world coordinates satisfy z=0. As a result, every cube appears to rest on the surface of the box.
See other final project: click here