Wednesday, December 10, 2014

Hyperlapse Video With Python

For my final project in Computational Photography, I wrote software that would emulate the Hyperlapse video effect found in Instagram's Hyperlapse App.

For those who are unaware, Hyperlapse video is the equivalent of a motion time-lapse picture. Typically, this is created with a series of image stabilization and subsampling algorithms. Below is an example of a Hyperlapse created with Instagram's app:



I did some digging to try to find out how these videos are created, and I found that Instagram uses the gyroscope built-into the iPhone for motion stabilization. As each frame is captured, the frame is translated in an amount and direction that is opposite to the motion of the camera, thus creating a synthetically smooth motion. In addition to the stabilization, the application crops, or in some cases, stretches each frame so that no black borders are visible from the translation of each frame.

Microsoft Research created a similar piece of software, but instead of using a gyroscope, they processed the video based on information from the image frames. Using a computation-heavy process of reconstructing a 3D point cloud of the entire scene, calculating an optimally smooth camera path, and then re-rendering the synthetic frames, they were able to output Hyperlapse-esque video.

---

Not having the resources of Microsoft nor an iPhone, I decided to try making a Hyperlapse effect with Python and OpenCV. In general, my algorithm was the following:
  1. Calculate motion between each video frame with sparse optical flow, then integrate the motion in the time-domain to calculate an estimated position of each frame at each time step
  2. Filter the position data with a low-pass filter to smooth the motion
  3. Subsample frames to speed up the footage
  4. Filter the subsampled position data with a low-pass filter again to remove high-frequency data that was added from subsampling
  5. Calculate how much to crop the video in order to remove the black borders from frame translations (if too much is cropped, overlay the frames)
  6. Translate and render the stabilized and subsampled frames
While not perfect, it ended up working pretty well! You can download the code for this from Github: https://github.com/sprestwood/Hyperlapse

Below are some videos I rendered with the program:















1 comment :

  1. Great piece of software! :D I wish Microsoft would release it's version to the public, but this will do nicely for now!

    ReplyDelete