- How motion vectors are calculated?
- OpenCV for calculating Optical Flow
In computer vision, there is always a need to track the motion of an object or blob across frames in a moving sequence. Various methods have been investigated in literature for this purpose. One very interesting method for motion estimation and tracking is optical flow. In simple terms, optical flow gives the measure of movement of a pixel or a block in two consecutive frame. This measure of movement is given in the form of a vector where the magnitude of the vector signifies the amount of motion and the angle of the vector specifies the direction of the motion. This vector is called motion vector.
Consider the image below:
Concentrate on the objects in the image above and the motion vectors drawn on them. Analyzing the motion vectors, we can see that a bicycle is moving towards the left while two pedestrians are moving towards the right. The speed of their motion is categorized by the magnitude of the motion vectors and their direction by the angel of the motion vectors.
Motion vectors can be defined at block level or sometimes at pixel level as well. The pixel level calculation is called dense optical flow. Usually in real-time computer vision applications, motion vectors are calculated at block level.
Lets see how the optical flow algorithm works. Consider two successive frames A and B. The current frame A is divided into small blocks (say of size 8x8). Now for each block in frame A, we try to find its closest match in frame B. For matching, various similarity or distance measures are used for calculating the similarity with each target block and finding the best match. These measures include euclidean distance, edit distance or correlation similarity. When a block is found, a vector is drawn in the direction of the found block from the location of the block in frame A.
This step is repeated for each block of frame A and as a result, we get an image where vectors are drawn on top to highlight the motion of objects in that image. The result of the optical flow algorithm can be seen in the sequence of pictures given below:
In the figure above, four successive frames are analyzed and optical flow is calculated for them. The green lines are the motion vectors with their end points shown in red dot. We can see that the objects that are stationary across frames have motion vectors of zero length (shown by only red dots). The blocks which are moving more have larger motion vectors.
This analysis of motion vectors is very useful in studying the motion in a video that can further be used for various applications. These applications include
In openCV, there are various implementations for calculating Optical Flow. The general principle is simple. The programs reads a video and then extracts its frames. The frames are passed to the Optical Flow function in pairs because an Optical Flow function takes previous and current Frames as input arguments along with few other parameters. For example, to find optical flow using a function that is based on Gunner Farneback's algorithm, the CV function is given below. It calculates dense optical flow i.e. motion vectors are calculated for each pixel in the frame.
void calcOpticalFlowFarneback(InputArray prev, InputArray next,InputOutputArray flow, double pyr_scale, int levels,int winsize, int iterations, int poly_n, double poly_sigma,int flags)
Similarly, there is another function in openCV that calculates Optical flow by employing sparse feature set using the iterative Lucas-Kanade algorithm. It is given here:
void calcOpticalFlowPyrLK(InputArray prevImg, InputArray nextImg,InputArray prevPts, InputOutputArray nextPts,OutputArray status, OutputArray err, Size winSize,int maxLevel, TermCriteria criteria, int flags,double minEigThreshold)
Here prevPts is a vector of 2D points for which the flow needs to be found and nextPts is the output vector of 2D points containing the calculated new positions of input features in the second image.
Details of the other function arguments present in above mentioned two optical flow functions is given here:
For beginners, following related functions are also useful:
To read a video:
vid = cv2.VideoCapture("test.avi");
To extract a frame from the video vid:
frame1 = vid.read();
Two extracted frames are passed to the Optical Flow function that calculates the relative motion between the two frames. Result of a dense optical flow function looks something like this:
In this overview, we have just learned about the optical flow, motion vectors and their calculation. We will look at its further applications in Video Compression and Video Stabilization articles. More details on calculating Optical Flow can seen on the openCV website.