High-Speed Tracking with Kernelized Correlation Filters

Abstract

The core component of most modern trackers is a discriminative classifier, tasked with distinguishing between the target and the surrounding environment. To cope with natural image changes, this classifier is typically trained with translated and scaled sample patches. Such sets of samples are riddled with redundancies—any overlapping pixels are constrained to be the same. Based on this simple observation, we propose an analytic model for datasets of thousands of translated patches. By showing that the resulting data matrix is circulant, we can diagonalize it with the discrete Fourier transform, reducing both storage and computation by several orders of magnitude. Interestingly, for linear regression our formulation is equivalent to a correlation filter, used by some of the fastest competitive trackers. For kernel regression, however, we derive a new kernelized correlation filter (KCF), that unlike other kernel algorithms has the exact same complexity as its linear counterpart. Building on it, we also propose a fast multi-channel extension of linear correlation filters, via a linear kernel, which we call dual correlation filter (DCF). Both KCF and DCF outperform top-ranking trackers such as Struck or TLD on a 50 videos benchmark, despite running at hundreds of frames-per-second, and being implemented in a few lines of code (Algorithm 1). To encourage further developments, our tracking framework was made open-source.

References

1. [1] Smeulders A., Chu D., Cucchiara R., Calderara S., Dehghan A., and Shah M., “Visual tracking: An experimental survey,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 36, no. 7, pp. 1442– 1468, Jul. 2014.
2. [2] Yang H., Shao L., Zheng F., Wang L., and Song Z., “Recent advances and trends in visual tracking: A review, ” Neurocomput., vol. 74, no. 18, pp. 3823 –3831, Nov. 2011.
3. [3] Zhang K., Zhang L., and Yang M.-H., “Real-time compressive tracking,” in Proc. Eur. Conf. Comput. Vis., 2012, pp. 864–877.
4. [4] Kalal Z., Mikolajczyk K., and Matas J., “ Tracking-learning-detection,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 34, no. 7, pp. 1409–1422, Jul. 2012.
5. [5] Babenko B., Yang M., and Belongie S., “Robust object tracking with online multiple instance learning,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 33, no. 8, pp. 1619–1632, Aug. 2011.
6. [6] Saffari A., Leistner C., Santner J., Godec M., and Bischof H., “On-line random forests,” in Proc. 3rd IEEE Int. Conf. Comput. Vis. Workshop On-line Comput. Vis., 2009, pp. 1393 –1400.
7. [7] Hare S., Saffari A., and Torr P., “Struck: Structured output tracking with kernels,” in Proc. Int. Conf. Comput. Vis., 2011, pp. 263–270.
8. [8] Rifkin R., Yeo G., and Poggio T., “Regularized least-squares classification,” Nato Sci. Ser. Sub Ser. III, vol. 190, pp. 131–154, 2003.
9. [9] Bolme D. S., Beveridge J. R., Draper B. A., and Lui Y. M. , “Visual object tracking using adaptive correlation filters,” in Proc. Comput. Vis. Pattern Recognit., 2010, pp. 2544–2550.
10. [10] Bolme D. S., Draper B. A., and Beveridge J. R., “ Average of synthetic exact filters,” in Proc. Comput. Vis. Pattern Recognit., 2009, pp. 2105–2112.
11. [11] Wu Y. , Lim J., and Yang M. H. , “Online object tracking: A benchmark,” in Proc. Comput. Vis. Pattern Recognit., 2013, pp. 2411–2418.
12. [12] Avidan S., “Support vector tracking,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 26, no. 8, pp. 1064 –1072, Aug. 2004.
13. [13] Grabner H., Leistner C., and Bischof H., “ Semi-supervised on-line boosting for robust tracking,” in Proc. Eur. Conf. Comput. Vis. , 2008, pp. 234–247.
14. [14] Wu Y. , Shen B., and Ling H. , “Online robust image alignment via iterative convex optimization,” in Proc. Comput. Vis. Pattern Recognit. , 2012, pp. 1808–1814.
15. [15] Sevilla-Lara L. and Learned-Miller E., “Distribution fields for tracking,” in Proc. Comput. Vis. Pattern Recognit., 2012, pp. 1910–1917.
16. [16] Lampert C., Blaschko M., and Hofmann T., “Beyond sliding windows: Object localization by efficient subwindow search,” in Proc. Comput. Vis. Pattern Recognit., 2008, pp. 1–8.
17. [17] Alexe B., Petrescu V., and Ferrari V., “Exploiting spatial overlap to efficiently compute appearance distances between image windows,” in Proc. Adv. Neural Inf. Processing Syst., 2011, pp. 2735–2743.
18. [18] Harzallah H., Jurie F., and Schmid C., “Combining efficient object localization and image classification,” in Proc. Int. Conf. Comput. Vis., 2009, pp. 237–244.
19. [19] Vedaldi A., Gulshan V., Varma M., and Zisserman A., “Multiple kernels for object detection, ” in Proc. Int. Conf. Comput. Vis., 2009, pp. 606 –613.
20. [20] Felzenszwalb P., Girshick R., McAllester D., and Ramanan D. , “Object detection with discriminatively trained part-based models,” IEEE Trans. Pattern Anal. Mach. Intell. , vol. 32, no. 9, pp. 1627–1645, Sep. 2010.
21. [21] Gonzalez R. C. and Woods R. E., Digital Image Processing. Upper Saddle River, NJ, USA: Prentice Hall, 2008.
22. [22] Dollár P., Appel R., Belongie S., and Perona P., “Fast feature pyramids for object detection,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 36, no. 8, pp. 1532–1545, Aug. 2014.
23. [23] Scholkopf B. and Smola A., Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond. Cambridge, MA, USA: MIT Press, 2002.
24. [24] Casasent D. and Patnaik R., “Analysis of kernel distortion-invariant filters,” Proc. SPIE , vol. 6764, p. 1, 2007.
25. [25] Patnaik R. and Casasent D., “Fast FFT-based distortion-invariant kernel filters for general object recognition, ” Proc. SPIE, vol. 7252, 2009, p. 1.
26. [26] Jeong K.-H., Pokharel P. P., Xu J.-W., Han S., and Principe J., “Kernel based synthetic discriminant function for object recognition,” in Proc. IEEE Int. Conf. Acoust. Speech Signal Process., 2006, pp. 1–5.
27. [27] Xie C. , Savvides M., and Kumar B. Vijaya, “Kernel correlation filter based redundant class-dependence feature analysis (KCFA) on FRGC2.0 data,” in Proc. 2nd Int. Conf. Anal. Model. Faces Gestures, 2005, pp. 32–43 .
28. [28] Mahalanobis A., Kumar B., and Casasent D., “Minimum average correlation energy filters,” Appl. Optics, vol. 26, pp. 3633–3640, 1987.
29. [29] Henriques J. F., Caseiro R., Martins P., and Batista J., “Exploiting the circulant structure of tracking-by-detection with kernels,” in Proc. Eur. Conf. Comput. Vis., vol. 7575, pp. 702–715, 2012.
30. [30] Revaud J., Douze M., Cordelia S., and Jégou H., “Event retrieval in large video collections with circulant temporal encoding,” in Proc. Comput. Vis. Pattern Recognit. , 2013, pp. 2459–2466.
31. [31] Henriques J. F., Carreira J., Caseiro R., and Batista J., “Beyond hard negative mining: Efficient detector learning via block-circulant decomposition,” in Proc. Int. Conf. Comput. Vis. , 2013, pp. 2760–2767.
32. [32] Galoogahi H. K., Sim T., and Lucey S., “ Multi-channel correlation filters,” in Proc. Int. Conf. Comput. Vis., 2013 , pp. 3072–3079.
33. [33] Boddeti V. N., Kanade T., and Kumar B. V., “ Correlation filters for object alignment,” in Proc. Comput. Vis. Pattern Recognit. , 2013, pp. 2291–2298.
34. [34] Gray R. M., Toeplitz and Circulant Matrices: A Review. Boston, MA, USA: Now Publishers, 2006.
35. [35] Davis P. J., Circulant Matrices. Providence, RI, USA: Amer. Math. Society, 1994.
36. [36] Vedaldi A. and Zisserman A. , “Efficient additive kernels via explicit feature maps,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 34, no. 3, pp. 480 –492, Mar. 2011.

Index Terms

• Computing methodologies

  • Artificial intelligence

    • Computer vision

      • Computer vision problems

        • Tracking

      • Computer vision tasks

        • Scene understanding

  • Machine learning