Real-time video abstraction

Abstract

We present an automatic, real-time video and image abstraction framework that abstracts imagery by modifying the contrast of visually important features, namely luminance and color opponency. We reduce contrast in low-contrast regions using an approximation to anisotropic diffusion, and artificially increase contrast in higher contrast regions with difference-of-Gaussian edges. The abstraction step is extensible and allows for artistic or data-driven control. Abstracted images can optionally be stylized using soft color quantization to create cartoon-like effects with good temporal coherence. Our framework design is highly parallel, allowing for a GPU-based, real-time implementation. We evaluate the effectiveness of our abstraction framework with a user-study and find that participants are faster at naming abstracted faces of known persons compared to photographs. Participants are also better at remembering abstracted images of arbitrary scenes in a memory task.

References

1. Arad, N., and Gotsman, C. 1999. Enhancement by image-dependent warping. IEEE Trans. on Image Processing 8, 9, 1063--1074.
2. Barash, D., and Comaniciu, D. 2004. A common framework for non-linear diffusion, adaptive smoothing, bilateral filtering and mean shift. Image and Video Computing 22, 1, 73--81.
3. Boomgaard, R. V. D., and de Weijer, J. V. 2002. On the equivalence of local-mode finding, robust estimation and mean-shift analysis as used in early vision tasks. 16th Internat. Conf. on Pattern Recog. 3, 927--390.
4. Canny, J. F. 1986. A computational approach to edge detection. IEEE Trans. on Pattern Analysis and Machine Intelligence 8, 769--798.
5. Collomosse, J. P., Rowntree, D., and Hall, P. M. 2005. Stroke surfaces: Temporally coherent artistic animations from video. IEEE Trans. on Visualization and Computer Graphics 11, 5, 540--549.
6. DeCarlo, D., and Santella, A. 2002. Stylization and abstraction of photographs. ACM Trans. Graph. 21, 3, 769--776.
7. Elder, J. H. 1999. Are edges incomplete? Internat. Journal of Computer Vision 34, 2-3, 97--122.
8. Fischer, J., Bartz, D., and Strasser, W. 2005. Stylized Augmented Reality for Improved Immersion. In Proc. of IEEE VR, 195--202.
9. Gooch, B., Reinhard, E., and Gooch, A. 2004. Human facial illustrations: Creation and psychophysical evaluation. ACM Trans. Graph. 23, 1, 27--44.
10. Hertzmann, A. 2001. Paint by relaxation. In CGI '01:Computer Graphics Internat. 2001, 47--54.
11. Itti, L., and Koch, C. 2001. Computational modeling of visual attention. Nature Reviews Neuroscience 2, 3, 194--203.
12. Loviscach, J. 1999. Scharfzeichner: Klare bilddetails durch verformung. Computer Technik 22, 236ff.
13. Marr, D., and Hildreth, E. C. 1980. Theory of edge detection. Proc. Royal Soc. London, Bio. Sci. 207, 187--217.
14. Palmer, S. E. 1999. Vision Science: Photons to Phenomenology. The MIT Press.
15. Perona, P., and Malik, J. 1991. Scale-space and edge detection using anisotropic diffusion. IEEE Trans. on Pattern Analysis and Machine Intelligence 12, 7.
16. Pham, T. Q., and Vliet, L. J. V. 2005. Separable bilateral filtering for fast video preprocessing. In IEEE Internat. Conf. on Multimedia & Expo, CD1-4.
17. Privitera, C. M., and Stark, L. W. 2000. Algorithms for defining visual regions-of-interest: Comparison with eye fixations. IEEE Trans. on Pattern Analysis and Machine Intelligence 22, 9, 970--982.
18. Raskar, R., Tan, K.-H., Feris, R., Yu, J., and Turk, M. 2004. Non-photorealistic camera: depth edge detection and stylized rendering using multi-flash imaging. ACM Trans. Graph. 23, 3, 679--688.
19. Saito, T., and Takahashi, T. 1990. Comprehensible rendering of 3-D shapes. In Proc. of ACM SIGGRAPH 90, 197--206.
20. Santella, A., and DeCarlo, D. 2004. Visual interest and NPR: an evaluation and manifesto. In Proc. of NPAR '04, 71--78.
21. Stevenage, S. V. 1995. Can caricatures really produce distinctiveness effects? British Journal of Psychology 86, 127--146.
22. Tomasi, C., and Manduchi, R. 1998. Bilateral filtering for gray and color images. In Proceedings of ICCV '98, 839.
23. Wang, J., Xu, Y., Shum, H.-Y., and Cohen, M. F. 2004. Video tooning. ACM Trans. Graph. 23, 3, 574--583.
24. Winkenbach, G., and Salesin, D. H. 1994. Computer-generated pen-and-ink illustration. In Proc. of ACM SIGGRAPH 94, 91--100.
25. Wyszecki, G., and Styles, W. 1982. Color Science: Concepts and Methods, Quantitative Data and Formulae. Wiley, New York, NY.

Index Terms

• Computing methodologies

  • Computer graphics

    • Image manipulation

    • Rendering