An invitation to discuss computer depiction

ABSTRACT

This paper draws from art history and perception to place computer depiction in the broader context of picture production. It highlights the often underestimated complexity of the interactions between features in the picture and features of the represented scene. Depiction is not always a unidirectional projection from a 3D scene to a 2D picture, but involves much feedback and influence from the picture space to the object space. Depiction can be seen as a pre-existing 3D reality projected onto 2D, but also as a 2D pictorial representation that is superficially compatible with an hypothetic 3D scene. We show that depiction is essentially an optimization problem, producing the best picture given goals and constraints.We introduce a classification of basic depiction techniques based on four kinds of issue. The spatial system deals with the mapping of spatial properties between 3D and 2D (including, but not restricted to, perspective projection). The primitive system deals with the dimensionality and mappings between picture primitives and scene primitives. Attributes deal with the assignment of visual properties such as colors, texture, or thickness. Finally, marks are the physical implementations of the picture (e.g. brush strokes, mosaic cells). A distinction is introduced between interaction and picture-generation methods, and techniques are then organized depending on the dimensionality of the inputs and outputs.

References

1. {3D} Deep Paint 3D. http://www.deeppaint3d.com.
2. {ABL95} M. Agrawala, A. Beers, and M. Levoy. 3d painting on scanned surfaces. ACM Symp. on Interactive 3D Graphics, 1995.
3. {Ada95} A. Adams. The Camera+The Negative+The Print. Little Brown and Co, 1995.
4. {Apo99} A. Apodaca. Photosurrealism. In A. Apodaca and L. Gritz, editors, Advanced Renderman : Creating CGI for Motion Pictures. Morgan Kaufmann, 1999.
5. {AS01} M. Agrawala and C. Stolte. Rendering effective route maps: Improving usability through generalization. In Computer Graphics (Proc. SIGGRAPH), 2001.
6. {Auc99} K. Aucoin. Making Faces. Little Brown and Co, 1999.
7. {AZM00} M. Agrawala, D. Zorin, and T. Munzner. Artistic multiprojection rendering. In Eurographics Workshop on Rendering, 2000.
8. {Bar97} R. Barzel. Lighting controls for computer cinematography. J. Graphics Tools, 2(1), 1997.
9. {BCD01} D. Bourguignon, M. P. Cani, and G. Drettakis. Drawing for illustration and annotation in 3d. Computer Graphics Forum, 20(3), 2001.
10. {BETT99} G. Di Battista, P. Eades, R. Tamassia, and I. Tollis. Graph Drawing: Algorithms for the Visualization of Graphs. Prentice Hall, 1999.
11. {BFR95} V. Bourgoin, N. Farenc, and M. Roelens. Creating special effects by ray-tracing with non classical perspectives. Technical Report 1995-13. École Nationale des Mines de St Etienne, 1995.
12. {BM92} Barbour and Meyer. Visual cues and pictorial limitations for computer generated photorealistic images. the Visual Computer, 9, 1992.
13. {Boo63} P. J. Booker. A History of Engineering Drawing. Chatto and Windus, 1963.
14. {BS00} J. Buchanan and M. Sousa. The edge buffer: A data structure for easy silhouette rendering. In Non-Photorealistic Animation and Rendering, 2000.
15. {BSLM01} B. Baxter, V. Scheib, M. Lin, and D. Manocha. Dab: Interactive haptic painting with 3d virtual brushes. Proc. SIGGRAPH, 2001.
16. {Can86} J. Canny. A computational approach to edge detection. IEEE Trans PAMI, 8(6):679-698, 1986.
17. {CAS+97} C. Curtis, S. Anderson, J. Seims, K. Fleischer, and D. Salesin. Computer-generated watercolor. Proc. SIGGRAPH, 1997.
18. {Che95} E. Chen. Quicktime VR - an image-based approach to virtual environment navigation. Proc. of SIGGRAPH, 1995.
19. {CHZ00} J. Cohen, J. Hughes, and R. Zeleznik. Harold: A world made of drawings. In Non-Photorealistic Animation and Rendering, 2000.
20. {CJTF98} W. Toledo Corrêa, R. Jensen, C. Thayer, and A. Finkelstein. Texture mapping for cel animation. Proc. SIGGRAPH, 1998.
21. {CL93} B. Cabral and L. (Casey) Leedom. Imaging vector fields using line integral convolution. In Proc. SIGGRAPH, 1993.
22. {Clo71} M. Clowes. On seeing things. Artificial Intelligence, 2(1):79-116, 1971.
23. {CP78} I. Carlbom and J. Paciorek. Planar geometric projections and viewing transformations. Comp. Surveys, 10:465-502, 1978.
24. {CT01} S.-H. Chu and C.-L. Tai. Animating chinese landscape paintings and panorama using multi-perspective modeling. In Computer Graphics International, 2001.
25. {Cur98} C. Curtis. Loose and Sketchy Animation. In SIGGRAPH: Conference Abstracts and Applications, 1998.
26. {DHvOS00} O. Deussen, S. Hiller, C. van Overveld, and T. Strothotte. Floating points: A method for computing stipple drawings. Computer Graphics Forum, 19(3), 2000.
27. {DOM+01} F. Durand, V. Ostromoukhov, M. Miller, F. Duranleau, and J. Dorsey. Decoupling strokes and high-level attributes for interactive traditional drawing. In Eurographics Workshop on Rendering, 2001.
28. {DTM96} P. Debevec, C. Taylor, and J. Malik. Modeling and rendering architecture from photographs: A hybrid geometry- and image-based approach. In Proc. of SIGGRAPH 96, 1996.
29. {DW83} F. Dubery and J. Willats. Perspective and Other Drawing Systems. Van Nostrand, 1983.
30. {EC90} G. Elber and E. Cohen. Hidden Curve Removal for Free Form Surfaces. In Computer Graphics (Proc. SIGGRAPH), 1990.
31. {EG01} J. Elder and R. Goldberg. Image editing in the contour domain. IEEE Trans. on Pattern Analysis and Machine Intelligence, 23(3), 2001.
32. {EHW97} P. Eades, M. E. Houle, and R. Webber. Finding the best viewpoints for three-dimensional graph drawings. Lecture Notes in Computer Science, 1353:87-??, 1997.
33. {Elb95} G. Elber. Line art rendering via a coverage of isoparametric curves. IEEE Trans. on Visualization and Computer Graphics, 1(3):231-239, September 1995.
34. {Elk94} J. Elkins. The Poetics of Perspective. Cornell U. Pr., 1994.
35. {Fai98} Fairchild. Color Appearance Models. Addison-Wesley, 1998.
36. {Fer99} J. Ferwerda. Varieties of realism. talk, Cornell Workshop on Rendering, Perception, and Measurement, http://www.graphics.cornell.edu/workshop/, 1999.
37. {FLR+95} O. Faugeras, S. Laveau, L. Robert, G. Csurka, and C. Zeller. 3-d reconstruction of urban scenes from sequences of images. In A. Gruen, O. Kuebler, and P. Agouris, editors, Automatic Extraction of Man-Made Objects from Aerial and Space Images. Birkhauser, 1995.
38. {FS94} A. Finkelstein and D. Salesin. Multiresolution curves. In Proc. SIGGRAPH, 1994.
39. {GDCV98} J. Gomes, L. Darsa, B. Costa, and L. Velho. Warping And Morphing Of Graphical Objects. Morgan Kaufman, 1998.
40. {GFMS95} A. Glassner, K. Fishkin, D. Marimont, and M. Stone. Device-directed rendering. ACM Trans. on Graphics, 14(1):58-76, January 1995.
41. {GG99} G. Glaeser and E. Gröller. Fast generation of curved perspective for ultra-wide-angle lenses in vr applications. Visual Computer, 15:365-376, 1999.
42. {GG01} Gooch and Gooch. Non-Photorealistic Rendering. AK-Peters, 2001.
43. {GGSC98} A. Gooch, B. Gooch, P. Shirley, and E. Cohen. A non-photorealistic lighting model for automatic technical illustration. Proc. SIGGRAPH, 1998.
44. {GH00} R. Gershbein and P. Hanrahan. A fast relighting engine for interactive cinematic lighting design. In Proc. SIGGRAPH, Computer Graphics, 2000.
45. {GKR95} B. Guenter, T. Knoblock, and E. Ruf. Specializing shaders. Proc. SIGGRAPH, 1995.
46. {Gla00} Andrew Glassner. Cubism and cameras: Free-form optics for computer graphics. Technical Report MSR-TR-2000-05, Microsoft Research, 2000.
47. {GM97} J. Geigel and K. Musgrave. A model for simulating the photographic development process on digital images. Proc. SIGGRAPH, 1997.
48. {Gom56} Gombrich. Art and Illusion. Princeton Press, 1956.
49. {Gom82} E. H. Gombrich. The image and the eye : further studies in the psychology of pictorial representation. Cornell U. Pr., 1982.
50. {Goo98} A. Gooch. Interactive non-photorealistic technical illustration. Master's thesis, U. of Utah, 1998.
51. {Gre99} S. Green. Beyond photorealism. In Eurographics Workshop on Rendering, 1999.
52. {GRMS01} B. Gooch, E. Reinhard, C. Moulding, and P. Shirley. Artistic composition for image creation. In Eurographics Workshop on Rendering, 2001.
53. {Grö94} Eduard Gröller. Nonlinear ray tracing: visualizing strange worlds. Visual Computer, 11(5), 1994.
54. {GSG+99} B. Gooch, P. Sloan, A. Gooch, P. Shirley, and R. Riesenfeld. Interactive Technical Illustration. ACM Symposium on Interactive 3D Graphics, 1999.
55. {GSS+99} S. Green, D. Salesin, S. Schofield, A. Hertzmann, P. Litwinowicz, A. Gooch, C. Curtis, and B. Gooch. Non-Photorealistic Rendering. SIGGRAPH Non-Photorealistic Rendering Course Notes, 1999.
56. {Guz71} A. Guzman. Analysis of curved line drawings using context and global information. In B. Meltzer D. Mitchie, editor, Machine Intelligence V6, pages 325-375. Edinburgh U. Pr., 1971.
57. {GW92} M. Gleicher and A. Witkin. Through-the-lens camera control. Computer Graphics (Proc. SIGGRAPH), 1992.
58. {HAA97} Y. Horry, K. Anjyo, and K. Arai. Tour into the picture: Using a spidery mesh interface to make animation from a single image. In Proc. of SIGGRAPH 97, 1997.
59. {Hae90} P. Haeberli. Paint by numbers: Abstract image representations. Proce. SIGGRAPH, 1990.
60. {Hag86} M. Hagen. Varieties of Realism: Geometries of Representational Art. Cambridge U. Pr., 1986.
61. {Hau01} A. Hausner. Simulating decorative mosaics. Proc. SIGGRAPH, 2001.
62. {HCS96} L. He, M. Cohen, and D. Salesin. The virtual cinematographer: A paradigm for automatic real-time camera control and directing. Computer Graphics (Proc. SIGGRAPH), 1996.
63. {Her98} A. Hertzmann. Painterly rendering with curved brush strokes of multiple sizes. Proc. SIGGRAPH, 1998.
64. {Her01} A. Hertzmann. Paint by relaxation. In CGI, 2001.
65. {HH90} P. Hanrahan and P. Haeberli. Direct WYSIWYG painting and texturing on 3D shapes. Computer Graphics (Proc. SIGGRAPH), 1990.
66. {Hoc01} D. Hockney. Secret Knowledge: Rediscovering the Lost Techniques of the Old Masters. Viking Press, 2001.
67. {Hog81} B. Hogarth. Dynamic Light and Shade. Watson Guptill, 1981.
68. {HOT98} J. Hodgins, J. O'Brien, and J. Tumblin. Perception of human motion with different geometric models. IEEE TVCG, 4(4), October 1998.
69. {HP00} A. Hertzmann and K. Perlin. Painterly rendering for video and interaction. In Non-Photorealistic Animation and Rendering, 2000.
70. {Huf71} D. Huffman. Impossible objects as nonsense sentences. In B. Meltzer and D. Michie, editors, Machine Intelligence 6, pages 295-324. Edinburgh U. Pr., 1971.
71. {Hun95} Hunt. The reproduction of Color (5th ed.). Kings Langley:Fountain Press, 1995.
72. {IMT99} T. Igarashi, S. Matsuoka, and H. Tanaka. Teddy: A sketching interface for 3d freeform design. Proc. SIGGRAPH, 1999.
73. {Kaj86} J. Kajiya. The rendering equation. In Computer Graphics (Proc. SIGGRAPH), 1986.
74. {Kem90} M. Kemp. The Science of Art. Yale University Press, 1990.
75. {KMN+99} M. Kowalski, L. Markosian, J. D. Northrup, L. Bourdev, R. Barzel, L. Holden, and J. Hughes. Art-based rendering of fur, grass, and trees. Proc. SIGGRAPH, 1999.
76. {Koe90} J. Koenderink. Solid Shape. MIT Press, 1990.
77. {KS00} C. Kaplan and D. Salesin. Escherization. Proc. SIGGRAPH, 2000.
78. {Kub86} Michael Kubovy. The Psychology of Perspective and Renissance Art. Cambridge University Press, 1986.
79. {Lan77} E. H. Land. The retinex theory of color vision. Scientific American, 237(6):108-128, December 1977.
80. {Lev98} Jonathan Levene. A framework for non-realistic projections. Master's thesis, Massachusetts Institute of Technology, 1998.
81. {LG96} H. Löffelmann and E. Gröller. Ray tracing with extended cameras. J. of Visualization and Comp. Animation, 7(4):211-227, 1996.
82. {Lit91} P. Litwinowicz. Inkwell: A 2 -d animation system. Computer Graphics (Proc. SIGGRAPH), 1991.
83. {Lit97} P. Litwinowicz. Processing images and video for an impressionist effect. Proc. SIGGRAPH, 1997.
84. {LS95} J. Lansdown and S. Schofield. Expressive Rendering: A Review of Nonphotorealistic Techniques. IEEE CG & A, 15(3):29-37, 1995.
85. {MAB+97} J. Marks, B. Andalman, P. Beardsley, W. Freeman, S. Gibson, J. Hodgins, T. Kang, B. Mirtich, H. Pfister, W. Ruml, K. Ryall, J. Seims, and S. Shieber. Design galleries: A general approach to setting parameters for computer graphics and animation. In Proc. SIGGRAPH 97, 1997.
86. {Max83} Nelson L. Max. Computer graphics distortion for IMAX and OMNIMAX projection. In Nicograph '83 Proceedings, pages 137-159, December 1983.
87. {Mei96} B. Meier. Painterly rendering for animation. Proc. SIGGRAPH, 1996.
88. {MGT00} D. Martin, S. Garcia, and J. C. Torres. Observer dependent deformations in illustration. In Non-Photorealistic Animation and Rendering, 2000.
89. {Mil91} Millerson. Lighting for Television and Film, 3rd ed. Focal Press, 1991.
90. {MKT+97} L. Markosian, M. Kowalski, S. Trychin, L. Bourdev, D. Goldstein, and J. Hughes. Real-time nonphotorealistic rendering. Proc. SIGGRAPH, 1997.
91. {MMK+00} L. Markosian, B. Meier, M. Kowalski, L. Holden, J. D. Northrup, and J. Hughes. Art-based rendering with continuous levels of detail. In Non-Photorealistic Animation and Rendering, 2000.
92. {NPA00} NPAR. Discussion. First International Symposium on Non Photorealistic Animation and Rendering, 2000.
93. {OCDD01} B. M. Oh, M. Chen, J. Dorsey, and F. Durand. Image-based modeling and photo editing. Proc. SIGGRAPH, 2001.
94. {OH95} V. Ostromoukhov and R. Hersch. Artistic screening. Proc. SIGGRAPH, 1995.
95. {Ost99} V. Ostromoukhov. Digital facial engraving. Proc. SIGGRAPH, 1999.
96. {Pan27} Erwin Panofsky. Perspective As Symbolic Form (La perspective comme forme symbolique). Zone Books, 1927.
97. {Par01} R. Parent. Computer Animation: Algorithms and Techniques,. Morgan-Kaufmann, 2001.
98. {PF92} P. Poulin and A. Fournier. Lights from highlights and shadows. In ACM Symposium on Interactive 3D Graphics, 1992.
99. {PFFL00} L. Petrovic, B. Fujito, A. Finkelstein, and L. Williams. Shadows for cel animation. Proc. SIGGRAPH, 2000.
100. {PFG00} F. Pellacini, J. Ferwerda, and D. Greenberg. Toward a psychophysically-based light reflection model for image synthesis. Proc. SIGGRAPH, 2000.
101. {PHM90} D. Pearson, E. Hanna, and K. Martinez. Computer generated cartoons. In Barlow, Blakemore, and Weston-Smith, editors, Images and Understanding. Cambrdge U. P., 1990.
102. {PHMF01} E. Praun, H. Hoppe, M., and A. Finkelstein. Real-time hatching. Proc. SIGGRAPH, 2001.
103. {Pir70} M. Pirenne. Optics, Painting and Photography. Cambridge U. Pr., 1970.
104. {PRJ97} P. Poulin, K. Ratib, and M. Jacques. Sketching shadows and highlights to position lights. In Proc. Computer Graphics International, 1997.
105. {Rad99} P. Rademacher. View-dependent geometry. Proc. SIGGRAPH, 1999.
106. {RAGS01} E. Reinhard, M. Adhikhmin, B. Gooch, and P. Shirley. Color transfer between images. IEEE Computer Graphics and Applications, 21(4):34-41, 2001.
107. {RB98} P. Rademacher and G. Bishop. Multiple-center-of-projection images. In Proc. SIGGRAPH, 1998.
108. {RC99} R. Raskar and M. Cohen. Image Precision Silhouette Edges. In Proc. 1999 ACM Symp. on Interactive 3D Graphics, 1999.
109. {Rey00} C. Reynolds. Stylized depiction in computer graphics non-photorealistic, painterly and 'toon rendering. http://www.red3d.com/cwr/npr/, 2000.
110. {RH99} V. S. Ramachandran and W. Hirstein. The science of art: A neurological theory of aesthetic experience. Journal of Consciousness Studies, 6(6/7), 1999.
111. {RK00} C. Rössl and L. Kobbelt. Line-art rendering of 3d models, 2000.
112. {RS56} T. Ryan and C. Schwartz. Speed of perception as a function of mode of representation. Am. J. of Psychology, 69:60-69, 1956.
113. {SABS94} Michael P. Salisbury, Sean E. Anderson, Ronen Barzel, and David H. Salesin. Interactive pen-and-ink illustration. Proceedings of SIGGRAPH 94, pages 101-108, July 1994. ISBN 0-89791-667-0. Held in Orlando, Florida.
114. {SB99} M. Sousa and J. Buchanan. Observational model of blenders and erasers in computer-generated pencil rendering. In Graphics Interface, 1999.
115. {SD96} S. Seitz and C. Dyer. View morphing: Synthesizing 3d metamorphoses using image transforms. Proc. SIGGRAPH, 1996.
116. {SDS+93} C. Schoeneman, J. Dorsey, B. Smits, J. Arvo, and D. Greenberg. Painting with light. Computer Graphics, 1993.
117. {SF91} Dorée Duncan Seligmann and Steven Feiner. Automated generation of intent-based 3d illustrations. Computer Graphics (Proceedings of SIGGRAPH 91), 25(4):123-132, July 1991. ISBN 0-201-56291-X. Held in Las Vegas, Nevada.
118. {SGG+00} P. Sander, X. Gu, S. Gortler, H. Hoppe, and J. Snyder. Silhouette clipping. Proc. SIGGRAPH, 2000.
119. {SK98} S. Seitz and K. Kutulakos. Plenoptic image editing. In Proc. 5th Int. Conf.on Computer Vision, 1998.
120. {SL01} R. Shacked and D. Lischinski. Automatic lighting design using a perceptual quality metric. Computer Graphics Forum, 20(3):215-226, 2001.
121. {SMGG01} P. Sloan, W. Martin, A. Gooch, and B. Gooch. The lit sphere: A model for capturing npr shading from art. In Proc. Graphics Interface, 2001.
122. {SSS74} Ivan E. Sutherland, Robert F. Sproull, and Robert A. Schumacker. A characterization of ten hidden-surface algorithms. ACM Computing Surveys, 6(1):1-55, March 1974.
123. {SST89} Saito, Shinya, and Takahashi. Highlighting rounded edges. In Computer Graphics International, 1989.
124. {ST90} T. Saito and T. Takahashi. Comprehensible Rendering of 3-D Shapes. In Computer Graphics (Proc. SIGGRAPH), 1990.
125. {SWHS97} M. Salisbury, M. Wong, J. Hughes, and D. Salesin. Orientable textures for image-based pen-and-ink illustration. Proc. SIGGRAPH, 1997.
126. {TB96} G. Turk and D. Banks. Image-guided streamline placement. In Proc. SIGGRAPH, 1996.
127. {TDM01} O. Tolba, J. Dorsey, and L. McMillan. A projective drawing system. In ACM Symp. on Interactive 3D Graphics, 2001.
128. {TFN99} Saeko Takagi, Issei Fujishiro, and Masayuki Nakajima. Volumetric modeling of artistic techniques in colored pencil drawing. In SIGGRAPH: Conf. abstracts and applications, 1999.
129. {TJ81} F. Thomas and O. Johnston. The Illusion of Life : Disney Animation. Abbeville Pr., 1981.
130. {TT99} Tumblin and Turk. LCIS: a boundary hierarchy for detail-preserving contrast reduction. In Proc. Siggraph, 1999.
131. {TTT91} Tanaka, Tokiichiro, and Takahashi. Precise rendering method for edge highlighting. In Computer Graphics International, 1991.
132. {Tum99} Jack Tumblin. Three methods of detail-preserving contrast reduction for displayed images. PhD thesis, College of Computing Georgia Inst. of Technology, September 1999.
133. {Uli87} R. Ulichney. Digital Halftoning. MIT Press, 1987.
134. {VB99} O. Veryovka and J. Buchanan. Halftoning with image-based dither screens. Graphics Interface, 1999.
135. {vH81} Hermann von Helmoholtz. On the relation of optics to painting. In Popular scientific lectures. Appleton, 1881.
136. {Web83} Webster. Ninth New Collegiate Dictionary. Merriam Webster, 1983.
137. {WFH+97} D. Wood, A. Finkelstein, J. Hughes, C. Thayer, and D. Salesin. Multiperspective panoramas for cel animation. Proc. SIGGRAPH, 1997.
138. {Wil90} J. Willats. The draughtsman's contract. In H. Barlow, C. Blakemore, and M. Weston-Smith, editors, Images and Understanding. Cambrdge U. Pr., 1990.
139. {Wil91} L. Williams. Shading in two dimensions. In Proc. of Graphics Interface '91, pages 143-151, June 1991.
140. {Wil97} J. Willats. Art and Representation. Princeton U. Pr., 1997.
141. {WS94} G. Winkenbach and D. Salesin. Computer-generated pen-and-ink illustration. Proc. SIGGRAPH, 1994.
142. {ZB95} D. Zorin and A. Barr. Correction of geometric perceptual distortion in pictures. In Proc. SIGGRAPH, 1995.
143. {ZBr} Pixelogic ZBrush. http://pixologic.com/.
144. {ZDPSS01} L. Zhang, G. Dugas-Phocion, J. Samson, and S. Seitz. Single view modeling of free-form scenes. In Proc. of CVPR, 2001.
145. {Zek00} S. Zeki. Inner Vision : An Exploration of Art and the Brain. Oxford U. Pr., 2000.
146. {ZH00} D. Zorin and A. Hertzmann. Illustrating smooth surfaces. Proc. SIGGRAPH, 2000.
147. {ZHH96} R. Zeleznik, K. Herndon, and J. Hughes. SKETCH: An interface for sketching 3D scenes. Proc. SIGGRAPH, 1996.

Index Terms

• Computing methodologies

  • Artificial intelligence

    • Computer vision

      • Image and video acquisition

        • 3D imaging

  • Computer graphics

    • Animation

    • Graphics systems and interfaces

• Human-centered computing

  • Human computer interaction (HCI)

    • Interaction devices

      • Graphics input devices