Probabilistic color-by-numbers: suggesting pattern colorizations using factor graphs

Abstract

We present a probabilistic factor graph model for automatically coloring 2D patterns. The model is trained on example patterns to statistically capture their stylistic properties. It incorporates terms for enforcing both color compatibility and spatial arrangements of colors that are consistent with the training examples. Using Markov Chain Monte Carlo, the model can be sampled to generate a diverse set of new colorings for a target pattern. This general probabilistic framework allows users to guide the generated suggestions via conditional inference or additional soft constraints. We demonstrate results on a variety of coloring tasks, and we evaluate the model through a perceptual study in which participants judged sampled colorings to be significantly preferable to other automatic baselines.

References

1. Albers, J. 1963. The interaction of color. Art news 62, 1.
2. Bribiesca, E. 1997. Measuring 2-d shape compactness using the contact perimeter. Computers & Mathematics with Applications 33, 11.
3. Carbonell, J., and Goldstein, J. 1998. The use of mmr, diversity-based reranking for reordering documents and producing summaries. In Proc. ACM SIGIR 1998.
4. Charpiat, G., Hofmann, M., and Schölkopf, B. 2008. Automatic image colorization via multimodal predictions. In Proc. ECCV 2008.
5. Cohen-Or, D., Sorkine, O., Gal, R., Leyvand, T., and Xu, Y.-Q. 2006. Color harmonization. In Proc. SIGGRAPH 2006.
6. Datta, R., Joshi, D., Li, J., and Wang, J. Z. 2006. Studying aesthetics in photographic images using a computational approach. In Proc. ECCV 2006.
7. Eitz, M., Hays, J., and Alexa, M. 2012. How do humans sketch objects? In Proc. SIGGRAPH 2012.
8. Geyer, C. 1991. Markov chain monte carlo maximum likelihood. In Proc. of the 23rd Symposium on the Inteface: Computing Science and Statistics, 156--163.
9. Hastings, W. K. 1970. Monte carlo sampling methods using markov chains and their applications. Biometrika 57, 1.
10. Heer, J., and Stone, M. 2012. Color naming models for color selection, image editing and palette design. In Proc. ACM CHI 2012.
11. Hinton, G. E. 2002. Training products of experts by minimizing contrastive divergence. Neural Computation.
12. Itten, J. 1974. The Art of Color: The Subjective Experience and Objective Rationale of Color. Wiley.
13. Jain, A., Thormählen, T., Ritschel, T., and Seidel, H.-P. 2012. Material memex: automatic material suggestions for 3d objects. In Proc. SIGGRAPH Asia 2012.
14. Levin, A., Lischinski, D., and Weiss, Y. 2004. Colorization using optimization. In Proc. SIGGRAPH 2004.
15. Lin, S., and Hanrahan, P. 2013. Modeling how people extract color themes from images. In Proc. of CHI 2013, ACM, New York, NY, USA, CHI '13.
16. Lübbe, E. 2010. Colours in the Mind - Colour Systems in Reality. Books on Demand.
17. Mccallum, A., Schultz, K., and Singh, S. 2009. Factorie: Probabilistic programming via imperatively defined factor graphs. In Proc. NIPS 2009.
18. Meier, B., Spalter, A., and Karelitz, D. 2004. Interactive color palette tools. Computer Graphics and Applications 24, 3.
19. Merrell, P., Schkufza, E., Li, Z., Agrawala, M., and Koltun, V. 2011. Interactive furniture layout using interior design guidelines. In Proc. SIGGRAPH 2011.
20. Metropolis, N., Rosenbluth, A. W., Rosenbluth, M. N., Teller, A. H., and Teller, E. 1953. Equation of State Calculations by Fast Computing Machines. The Journal of Chemical Physics 21, 6.
21. Munsell, A., and Birren, F. 1969. A Grammar of Color: A Basic Treatise on the Color System of Albert H. Munsell. Van Nostrand Reinhold.
22. Neal, R. M. 2010. Mcmc using hamiltonian dynamics. In Handbook of Markov Chain Monte-Carlo (Steve Brooks, Andrew Gelman, Gailin Jones and Xiao-Li Meng, Eds.
23. Nguyen, C. H., Ritschel, T., Myszkowski, K., Eisemann, E., and Seidel, H.-P. 2012. 3d material style transfer. Computer Graphics Forum.
24. O'Donovan, P., Agarwala, A., and Hertzmann, A. 2011. Color Compatibility From Large Datasets. ACM Transactions on Graphics 30, 4.
25. Palmer, S. E., and Schloss, K. B. 2010. An ecological valence theory of human color preference. Proceedings of the National Academy of Sciences 107, 19, 8877--8882.
26. Robert, C. P. 1995. Simulation of truncated normal variables. Statistics and Computing 5.
27. Sauvaget, C., Manuel, S., Vittaut, J.-N., Suarez, J., and Boyer, V. 2010. Segmented images colorization using harmony. In Proc. of Signal-Image Technology and Internet-Based Systems (SITIS) 2010, vol. 1.
28. Sharma, G., Wu, W., and Dalal, E. N. 2005. The CIEDE2000 color-difference formula: implementation notes, supplementary test data, and mathematical observations. Color research and application 30, 1.
29. Sutton, T., and Whelan, B. 2004. The Complete Color Harmony: Expert Color Information for Professional Color Results. Color Harmony Series. Quayside Publishing Group.
30. Wang, B., Yu, Y., Wong, T.-T., Chen, C., and Xu, Y.-Q. 2010. Data-driven image color theme enhancement. In Proc. SIGGRAPH Asia 2010.
31. Welsh, T., Ashikhmin, M., and Mueller, K. 2002. Transferring color to greyscale images. In Proc. SIGGRAPH 2002.
32. Yeh, Y.-T., Breeden, K., Yang, L., Fisher, M., and Hanrahan, P. 2012. Synthesis of tiled patterns using factor graphs. ACM Transactions on Graphics.
33. Yeh, Y.-T., Yang, L., Watson, M., Goodman, N. D., and Hanrahan, P. 2012. Synthesizing open worlds with constraints using locally annealed reversible jump mcmc. In Proc. SIGGRAPH 2012.
34. Yu, L.-F., Yeung, S.-K., Terzopoulos, D., and Chan, T. F. 2012. Dressup!: outfit synthesis through automatic optimization. In Proc. SIGGRAPH Asia 2012.