Soheil Darabi
Dan B. Goldman
Abstract
Current methods for combining two different images produce visible artifacts when the sources have very different textures and structures. We present a new method for synthesizing a transition region between two source images, such that inconsistent color, texture, and structural properties all change gradually from one source to the other. We call this process image melding. Our method builds upon a patch-based optimization foundation with three key generalizations: First, we enrich the patch search space with additional geometric and photometric transformations. Second, we integrate image gradients into the patch representation and replace the usual color averaging with a screened Poisson equation solver. And third, we propose a new energy based on mixed L2/L0 norms for colors and gradients that produces a gradual transition between sources without sacrificing texture sharpness. Together, all three generalizations enable patch-based solutions to a broad class of image melding problems involving inconsistent sources: object cloning, stitching challenging panoramas, hole filling from multiple photos, and image harmonization. In several cases, our unified method outperforms previous state-of-the-art methods specifically designed for those applications.
Supplemental Material
Available for Download
Supplemental material.
- Adobe, 2010. Photoshop cs5 content-aware fill. http://www.adobe.com/technology/projects/content-aware-fill.html.Google Scholar
- Agarwala, A., Dontcheva, M., Agrawala, M., Drucker, S., Colburn, A., Curless, B., Salesin, D., and Cohen, M. 2004. Interactive digital photomontage. In ACM SIGGRAPH, vol. 23, 294--302. Google ScholarDigital Library
- Arias, P., Facciolo, G., Caselles, V., and Sapiro, G. 2011. A variational framework for exemplar-based image in-painting. IJCV 93 (July), 319--347. Google ScholarDigital Library
- Barnes, C., Shechtman, E., Finkelstein, A., and Goldman, D. B. 2009. PatchMatch: A randomized correspondence algorithm for structural image editing. In ACM SIGGRAPH, vol. 28, 24:1--24:11. Google ScholarDigital Library
- Barnes, C., Shechtman, E., Goldman, D. B., and Finkelstein, A. 2010. The Generalized PatchMatch correspondence algorithm. In ECCV. Google ScholarDigital Library
- Bhat, P., Curless, B., Cohen, M., and Zitnick, L. 2008. Fourier analysis of the 2D screened Poisson equation for gradient domain problems. In ECCV. Google ScholarDigital Library
- Bhat, P., Zitnick, C. L., Cohen, M., and Curless, B. 2010. Gradientshop: A gradient-domain optimization framework for image and video filtering. ACM Trans. Graphics 29 (April), 10:1--10:14. Google ScholarDigital Library
- Bugeau, A., Bertalmío, M., Caselles, V., and Sapiro, G. 2010. A comprehensive framework for image inpainting. IEEE Trans. on Image Processing 19, 10 (oct.), 2634--2645. Google ScholarDigital Library
- Burt, P. J., and Adelson, E. H. 1983. A multiresolution spline with application to image mosaics. ACM Trans. Graphics 2 (October), 217--236. Google ScholarDigital Library
- Candes, E., Rudelson, M., Tao, T., and Vershynin, R. 2005. Error correction via linear programming. In IEEE Symposium on Foundations of Computer Science, 668--681.Google Scholar
- Efros, A. A., and Leung, T. K. 1999. Texture synthesis by non-parametric sampling. IEEE Computer Society, Los Alamitos, CA, USA.Google Scholar
- Fang, H., and Hart, J. C. 2007. Detail preserving shape deformation in image editing. In ACM SIGGRAPH, vol. 26, 1--5. Google ScholarDigital Library
- Farbman, Z., Fattal, R., and Lischinski, D. 2011. Convolution pyramids. In ACM SIGGRAPH Asia, vol. 30, 175:1--175:8. Google Scholar
- HaCohen, Y., Shechtman, E., Goldman, D. B., and Lischinski, D. 2011. Non-rigid dense correspondence with applications for image enhancement. In ACM SIGGRAPH, vol. 30, 70:1--70:10. Google ScholarDigital Library
- Hays, J., and Efros, A. A. 2007. Scene completion using millions of photographs. In ACM SIGGRAPH, vol. 26, 4:1--4:7. Google ScholarDigital Library
- Kaneva, B., Sivic, J., Torralba, A., Avidan, S., and Freeman, W. T. 2010. Infinite images: Creating and exploring a large photorealistic virtual space. In Proceedings of the IEEE.Google Scholar
- Kwatra, V., Schödl, A., Essa, I., Turk, G., and Bobick, A. 2003. Graphcut textures: image and video synthesis using graph cuts. In ACM SIGGRAPH, vol. 22, 277--286. Google ScholarDigital Library
- Kwatra, V., Essa, I., Bobick, A., and Kwatra, N. 2005. Texture optimization for example-based synthesis. In ACM SIGGRAPH, vol. 24, 795--802. Google ScholarDigital Library
- Lin, W.-Y., Liu, S., Matsushita, Y., Ng, T.-T., and Cheong, L.-F. 2011. Smoothly varying affine stitching. In CVPR.Google Scholar
- Mansfield, A., Prasad, M., Rother, C., Sharp, T., Kohli, P., and Van Gool, L. 2011. Transforming image completion. In Proc. BMVC.Google ScholarCross Ref
- Pérez, P., Gangnet, M., and Blake, A. 2003. Poisson image editing. In ACM SIGGRAPH, vol. 22, 313--318. Google ScholarDigital Library
- Pritch, Y., Kav-Venaki, E., and Peleg, S. 2009. Shift-map image editing. In ICCV.Google Scholar
- Rother, C., Bordeaux, L., Hamadi, Y., and Blake, A. 2006. Autocollage. In ACM SIGGRAPH, vol. 25, 847--852. Google ScholarDigital Library
- Ruiters, R., Schnabel, R., and Klein, R. 2010. Patch-based texture interpolation. Computer Graphics Forum 29, 4 (June), 1421--1429. Google ScholarDigital Library
- Shechtman, E., Rav-Acha, A., Irani, M., and Seitz, S. 2010. Regenerative morphing. In CVPR.Google Scholar
- Simakov, D., Caspi, Y., Shechtman, E., and Irani, M. 2008. Summarizing visual data using bidirectional similarity. In CVPR.Google Scholar
- Sunkavalli, K., Johnson, M. K., Matusik, W., and Pfister, H. 2010. Multi-scale image harmonization. In ACM SIGGRAPH, vol. 29, 125:1--125:10. Google ScholarDigital Library
- Szeliski, R., and Shum, H.-Y. 1997. Creating full view panoramic image mosaics and environment maps. In ACM SIGGRAPH, 251--258. Google ScholarDigital Library
- Tappen, M., Freeman, W., and Adelson, E. 2005. Recovering intrinsic images from a single image. IEEE Trans. PAMI 27, 9 (sept.), 1459--1472. Google ScholarDigital Library
- Tropp, J., and Gilbert, A. 2007. Signal recovery from random measurements via orthogonal matching pursuit. IEEE Trans. Information Theory 53, 12 (dec.), 4655--4666. Google ScholarDigital Library
- Wei, L. Y., and Levoy, M. 2000. Fast texture synthesis using tree-structured vector quantization. In ACM SIGGRAPH, 479--488. Google ScholarDigital Library
- Wexler, Y., Shechtman, E., and Irani, M. 2007. Space-time completion of video. IEEE Trans. PAMI 29, 3 (march), 463--476. Google ScholarDigital Library
- Whyte, O., Sivic, J., and Zisserman, A. 2009. Get out of my picture! internet-based inpainting. In BMVC.Google Scholar
- Xu, L., Lu, C., Xu, Y., and Jia, J. 2011. Image smoothing via L0 gradient minimization. In ACM SIGGRAPH Asia, vol. 30, 174:1--174:12. Google Scholar
Index Terms
Image melding: combining inconsistent images using patch-based synthesis
Recommendations
Image completion based on views of large displacement
This paper presents an algorithm for image completion based on the views of large displacement. A distinct from most existing image completion methods, which exploit only the target image’s own information to complete the damaged regions, our algorithm ...
On texture and image interpolation using Markov models
Markov-type models characterize the correlation among neighboring pixels in an image in many image processing applications. Specifically, a wide-sense Markov model, which is defined in terms of minimum linear mean-square error estimates, is applicable ...
Image completion with variable scope patch sampling
ICAIR-CACRE '16: Proceedings of the International Conference on Artificial Intelligence and Robotics and the International Conference on Automation, Control and Robotics EngineeringThis paper presents a simple but effective way of patch sampling for image completion which changes the assignment scope in different pixels when sampling patches during completion. In each pixel of missing regions, the scope expands from narrow to wide ...
Comments