MaeSTrO: mobile style transfer orchestration using adaptive neural networks

ABSTRACT

We present MaeSTrO, a mobile app for image stylization that empowers users to direct, edit and perform a neural style transfer with creative control. The app uses iterative style transfer, multi-style generative and adaptive networks to compute and apply flexible yet comprehensive style models of arbitrary images at run-time. Compared to other mobile applications, MaeSTrO introduces an interactive user interface that empowers users to orchestrate style transfers in a two-stage process for an individual visual expression: first, initial semantic segmentation of a style image can be complemented by on-screen painting to direct sub-styles in a spatially-aware manner. Second, semantic masks can be virtually drawn on top of a content image to adjust neural activations within local image regions, and thus direct the transfer of learned sub-styles. This way, the general feed-forward neural style transfer is evolved towards an interactive tool that is able to consider composition variables and mechanisms of general artwork production, such as color, size and location-based filtering. MaeSTrO additionally enables users to define new styles directly on a device and synthesize high-quality images based on prior segmentations via a service-based implementation of compute-intensive iterative style transfer techniques.

References

1. Kapil Dev. 2013. Mobile Expressive Renderings: The State of the Art. IEEE Computer Graphics and Applications 33, 3 (May/June 2013), 22--31.
2. Leon A. Gatys, Alexander S. Ecker, and Matthias Bethge. 2016. Image Style Transfer Using Convolutional Neural Networks. In Proc. CVPR. IEEE Computer Society, Los Alamitos, 2414--2423.
3. Xun Huang and Serge Belongie. 2017. Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization. arXiv.org report 1703.06868. arXiv. https://arxiv.org/abs/1703.06868
4. Justin Johnson, Alexandre Alahi, and Li Fei-Fei. 2016. Perceptual Losses for Real-Time Style Transfer and Super-Resolution. In Proc. ECCV. Springer International, Cham, Switzerland, 694--711.
5. Fujun Luan, Sylvain Paris, Eli Shechtman, and Kavita Bala. 2017. Deep Photo Style Transfer. CoRR abs/1703.07511. arXiv. http://arxiv.org/abs/1703.07511
6. Amir Semmo, Tobias Isenberg, and Jürgen Döllner. 2017a. Neural Style Transfer: A Paradigm Shift for Image-based Artistic Rendering?. In Proc. NPAR, Holger Winnemöller and Lyn Bartram (Eds.). ACM, New York, 5:1--5:13.
7. Amir Semmo, Matthias Trapp, Jürgen Döllner, and Mandy Klingbeil. 2017b. Pictory: Combining Neural Style Transfer and Image Filtering. In Proc. SIGGRAPH Appy Hour. ACM, New York, NY, USA, 5:1--5:2.
8. Hang Zhang and Kristin Dana. 2017. Multi-style Generative Network for Real-time Transfer. arXiv.org report 1703.06953. arXiv. https://arxiv.org/abs/1703.06953

Index Terms

• Computing methodologies

  • Computer graphics

    • Image manipulation

      • Image processing

    • Rendering

      • Non-photorealistic rendering