Benito Buchheim
ABSTRACT
We present StyleTune, a mobile app for interactive style transfer enhancement that enables global and spatial control over stroke elements and can generate high fidelity outputs. The app uses adjustable neural style transfer (NST) networks to enable art-direction of stroke size and orientation in the output image. The implemented approach enables continuous and seamless edits through a unified stroke-size representation in the feature space of the style transfer network. StyleTune introduces a three-stage user interface, that enables users to first explore global stroke parametrizations for a chosen NST. They can then interactively locally retouch the stroke size and orientation using brush metaphors. Finally, high resolution outputs of 20 Megapixels and more can be obtained using a patch-based upsampling and local detail transfer approach, that transfers small-scale details such as paint-bristles and canvas structure. The app uses Apple’s CoreML and Metal APIs for efficient on-device processing.
Supplemental Material
- Leon A. Gatys, Alexander S. Ecker, and Matthias Bethge. 2016. Image Style Transfer Using Convolutional Neural Networks. In Proc. CVPR. IEEE Computer Society, Los Alamitos, 2414–2423. https://doi.org/10.1109/CVPR.2016.265Google Scholar
Cross Ref
- Yongcheng Jing, Yang Liu, Yezhou Yang, Zunlei Feng, Yizhou Yu, and Mingli Song. 2018. Stroke Controllable Fast Style Transfer with Adaptive Receptive Fields. CoRR abs/1802.07101(2018), 244–260. arxiv:1802.07101http://arxiv.org/abs/1802.07101Google Scholar
- Justin Johnson, Alexandre Alahi, and Li Fei-Fei. 2016. Perceptual Losses for Real-Time Style Transfer and Super-Resolution. In Proc. ECCV. Springer International, Cham, Switzerland, 694–711. https://doi.org/10.1007/978-3-319-46475-6_43Google Scholar
- Sebastian Pasewaldt, Amir Semmo, Jürgen Döllner, and Frank Schlegel. 2016. Becasso: Artistic Image Processing and Editing on Mobile Devices. In Proc. MGIA (Macau). ACM, New York, NY, USA, 14:1–14:1. https://doi.org/10.1145/2999508.2999518Google Scholar
- Max Reimann, Mandy Klingbeil, Sebastian Pasewaldt, Amir Semmo, Matthias Trapp, and Jürgen Döllner. 2019. Locally controllable neural style transfer on mobile devices. The Visual Computer 35, 11 (2019), 1531–1547. https://doi.org/10.1007/s00371-019-01654-1Google ScholarCross Ref
- Ondřej Texler, Jakub Fišer, Michal Lukáč, Jingwan Lu, Eli Shechtman, and Daniel Sýkora. 2019. Enhancing Neural Style Transfer using Patch-Based Synthesis. In Proceedings of the 8th ACM/EG Expressive Symposium. Eurographics Association, Goslar, DEU, 43–50.Google Scholar
Index Terms
StyleTune: Interactive Style Transfer Enhancement on Mobile Devices
Recommendations
MaeSTrO: mobile style transfer orchestration using adaptive neural networks
SIGGRAPH '18: ACM SIGGRAPH 2018 Appy HourWe present MaeSTrO, a mobile app for image stylization that empowers users to direct, edit and perform a neural style transfer with creative control. The app uses iterative style transfer, multi-style generative and adaptive networks to compute and ...
Conditional Fast Style Transfer Network
ICMR '17: Proceedings of the 2017 ACM on International Conference on Multimedia RetrievalIn this paper, we propose a conditional fast neural style transfer network. We extend the network proposed as a fast neural style transfer network by Johnson et al. [1] so that the network can learn multiple styles at the same time. To do that, we add a ...
Approaches for local artistic control of mobile neural style transfer
Expressive '18: Proceedings of the Joint Symposium on Computational Aesthetics and Sketch-Based Interfaces and Modeling and Non-Photorealistic Animation and RenderingThis work presents enhancements to state-of-the-art adaptive neural style transfer techniques, thereby providing a generalized user interface with creativity tool support for lower-level local control to facilitate the demanding interactive editing on ...
Comments