Pseudo-3D Scene Modeling for Virtual Reality Using Stylized Novel View Synthesis

ABSTRACT

Stylized Novel View Synthesis is an emerging technique that combines style transfer and view synthesis. However, none of the existing works explore their applications in Virtual Reality (VR). This work devises a novel application for stylized novel view synthesis. We propose to replace actual 3D scene models or 360 images with stylized stereoscopic images for the areas outside the major play area but are still visible to the user. User study results reveal that users can feel 3D sense and tell them from plane texture. Codes and other materials are available at: kuan-wei-tseng.github.io/ArtNV

References

1. Pei-Ze Chiang, Meng-Shiun Tsai, Hung-Yu Tseng, Wei-Sheng Lai, and Wei-Chen Chiu. 2022. Stylizing 3D Scene via Implicit Representation and HyperNetwork. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). 1475–1484.
2. Leon A. Gatys, Alexander S. Ecker, and Matthias Bethge. 2016. Image Style Transfer Using Convolutional Neural Networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
3. Hsin-Ping Huang, Hung-Yu Tseng, Saurabh Saini, Maneesh Singh, and Ming-Hsuan Yang. 2021. Learning to Stylize Novel Views. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV).
4. Fangzhou Mu, Jian Wang, Yicheng Wu, and Yin Li. 2022. 3D Photo Stylization - Learning to Generate Stylized Novel Views from a Single Image. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
5. Zachary Teed and Jia Deng. 2020. Raft: Recurrent all-pairs field transforms for optical flow. In Proceedings of the European Conference on Computer Vision (ECCV). Springer, 402–419.
6. Kuan-Wei Tseng, Yao-Chih Lee, and Chu-Song Chen. 2022. Artistic Style Novel View Synthesis Based on A Single Image. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops.
7. Olivia Wiles, Georgia Gkioxari, Richard Szeliski, and Justin Johnson. 2020. SynSin: End-to-End View Synthesis From a Single Image. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).