Creating and chaining camera moves for quadrotor videography

Abstract

Capturing aerial videos with a quadrotor-mounted camera is a challenging creative task, as it requires the simultaneous control of the quadrotor's motion and the mounted camera's orientation. Letting the drone follow a pre-planned trajectory is a much more appealing option, and recent research has proposed a number of tools designed to automate the generation of feasible camera motion plans; however, these tools typically require the user to specify and edit the camera path, for example by providing a complete and ordered sequence of key viewpoints.

In this paper, we propose a higher level tool designed to enable even novice users to easily capture compelling aerial videos of large-scale outdoor scenes. Using a coarse 2.5D model of a scene, the user is only expected to specify starting and ending viewpoints and designate a set of landmarks, with or without a particular order. Our system automatically generates a diverse set of candidate local camera moves for observing each landmark, which are collision-free, smooth, and adapted to the shape of the landmark. These moves are guided by a landmark-centric view quality field, which combines visual interest and frame composition. An optimal global camera trajectory is then constructed that chains together a sequence of local camera moves, by choosing one move for each landmark and connecting them with suitable transition trajectories. This task is formulated and solved as an instance of the Set Traveling Salesman Problem.

References

1. Helmut Alt and Emo Welzl. 1988. Visibility graphs and obstacle-avoiding shortest paths. Mathematical Methods of Operations Research 32, 3 (1988), 145--164.
2. Olov Andersson, Mariusz Wzorek, and Patrick Doherty. 2017. Deep Learning Quad-copter Control via Risk-Aware Active Learning. In Proc. AAAI Conf. on Artificial Intelligence, Vol. 5. 3812--3818.
3. Carlos Gran Andújar, Pere Pau Alcocer Vázquez, and Marta González Fairén. 2004. Way-Finder: Guided Tours Through Complex Walkthrough Models. Computer Graphics Forum (Proc. of Eurographics) 23, 3 (2004), 499--508.
4. William H. Bares, Somying Thainimit, and Scott McDermott. 2000. A Model for Constraint-based Camera Planning. In Smart Graphics AAAI Spring Symposium. 84--91.
5. Jim Blinn. 1988. Where am I? What am I looking at? IEEE Computer Graphics and Applications 8, 4 (1988), 76--81.
6. Luca Chittaro, Roberto Ranon, and Lucio Ieronutti. 2003. Guiding Visitors of Web3D Worlds Through Automatically Generated Tours. In Proc. Conf. on 3D Web Technology. 27--38.
7. Marc Christie and Patrick Olivier. 2009. Camera control in computer graphics: models, techniques and applications. In ACM SIGGRAPH ASIA 2009 Courses.
8. Marc Christie, Patrick Olivier, and Jean-Marie Normand. 2008. Camera control in computer graphics. Computer Graphics Forum 27, 8 (2008), 2197--2218.
9. T. J. Diaz. 2015. Lights, drone... action. IEEE Spectrum 52, 7 (2015), 36--41.
10. Steven M Drucker and David Zeltzer. 1994. Intelligent camera control in a virtual environment. In Proc. of Graphics Interface. 190--190.
11. Oliver Dunkley, Jakob Engel, Jürgen Sturm, and Daniel Cremers. 2014. Visual-Inertial Navigation for a Camera-Equipped 25g Nano-Quadrotor. In Aerial Open Source Robotics Workshop.
12. Niklas Elmqvist, M. Eduard Tudoreanu, and Philippas Tsigas. 2007. Tour generation for exploration of 3D virtual environments. In Proc. ACM symposium on Virtual reality software and technology. 207--210.
13. Jiankun Fan. 2014. Optimal path planning and control of quadrotor unmanned aerial vehicle for area coverage. Ph.D. Dissertation. The University of Toledo.
14. Julien Fleureau, Quentin Galvane, Francois-Louis Tariolle, and Philippe Guillotel. 2016. Generic Drone Control Platform for Autonomous Capture of Cinema Scenes. In Proc. Workshop on Micro Aerial Vehicle Networks, Systems, and Applications for Civilian Use. 35--40.
15. Quentin Galvane, Marc Christie, Rémi Ronfard, Chen-Kim Lim, and Marie-Paule Cani. 2013. Steering behaviors for autonomous cameras. In Proc. Motion on Games. 93--102.
16. Quentin Galvane, Julien Fleureau, Francois-Louis Tariolle, and Philippe Guillotel. 2016. Automated Cinematography with Unmanned Aerial Vehicles. In Proc. Eurographics Workshop on Intelligent Cinematography and Editing. 23--30.
17. Christoph Gebhardt, Benjamin Hepp, Tobias Nägeli, Stefan Stevšić, and Otmar Hilliges. 2016. Airways: Optimization-Based Planning of Quadrotor Trajectories According to High-Level User Goals. In Proc. CHI Conf. on Human Factors in Computing Systems. 2508--2519.
18. Nicolas Halper, Ralf Helbing, and Thomas Strothotte. 2001. A Camera Engine for Computer Games: Managing the Trade-Off Between Constraint Satisfaction and Frame Coherence. Computer Graphics Forum (Proc. of Eurographics) 20 (2001), 174--183. Issue 3.
19. Li-Wei He, Michael F Cohen, and David H Salesin. 1996. The virtual cinematographer: a paradigm for automatic real-time camera control and directing. In Proc. of SIGGRAPH. 217--224.
20. Keld Helsgaun. 2015. Solving the equality generalized traveling salesman problem using the Lin-Kernighan-Helsgaun algorithm. Mathematical Programming Computation 7, 3 (2015), 269--287.
21. Wei-Hsien Hsu, Yubo Zhang, and Kwan-Liu Ma. 2013. A Multi-Criteria Approach to Camera Motion Design for Volume Data Animation. IEEE Trans. Visualization & Computer Graphics 19, 12 (2013), 2792--2801.
22. Niels Joubert, L. E. Jane, Dan B. Goldman, Floraine Berthouzoz, Mike Roberts, James A. Landay, and Pat Hanrahan. 2016. Towards a Drone Cinematographer: Guiding Quadrotor Cameras using Visual Composition Principles. ArXiv e-prints (Oct. 2016). arXiv:cs.GR/1610.01691
23. Niels Joubert, Mike Roberts, Anh Truong, Floraine Berthouzoz, and Pat Hanrahan. 2015. An interactive tool for designing quadrotor camera shots. ACM Trans. on Graphics (Proc. of SIGGRAPH Asia) 34, 6 (2015), 238:1--238:11.
24. Tsai-Yen Li and Chung-Chiang Cheng. 2008. Real-Time Camera Planning for Navigation in Virtual Environments. In Smart Graphics. Lecture Notes in Computer Science, Vol. 5166. Springer Berlin Heidelberg.
25. Christophe Lino and Marc Christie. 2015. Intuitive and efficient camera control with the toric space. ACM Trans. on Graphics (Proc. of SIGGRAPH) 34, 4 (2015), 82:1--82:12.
26. Christophe Lino, Marc Christie, Fabrice Lamarche, Guy Schofield, and Patrick Olivier. 2010. A Real-time Cinematography System for Interactive 3D Environments. In Proc. ACM SIGGRAPH/Eurographics Symp. on Computer Animation. 139--148.
27. Daniel Mellinger and Vijay Kumar. 2011. Minimum snap trajectory generation and control for quadrotors. In Proc. IEEE Int. Conf. on Robotics & Automation. 2520--2525.
28. A. Messina et al. 2017. Multidrone media production requirements. Technical Report. University of Bristol.
29. Tobias Nägeli, Lukas Meier, Alexander Domahidi, Javier Alonso-Mora, and Otmar Hilliges. 2017. Real-time Planning for Automated Multi-view Drone Cinematography. ACM Trans. on Graphics (Proc. of SIGGRAPH) 36, 4 (2017), 132:1--132:10.
30. Dennis Nieuwenhuisen and Mark H. Overmars. 2004. Motion Planning for Camera Movements. In Proc. IEEE Int. Conf. on Robotics & Automation, Vol. 4. 3870--3876.
31. Thomas Oskam, Robert W Sumner, Nils Thuerey, and Markus Gross. 2009. Visibility transition planning for dynamic camera control. In Proc. ACM SIGGRAPH/Eurographics Symp. on Computer Animation. 55--65.
32. Roberto Ranon and Tommaso Urli. 2014. Improving the Efficiency of Viewpoint Composition. IEEE Trans. Visualization & Computer Graphics 20, 5 (2014), 795--807.
33. Charles Richter, Adam Bry, and Nicholas Roy. 2016. Polynomial Trajectory Planning for Aggressive Quadrotor Flight in Dense Indoor Environments. Robotics Research 114 (2016), 649--666.
34. Mike Roberts and Pat Hanrahan. 2016. Generating Dynamically Feasible Trajectories for Quadrotor Cameras. ACM Trans. on Graphics (Proc. of SIGGRAPH) 35, 4 (2016), 61:1--61:11.
35. Brian Salomon, Maxim Garber, Ming Lin, and Dinesh Manocha. 2003. Interactive navigation in complex environments using path planning. In Proc. Sym. on Interactive 3D Graphics. 41--50.
36. Ekrem Serin, Serdar Hasan Adali, and Selim Balcisoy. 2012. Automatic path generation for terrain navigation. Computers & Graphics 36, 8 (2012), 1013--1024.
37. Dmitry Sokolov and Dimitri Plemenos. 2008. Virtual world explorations by using topological and semantic knowledge. The Visual Computer 24, 3 (2008), 173--185.
38. Pere-Pau Vázquez, Miquel Feixas, Mateu Sbert, and Wolfgang Heidrich. 2001. Viewpoint Selection using Viewpoint Entropy. In Proc. of Vision Modeling and Visualization Conference, Vol. 1. 273--280.
39. Georgios N. Yannakakis and John Hallam. 2011. Rating vs. Preference: a comparative study of self-reporting. In Proc. Int. Conf. on Affective Computing and Intelligent Interaction. 437--446.

Index Terms

• Computing methodologies

  • Artificial intelligence

    • Computer vision

      • Computer vision problems

        • Interest point and salient region detections

  • Computer graphics

    • Graphics systems and interfaces