Toward wave-based sound synthesis for computer animation

Abstract

We explore an integrated approach to sound generation that supports a wide variety of physics-based simulation models and computer-animated phenomena. Targeting high-quality offline sound synthesis, we seek to resolve animation-driven sound radiation with near-field scattering and diffraction effects. The core of our approach is a sharp-interface finite-difference time-domain (FDTD) wavesolver, with a series of supporting algorithms to handle rapidly deforming and vibrating embedded interfaces arising in physics-based animation sound. Once the solver rasterizes these interfaces, it must evaluate acceleration boundary conditions (BCs) that involve model-and phenomena-specific computations. We introduce acoustic shaders as a mechanism to abstract away these complexities, and describe a variety of implementations for computer animation: near-rigid objects with ringing and acceleration noise, deformable (finite element) models such as thin shells, bubble-based water, and virtual characters. Since time-domain wave synthesis is expensive, we only simulate pressure waves in a small region about each sound source, then estimate a far-field pressure signal. To further improve scalability beyond multi-threading, we propose a fully time-parallel sound synthesis method that is demonstrated on commodity cloud computing resources. In addition to presenting results for multiple animation phenomena (water, rigid, shells, kinematic deformers, etc.) we also propose 3D automatic dialogue replacement (3DADR) for virtual characters so that pre-recorded dialogue can include character movement, and near-field shadowing and scattering sound effects.

References

1. T. Akenine-Möller. 2002. Fast 3D Triangle-box Overlap Testing. J. Graph. Tools 6, 1 (2002).
2. A. Allen and N. Raghuvanshi. 2015. Aerophones in Flatland: Interactive Wave Simulation of Wind Instruments. ACM Transactions on Graphics (Proceedings of SIGGRAPH 2015) 34, 4 (Aug. 2015).
3. S. S. An, D. L. James, and S. Marschner. 2012. Motion-driven Concatenative Synthesis of Cloth Sounds. ACM Transactions on Graphics (SIGGRAPH 2012) (Aug. 2012).
4. Avid Technology. 2018. Pro Tools. (2018). http://www.avid.com/pro-tools.
5. D. R. Begault. 1994. 3-D Sound for Virtual Reality and Multimedia. Academic Press Professional, Cambridge, MA.
6. S. Bilbao. 2009. Numerical Sound Synthesis: Finite Difference Schemes and Simulation in Musical Acoustics. John Wiley and Sons.
7. S. Bilbao. 2011. Time domain simulation and sound synthesis for the snare drum. J. Acoust. Soc. Am. 131, 1 (2011).
8. Stefan Bilbao. 2013. Modeling of complex geometries and boundary conditions in finite different/finite volume time domain room acoustics simulation. IEEE Transactions on Audio, Speech, and Language Processing 21 (2013).
9. S. Bilbao and C. J. Webb. 2013. Physical modeling of timpani drums in 3D on GPGPUs. Journal of the Audio Engineering Society 61, 10 (2013), 737--748.
10. N. Bonneel, G. Drettakis, N. Tsingos, I. Viaud-Delmon, and D. James. 2008. Fast Modal Sounds with Scalable Frequency-Domain Synthesis. ACM Transactions on Graphics 27, 3 (Aug. 2008), 24:1--24:9.
11. D. Botteldooren. 1994. Acoustical finite-difference time-domain simulation in a quasi-cartesian grid. Journal of the Acoustical Society of America 95 (1994).
12. D. Botteldooren. 1997. Time-domain simulation of the influence of close barriers on sound propagation to the environment. The Journal of the Acoustical Society of America 101, 3 (1997), 1278--1285.
13. J. N. Chadwick, S. S. An, and D. L. James. 2009. Harmonic Shells: A Practical Nonlinear Sound Model for Near-Rigid Thin Shells. ACM Transactions on Graphics (Aug. 2009).
14. J. N. Chadwick and D. L. James. 2011. Animating Fire with Sound. ACM Transactions on Graphics 30, 4 (Aug. 2011).
15. J. N. Chadwick, C. Zheng, and D. L. James. 2012a. Faster Acceleration Noise for Multi-body Animations using Precomputed Soundbanks. ACM Eurographics Symposium on Computer Animation (2012).
16. J. N. Chadwick, C. Zheng, and D. L. James. 2012b. Precomputed Acceleration Noise for Improved Rigid-Body Sound. ACM Transactions on Graphics (Proceedings of SIGGRAPH 2012) 31, 4 (Aug. 2012).
17. A. Chaigne, C. Touzé, and O. Thomas. 2005. Nonlinear vibrations and chaos in gongs and cymbals. Acoustical science and technology 26, 5 (2005), 403--409.
18. A. Chandak, C. Lauterbach, M. Taylor, Z. Ren, and D. Manocha. 2008. Ad-frustum: Adaptive frustum tracing for interactive sound propagation. IEEE Transactions on Visualization and Computer Graphics 14, 6 (2008), 1707--1722.
19. G. Cirio, D. Li, E. Grinspun, Mi. A. Otaduy, and C. Zheng. 2016. Crumpling sound synthesis. ACM Transactions on Graphics (TOG) 35, 6 (2016), 181.
20. R. Clayton and B. Engquist. 1977. Absorbing boundary conditions for acoustic and elastic wave equations. Bulletin of the Seismological Society of America 67, 6 (1977), 1529.
21. M. Cook. 2015. Pixar, 'The Road to Point Reyes' and the long history of landscape in new visual technologies. (2015).
22. P. R. Cook. 2002. Sound Production and Modeling. IEEE Computer Graphics & Applications 22, 4 (July/Aug. 2002), 23--27.
23. R. L. Cook, L. Carpenter, and E. Catmull. 1987. The Reyes Image Rendering Architecture. In Proceedings of the 14th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH '87). ACM, New York, NY, USA, 95--102.
24. M. Ducceschi and C. Touzé. 2015. Modal approach for nonlinear vibrations of damped impacted plates: Application to sound synthesis of gongs and cymbals. Journal of Sound and Vibration 344 (2015), 313--331.
25. B. Engquist and A. Majda. 1977. Absorbing boundary conditions for numerical simulation of waves. Proceedings of the National Academy of Sciences 74, 5 (1977), 1765--1766.
26. Ronald P Fedkiw, Tariq Aslam, Barry Merriman, and Stanley Osher. 1999. A Non-oscillatory Eulerian Approach to Interfaces in Multimaterial Flows (the Ghost Fluid Method). J. Comput. Phys. 152, 2 (1999), 457 -- 492.
27. T. Funkhouser, I. Carlbom, G. Elko, G. Pingali, M. Sondhi, and J. West. 1998. A Beam Tracing Approach to Acoustic Modeling for Interactive Virtual Environments. In Proceedings of SIGGRAPH 98 (Computer Graphics Proceedings, Annual Conference Series). 21--32.
28. T. A. Funkhouser, P. Min, and I. Carlbom. 1999. Real-Time Acoustic Modeling for Distributed Virtual Environments. In Proceedings of SIGGRAPH 99 (Computer Graphics Proceedings, Annual Conference Series). 365--374.
29. W. W. Gaver. 1993. Synthesizing auditory icons. In Proceedings of the TNTERACT'93 and CHI'93 conference on Human factors in computing systems. ACM, 228--235.
30. Y. I. Gingold, A. Secord, J. Y. Han, E. Grinspun, and D. Zorin. 2004. A Discrete Model for Inelastic Deformation of Thin Shells.
31. G. Guennebaud, B.Jacob, et al. 2010. Eigen v3. http://eigen.tuxfamily.org. (2010).
32. Jon Häggblad and Björn Engquist. 2012. Consistent modeling of boundaries in acoustic finite-difference Time-domain simulations. Journal of the Acoustical Society of America 132 (2012).
33. P. S. Heckbert. 1987. Ray tracing Jell-O brand gelatin. In ACM SIGGRAPH Computer Graphics, Vol. 21. ACM, 73--74.
34. A. Jacobson, D. Panozzo, et al. 2017. libigl: A simple C++ geometry processing library. (2017). http://libigl.github.io/libigl/.
35. D. L. James, J. Barbie, and D. K. Pai. 2006. Precomputed Acoustic Transfer: Output-sensitive, accurate sound generation for geometrically complex vibration sources. ACM Transactions on Graphics 25, 3 (July 2006), 987--995.
36. D. L. James and D. K. Pai. 2002. DyRT: Dynamic Response Textures for Real Time Deformation Simulation with Graphics Hardware. ACM Trans. Graph. 21, 3 (July 2002), 582--585.
37. M. Kleiner, B.-I. Dalenbäck, and P. Svensson. 1993. Auralization-An Overview. J. Audio Engineering Society 41 (1993), 861--861. Issue 11.
38. D. Komatitsch, G. Erlebacher, D. Göddeke, and D. Michéa. 2010. High-order finite-element seismic wave propagation modeling with MPI on a large GPU cluster. Journal of computational physics 229, 20 (2010), 7692--7714.
39. T. R. Langlois, S. S. An, K. K. Jin, and D. L. James. 2014. Eigenmode Compression for Modal Sound Models. ACM Trans. Graph. 33, 4, Article 40 (July 2014), 9 pages.
40. T. R Langlois and D. L James. 2014. Inverse-foley animation: Synchronizing rigid-body motions to sound. ACM Transactions on Graphics (TOG) 33, 4 (2014), 41.
41. T. R. Langlois, C. Zheng, and D. L. James. 2016. Toward Animating Water with Complex Acoustic Bubbles. ACM Trans. Graph. 35, 4, Article 95 (July 2016), 13 pages.
42. S. Larsson and V Thomée. 2009. Partial Differential Equations with Numerical Methods. Springer.
43. Q.-H. Liu and J. Tao. 1997. The perfectly matched layer for acoustic waves in absorptive media. The Journal of the Acoustical Society of America 102, 4 (1997), 2072--2082.
44. S. Marburg and B. Nolte. 2008. Computational acoustics of noise propagation in fluids: finite and boundary element methods. Vol. 578. Springer.
45. R. Mehra, N. Raghuvanshi, L. Antani, A. Chandak, S. Curtis, and D. Manocha. 2013. Wave-based sound propagation in large open scenes using an equivalent source formulation. ACM Transactions on Graphics (TOG) 32, 2 (2013), 19.
46. R. Mehra, N. Raghuvanshi, L. Savioja, M. C. Lin, and D. Manocha. 2012. An efficient GPU-based time domain solver for the acoustic wave equation. Applied Acoustics 73, 2(2012), 83--94.
47. A. Meshram, R. Mehra, H. Yang, E. Dunn, J.-M. Frahm, and D. Manochak. 2014. P-hrtf: Efficient personalized hrtf computation for high-fidelity spatial sound. Mixed and Augmented Reality (ISMAR), 2014 IEEE International Symposium on (2014).
48. P. Micikevicius. 2009. 3D Finite Difference Computation on GPUs Using CUDA. In Proceedings of 2Nd Workshop on General Purpose Processing on Graphics Processing Units (GPGPU-2). ACM, New York, NY, USA, 79--84.
49. M. Minnaert. 1933. XVI. On musical air-bubbles and the sounds of running water. The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science 16, 104 (1933), 235--248.
50. R. Mittal, H. Dong, M. Bozkurttas, F. M. Najjar, A. Vargas, and A. Loebbecke. 2008. A versatile sharp interface immersed boundary method for incompressible flows with complex boundaries. J. Comput. Phys. 227 (2008).
51. R. Mittal and G. Iaccarino. 2005. Immersed Boundary Methods. Annual Review of Fluid Mechanics 37 (2005).
52. P. Morse and K. U. Ingard. 1968. Theoretical Acoustics. Princeton University Press, Princeton, New Jersey.
53. W. Moss, H. Yeh, J.-M. Hong, M. C. Lin, and D. Manocha. 2010. Sounding Liquids: Automatic Sound Synthesis from Fluid Simulation. ACM Trans. Graph. 29, 3 (2010).
54. J. F. O'Brien, P. R. Cook, and G. Essl. 2001. Synthesizing Sounds From Physically Based Motion. In Proceedings of SIGGRAPH 2001. 529--536.
55. J. F. O'Brien, C. Shen, and C. M. Gatchalian. 2002. Synthesizing sounds from rigid-body simulations. In The ACM SIGGRAPH 2002 Symposium on Computer Animation. ACM Press, 175--181.
56. C. S. Peskin. 1981. The fluid dynamics of heart valves: experimental, theoretical and computational methods. Annual Review of Fluid Mechanics 14 (1981).
57. N. Raghuvanshi, R. Narain, and M. C. Lin. 2009. Efficient and Accurate Sound Propagation Using Adaptive Rectangular Decomposition. IEEE Trans. Vis. Comput. Graph. 15, 5 (2009), 789--801.
58. N. Raghuvanshi and J. Snyder. 2014. Parametric wave field coding for precomputed sound propagation. ACM Transactions on Graphics (TOG) 33, 4 (2014), 38.
59. J. Saarelma, J. Botts, B. Hamilton, and L. Savioja. 2016. Audibility of dispersion error in room acoustic finite-difference time-domain simulation as a function of simulation distance. The Journal of the Acoustical Society of America 139, 4 (2016), 1822--1832.
60. C. Schissler, R. Mehra, and D. Manocha. 2014. High-order diffraction and diffuse reflections for interactive sound propagation in large environments. ACM Transactions on Graphics (TOG) 33, 4 (2014), 39.
61. C. Schreck, D. Rohmer, D. James, S. Hahmann, and M.-P. Cani. 2016. Real-time sound synthesis for paper material based on geometric analysis. In Eurographics/ACM SIGGRAPH Symposium on Computer Animation (2016).
62. E. Schweickart, D. L. James, and S. Marschner. 2017. Animating Elastic Rods with Sound. ACM Transactions on Graphics 36, 4 (July 2017).
63. A. A. Shabana. 2012. Theory of Vibration: An Introduction. Springer Science & Business Media.
64. A. A. Shabana. 2013. Dynamics of multibody systems. Cambridge university press.
65. Side Effects. 2018. Houdini Engine. (2018). http://www.sidefx.com.
66. J. O. Smith. 1992. Physical modeling using digital waveguides. Computer music journal 16, 4 (1992), 74--91.
67. A. Taflove and S. C. Hagness. 2005. Computational Electrodynamics: The Finite-Difference Time-Domain Method. Artech House.
68. T. Takala and J. Hahn. 1992. Sound rendering. In Computer Graphics (Proceedings of SIGGRAPH 92). 211--220.
69. J. G. Tolan and J. B. Schneider. 2003. Locally conformal method for acoustic finite-difference time-domain modeling of rigid surfaces. Journal of the Acoustical Society of America 114 (2003).
70. N. Tsingos, T. Funkhouser, A. Ngan, and I. Carlbom. 2001. Modeling acoustics in virtual environments using the uniform theory of diffraction. In SIGGRAPH '01: Proceedings of the 28th annual conference on Computer graphics and interactive techniques. ACM, New York, NY, USA, 545--552.
71. K. van den Doel. 2005. Physically Based Models for Liquid Sounds. ACM Trans. Appl. Percept. 2, 4 (Oct. 2005), 534--546.
72. K. van den Doel, P. G. Kry, and D. K. Pai. 2001. FoleyAutomatic: Physically-based Sound Effects for Interactive Simulation and Animation. (2001), 537--544.
73. K. van den Doel and D. K. Pai. 1998. The sounds of physical shapes. Presence: Teleoperators and Virtual Environments 7, 4 (1998), 382--395.
74. M. Vorländer. 2008. Auralization. Aachen: Springer (2008).
75. C.J. Webb. 2014. Parallel computation techniques for virtual acoustics and physical modelling synthesis. Ph.D. Dissertation.
76. C. J. Webb and S. Bilbao. 2011. Computing room acoustics with CUDA - 3D FDTD schemes with boundary losses and viscosity. 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (2011), 317--320.
77. H. Yeh, R. Mehra, Z. Ren, L. Antani, D. Manocha, and M. Lin. 2013. Wave-ray Coupling for Interactive Sound Propagation in Large Complex Scenes. ACM Trans. Graph. 32, 6, Article 165 (Nov. 2013), 11 pages.
78. C. Zheng and D. L. James. 2009. Harmonic Fluids. ACM Transactions on Graphics (SIGGRAPH 2009) 28, 3 (Aug. 2009).
79. C. Zheng and D. L. James. 2010. Rigid-Body Fracture Sound with Precomputed Sound-banks. ACM Transactions on Graphics (SIGGRAPH 2010) 29, 3 (July 2010).
80. C. Zheng and D. L. James. 2011. Toward High-Quality Modal Contact Sound. ACM Transactions on Graphics (Proceedings of SIGGRAPH 2011) 30, 4 (Aug. 2011).

Index Terms

• Computing methodologies

  • Computer graphics

    • Animation

      • Physical simulation