How do spatial audio plugins work and what functionalities do they offer: a comparative perspective

Rosero, K. Faculdade de Engenharia Elétrica e de Computação (FEEC), Universidade Estadual de Campinas (UNICAMP), Campinas, SP, Brasil, k264373@g.unicamp.br

Abreu, D. Faculdade de Engenharia Elétrica e de Computação (FEEC), Universidade Estadual de Campinas (UNICAMP), Campinas, SP, Brasil, douglashsabreu@gmail.com

Grijalva, F. Faculty of Engineering and Applied Sciences (FICA), Telecommunications Engineering, Universidad de Las Américas (UDLA), Departamento de Electrónica, Telecomunicaciones y Redes de Información (DETRI), Escuela Politécnica Nacional, Quito 170125, Ecuador, felipe.grijalva@udla.edu.ec, felipe.grijalva@epn.edu.ec

Masiero, B. Faculdade de Engenharia Elétrica e de Computação (FEEC), Universidade Estadual de Campinas (UNICAMP), Campinas, SP, Brasil, masiero@unicamp.br

Abstract

Audio systems that reproduce spatial information are increasingly spreading across the market, aiming at offering high fidelity in the perception of timbres and high quality in the perception of spatial content. Special equipment, like microphones and loudspeaker arrays are commonly required for recording and reproducing spatial sound. Similar to traditional audio formats, the manipulation of immersive sound employs digital audio workstations (DAWs) with software extensions installed to expand the DAW’s functionality, named “spatial audio plugins”. As a result, it is necessary to disseminate among researchers and practitioners the most relevant spatial audio plugins available. Based on the production chain of immersive sound, we present plugins for the encoding and decoding stages with a comparative perspective of its features. Common operations provided by the available plugins encode regular audio formats such as mono, stereo or multi-channel sound into the Ambisonics spatial audio format of different orders. Decoding plugins process Ambisonic signals into binaural outputs or loudspeaker array formats. This paper summarizes the available spatial audio plugins and the recent developments they provide, briefly describing the parametric and non-parametric methods in which they are based. We hope this manuscript contributes to a better navigation through this large range of options regarding spatial audio plugins.

Keywords:

spatial audio

ambisonics

digital audio workstations

TV 3.0

MPEG-H.

References

[1] Stephan Paul. Binaural recording technology: A historical review and possible future developments. Acta acustica united with Acustica, 95(5):767–788, 2009.

[2] Kristofer Kjörling, Jonas Rödén, Martin Wolters, Jeff Riedmiller, Arijit Biswas, Per Ekstrand, Alexander Gröschel, Per Hedelin, Toni Hirvonen, Holger Hörich, et al. Ac-4–the next generation audio codec. In Audio Engineering Society Convention 140. Audio Engineering Society, 2016.

[3] Jürgen Herre, Johannes Hilpert, Achim Kuntz, and Jan Plogsties. Mpeg-h 3d audio—the new standard for coding of immersive spatial audio. IEEE Journal of Selected Topics in Signal Processing, 9(5):770– 779, 2015. doi:10.1109/JSTSP.2015.2411578.

[4] Rich Chernock, David Gomez-Barquero, Jerry Whitaker, Sung-Ik Park, and Yiyan Wu. Atsc 3.0 next generation digital tv standard—an overview and preview of the issue. IEEE Transactions on Broadcasting, 62(1):154–158, 2016.

[5] StephenMTown,WOwen Brimijoin, and Jennifer K Bizley. Egocentric and allocentric representations in auditory cortex. PLoS Biol, 15(6):e2001878, June 2017. doi:10.1371/journal.pbio.2001878.

[6] William Yost and Raymond Dye. Fundamentals of Directional Hearing. Seminars in Hearing, 18: 321–344, 11 1997. doi:10.1055/s-0028-1083035.

[7] Henrik Møller, Michael Friis Sørensen, Dorte Hammershøi, and Clemen Boje Jensen. Head-Related Transfer Functions of Human Subjects. Journal of the Audio Engineering Society, 43(5):300–321, May 1995.

[8] Y. Suzuki, D. Brungart, Y. Iwaya, K. Iida, D. Cabrera, and H. Kato. Principles and Applications of Spatial Hearing. World Scientific Publishing Co. Pte. Ltd., Danvers, 2011. ISBN 981-4313-87-4.

[9] Francis Rumsey. Spatial Audio, volume 2nd of Music Technology Series. Focal Press, London, England, 1st edition, 2001. ISBN 0240516230.

[10] Silva, R. Surround Sound: The Audio Side of Home Theater. https://www.lifewire.com/surround-soundhome- theater-1846357, December 2021.

[11] Chungeun Kim. Object-Based Spatial Audio: Concept, Advantages, and Challenges, pages 79–84. Springer New York, New York, NY, 2014. ISBN 978-1-4614-8373-1. doi:10.1007/978-1-4614-8373- 1_5.

[12] Ville Pulkki. Virtual sound source positioning using vector base amplitude panning. Journal of the Audio Engineering Society, 45(6):456–466, June 1997.

[13] Ville Pulkki. Generic panning tools for max/msp. In Proceedings of International Computer Music Conference, page 304307, 2000.

[14] Konrad Kowalczyk, Oliver Thiergart, Maja Taseska, Giovanni Del Galdo, Ville Pulkki, and Emanuel AP Habets. Parametric spatial sound processing: A flexible and efficient solution to sound scene acquisition, modification, and reproduction. IEEE Signal Processing Magazine, 32(2):31–42, 2015.

[15] Ville Pulkki, Symeon Delikaris-Manias, and Archontis Politis. Higher-Order Directional Audio Coding, pages 141–159. 2018. doi:10.1002/9781119252634.ch6.

[16] Archontis Politis, Sakari Tervo, and Ville Pulkki. Compass: Coding and multidirectional parameterization of ambisonic sound scenes. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6802–6806. IEEE, 2018.

[17] Franz. Zotter. Ambisonics A Practical 3D Audio Theory for Recording, Studio Production, Sound Reinforcement, and Virtual Reality . Springer Topics in Signal Processing, 19. Springer International Publishing, Cham, 1st ed. 2019. edition, 2019. ISBN 3-030-17207-4.

[18] C. NACHBAR, F. ZOTTER, E. DELEFLIE, and A. SONTACCHI. ambix - a suggested ambisonics format. In Proceedings of the Ambisonics Symposium 2011, Lexington KY, July 2011.

[19] Boaz Rafaely. Fundamentals of Spherical Array Processing. Springer Nature Switzerland AG, 2019. ISBN 1866-2609. doi:10.1007/978-3-319-99561-8.

[20] Leo McCormack, Symeon Delikaris-Manias, Angelo Farina, Daniel Pinardi, and Ville Pulkki. Real-time conversion of sensor array signals into spherical harmonic signals with applications to spatially localized sub-band sound-field analysis. In Audio Engineering Society Convention 144. Audio Engineering Society, 2018.

[21] Franz Zotter and Matthias Frank. All-round ambisonic panning and decoding. Journal of the Audio Engineering Society, 60(10):807–820, October 2012.

[22] Christian Schörkhuber, Markus Zaunschirm, and Robert Höldrich. Binaural rendering of ambisonic signals via magnitude least squares. In Fortschritter der Akustik (DAGA) 2018, March 2018. ISBN 978-3-939296-13-3.

[23] Leo McCormack and Symeon Delikaris-Manias. Parametric first-order ambisonic decoding for headphones utilising the cross-pattern coherence algorithm. In EAA Spatial Audio Signal Processing Symposium, pages 173–178, Paris, France, September 2019. doi:10.25836/sasp.2019.26.

[24] Markus Zaunschirm, Christian Schörkhuber, and Robert Höldrich. Binaural rendering of ambisonic signals by head-related impulse response time alignment and a diffuseness constraint. The Journal of the Acoustical Society of America, 143(6):3616, June 2018. doi:10.1121/1.5040489.

[25] R. Schmidt. Multiple emitter location and signal parameter estimation. IEEE Transactions on Antennas and Propagation, 34(3):276–280, 1986. doi:10.1109/TAP.1986.1143830.

[26] R. Roy and T. Kailath. Esprit-estimation of signal parameters via rotational invariance techniques. IEEE Transactions on Acoustics, Speech, and Signal Processing, 37(7):984–995, 1989. doi:10.1109/29.32276.

[27] Symeon Delikaris-Manias and Ville Pulkki. Cross pattern coherence algorithm for spatial filtering applications utilizing microphone arrays. IEEE Transactions on Audio, Speech, and Language Processing, 21(11):2356–2367, 2013. doi:10.1109/TASL.2013.2277928.

[28] Jacek P. Dmochowski, Jacob Benesty, and SofiÈne Affes. A generalized steered response power method for computationally viable source localization. IEEE Transactions on Audio, Speech, and Language Processing, 15(8):2510–2526, 2007. doi:10.1109/TASL.2007.906694.

[29] Franz Zotter, Hannes Pomberger, and Markus Noisternig. Energy-preserving ambisonic decoding. Acta Acustica united with Acustica, 98:37–47, 01 2012. doi:10.3813/AAA.918490.

[30] Tianshu Qu, Huang Zhichao, Yue Qiao, and Xihong Wu. Matching projection decoding method for ambisonics system. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 561–565, 04 2018. doi:10.1109/ICASSP.2018.8461515.

[31] Andrew Wabnitz, Nicolas Epain, and Craig T. Jin. A frequency-domain algorithm to upscale ambisonic sound scenes. In 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 385–388, 2012. doi:10.1109/ICASSP.2012.6287897.

[32] Leo McCormack and Archontis Politis. SPARTA & COMPASS: Real-Time Implementations of Linear and Parametric Spatial Audio Reproduction and Processing Methods. 2019 AES International Conference on Immersive and Interactive Audio;, 2019. ISBN 978-1-942220-27-5. URL http://urn.fi/URN: NBN:fi:aalto-201906203881.

[33] Leo McCormack, Archontis Politis, Ville Pulkki, et al. Parametric spatial audio effects based on the multi-directional decomposition of ambisonic sound scenes. In 23rd International Conference on Digital Audio Effects (DAFx2020), 2021.

[34] Leo McCormack, Archontis Politis, and Ville Pulkki. Rendering of source spread for arbitrary playback setups based on spatial covariance matching. In 2021 IEEEWorkshop on Applications of Signal Processing to Audio and Acoustics (WASPAA), pages 371–375. IEEE, 2021.

[35] M. Kajala and M. Hamalainen. Filter-and-sum beamformer with adjustable filter characteristics. In 2001 IEEE International Conference on Acoustics, Speech, and Signal Processing. Proceedings (Cat. No.01CH37221), volume 5, pages 2917–2920 vol.5, 2001. doi:10.1109/ICASSP.2001.940257.

[36] Brian G. Ferguson. Minimum variance distortionless response beamforming of acoustic array data. The Journal of the Acoustical Society of America, 104(2):947–954, aug 1998. ISSN 0001-4966. doi:10.1121/1.423311.

[37] J. Fernande, L. McCormack, P. Hyvärinen, A. Politis, and V. Pulkki. Enhancing binaural rendering of head-worn microphone arrays through the use of adaptive spatial covariance matching. Journal of the Acoustic Society of America, 151:2624–2635, April 2022. doi:10.1121/10.0010109.

[38] V. Pulkki, L. McCormack, and R. Gonzalez. Superhuman spatial hearing technology for ultrasonic frequencies. Nature, 11(11608), June 2021. doi:10.1038/s41598-021-90829-9.

[39] Archontis Politis, Juha Vilkamo, and Ville Pulkki. Sector-based parametric sound field reproduction in the spherical harmonic domain. IEEE Journal of Selected Topics in Signal Processing, 9(5):852–866, 2015.

[40] The Advanced Television Systems Committee. ATSC 3.0: Standards. URL https://www.atsc.o rg/atsc-documents/type/3-0-standards. Accessed: 2022-06-14.

[41] Dolby Laboratories. Dolby Encoding Engine User’s Guide. February 2019. URL https://develope r.dolby.com/globalassets/professional/productmanuals/ dolbyencodingengineusersguide.pdf.

[42] Fraunhofer Institute for Integrated Circuits IIS. MPEG-H authoring suite. URL https://www.iis. fraunhofer.de/en/ff/amm/dl/software/mas.html. Accessed: 2022-06-14.