[x3d-public] suggested approaches to improve X3D v4 Sound model

Sat Jul 29 17:14:34 PDT 2017

Summary. Sound support in virtual environments (VEs) is a complex topic.  Historically audio models are oversimplified and not very realistic, for hardware and software and also X3D.  Nevertheless many components appear to be available for composition in X3D to create a much better sound model.  This note attempts to organize the variety of factors involved so that we might improve X3D sound support, in scenes and players (for X3Dv4.0) and head-mounted devices (X3Dv4.1).

Background.  Recently read a wonderful dissertation, highest recommendation:

	Scott McDermott, "An Analysis of Accurate, Real-Time Reproduction of 3D Acoustics in Virtual Environments,"
	Ph.D. Dissertation, University of Louisiana at Lafayette, Fall 2014.
	http://cas.loyno.edu/mathematics/bios/scott-mcdermott
	http://loyno.edu/~smcdermo
	http://pqdtopen.proquest.com/doc/1669973288.html?FMT=ABS&pubnum=3687696

> Abstract.  Many of the applications, virtual environments, and video games available to average computer users integrate stunning three-dimensional (3D) graphics and real-world visualizations. Developers spend an extraordinary amount of time and effort creating these immersive, realistic virtual environments, primarily focusing on the graphics components. Within these virtual realities, the user should easily perceive the locations of sound sources accurately, as well as the acoustic nature of the environment. However, for reasons of economy and simplicity, most developers apply readily available industry standards for generating pseudo-3D sounds in their applications. This research explores the shortcomings of these standards, proposes an effective alternative, and provides a detailed analysis of the various possible approaches.
> 
> This project includes a number of computationally efficient, physics-based 3D acoustics simulations, each of which will produce realistic aural reproductions. The primary goal is to evaluate and compare these algorithms against each other, non-3D sound reproduction, and the current industry standards (e.g. Microsoft's DirectX® pseudo-3D algorithm). We will test three hypotheses. First, users will find that physics-based 3D algorithms will render improved auralization reproductions compared against industry standards like DirectX® and/or OpenAL. Second, localization and spatialization will improve with user training when using these algorithms. Finally, we should discover an unambiguous ranking system for the quality of each tested algorithm.

	Earlier paper:
	Scott McDermott, Cheehung Henry Chu, "An Exploration of Virtual 3D Sound," University of Louisiana at Lafayette,
	154th Acoustical Society of America (ASA) Meeting, 30 November 2007
	http://www.loyno.edu/~smcdermo/papers/ASA4aAA7Lay

This work holds excellent potential value for us because it provides both methodology and analysis that can help provide a solid basis for improvements we hope to accomplish for sound spatialization in the X3D.

References.  Supporting materials for the current simplistic X3D sound model include

	X3D abstract specification: Sound component
	http://www.web3d.org/documents/specifications/19775-1/V3.3/Part01/components/sound.html

	X3D Graphics for Web Authors, Chapter 12 slideset: Environment Sensor and Sound Nodes
	http://x3dgraphics.com/slidesets/X3dForWebAuthors/Chapter12EnvironmentSensorSound.pdf

	X3D Tooltips: AudioClip and Sound nodes
	http://www.web3d.org/x3d/content/X3dTooltips.html#AudioClip
	http://www.web3d.org/x3d/content/X3dTooltips.html#Sound

	X3D Example Archives: VRML 2 Sourcebook, Chapter 24 Sound
	http://www.web3d.org/x3d/content/examples/Vrml2Sourcebook/Chapter24Sound

	X3D Example Archives: X3D for Web Authors, Chapter 12 Environment Sensor Sound
	http://x3dgraphics.com/examples/X3dForWebAuthors/Chapter12EnvironmentSensorSound

	X3D Scene Authoring Hints: Audio and Sound Tools
	http://www.web3d.org/x3d/content/examples/X3dSceneAuthoringHints.html#Audio

Proposed, first draft. _X3D v4 Strategies to Improve Sound Component_

1. 3D spatialization algorithms for Sound node  attenuation ellipsoids (see)
a. McDermott 2014: Bouncing Reflections, Reflected Path
b. RESound: Interactive Sound Rendering for Dynamic Virtual Environments, http://gamma.cs.unc.edu/Sound/RESound
c. Others, perhaps including parameterized/extensible algorithms
d. Consider ordered list of supported algorithms, corresponding to computational complexity and X3D player support
e. Define corresponding list of enumerations for each spatialization algorithm
f. Scoping within scene graph: boolean global variable for localization, computational efficiency?
g. Boolean attributes to enable/disable, also to facilitate disabling attenuation effects
h. Consider composable regions of higher/lower fidelity or salience

2. Audio material properties
a. Add to Material node, or add new AcousticMaterial within Shape node
b. Properties: absorption, dispersion, others
c. Frequency distribution of attenuation
d. Characteristics of transmission media, perhaps refraction

3. Scene graph integration: simplified geometry for satisfactory sound reflection
a. Direct integration within Shape node
b. Consider further restrictions via Collision-proxy or Collision-acoustic
c. Consider bounding box restrictions in addition to (overriding) outer attenuation ellipsoid
d. Fast Fourier Transform (FFT) precomputations: allow? How to include?

4. World Wide Web Consortium (W3C)
a. W3C Audio Group https://www.w3.org/2011/audio
b. W3C Audio Publications, Milestones https://www.w3.org/2011/audio/wiki/PubStatus
c. File formats: match accepted standards, royalty-bearing formats can only be optional
d. Streamable input: simply file-format capability? perhaps other aspects with common metadata
e. Filter chains, Web audio models http://www.webaudiomodules.org
f. W3C Audio and Video https://www.w3.org/standards/webdesign/audiovideo.html
g. W3C Accessibility https://www.w3.org/standards/webdesign/accessibility.html
h. Compatibility with hardware acceleration developments
i. Follow WebVR developments https://w3c.github.io/webvr to ensure matching (or at least compatible) aspects related to wearable devices and user settings/preferences.

5. X3D Standards Evolution
a. X3D v4.0 scene-related acoustics model
b. X3D v4.1 (MAR) user customization for AR/VR
    i. Head shadow, head motion, pinna and shoulder response, interaural distance
    ii. Head related transfer functions (HRTF) https://en.wikipedia.org/wiki/Head-related_transfer_function
c. Other ISO and industry standardization efforts?
d. X3D player implementations in C/C++, Java, Javascript
e. Example scenes demonstrating auralization effects with corresponding visualization assists
f. Three Dimensional Sound (TDS) Simulator improvements to support/evaluate X3D
g. Careful observance of Web3D Intellectual Property Rights (IPR) Policy to achieve royalty free (RF) solution

Questions, feedback and improvements welcome.

all the best, Don
-- 
Don Brutzman  Naval Postgraduate School, Code USW/Br       brutzman at nps.edu
Watkins 270,  MOVES Institute, Monterey CA 93943-5000 USA   +1.831.656.2149
X3D graphics, virtual worlds, navy robotics http://faculty.nps.edu/brutzman
-------------- next part --------------
A non-text attachment was scrubbed...
Name: X3Dv4StrategiesToImproveSoundComponent.pdf
Type: application/pdf
Size: 91910 bytes
Desc: not available
URL: <http://web3d.org/pipermail/x3d-public_web3d.org/attachments/20170729/d27ab86a/attachment-0001.pdf>