[x3d-public] meeting minutes 21 May 2020: X3D4 Audio and Sound Component
GPU Group
gpugroup at gmail.com
Thu May 21 16:24:15 PDT 2020
Hypothesis: AudioListener pose _is_ the avatar pose.
- like rendering graphics - you could render somewhere the avatar isn't
pointing, but for web3d immersive experience, you want to see the scene in
front of you rendered.
- same with the Listener pose (xyz,orentation) parameters Hypothesis: they
should be implied / automatically taken from the avatar pose.
-Doug Sanden
On Thu, May 21, 2020 at 12:40 PM Don Brutzman <brutzman at nps.edu> wrote:
> Attendees: Efi, Thanos, Dick, Don
>
> No member-only information is contained in these minutes.
>
> 1. We reviewed Efi's latest update, with thanks for progress and great
> interest. 8)
>
> Extracted from attachment, initial fragment improvement regarding DEF and
> USE:
> =========================
> <Transform USE='Audio1'/><!-- USE is a copy by reference, so it cannot
> have children -->
> <X3DSpatialAudioNode><!-- X3DSpatialAudioNode is an abstract
> class, not a concrete node -->
> <PannerNode position='0 0 0' orientation='1 0 0'
> velocity='0 0 0' coneInnerAngle='360' coneOuterAngle='360'
> coneOuterGain='0' distanceModel='inverse' maxDistance='1'
> panningModel='HRTF' refDistance='1' rolloffFactor='1' gain =1 >
> </PannerNode>
> <AudioClip loop='true' url='"sound/africa/beat.mp3" gain =
> 0.5 isViewpoint = true />
> </X3DSpatialAudioNode>
> =========================
> <Transform DEF='Audio1'>
> <!-- is there a parent node to connect these? otherwise how to
> connect them? -->
> <PannerNode position='0 0 0' orientation='1 0 0'
> velocity='0 0 0' coneInnerAngle='360' coneOuterAngle='360'
> coneOuterGain='0' distanceModel='inverse' maxDistance='1'
> panningModel='HRTF' refDistance='1' rolloffFactor='1' gain =1 >
> </PannerNode>
> <AudioClip loop='true' url='"sound/africa/beat.mp3" gain =
> 0.5 isViewpoint = true />
> <!-- closing tag for parent node? -->
> </Transform>
> =========================
>
> Hmmm... Step by step we continued.
>
> -----------------------------------
>
> 2. AudioListener cannot be the same level in object hierarchy diagram as
> X3DSoundSourceNode.
>
> Is AudioListener similar to AudioClip, providing a source of sound, as an
> X3DSoundSourceNode? __(yes according to today's attached draft report "2nd
> approach" on page 7)__ If so then they should be the same level in
> interface hierarchy...
>
> Much discussion, BUT actual answer is: *NO* AudioClip is not the same type
> of functionality.
>
> -----------------------------------
>
> 3. We next considered patterns is further detail, comparing to X3D
> Specification.
>
> From X3D Specification there is an essential ASCII-art diagram, excerpted
> here:
>
> [1] X3D Architecture, 4.4.2.3 Interface hierarchy
>
> https://www.web3d.org/documents/specifications/19775-1/V3.3/Part01/concepts.html#InterfaceHierarchy
>
> +- X3DSoundNode -+- Sound
>
> +- X3DTimeDependentNode -+- TimeSensor (X3DSensorNode)*
> | |
> | +- X3DSoundSourceNode -+- AudioClip
> (X3DUrlObject)*
> | +- MovieTexture
> (X3DTexture2DNode, X3DUrlObject)*
>
> We looked very hard at this.
>
> Wondered, is AudioListener perhaps the missing parent node in the previous
> section?? Much discussion, again answer: NO.
>
> If not, then what is the node that takes an AudioClip source and modifies
> it according to the Web Audio API, then exposes result to X3D scene graph?
>
> Current X3D pattern is two nodes working together, Sound and AudioClip:
>
> <Transform DEF="OptionalLocalReferenceFrame">
> <Sound>
> <AudioClip/>
> <Sound>
> </Transform>
>
> For example, from X3D encoding specifications,
>
> [2] X3D ClassicVRML Encoding, Examples, B.13.3 Alarm clock
>
> https://www.web3d.org/documents/specifications/19776-2/V3.3/Part02/examples.html#AlarmClock
>
> Sound {
> source DEF Sounder AudioClip {
> description "chime"
> url [ "click.wav" ]
> }
>
> [3] X3D XML Encoding, Examples, C.13.3 Alarm clock
>
> https://www.web3d.org/documents/specifications/19776-1/V3.3/Part01/examples.html#AlarmClock
>
> [4] X3D Example Archives: Basic, X3D Specifications, Alarm Clock
>
> https://www.web3d.org/x3d/content/examples/Basic/X3dSpecifications/AlarmClockIndex.html
>
> <!--
> https://www.web3d.org/x3d/content/examples/Basic/X3dSpecifications/AlarmClock.x3d
> -->
> <Sound>
> <AudioClip DEF='Sounder' description='chime'
> url='"click.wav" "
> https://www.web3d.org/x3d/content/examples/Basic/X3dSpecifications/click.wav
> "'/>
> </Sound>
>
> Next.... Basic Web Audio API pattern is multiple nodes working together.
> From attached current-draft report it seems the primary nodes are:
>
> PannerNode
> Web Audio API sound graph (might get a synthetic signal or use
> AudioClip, many internal filters etc.)
>
> Naming note: ending a node's name with 'Node' is undesirable since we do
> that for abstract node types, e.g. X3DSoundSourceNode.
>
> Sticking point... do we have a single node for composite Web Audio API
> signal?
>
> After much discussion, perhaps SpatialSound as a better name for the
> PannerNode functionality being discussed?
>
> Possible scene graph structure (not hierarchy!) might be
>
> =============================================
> <Transform DEF="LocalReferenceFrame">
> <PannerNode>
> <AudioClip/>
> </PannerNode>
> </Transform>
> =============================================
> or
> <Transform DEF="LocalReferenceFrame">
> <Sound>
> <PannerNode/>
> <AudioClip/>
> </Sound>
> </Transform>
> =============================================
> alternatively, perhaps best:
>
> <Transform DEF="LocalReferenceFrame">
> <SpatialSound>
> <!-- Web Audio API library of effects, etc. would
> be invoked as child/grandchild field nodes here -->
> <AudioClip/>
> </SpatialSound>
> </Transform>
> =============================================
>
> which in turn would lead to abstract hierarchy change, adding SpatialSound
> as follows:
>
> +- X3DSoundNode -+- Sound
> +- SpatialSound
>
> ... where SpatialSound contains useful WebAudio API nodes, panner, etc.
> etc.
>
> =============================================
>
> This is an important question to consider in preparation for next week's
> meeting.
>
> Of note is this may be our first potential solution considered so far that
> is relatively simple to understand, from an authoring and implementation
> perspective.
>
> We continue proceed carefully. Inspired by the magnificence of Greek
> cuisine, we are hoping to avoid creating the special culinary dish known as
> "soup sandwich" .... :0
>
> [5] Soup sandwich
> https://en.wiktionary.org/wiki/soup_sandwich
>
> -----------------------------------
>
> 4. Aural rendering and separation of concerns.
>
> We also discussion how user's "camera" can navigate around the scene,
> often (but not necessarily) at a given Viewpoint location. The scene is
> visually rendered at that current user location.
>
> Similarly, audio is aurally rendered from perspective of (a) where is
> sounds source, and (b) where are the user's ear locations, once again
> centered about current user's point of view.
>
> Thus there is a logical separation between
> - spatialized sound source(s),
> - how they are aurally rendered,
> - how they are propagated through scene, and
> - where they are perceived by user.
>
> SpatialSound is different than existing Sound node because it can include
> more sophisticated panning effects, as well as all the richness of Web
> Audio API.
>
> -----------------------------------
>
> Wow what a productive session!
>
> We will meet again in one week, 0930-1100 pacific.
>
> v/r Don, Dick, Efi, Thanos
>
>
> On 4/19/2020 6:41 PM, Don Brutzman wrote:
> > 1. *Summary*. Today's session focused on X3D Sound Component and work by
> Efi Lakka and Athanasios Malamos to gain compatibility with W3C Web Audio
> components that are commonly being implemented in HTML5 browsers. Numerous
> references are also included.
> >
> > We met Friday 17 April 2020 0800-0940 for regular weekly meeting of X3D
> Working Group.
> >
> > Attendees: Efi Lakka, Athanasios Malamos, Vince Marchetti, Nicholas
> Polys, Dick Puk, Anita Havele, Don Brutzman.
> >
> > [1.1] X3D Graphics Working Group
> > https://www.web3d.org/working-groups/x3d
> >
> > Confirmed that no member-only information is present in these minutes.
> Noted that the information here:
> > - helps members pay full attention to the detailed improvement of X3D4,
> > - helps community understand and trust the diligence of the process, and
> > - helps everyone understand that Web3D Consortium offers serious member
> value!
> > https://www.web3d.org/join
> >
> > Note that all Web3D zoom conference lines are now protected by password.
> >
> > [1.2] Web3D Teleconference Information
> > https://www.web3d.org/member/teleconference-information
> >
> > - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
> >
> > 2. *ISO Publication of HAnim*
> >
> > Good news! Dick and I worked for about an hour during one of our
> twice-weekly X3D Specification Editor teleconferences to confirm that the
> ISO version (published November 2019) indeed matches (a) what we submitted,
> and (b) the Web3D version. Everything looked great. 8)
> >
> > As most people know, these documents are available to members on github.
> >
> > Github links:
> >
> > [2.0] GitHub: Web3D Consortium
> > https://github.com/Web3DConsortium
> >
> > [2.1] GitHub: Web3D Consortium members, HAnim
> > https://github.com/Web3DConsortium/HAnim
> >
> > [2.2] GitHub: Web3D Consortium members, HAnim Specifications combined
> parts 1 and 2
> >
> https://github.com/Web3DConsortium/HAnim/raw/master/ISO-IEC19774-combined-IS.Web3D.zip
> >
> > We've handed them off to Executive Director Anita and the Webmaster to
> post on web3d.org pages. These go on web3d.org under STANDARDS (All,
> Recommended, HAnim) replacing the v2.0 DIS version with this new v2.0
> International Standard (IS) version.
> >
> > Next steps:
> > - publish on web3d.org,
> > - public announcement (Anita will give initial draft to HAnim
> principals) with completion goal next week,
> > - track ISO correction of their catalog pages (H-Anim to HAnim),
> > - track ISO addition of HAnim specifications to their freely available
> ISO specifications,
> > - Joe and Don are now reviewing HAnim 1 and 2 examples for correctness.
> This take a number of weeks because the example version numbers need close
> review, and because not all prior changes were applied thoroughly).
> > - if needed, submit a corrigendum to ISO for updated HAnim examples.
> Dick can help with the process.
> >
> > Suggested: followup with Medical Working Group about future strategies
> taking advantage of new HAnim capabilities; also DICOM HealthLevel7 and
> IEEE 3D Body Processing (3DBP). Perhaps OGC GeoPose too?
> >
> > - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
> >
> > 2.5 *Web3DUX Working Group*
> >
> > Announcement needed, Anita will contact working group chairs. Nicholas
> Anita Don and Dick will continue helping to encourage broader activity.
> They meet every other Wednesday on Web3D Teleconference line 1015-1100
> pacific.
> >
> > - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
> >
> > 3. *Audio Improvements and Sound Component*
> >
> > Summary: Prior review by X3D Working Group accepted this work for X3D4.
> We are thrilled to see much progress today. Much detail follows.
> >
> > This week's X3D Working Group teleconference features Efi Lakka and
> Professor Athanasios Malamos reviewing significant changes to X3D Sound
> Component for spatialization of sound and alignment of audio capabilities
> with current Web capabilities. This builds on capabilities first reported
> in the Web3D Conference held in Crete a few years ago. Her work includes
> updates to the X3D Sound Component (as a github pull request) plus demo
> scenes implemented in X3DOM.
> >
> > Efi Lakka reviewed the overall design and summarized her prior work.
> Pull Request 9 information follows, also see attached notification from
> github.
> >
> > [3.1] X3D4 PR9
> > ===============================================================
> > You can view, comment on, or merge this pull request online at:
> > https://github.com/Web3DConsortium/X3D/pull/9
> >
> > "Draft replacement to Sound Component for X3D4 Abstract Architecture,
> adding improved spatial audio rendering capabilities that take advantage of
> W3C Audio recommendations."
> >
> > Patch Links:
> > * https://github.com/Web3DConsortium/X3D/pull/9.patch
> > * https://github.com/Web3DConsortium/X3D/pull/9.diff
> > ===============================================================
> >
> > [3.2] Multimedia Laboratory, Hellenic Mediterranean University
> > http://medialab.hmu.gr
> >
> > [3.3] For everyone who wants to see how this first approach works,
> visit
> > http://medialab.hmu.gr/minipages/x3domAudio/index.html
> >
> > Background document on X3D4 Implementations page is quite thorough.
> Small changes as version 3 will be released shortly.
> >
> > [3.3] X3Dv4 Implementations Status
> > https://www.web3d.org/x3dv4-implementations
> >
> > [3.3] Attachment X3dAudioExtension_v02.pdf
> >
> https://www.web3d.org/sites/default/files/page/X3Dv4%20Implementations/X3dAudioExtension_v02.pdf
> >
> > [3.4] TODO needed: slideset shown during meeting
> > ___________
> >
> > Structure of hooking together various audio filters and sources is
> interesting. Generally design of an audio "chain" is accomplished
> primarily by hierarchical parent-child field references, not ROUTE
> connections.
> >
> > Addition of AcousticMaterial and properties will soon be included -
> seems like the right idea to do now since it is part of the spatialization
> computations. Are extra properties needed? an important area for
> discussion and continued work. We will temporarily add to Sound component
> for initial review, then into Shape component when confirmed robust.
> >
> > Implementation and evaluation will guide dialog and decision. Suggest
> specification prose sooner (at least as placeholders) and any additional
> implementation to follow shortly after. For now, "Better is the enemy of
> Good Enough" to keep building understanding and momentum.
> >
> > X3DOM examples are more mature now, and can be found at
> >
> > [3.5] TODO needed: latest examples link
> > ___________
> >
> > /Discussion/
> >
> > Q: When can X_ITE plan to begin implementation work? Likely in another
> few months, probably best for everyone to focus completely on X3DOM
> implementation completely. (Andreas Plesch has visibility into both
> codebases and can advise.)
> >
> > Q: Is JavaScript necessary for implementation support? Not
> necessarily. Most code so far has been moved into the X3DOM
> implementation. Some might be used for audio animation-style effects, see
> examples.
> >
> > Q: So far, are you only using browser audio?
> > A: yes.
> >
> > Q: Have you seen any audio implementations other than HTML browsers?
> This might be helpful for some of the non-HTML X3D players (view3dscene,
> FreeWrl, instantreality et al.). A: have not looked at that. More
> investigation by others is appropriate.
> >
> > Q: How close is X3D audio content to HTML5 audio code? Can similar
> effects be achieved either inside X3D scene or above in enclosing HTML
> page? Will HTML audio authors find it familiar?
> > A: X3D4 authors will find the new component as easy to use as the
> existing X3D3 component. Implementation complexity is mostly hidden.
> >
> > Q: What is your assessment of maturity of the W3C Audio candidate
> recommendations (CRs)?
> > A: They seem stable but not always complete. More study and feedback
> appears appropriate.
> >
> > Background W3C resources of interest:
> >
> > -----
> >
> > [a.1] W3C » Standards » Web Design and Applications » Audio and
> Video
> > https://www.w3.org/standards/webdesign/audiovideo
> >
> > [a.2] W3C Wiki: Audio
> > https://www.w3.org/wiki/Audio
> >
> > [a.3] Making Audio and Video Media Accessible
> > https://www.w3.org/WAI/media/av
> >
> > [a.4] W3C Standards and Drafts: Audio
> > https://www.w3.org/TR/?title=audio
> >
> > [a.5] Web Audio API
> > W3C Candidate Recommendation, 18 September 2018
> > Latest published version:
> > https://www.w3.org/TR/webaudio
> >
> > "Abstract. This specification describes a high-level Web API for
> processing and synthesizing audio in web applications. The primary paradigm
> is of an audio routing graph, where a number of AudioNode objects are
> connected together to define the overall audio rendering. The actual
> processing will primarily take place in the underlying implementation
> (typically optimized Assembly / C / C++ code), but direct script processing
> and synthesis is also supported.
> > The Introduction section covers the motivation behind this specification.
> > This API is designed to be used in conjunction with other APIs and
> elements on the web platform, notably: XMLHttpRequest [XHR] (using the
> responseType and response attributes). For games and interactive
> applications, it is anticipated to be used with the canvas 2D [2dcontext]
> and WebGL [WEBGL] 3D graphics APIs."
> >
> > "Status of this document. [...] This document was published by the the
> Web Audio Working Group as a Candidate Recommendation. This document is
> intended to become a W3C Recommendation. [...] A preliminary implementation
> report is available."
> >
> > [a.6] "CanIUseWebAudio"
> > https://caniuse.com/#feat=audio-api
> >
> > [a.7] Audio Output Devices API
> > W3C Candidate Recommendation 03 October 2017
> > Latest published version:
> > https://www.w3.org/TR/audio-output
> >
> > "Abstract. This document defines a set of JavaScript APIs that let a Web
> application manage how audio is rendered on the user audio output devices."
> >
> > "Status of This Document. [...] While the specification is feature
> complete and is expected to be stable, it might evolve based on
> implementation feedback that will be gathered as its associated test suite
> evolves. This test suite will be used to build an implementation report of
> the API; to go into Proposed Recommendation status, the group expects to
> demonstrate implementation of each feature in at least two deployed
> browsers."
> >
> > [a.8] Web MIDI API
> > W3C Working Draft 17 March 2015
> > Latest published version:
> > http://www.w3.org/TR/webmidi
> >
> > [a.9] Web Audio Processing: Use Cases and Requirements
> > W3C Working Group Note 29 January 2013
> > Latest published version:
> > http://www.w3.org/TR/webaudio-usecases
> > includes
> > 2.2 3D game with music and convincing sound effects
> >
> https://www.w3.org/TR/webaudio-usecases/#x3d-game-with-music-and-convincing-sound-effects
> > and
> > 2.11 User Control of Audio
> > https://www.w3.org/TR/webaudio-usecases/#user-control-of-audio
> >
> > [a.9] Audiobooks
> > W3C Candidate Recommendation 17 March 2020
> > Latest published version:
> > https://www.w3.org/TR/audiobooks
> >
> > "Abstract. This specification describes the requirements for the
> creation of audiobooks, using a profile of the Publication Manifest
> specification."
> >
> > [a.10] HTML5.2, section 4.7.11 The audio element
> >
> https://www.w3.org/TR/html52/semantics-embedded-content.html#the-audio-element
> >
> > "An audio element represents a sound or audio stream." [...]
> >
> > [a.11] Audio Working Group
> > https://www.w3.org/2011/audio
> >
> > "Mission. The mission of the Audio Working Group is to add advanced
> sound and music synthesis capabilities to the Open Web Platform."
> >
> > [a.12] Publishing Working Group
> > https://www.w3.org/publishing/groups/publ-wg
> >
> > "Mission. The mission of the Publishing Working Group is to enable all
> publications—with all their specificities and traditions—to become
> first-class entities on the Web. The group will provide the necessary
> technologies on the Open Web Platform to make the combination of
> traditional publishing and the Web complete in terms of accessibility,
> usability, portability, distribution, archiving, offline access, and
> reliable cross referencing."
> >
> > -----
> >
> > Q: Is "chain" the right term for an effect? We should likely use same
> term as W3C Audio group uses.
> > A. TODO check/confirm.
> >
> > Within the Web Audio CR, looks like the proper term is "modular routing"
> for composition of AudioNode source-node outputs and destination-object
> inputs.
> >
> > [a.13] Web Audio API, Features: Modular Routing
> > https://www.w3.org/TR/webaudio/#ModularRouting
> >
> > Q: Is the audio "chain" of source-filter-effect-spatialization effect
> represented as your .mp3 file?
> > A: TODO explain relationship.
> >
> > Tool of interest:
> >
> > [a.14] Web Audio Inspector (Chrome plugin)
> > https://opensource.google/projects/audion
> >
> > "Audion (Web Audio Inspector) is a Chrome extension that adds a Web
> Audio panel to dev tools. This panel visualizes the web audio graph in
> real-time and lets users inspect nodes."
> >
> > Erratum: for example shown during screenshare, <X3DSoundSourceNode>
> should be <AudioClip> (or perhaps another newer node)... the object
> interface might be functionally correct but we need to be careful about
> syntax, can't use abstract node types in X3D model files.
> >
> > Don and Dick plan to work during Tuesday's X3D4 Specification Editors
> call to drop Efi's draft on top of existing X3D Sound component, see if we
> are ready to check in, if good enough then accept the merge and keep moving
> forward.
> >
> > Next steps for action:
> > a. Marking up specification changes (css for proposed/yellow,
> deleted/orange) after including the Pull Request (PR9) in draft spec.
> > b. Are there any node name changes to consider? Create summary list for
> confirmation. Use Mantis to track specific changes.
> > c. Describing X3DOM implementation
> > d. Publishing example models, evaluate implementation correctness.
> > e. Encourage additional implementations X_iTE and other players - Web3D
> Call for Implementations in June, perhaps?
> > f. demonstration video for "Learning X3D4" later when stable, a draft
> video was created today. Thanks! Anita will publish privately with X3D
> Working Group.
> > g. prepare paper for Web3D 2020 in November? we hope so.
> >
> > Of related note: Web3D Consortium has the opportunity to nominate
> individual member(s) to participation in W3C Spatial Audio group. This can
> really help us achieve all of our goals effectively.
> >
> > [5.1] [x3d-public] Fwd: Audio Working Group Charter Approved; Call for
> Participation
> >
> https://web3d.org/pipermail/x3d-public_web3d.org/2020-April/012060.html
> >
> > Subject to level of effort while implementing X3D4 Sound, Thanos and Efi
> are willing to participate. Don will participate as well and lead the
> initial engagement. Anyone else?
> >
> > Time expectation for completion: primary improvements are complete and
> demonstrated. More incremental work will continue over a number of
> months. Goal completion for initial examples and specification is June
> with announcement of near-completion at SIGGRAPH in July. Working together
> with W3C colleagues will help.
> >
> > Our plan is to check status on integrating PR9 Audio into the X3D4 draft
> in 2 weeks, 1 May 2000, at 0900 pacific, one hour into the regular X3D
> Working Group teleconference. (Please note that it is a bank holiday in
> Crete!)
> >
> > - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
> >
> > 4. Fixing Mantis access: partially complete.
> >
> > Webmaster has provided access to Mantis for following contributors:
> >
> >> On 4/7/2020 11:54 AM, Don Brutzman wrote:
> >>> Webmaster, please add the following members to Mantis issue tracker:
> >>>
> >>> Athanasios Malamos <athanasios.malamos at gmail.com>
> >>>
> >>> Efi Lakka <efilakka at gmail.com>
> >>>
> >>> Jakub Flotyński <flotynski at kti.ue.poznan.pl>
> >>>
> >>> Thanks for all support.
> >
> > However need to add them to X3D4 project. Didn't work for me so far,
> apparently due to lack of permissions. Have again requested webmaster
> support.
> >
> > - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
> >
> > Suggested topic for next week: Cross Origin Resource Sharing (CORS),
> following up on prior meeting topics.
> >
> > Onward we go. Have fun with X3D audio and sound! 8)
> >
> > all the best, Don
>
> all the best, Don
> --
> Don Brutzman Naval Postgraduate School, Code USW/Br
> brutzman at nps.edu
> Watkins 270, MOVES Institute, Monterey CA 93943-5000 USA +1.831.656.2149
> X3D graphics, virtual worlds, navy robotics
> http://faculty.nps.edu/brutzman
> _______________________________________________
> x3d-public mailing list
> x3d-public at web3d.org
> http://web3d.org/mailman/listinfo/x3d-public_web3d.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://web3d.org/pipermail/x3d-public_web3d.org/attachments/20200521/0d3d55fc/attachment-0001.html>
More information about the x3d-public
mailing list