[x3d-public] X3D4 Sound meeting 20 SEP 2020: Web3D 2020 preparations, Gain and ChannelSelector nodes, avoiding channel indices via parent-child MFNode field relationships

Don Brutzman brutzman at nps.edu
Wed Sep 30 11:02:54 PDT 2020


Attendees: Efi, Thanos, Dick, Don

Prior minutes:

[0] [x3d-public] X3D Audio and Sound meeting minutes, 17 SEP 2020: channel splitting and selecting single channel
     http://web3d.org/pipermail/x3d-public_web3d.org
     http://web3d.org/pipermail/x3d-public_web3d.org/2020-September/013609.html

---

1. "bad news = good news" department  8)

X3D4 glTF design is done, so... we need to finish sound component changes!

Also: program is not yet online, but please register for Web3D 2020:

* https://web3d.siggraph.org

---

2. W3C Audio Group

Efi is in the group.  Will check on other status...

When we have our spec changes and tutorial ready, we are ready to ask to brief the W3C Audio Working Group.

We will let the completion of materials for our tutorial be the threshold for what we provide them.  Having something stable helps discussion.  Thanos thinks we might attract some more people too.

---

3.  Slideset

Efi has an initial draft slideset for tutorial.  Work continues.

Planned tutorial recording: we set a goal to be ready for 3rd or 4th week of October.

---

4. Examples and Node signatures

 From last week's minutes:

> 4. Tricky problem: channel subselection (single or multiple).  We talked about this a lot.
> 
> Having a field on ChannelSplitter to select a specific channel is appealing, but then may be difficult to reuse because changing values in an MFInt32 array typically requires a Script to accomplish.  We are being exceedingly careful to stay aligned as closely as possible with Web Audio API [1].
> 
> Possible new utility node to augment ChannelSplitter:  maybe a ChannelSelector node.
> 
> ChannelSelector : X3DSoundSourceNode {
>   SFNode  [in]     source        [X3DSoundSource]
>   SFInt32 [in out] channelNumber 0
> 
> Conceivably any source might be provided but typically this would be ChannelSplitter.
> 
> a. ChannelSplitter can provide access to multiple channels at once, might allow selection of several channels, and preserves correspondence to WebAudioApi.  Does that have essential completeness for our use cases?
> 
> b. ChannelSelector can be defined to only give a single channel always, making semantics clear and addressing of individual channels easy.
> 
> We discussed
> - whether channel selection might be in Sound and SpatialSound nodes,
> - noted that .wav and .midi can be stereo - hence Sound and SpatialSound nodes are also,
> - keeping sound nodes separate from processing nodes,
> - ChannelSplitter only accepts single multichannel input,
> - ChannelMerger   only accepts multiple singlechannel inputs.
> 
> ChannelSplitter needs a selection field.  Key question, can:
> - multiple indexes be used, providing a subset of several input channels, or
> - only a single index for a single channel is used?
> 
> If there is no way to directly select a single channel from a ChannelSplitter node, then we need a mechanism to do that (such as ChannelSelector node).  We have not yet detected such a mechanism directly in Web Audio API.
> 
> Recommended next step: model prior incomplete X3D example using ChannelSelector node; and if subsequently we find a way to do it using WebAudioAPI nodes/fields then it will be easy to converge. 

Here is Example 4.  Previous existing version, with difficulties reconstructing the audio graph topology:

> http://www.medialab.teicrete.gr/minipages/x3domAudio/splitChannels.xhtml
> Before the update:
> <ChannelSplitter DEF='ChannelSplitter2' channelCountMode = 'explicit'>
>     <AudioBufferSource DEF='AudioBufferSource2' containerField='inputs'/>
> <ChannelSplitter/>
> 
> <AudioDestination DEF='AudioDestination2' channelInterpretation= 'speakers' gain= '0.7'>
>     <Gain DEF='Gain2_4' containerField='inputs'>
>         <Gain DEF='Gain2_3' containerField='inputs'>
>             <ChannelMerger DEF='ChannelMerger2' containerField='inputs'>
>                 <Gain DEF='Gain2_0' containerField='inputs'/>
>                     <ChannelSplitter USE= 'ChannelSplitter2' containerField='inputs'>
>                 <Gain DEF='Gain2_1' containerField='inputs'/>
>                     <ChannelSplitter USE= 'ChannelSplitter2' containerField='inputs'>
>             </ChannelMerger>
>         </Gain>
>     </Gain>
> </AudioDestination>

Hmmm....

It seems we might avoid any explicit indexing whatsoever by taking advantage of ordering of input nodes and output nodes.  It is helpful to avoid explicit indexing to avoid possible counting errors or need for Script animation of selections.

So... looks like we avoided problems with <ChannelSplitter USE= 'ChannelSplitter2' containerField='inputs'> through modifications to our tricky example as follows:

===============================
<AudioDestination DEF='AudioDestination2' channelInterpretation= 'speakers' gain= '0.7'>
     <Gain DEF='Gain2_4' containerField='inputs'>
         <Gain DEF='Gain2_3' containerField='inputs'>
             <ChannelMerger DEF='ChannelMerger2' containerField='inputs'>
                 <Gain DEF='Gain2_0' containerField='inputs'/><-- gets a channel signal below -->
                 <Gain DEF='Gain2_1' containerField='inputs'/><-- gets a channel signal below -->
             </ChannelMerger>
         </Gain>
     </Gain>
</AudioDestination>

<ChannelSplitter DEF='ChannelSplitter2' channelCountMode = 'explicit'>
     <AudioBufferSource DEF='AudioBufferSource2' containerField='inputs'/>
     <Gain USE='Gain2_0' containerField='outputs'/> <!-- initial output is audio channel 0 -->
     <Gain USE='Gain2_1' containerField='outputs'/> <!-- second  output is audio channel 1 -->
<ChannelSplitter/>
===============================

If we define containerField='inputs' as X3D XML default, we can omit it from XML without ambiguity.

Corresponding terse form, with some further helpful comments added:

===============================
<AudioDestination DEF='AudioDestination2' channelInterpretation='speakers' gain= '0.7'>
     <Gain DEF='Gain2_4'>
         <Gain DEF='Gain2_3'>
             <ChannelMerger DEF='ChannelMerger2'>
                 <!-- order of input children channels is correspondence to channel index 0, 1, 2, etc. -->
                 <Gain DEF='Gain2_left'/> <-- initial input gets a channel signal from below -->
                 <Gain DEF='Gain2_right'/><-- second  input gets a channel signal from below -->
		<!-- interestingly you might drop a channel here, Gain2_other is not utilized -->
		<!-- interestingly you might even swap left/right here simply by changing order of Gain nodes -->
             </ChannelMerger>
         </Gain>
     </Gain>
</AudioDestination>

<ChannelSplitter DEF='ChannelSplitter2' channelCountMode = 'explicit'>
     <AudioBufferSource DEF='AudioBufferSource2'/>
     <Gain USE='Gain2_left'  containerField='outputs'/> <!-- initial output is audio channel 0 -->
     <Gain USE='Gain2_right' containerField='outputs'/> <!-- second  output is audio channel 1 -->
     <Gain USE='Gain2_other' containerField='outputs'/> <!-- third   output is audio channel 2 -->
<ChannelSplitter/>
===============================

We think that ChannelSplitter is useful to include in specification, and more study (perhaps with more examples) is a good idea too.

---

b. Any other field names as containerField candidates?  Currently inputs/outputs as defined in spec.

We can set up XML Schema validation so that only correct values are allowed.  Now on Don's (exceedingly long) TODO list.

---

c. Sound location: as we've seen repeatedly, conflating audio with spatial gets confusing.

We can simply keep audio-graph topology (for connections) separate from scene-graph topology for transformation hierarchy.  For example,

<Transform DEF='locationFirstSpeaker' translation='1000 2 1000'>
    <AudioDestination USE='AudioDestination1'/>
</Transform

<Transform DEF='locationSecondSpeaker' translation='0 2 0'>
    <AudioDestination USE='AudioDestination2'/>
</Transform>

Efi you will want to have a representative example or two, in your code and slides.

---

d. Updating X3D4 specification:

We would like to still include a gain field as part of nodes (renamed outputGain) as part of 16.3.4 X3DSoundProcessingNode.

Need node signatures for the following:

- insert ChannelSelector as 16.4.7, and
- insert Gain as 16.4.11

Efi will send drafts for following nodes, Don will refine and put in draft spec for review during next meeting:

- Gain
- ChannelSelector

That should complete our list of Audio and Sound nodes needed for X3D4 Sound component... hooray, we hope!

Updating the specification will be good for reviewing this thoroughly.

---

5. Publishing examples

Efi's examples are our primary source, and found online at

[5] Spatial Sound in X3DOM with Web Audio API
     http://www.medialab.teicrete.gr/minipages/x3domAudio/index.html

To support testing of tools, Don is ready to add some of these to X3D Schema, DOCTYPE, X3DUOM, Tooltips, X3DJSAIL Java, X3DPSAIL Python (similar to what we just did for PBR nodes).

TODO "Real Soon Now" when a little bit more stable: adding example scenes to Web3D Examples Archives for regression testing.

---

High doppler shifts, we are accelerating and moving out smartly...

Have fun with X3D4 Audio and Sound!  8)

all the best, Don
-- 
Don Brutzman  Naval Postgraduate School, Code USW/Br       brutzman at nps.edu
Watkins 270,  MOVES Institute, Monterey CA 93943-5000 USA   +1.831.656.2149
X3D graphics, virtual worlds, navy robotics http://faculty.nps.edu/brutzman



More information about the x3d-public mailing list