[x3d-public] Sound v4 > Convolver, BufferAudioNode RECOMMENDATIONS
Brutzman, Donald (Don) (CIV)
brutzman at nps.edu
Thu Feb 23 21:50:01 PST 2023
Doug, thanks for continuing implementation efforts and close scrutiny. Very important.
Dr. Malamos is not available this week so we’ll push audio discussion back to Friday 3 MAR.
Seems like a lot has accumulated, should you consolidate your recommendations? Am thinking we should collectively review and respond to the full set.
We can talk during tomorrow’s call about how to document all this – perhaps in Mantis with periodic email updates, or a dedicated issues page.
In any case, TIA.
all the best, Don
--
Don Brutzman Naval Postgraduate School, Code USW/Br brutzman at nps.edu
Watkins 270, MOVES Institute, Monterey CA 93943-5000 USA +1.831.656.2149
X3D graphics, virtual worlds, Navy robotics https:// faculty.nps.edu/brutzman
From: x3d-public <x3d-public-bounces at web3d.org> On Behalf Of GPU Group
Sent: Thursday, February 23, 2023 2:46 PM
To: X3D Graphics public mailing list <x3d-public at web3d.org>
Subject: [x3d-public] Sound v4 > Convolver, BufferAudioNode RECOMMENDATIONS
RECOMMENDATIONS
1. Convolver: change MFFloat buffer to SFNode buffer [AudioBuffer]
2. Add AudioBuffer node type
3. Refactor BufferAudioSource to delegate loading to AudioBuffer
4. browser developers: beware implementing BufferAudioSource via web audio AudioBufferSource
-Doug Sanden
Explanation:
Convolver is incomplete because it doesn't have a field to specifiy the number of channels in the buffer. Without that, there's no way to know how many channel chunks to split the MFFloat PCM32 data into. Yet the documentation claims it can have any number of channels.
3 options to fix:
a) in text, say single channel only
b) add a bufferChannels field
c) delegate loading to AudIoBuffer node
I recommend c) to gain the additional benefits of URL loading. Most web audio Convolver examples load impulse response data from .wav files.
https://middleearmedia.com/web-audio-api-convolver-node/ <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmiddleearmedia.com%2Fweb-audio-api-convolver-node%2F&data=05%7C01%7Cbrutzman%40nps.edu%7Cea6199f665ae4be3a9c108db15efe48f%7C6d936231a51740ea9199f7578963378e%7C0%7C0%7C638127893310275903%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=OQwJwzS5WRuG6CmBOYdhvMYSUVJW3Jm3XMYonwpxeUE%3D&reserved=0>
In web audio, both Convolver and AudioBufferSource have a .buffer field that takes an AudioBuffer node.
https://developer.mozilla.org/en-US/docs/Web/API/ConvolverNode <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdeveloper.mozilla.org%2Fen-US%2Fdocs%2FWeb%2FAPI%2FConvolverNode&data=05%7C01%7Cbrutzman%40nps.edu%7Cea6199f665ae4be3a9c108db15efe48f%7C6d936231a51740ea9199f7578963378e%7C0%7C0%7C638127893310275903%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=KKOT0ipVtlTlmI8%2FvGJcUJlzXW4dgsqhWXcPq%2BYgcww%3D&reserved=0>
- example code shows loading AudioBuffer from .wav
https://developer.mozilla.org/en-US/docs/Web/API/AudioBufferSourceNode <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdeveloper.mozilla.org%2Fen-US%2Fdocs%2FWeb%2FAPI%2FAudioBufferSourceNode&data=05%7C01%7Cbrutzman%40nps.edu%7Cea6199f665ae4be3a9c108db15efe48f%7C6d936231a51740ea9199f7578963378e%7C0%7C0%7C638127893310275903%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=R%2FFHxQcfqphgzKiTcFHWVLnGopvKVm4HdoEh%2FDRUTSI%3D&reserved=0>
- example code shows loading PCM data in AudioBuffer algorithmically at run time
https://developer.mozilla.org/en-US/docs/Web/API/AudioBuffer <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdeveloper.mozilla.org%2Fen-US%2Fdocs%2FWeb%2FAPI%2FAudioBuffer&data=05%7C01%7Cbrutzman%40nps.edu%7Cea6199f665ae4be3a9c108db15efe48f%7C6d936231a51740ea9199f7578963378e%7C0%7C0%7C638127893310275903%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=I2IYy7jXY%2BIkT1kYY6v%2BBtYf3xpF%2BzNjV8b4gGJt8mM%3D&reserved=0>
- explains 2 ways to load data, from .wav or from raw data (PCM)
Having both BufferAudioSource and Convolver delegate loading to AudioBuffer will reduce code needed, match web audio, while fixing Convolver shortcoming and adding url loading to Convolver through AudioBuffer
i have the recommendations working in freewrl with these scene files:
https://freewrl.sourceforge.io/tests/16_Sound/convolver_control.wrl <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Ffreewrl.sourceforge.io%2Ftests%2F16_Sound%2Fconvolver_control.wrl&data=05%7C01%7Cbrutzman%40nps.edu%7Cea6199f665ae4be3a9c108db15efe48f%7C6d936231a51740ea9199f7578963378e%7C0%7C0%7C638127893310275903%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=NboaUmYfeyYfgYIOIXKUk6Sm34n2G617tJ%2FDv6%2FZpnY%3D&reserved=0>
https://freewrl.sourceforge.io/tests/16_Sound/convolver_control_PCM.wrl <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Ffreewrl.sourceforge.io%2Ftests%2F16_Sound%2Fconvolver_control_PCM.wrl&data=05%7C01%7Cbrutzman%40nps.edu%7Cea6199f665ae4be3a9c108db15efe48f%7C6d936231a51740ea9199f7578963378e%7C0%7C0%7C638127893310275903%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=F72OplQ93ljYIgGDnDPcO1W%2Bassh6CWl6y8YXP%2BX9jY%3D&reserved=0>
https://freewrl.sourceforge.io/tests/16_Sound/BufferAudioSource.x3d <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Ffreewrl.sourceforge.io%2Ftests%2F16_Sound%2FBufferAudioSource.x3d&data=05%7C01%7Cbrutzman%40nps.edu%7Cea6199f665ae4be3a9c108db15efe48f%7C6d936231a51740ea9199f7578963378e%7C0%7C0%7C638127893310275903%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=F1%2Fg3Xtx4cmBaW3orMWW0VA6L3QrGLrrRGXClp1SL3A%3D&reserved=0>
Implementing BufferAudioSource
https://developer.mozilla.org/en-US/docs/Web/API/AudioBufferSourceNode <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdeveloper.mozilla.org%2Fen-US%2Fdocs%2FWeb%2FAPI%2FAudioBufferSourceNode&data=05%7C01%7Cbrutzman%40nps.edu%7Cea6199f665ae4be3a9c108db15efe48f%7C6d936231a51740ea9199f7578963378e%7C0%7C0%7C638127893310275903%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=R%2FFHxQcfqphgzKiTcFHWVLnGopvKVm4HdoEh%2FDRUTSI%3D&reserved=0>
"It will automatically be garbage-collected" sometime after it quits playing.
The unusual nature of automatic garbage collection in procedural code doesn't apply well to declarative web3d syntax where nodes perisist for the life of the scene run. Labsound native library for web audio has no AudioBufferSourceNode - the one node its missing - due to the garbage collection behavior, and they've had no requests for it. So I can't test implementing BufferAudioSource via web audio AudioBufferSource and can't recommend either way.
In native code via Labsound I'm able to implement BufferAudioSource the same way as AudioClip --Labsound SampledAudioSource which has detune and playbackRate, stop,start,loop-- and have that working in FreeWRL.
ORIGINAL BUFFERAUDIOSOURCE
BufferAudioSource : X3DSoundSourceNode, X3DUrlObject {
SFTime [in,out] autoRefresh 0.0 [0,∞)
SFTime [in,out] autoRefreshTimeLimit 3600.0 [0,∞)
MFFloat [in,out] buffer [] [−1,1]
SFTime [in,out] bufferDuration 0 [0,∞)
SFString [in,out] channelCountMode "max" ["max", "clamped-max", "explicit"]
SFString [in,out] channelInterpretation "speakers" ["speakers", "discrete"]
SFString [in,out] description ""
SFFloat [in,out] detune 0 [0,∞)
SFBool [in,out] enabled TRUE
SFFloat [in,out] gain 1 (-∞,∞)
SFBool [in,out] load TRUE
SFBool [in,out] loop FALSE
SFTime [in,out] loopEnd 0 [0,∞)
SFTime [in,out] loopStart 0 [0,∞)
SFNode [in,out] metadata NULL [X3DMetadataObject]
SFInt32 [in,out] numberOfChannels 0 [0,∞)
SFTime [in,out] pauseTime 0 (-∞,∞)
SFFloat [in,out] playbackRate 1 (-∞,∞)
SFTime [in,out] resumeTime 0 (-∞,∞)
SFFloat [in,out] sampleRate 0 [0,∞)
SFTime [in,out] startTime 0 (-∞,∞)
SFTime [in,out] stopTime 0 (-∞,∞)
MFString [in,out] url [] [URI]
SFInt32 [out] bufferLength 0 [0,∞)
SFInt32 [out] channelCount [0,∞)
SFTime [out] elapsedTime
SFBool [out] isActive
SFBool [out] isPaused
}
/ORIGINAL BUFFERAUDIOSOURCE
REFACTORED BUFFERAUDIOSOURCE
AudioBufferSource : X3DSoundSourceNode {
SFNode [in,out] buffer NULL [AudioBuffer]
SFString [in,out] channelCountMode "max" ["max", "clamped-max", "explicit"]
SFString [in,out] channelInterpretation "speakers" ["speakers", "discrete"]
SFString [in,out] description ""
SFFloat [in,out] detune 0 [0,∞)
SFBool [in,out] enabled TRUE
SFBool [in,out] loop FALSE
SFTime [in,out] loopEnd 0 [0,∞)
SFTime [in,out] loopStart 0 [0,∞)
SFNode [in,out] metadata NULL [X3DMetadataObject]
SFTime [in,out] pauseTime 0 (-∞,∞)
SFFloat [in,out] playbackRate 1 (-∞,∞)
SFTime [in,out] resumeTime 0 (-∞,∞)
SFTime [in,out] startTime 0 (-∞,∞)
SFTime [in,out] stopTime 0 (-∞,∞)
SFTime [out] elapsedTime
SFBool [out] isActive
SFBool [out] isPaused
}
/REFACTORED BUFFERAUDIOSOURCE
AUDIOBUFFER
BufferAudioSource : X3DUrlObject {
SFTime [in,out] autoRefresh 0.0 [0,∞)
SFTime [in,out] autoRefreshTimeLimit 3600.0 [0,∞)
MFFloat [in,out] bufferData [] [−1,1]
SFInt32 [in,out] bufferChannels 1 [0,∞)
SFString [in,out] channelCountMode "max" ["max", "clamped-max", "explicit"]
SFString [in,out] channelInterpretation "speakers" ["speakers", "discrete"]
SFString [in,out] description ""
SFBool [in,out] load TRUE
SFNode [in,out] metadata NULL [X3DMetadataObject]
SFFloat [in,out] sampleRate 0 [0,∞)
MFString [in,out] url [] [URI]
SFInt32 [out] bufferLength 0 [0,∞)
SFInt32 [out] channelCount [0,∞)
}
Note on field names bufferData and bufferChannels
- web audio uses numberOfChannels as does current BufferAudioNode
- but its not clear numberOfChannels only applies the the raw PCM32 data -- not to the URL loaded .wav file which will get its channel count from the .wav file.
- to make the association clearer I recommend the names for buffer data and channel count have the same prefix, so will be sequential in specs alphabetically listed fields
/AUDIOBUFFER
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://web3d.org/pipermail/x3d-public_web3d.org/attachments/20230224/f0637eb8/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 5353 bytes
Desc: not available
URL: <http://web3d.org/pipermail/x3d-public_web3d.org/attachments/20230224/f0637eb8/attachment-0001.p7s>
More information about the x3d-public
mailing list