[x3d-public] towards concurrent X3D processing

[Andreas Plesch]

Hi John,

On Sun, May 1, 2022 at 11:59 PM John Carlson <yottzumm at gmail.com> wrote:
>
> I couldn’t read all of this, but I recall keeping a counter of “children loaded”. When I reached the number of children = counter, I would exit loading.  The thing was, I would often hang on the last child.  I do not recall how I solved this.
>
> I can probably dig into this.  I don’t recall adding extra state to nodes, rather I used a local variable, possibly passed as a parameter. I recall the number of callback functions as horrendous, so doing something with await async would probably be preferred?  Idk, sorry.
>
> I get the heebee jeebees just thinking about it.

I think it is too early to dig into code for this. It is called
callback hell for a reason. In JS async await is preferred by most; I
got almost used to Promise. then which I think in the end is
equivalent.

> I think your containerFieldIndex may be a good idea, but I worry about missing or removed indexes.

At this point, it is about thinking through such a feature. Another
idea is to investigate how generic xml parent-child hierarchies can be
parsed in parallel. This seems complicated but a common, and possibly
solved problem.

True, I think being able to fill array slots out of sequence but not
earlier ones would require handling of empty or null nodes, mostly by
doing nothing when encountering those.

Thanks, Andreas

> John
>
> On Sun, May 1, 2022 at 8:48 PM Andreas Plesch <andreasplesch at gmail.com> wrote:
>>
>> Please let me outline an idea which would help with concurrent and
>> asynchronous processing by making the position of a node in an MFNode
>> array explicit. A new attribute similar to "containerField" would hold
>> the index of the node in the array, a "containerFieldIndex" attribute.
>> This would probably only apply to MFNode fields, and allow a browser
>> to easily assign the intended array position to a node, even if its
>> loading and processing is delayed, and thus support parallel and out
>> of sequence processing better.
>>
>> Feel free to skip the longer explanation below, Any feedback welcome.
>>
>> X3D relies on the idea of sequential processing of a file. For
>> example, USE nodes can only occur after their DEFinitions, or
>> ProtoInstances can only occur after the corresponding
>> ProtoDeclarations.
>>
>> This helps with processing since the browser can assume a sensible
>> sequence in the file, or complain otherwise to the author. It also
>> helps with authoring since it provides a clear paradigm an author can
>> understand and follow.
>>
>> However, for nodes using remote resources sequential processing can
>> lead to complete blocking while waiting for timeouts, or due to slow
>> or unreliable server responses. Also, some nodes which use a lot of
>> data, eg. large IFS, may take significant time to parse and add to the
>> scene. In other words, for fast loading of large scenes it is
>> desirable to process nodes in parallel and out of order or
>> asynchronously.
>>
>> This is possible even with a sequential loading paradigm, the browser
>> just has to carefully keep track of things, use placeholders etc. In
>> fact, x3dom loads Inline and ExternProtos asynchronously, adding
>> content and instances out of sequence to an already running scene.
>> This way a user can start to see and interact with a scene earlier.
>>
>> To do this, x3dom uses the "containerFieldIndex" idea internally, to
>> first synchronously record the index position of an ExternProto
>> instance during sequential loading. Then, after out of sequence, slow
>> downloading and processing, the instance is added to the parent MFNode
>> field in the standard manner, eg. at the end of the array. But in a
>> final step, x3dom can use the recorded containerFieldIndex to
>> rearrange the array to have the correct order.
>>
>> While this can be done internally, exposing a "containerFieldIndex"
>> attribute would have benefits. It would help authors to be both clear
>> and flexible about how they want nodes ordered, especially when
>> programmatically constructing scenes. It helps browsers since they
>> could rely on such an attribute in their processing. It would be
>> universal and apply to MFNode fields of any node. And it would promote
>> parallel processing since nodes become more self-describing. For
>> backwards compatibility, the default value of a containedFieldIndex
>> attribute would be the index of the node as given by its position in
>> the file.
>>
>> "containerField" is already a strange concept (since it is unusual to
>> refer to something in parents), so "containerFieldIndex" may be even
>> stranger. But I could not think of a better idea to help with
>> asynchronous loading. I am pretty sure there are better ones.
>>
>> Thanks for reading to the end ! Cheers, Andreas
>>
>> --
>> Andreas Plesch
>> Waltham, MA 02453
>>
>> _______________________________________________
>> x3d-public mailing list
>> x3d-public at web3d.org
>> http://web3d.org/mailman/listinfo/x3d-public_web3d.org



-- 
Andreas Plesch
Waltham, MA 02453