[x3d-public] Warning: controversial and disturbing

[John Carlson]

Streaming XML validation (answers anyone?)

https://stackoverflow.com/questions/11776056/how-to-validate-large-xml-streams-in-a-way-that-it-creates-chunk-of-stream-read

https://www.di.ens.fr/~segoufin/Papers/Mypapers/streaming-pods.pdf

https://stackoverflow.com/questions/66003045/streaming-xml-schema-validation-using-xsd-files-in-databricks

Streaming JSON validation:

https://arxiv.org/abs/2211.08891

https://github.com/awwright/jsonschemaparse

When will we see streaming parsers and validators on the web and in
Saxon-HE?

Have you tried converting an 10GB XML file to Python?  Or convert a large
JSON file to Java? (Hint, X3DJSONLD requires Ajv, Node.js or an ordinary
web JSON parser)

Anyone?

No one?

Am I alone?

John

On Mon, Oct 2, 2023 at 10:26 PM John Carlson <yottzumm at gmail.com> wrote:

> Document model seen as fundamentally limited.  The Metaverse is too big to
> fit in a Document.  Request/Response of HTTP/HTTPS seen as flawed,
> multi-request/multi-response (DIS?) preferred.
>
> We should move post-haste to solutions like DIS, VRML, SAI, Streaming API
> for XML (StAX), and streaming JSON parsers, and streaming validators.
>
> Things like DOM are unworkable for large streaming assets. Imagine a live
> feed with real-time animation over a period of hours.  A concert, for
> example.  Imagine several BVH files being streamed live—you’ve got to drop
> some portion or compress the file.
>
> Think of streaming solutions like JavaScript and Python.  Not something
> that fits in a single file, or outputs a single document, but something
> that consumes or emits a stream.  SAI is the right approach, but call it
> the Streaming Access Interface.
>
> Ever have a JSON file that was too large to parse?  Many JSON parsers
> assume that they can pass back a single object.  What if that object is too
> big for memory?  What happens?  We should focus on Streaming parsers for
> JSON and XML.  Don’t worry about duplicating keys!
>
> Current “create only” solutions do not fit the real world.  Updates and
> deletes with SAI and DIS should be considered.  Reduce, Reuse, Recycle.
> Systems that do not provide update and delete are flawed.  No plastics
> filling up my SSD, please!
>
> Yes, we will still support non-streaming technology, with a limited file
> size.  I recommend specifying a Content-Length on limited sized files.
>
> Maybe I should get a faster bus/disk?
>
> Is MSF listening, or are they doomed too?
>
> John
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://web3d.org/pipermail/x3d-public_web3d.org/attachments/20231002/3c3ebac9/attachment.html>