[x3d-public] x3d.py problem validating MFVec2f default value, other fixes applied, version 4.0.64.1 released

[John Carlson]

Don, your work looks good, will try to check online soon, your review and
response to this letter is most appreciated, as I have a lot to discuss on
one small topic you brought up (included below).

Apologies for a draft email.

On Sun, Jun 12, 2022 at 8:40 PM Brutzman, Donald (Don) (CIV) <
brutzman at nps.edu> wrote:

 I believe such conversions from long arrays to arrays of typed tuples is
already working in X3dToJson.xslt conversions, so this resolution improved
x3d.py itself.
==
I don't know if we agreed to changing X3dToJson.xslt to output nested
arrays for MF fields, particularly MFVec* fields. I believe we did agree to
change one MFString from an array to a string.   I'm pretty sure that
x3djsonld.py does not accept JSON arrays of arrays and would probably have
to change, probably for the better.  x3djsonld.py is a divergence from
historical X3DJSONLD in that x3djsonld.py skips XML.  in X3DJSONLD, there's
a preprocessor that converts to XML/DOM and then a postprocessing
serializer that converts to SAI.  Other approaches would read X3D JSON into
a typed scenegraph at runtime in a derivative of x3d.py, as previously
discussed before x3djsonld.py was created.

The main proposal for not having nested arrays was to smooth the transition
to WebGL typed arrays (what does glTF do?), but since we do not have native
X3D JSON in X3DOM and X_ITE, that issue is probably moot--more important
there is conversion to XML.  I believe that adding native JSON parsing
(beyond using X3DJSONLD/JSONParser.js) may be a significant effort, but
probably well worth it--We can follow the x3djsonld.py code and convert to
JavaScript/TypeScript/ECMAScript, then do an eval.   I am not a library
developer, and finding someone to implement a full native X3D JSON parser
may be problematic, we should discuss design with people knowledgeable of
X3DOM and X_ITE (Perhaps Andreas and Holger initially), and get their buy
off before disabling and changing X3D JSON for those platforms.  A good way
to support both typed and non-typed arrays is to use a file extension (not
perfect).  Another way to support is to test for nested arrays in X3DJSONLD
and friends, then flatten the array to XML/DOM to go to X_ITE and X3DOM.
 I was able to shim in X3D JSON with flat arrays fairly easily without a
lot of disruption of current design. Primarily because we were going from
JSON to XML/DOM, which had flat arrays.  We might have to add code for each
MFVec* field type, for example, if we go with a nested design.

Arrays are treated differently in X3DOM Scripts and X_ITE VRMLScript last I
checked.   I believe that X_ITE follows the SAI.

I do not know if X3D JSON needs to follow the SAI (unlike x3d.py), and
creating "typed" tuples in JSON is very suspect. I agree that nested arrays
may be feasible, but I'm unsure about how much rework needs to be done in
X3DJSONLD (all languages, X3DOM and X_ITE JSONParser.js).  Tread carefully,
or the "lion," as Joe puts it, may reemerge.  I recommend we solidify the
X3D JSON working draft before proceeding, to avoid even more work when
people try to implement X3D JSON.

I do not know if anyone is using arrays of typed tuples in X3dToJson.xslt,
as opposed to a flat array.

Probably the X3D JSON Working Draft needs to be changed for all MFVec*
field types.

I don't even know if your proposed solution for typed arrays is truly
typesafe (each inner tuple should be converted, if we follow X_ITE's lead).

Please talk through design reasoning and address customer issues (if any),
for supporting "typed" arrays in JSON (even saying it makes my mouth go
foul).

I agreee that ECMAScript or TypeScript SAI implemenations should include
typed arrays.  Those languages are not JSON.

I suggest we synchronize the output of my XML to JSON serializer and
X3dToJson.xslt.

I do not know why we are diverging in the X3D JSON encoding from XML
encoding, except to mirror VRML, which I do not know.  Older versions of
X3DJSONLD target XML, and x3d.py is the new kid on the block, where we are
targeting python, and I have not even seen a Python SAI working draft?  I
agree that targetting type safe JavaScript and friends from JSON in an
offline fashion may be desireable.  I have already tried to target
ECMAScript SAI, but I'm unsure whether to support X3DOM, X_ITE, or some
future SAI implementation.

Future unilateral changes should go through a test scenario, or at least
schema/validation checks before being implemented/committed.  That is, we
should either validate X3D JSON output from X3dToJson.xslt with JSON
Schema, X3DJSAIL, x3djsonld.py/x3d.py, plus end-to-end checks with major
browsers.

Can you show me what you do to validate JSON?  Please not
jshint/jslint...that's for JavaScript.   Even something like jsonlint would
be better, but not really validating.

I will try to contact you today to discuss the above in a meeting.  I know
I've been missing lots of meetings. When X3dToJson.xslt output is changed
significantly, we should agree to running through scenarios, probably F2F.
At least we should agree that output JSON should be tested with X3DJSAIL,
if not JSON schema.

Not only should we discuss JSON validation and how to proceed, we should
start reviewing x3djsonld.py output (with historical JSON output for now).
If you wish to review the x3djsonld.py or test, look here:
https://github.com/coderextreme/X3DJSONLD/blob/master/src/main/python/x3djsonld.py

John
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://web3d.org/pipermail/x3d-public_web3d.org/attachments/20220613/ccdd5f60/attachment.html>