[X3D-Ecosystem] RawKee Update - Implemented a companion export script for x3d.py

Wed Mar 5 11:07:34 PST 2025

Note that this is not a panacea, and node order is still important in X3D
JSON, especially when converting to XML.

I do like Castle for validation, but it doesn’t do JSON yet.

There are tools available to convert JSON to XML, including create3000’s
x3d-tidy.  Then one can apply XML schema, schematron, etc.

This might be a start on getting a command-line tool to apply schematron to
JSON:

https://github.com/coderextreme/ci2had/blob/main/build.xml

If anyone knows an API to apply schematron to DOM, clue me in.

John

On Wed, Mar 5, 2025 at 12:04 PM John Carlson <yottzumm at gmail.com> wrote:

> My suggestion is to validate your code here:
>
> https://coderextreme.net/X3DJSONLD/src/main/html/validator.html
>
> before using jsonlint.  Jsonlint is used for figuring out location of
> syntax errors.  X3D is more nuanced, and I’m not sure that Sunrize
> completely tests everything completely.  Vince pointed out that Sunrize may
> ignore invalid fields, similar to HTML5.
>
> Good luck,
>
> John
>
> On Wed, Mar 5, 2025 at 10:57 AM Bergstrom, Aaron <aaron.bergstrom at und.edu>
> wrote:
>
>> John,
>>
>>
>>
>> Within the RawKee plugin, I’ve implemented the export functions in two
>> stages.
>>
>>    - Stage One - the Maya data gets stored in the x3d.py Scene.
>>    - Stage Two – the X3D contents get saved to disk.
>>
>>
>>
>> I did this because I have future plans to offer additional functionality
>> before the information is written to disk.
>>
>>
>>
>> While I haven’t yet fully written the code to export the HAnimHumanoid
>> node, I’ve run some tests. In my testing of HAnim data storage during Stage
>> One, I store the Maya joint nodes as DEF’d HAnimJoint nodes in the
>> ‘skeleton’ field first before I add them to the “joints” field as USE’d
>> HAnimJoint nodes.
>>
>>
>>
>> And then at the time of export, during Stage Two (at the very end of the
>> field sorting function in the RKSceneTraversal.py script), I flip the
>> position of the ‘joints’ and ‘skeleton’ fields in the array in which I hold
>> the names of the fields for the node that is about to be fed to the
>> ‘processSortedNode()’ method – which actually begins the process of writing
>> the node to disk.
>>
>>
>>
>> The Stage Two field flipping code is fairly simple, so I put it here –
>> where ‘nType’ is the type of X3D node that is about to exported and
>> ‘mNodeList’ is the list of fields of that X3D node that contain multiple
>> X3D nodes. The ‘HAnimHumanoid’ node type is the only node type in the
>> script that receives special treatment. The script is as follows:
>>
>>
>>
>> if nType == "HAnimHumanoid":
>>
>> jIdx = -1
>>
>> sIdx = -1
>>
>> for idx in range(len(mNodeList)):
>>
>> if mNodeList[idx] == "joints":
>>
>> jIdx = idx
>>
>> elif mNodeList[idx] == "skeleton":
>>
>> sIdx = idx
>>
>> if sIdx != -1 and jIdx != -1:
>>
>> mNodeList[jIdx] = "skeleton"
>>
>> mNodeList[sIdx] = "joints"
>>
>>
>>
>> While I haven’t finished the code for HAnimHumanioid export during Stage
>> One, the method described above aligns with how I exported HAnim content in
>> the old C++ version of RawKee, so I am fairly confident that it will work
>> as intended.
>>
>>
>>
>> I have verified that the JSON encoding produced by the
>> RKSceneTraversal.py script is valid JSON by pasting the code into the
>> online JSON Validator found here:
>>
>> https://jsonlint.com/
>>
>>
>>
>> I have verified that the scene graph is valid by loading the resulting
>> x3dj file into Sunrize.
>>
>>
>>
>> So as far as I can tell, the JSON code is correct, both for JSON validity
>> and X3D scene graph structure… at least for the node types that RawKee is
>> currently able to export.
>>
>>
>>
>> The ‘containerField’ field value is added as the custom Python attribute
>> ‘_RK__containerField’ to X3D node during Stage One, and is queried during
>> Stage Two when exporting to the XML encoding. ‘x3d.py’ has not be altered
>> in any way.
>>
>>
>>
>> For VMRL and JSON encodings, the ‘_RK__containerField’ attribute is
>> ignored.
>>
>>
>>
>> The missing ‘mapping’ field of TextureTransform is handled in a similar
>> manner. During Stage One, where I should be able to do something like:
>>
>> TextureTransform.mapping = “somestring”
>>
>>
>>
>> I instead do:
>>
>> TextureTransform._RK__mapping = “somestring”
>>
>>
>>
>> And then during Stage Two I query ‘_RK__mapping’ when it appears in the
>> field list.
>>
>>
>>
>> The field list for each node is produced by querying the Python
>> dictionary for that node using the following code:
>>
>> nDict = vars(node)
>>
>> for key, value in nDict.items():
>>
>>               …some more code…
>>
>>
>>
>> -- All of that said, I am a bit surprised at how well the
>> RKSceneTraversal.py script works. I started out hard coding the export for
>> each node type, and when I realized that was going to take me forever and
>> ever and ever to implement, I decided there had to be a way I could
>> implement it generically.
>>
>>
>>
>> I expected it to take me several weeks of troubleshooting to get it
>> right, but surprisingly, after I changed my approach, I was able to get it
>> to export correctly after only two days of work and in less than 700 lines
>> of code.
>>
>>
>>
>> It relies fairly heavily on x3d.py, so I would not be able to get it to
>> work with such a small amount of code without x3d.py. But from what I can
>> tell, at this point it looks to me like it should work with every X3D node
>> type in x3d.py.
>>
>>
>>
>> Also, I had a real hard time figuring out the JSON encoding, but Sunrize
>> does a very good job of this. So studied x3dj files created by Sunrize and
>> did my best to mimic this structure in the RawKee JSON export. Doing so
>> went surprisingly well. Sunrize has been key to helping work through many
>> exporter development problems, and verify that my exported encoding
>> actually functions correctly. I think without it, I would be lost.
>>
>>
>>
>> But I do make a lot of coding errors, so it’s quite possible that there
>> are use cases where my code fails. If anyone has the time and inclination,
>> I would appreciate it if others could take a look at the
>> RKSceneTraversal.py script and provide additional feedback. It can be found
>> here:
>>
>>
>> https://github.com/und-dream-lab/rawkee/blob/main/rawkee/RKSceneTraversal.py
>>
>>
>>
>> Thanks,
>>
>>
>>
>> Aaron
>>
>>
>>
>> *From:* John Carlson <yottzumm at gmail.com>
>> *Sent:* Wednesday, March 5, 2025 4:03 AM
>> *To:* X3D Ecosystem public discussion <x3d-ecosystem at web3d.org>
>> *Cc:* Bergstrom, Aaron <aaron.bergstrom at und.edu>
>> *Subject:* Re: [X3D-Ecosystem] RawKee Update - Implemented a companion
>> export script for x3d.py
>>
>>
>>
>> 1.    JSON export was completely broken.
>>
>> I tried fixing this in x3d.py and failed.  I have worked on python
>> validators for JSON export, but it’s slow in Python.  Validating the
>> scenegraph isn’t enough.
>>
>> 1.    The skeleton and joint export order of the HAnimHumanoid node
>> fields using x3d.py would theoretically dump all the DEF’d joints into the
>> ‘joints’ field, and thus only produce one USE’d root joint in the 'skeleton
>> field. Though I haven’t implemented HAnimHumanoid yet, I re-ordered the
>> field export so that the ‘skeleton’ field will get writen to the file
>> before the ‘joints’ field.
>>
>> You’ll also want to verify skin and skinCoord field order, plus sites and
>> segments field order with skeleton.
>>
>> 1.    ‘containerField’ was not implemented for XML export, which caused
>> the textures to not work because they weren’t allocated to the proper
>> texture field within the Material node.
>>
>> Andreas has a good patch for that.
>>
>> 1.    The ‘mapping’ field was missing from the TextureTransform object,
>> so it wouldn’t export.
>>
>>
>>
>> Good catch.
>>
>>
>>
>> Also,  how did you solve the DEF/USE ordering issue?
>>
>>
>>
>> I’ll post what i have for JSON elsewhere.
>>
>>
>>
>> John
>>
>> I will do a bit of a search for validation.
>>
>> On Wed, Mar 5, 2025 at 3:39 AM Bergstrom, Aaron via X3D-Ecosystem <
>> x3d-ecosystem at web3d.org> wrote:
>>
>> I was having issues with some of the export functionality of x3d.py.
>>
>>
>>
>> I implemented a companion script. And wrote about it on the Castle Game
>> Engine Website:
>>
>>
>> https://forum.castle-engine.io/t/rawkee-x3d-exporter-plugin-for-maya-repost/1804/3
>>
>>
>>
>> Aaron
>>
>> --
>> X3D-Ecosystem mailing list
>> X3D-Ecosystem at web3d.org
>> http://web3d.org/mailman/listinfo/x3d-ecosystem_web3d.org
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://web3d.org/pipermail/x3d-ecosystem_web3d.org/attachments/20250305/dec44f7b/attachment-0001.html>