[x3d-public] [x3dom-users] simple X3D -> JSON -> X3DOM, D3.js

Don Brutzman brutzman at nps.edu
Mon Feb 23 06:41:12 PST 2015


Thanks for the great analysis Kristian, sounds like we are all gradually understanding these worthy design challenges more and more deeply.  We need to review each of your points in detail.

Whether containerField information is implicit (as with XML encoding) or explicit (as with ClassicVRML/VRML encodings) is indeed a key design choice.  Both are certainly workable and each has tradeoffs such as you've listed.  I suggest that eventual consensus this design choice is primarily based on what approach is most useful to Javascript programmers, especially those writing loaders for Javascript APIs such as X3DOM, three.js, D3.js, et al.

Meanwhile I did some more work on the experimental X3dToJson.xslt stylesheet.  Realized from earlier email with John that the current online conversion examples should all be fully round-trippable from the root X3D element and not start at the Scene element... now fixed, parameter option remains available in stylesheet if someone someday wants to ignore header (head/component/unit/meta) information.  Still having trouble stripping DTD defaults, the SAX parser is ignoring an option... will continue troubleshooting.

Also updated the HelloWorld.json example and design commentary found in the stylesheet.  Corrected conversion of all the other examples is complete, will upload them.

Following is excerpt from the stylesheet documentation explaining current status and my latest attempt at describing our shared next-step challenges to build the X3D JSON encoding.  Have integrated several of your points and those recently raised, also trying to note open questions and alternatives.  Glad we are all continuing with this excellent dialog.

http://www.web3d.org/x3d/stylesheets/X3dToJson.xslt
https://sourceforge.net/p/x3d/code/HEAD/tree/www.web3d.org/x3d/stylesheets/X3dToJson.xslt

=======================================================================
X3D JSON Design Considerations and X3dToJson.xslt Converter Status

References and resources:
http://www.web3d.org/wiki/index.php/X3D_JSON_Encoding

1. Working features in this converter:
- XML elements (X3D nodes), XML attributes (X3D simple-type fields), comments.
- Square and squiggly brackets, commas.
- Escaping quotation marks.
- Handling of singleton numeric values.
- Handling of MFVec arrays (simple-array form).
- Suppression of default attribute values inserted by XML parser reading DOCTYPE
   http://stackoverflow.com/questions/11749319/how-can-i-ignore-the-doctype-declaration-with-xsl
   Saxon command-line option expand:off applied (but apparently doesn't always work)
   http://saxonica.com/documentation9.0/using-xsl/commandline.html
- Embedded support in X3D-Edit found under X3D Conversions menu list.
- Conversion applied and results available in X3D Examples Archives.
   http://www.web3d.org/x3d-resources/content/examples/X3dResources.html#Examples

2. TODO and regression testing:
- Strictly differentiate typing of string/number/boolean values and arrays.
   Currently accomplished heuristically, perhaps that is sufficient. Typing can
   allow Javascript type promotion/demotion as needed and make parsing simpler.
- Handling special characters (UTF-16, Unicode) in strings.
- Preserve contained CDATA text (principal example: contained Script source code).
- Round-trip testing using a JSON-to-XML converter to ensure completeness and
   correctness.
- Testing using X3DOM parsing, rendering, serialization.
- Testing using three.js, XML3D, webgl source code, other javascript libraries
   is also important and welcome.

3. What X3D JSON specification will need to specify:
    http://www.web3d.org/specifications/X3dSpecificationRelationships.png

- Round-trip conversion capability as primary requirement to show that features
   in the X3D Abstract Specification can all be represented in an X3D JSON file.
- @attributeName prefix using @ character.
- "#comment" preservation of comments as an allowed option.
- "#CDATA"   preservation of character data as a requirement.
- Handling of values and arrays for numeric, boolean and string types.
- Use of "null" (if any).
- Handling of singleton numeric values, e.g.
	"transparency": 1,
   rather than
	"transparency": [1],
- Handling of nested vector values in MFVec arrays, for example
	"keyValue": [0, 0, 0, 1, 1, 1, 2, 2, 2],
         (simpler form is a better match for rendering usage)
   or else
	"keyValue": [ [0, 0, 0], [1, 1, 1], [2, 2, 2] ],
         (structured form is easier for programmers to manipulate)
   or perhaps either?
- Omitting default attribute values allowed?
- Omitting or including default containerField values?  These might be the
   basis for restructuring the JSON to insert explicit containerField values
   prior to each node definition.

4. Functional goals and use cases:

Overview. JSON is a built-in JavaScript object. This works out of the box:
the parse method simply creates a valid JS object from a JSON string.
Once that is done, the resulting object tree still has to be traversed
in order to attach the renderable presentation and behavior if the scene graph.

The exact manner in which an Javascript engine chooses to interpret the JSON and
render the X3D scenegraph is an independent decision for each Javascript library.
The JSON representation is only meant to provide the scenegraph for appropriate
initialization in the 3D engine (and optionally export its state at a given
moment if needed).

5. Specific design goals and use cases include:
- X3D JSON is written in same javascript language as the rest of the application.
- it can be optimized with the rest of the application.
- it can be tested with the rest of the application.
- it can refer to and interact with non-3D modules (e.g. a third-party library
   for formatting Date timestamps).
- no eval() method needs to be applied.
- lightweight to parse.
- meaningful and low complexity because it's simpler to think with, easier to read,
   easier to diff between two states, and easier to synchronize between devices.
- easy to require(), to use the same package management as the rest of the
   javascript application code and assets.
- Easy for programmers to manipulate and adapt.
- Strive for reusability, avoid hard-wiring the object syntax to one exclusive
   API approach if possible (there are many "similar but different" scene graphs).
- Future ISO 19776-5 X3D JSON Encoding has to capture everything representable
   in a scene graph defined by the X3D Abstract Specification.  Full expressoin
   of all scene graph information is commonly referred to as "round trippable"
   property.  The round-trippable requirement is shared by all X3D encodings:
   XML .x3d, ClassicVRML .x3dv, Compressed Binary .x3db.
- TBD what is use-case requirement for strict or flexible typing of data values?

6. Examples

- HelloWorld.json and HelloWorld.scene.json
   http://www.web3d.org/x3d/content/examples/HelloWorld.x3d
   http://www.web3d.org/x3d/content/examples/HelloWorld.json
- All Web3D archive examples are automatically converted and published:
   http://www.web3d.org/x3d/content/examples/X3dResources.html#Examples
- Need X3DOM examples with corresponding parsing of someScene.json
- Other example uses welcome.

7. Apparently not possible

- Embedded JSON comments are specifically disallowed by JSON specification
   and so round-trippable inclusion must be a custom feature of this encoding.
   http://www.quora.com/How-do-I-write-comments-inside-a-JSON-document
- No JSON-unique header. Can optionally use X3D/head/meta name=value pairs
   (if capturing full document) or comments in a scene-graph fragment
   or even an X3D Metadata node.  However for anything intended to be
   interoperable/reusable, consistency and repeatability is needed.
=======================================================================

all the best, Don
-- 
Don Brutzman  Naval Postgraduate School, Code USW/Br       brutzman at nps.edu
Watkins 270,  MOVES Institute, Monterey CA 93943-5000 USA   +1.831.656.2149
X3D graphics, virtual worlds, navy robotics http://faculty.nps.edu/brutzman
-------------- next part --------------
{
  "X3D": [
  {
    "@profile": "Immersive",
    "@version": 3.3,
    "@xsd:noNamespaceSchemaLocation": "http://www.web3d.org/specifications/x3d-3.3.xsd",
    "head": [
    {
      "meta": {
        "@content": "HelloWorld.x3d",
        "@name": "title"
      },
      "meta": {
        "@content": "Simple X3D scene example: Hello World!",
        "@name": "description"
      },
      "meta": {
        "@content": "30 October 2000",
        "@name": "created"
      },
      "meta": {
        "@content": "23 November 2014",
        "@name": "modified"
      },
      "meta": {
        "@content": "Don Brutzman",
        "@name": "creator"
      },
      "meta": {
        "@content": "HelloWorld.tall.png",
        "@name": "Image"
      },
      "meta": {
        "@content": "http://en.wikipedia.org/wiki/Hello_world",
        "@name": "reference"
      },
      "meta": {
        "@content": [ en.wikipedia.org/wiki/Hello#"Hello,_World"_computer_program ],
        "@name": "reference"
      },
      "meta": {
        "@content": "http://en.wikibooks.org/w/index.php?title=Computer_Programming/Hello_world",
        "@name": "reference"
      },
      "meta": {
        "@content": "http://www.HelloWorldExample.net",
        "@name": "reference"
      },
      "meta": {
        "@content": "http://www.web3D.org",
        "@name": "reference"
      },
      "meta": {
        "@content": "http://www.web3d.org/realtime-3d/news/internationalization-x3d",
        "@name": "reference"
      },
      "meta": {
        "@content": "http://www.web3d.org/x3d/content/examples/HelloWorld.x3d",
        "@name": "reference"
      },
      "meta": {
        "@content": "http://X3dGraphics.com/examples/X3dForAdvancedModeling/HelloWorldScenes",
        "@name": "reference"
      },
      "meta": {
        "@content": "http://X3dGraphics.com/examples/X3dForWebAuthors/Chapter01-TechnicalOverview/HelloWorld.x3d",
        "@name": "identifier"
      },
      "meta": {
        "@content": "http://www.web3d.org/x3d/content/examples/license.html",
        "@name": "license"
      },
      "meta": {
        "@content": "X3D-Edit 3.3, https://savage.nps.edu/X3D-Edit",
        "@name": "generator"
      },
      "#comment": "Alternate encodings: VRML97, X3D ClassicVRML Encoding, X3D Compressed Binary Encoding (CBE), X3DOM, JSON",
      "meta": {
        "@content": "HelloWorld.wrl",
        "@name": "reference"
      },
      "meta": {
        "@content": "HelloWorld.x3dv",
        "@name": "reference"
      },
      "meta": {
        "@content": "HelloWorld.x3db",
        "@name": "reference"
      },
      "meta": {
        "@content": "HelloWorld.xhtml",
        "@name": "reference"
      },
      "meta": {
        "@content": "HelloWorld.json",
        "@name": "reference"
      }
    }
    ],
    "Scene": [
    {
      "#comment": "Example scene to illustrate X3D nodes and fields (XML elements and attributes)",
      "Group": [
      {
        "@bboxCenter": [ 0, 0, 0 ],
        "@bboxSize": [ -1, -1, -1 ],
        "@containerField": "children",
        "Viewpoint": {
          "@DEF": "ViewUpClose",
          "@centerOfRotation": [ 0, -1, 0 ],
          "@description": "Hello world!",
          "@position": [ 0, -1, 7 ],
          "@fieldOfView": 0.7854,
          "@jump": true,
          "@orientation": [ 0, 0, 1, 0 ],
          "@retainUserOffsets": false,
          "@containerField": "children"
        },
        "Transform": [
        {
          "@rotation": [ 0, 1, 0, 3 ],
          "@center": [ 0, 0, 0 ],
          "@scale": [ 1, 1, 1 ],
          "@scaleOrientation": [ 0, 0, 1, 0 ],
          "@translation": [ 0, 0, 0 ],
          "@bboxCenter": [ 0, 0, 0 ],
          "@bboxSize": [ -1, -1, -1 ],
          "@containerField": "children",
          "Shape": [
          {
            "@containerField": "children",
            "@bboxCenter": [ 0, 0, 0 ],
            "@bboxSize": [ -1, -1, -1 ],
            "Sphere": {
              "@radius": 1,
              "@solid": true,
              "@containerField": "geometry"
            },
            "Appearance": [
            {
              "@containerField": "appearance",
              "Material": {
                "@DEF": "MaterialLightBlue",
                "@diffuseColor": [ 0.1, 0.5, 1 ],
                "@ambientIntensity": 0.2,
                "@emissiveColor": [ 0, 0, 0 ],
                "@shininess": 0.2,
                "@specularColor": [ 0, 0, 0 ],
                "@transparency": 0,
                "@containerField": "material"
              },
              "ImageTexture": {
                "@DEF": "ImageCloudlessEarth",
                "@url": [ "earth-topo.png", "earth-topo.jpg", "earth-topo-small.gif", "http://www.web3d.org/x3d/content/examples/Basic/earth-topo.png", "http://www.web3d.org/x3d/content/examples/Basic/earth-topo.jpg", "http://www.web3d.org/x3d/content/examples/Basic/earth-topo-small.gif" ],
                "@repeatS": true,
                "@repeatT": true,
                "@containerField": "texture"
              }
            }
            ]
          }
          ]
        }
        ],
        "Transform": [
        {
          "@translation": [ 0, -2, 0 ],
          "@center": [ 0, 0, 0 ],
          "@rotation": [ 0, 0, 1, 0 ],
          "@scale": [ 1, 1, 1 ],
          "@scaleOrientation": [ 0, 0, 1, 0 ],
          "@bboxCenter": [ 0, 0, 0 ],
          "@bboxSize": [ -1, -1, -1 ],
          "@containerField": "children",
          "Shape": [
          {
            "@containerField": "children",
            "@bboxCenter": [ 0, 0, 0 ],
            "@bboxSize": [ -1, -1, -1 ],
            "Text": [
            {
              "@DEF": "TextMessage",
              "@string": [ "Hello", "world!" ],
              "@maxExtent": 0.0,
              "@solid": false,
              "@containerField": "geometry",
              "FontStyle": {
                "@justify": [ "MIDDLE", "MIDDLE" ],
                "@family": [ "SERIF" ],
                "@horizontal": true,
                "@leftToRight": true,
                "@size": 1.0,
                "@spacing": 1.0,
                "@style": "PLAIN",
                "@topToBottom": true,
                "@containerField": "fontStyle"
              }
            }
            ],
            "Appearance": [
            {
              "@containerField": "appearance",
              "Material": {
                "@USE": "MaterialLightBlue",
                "@ambientIntensity": 0.2,
                "@diffuseColor": [ 0.8, 0.8, 0.8 ],
                "@emissiveColor": [ 0, 0, 0 ],
                "@shininess": 0.2,
                "@specularColor": [ 0, 0, 0 ],
                "@transparency": 0,
                "@containerField": "material"
              }
            }
            ]
          }
          ]
        }
        ]
      }
      ]
    }
    ]
  }
  ]
}


More information about the x3d-public mailing list