[x3d-public] X3D JSON schema, identifying X3D type for items within an array. (was element)

[John Carlson]

I had a great reply set up for this, but I fumble fingered it.

Please present a table of MF field type to SF field type mappings. Also 
SF types to types inside SF types (not JavaScript types), these are the 
type mappings that are difficult to automate. This is obviously dead 
simple stuff, and I shouldn't be asking a PhD to do it.  I don't know 
much about X3DUOM generation.  Once the table is constructed, we can 
discuss whether to put it in X3DUOM or the X3D JSON schema generator.

If you have an example where #comment is not working, please share.   We 
can discuss.

I won't do accessTypes on shared fields between nodes, since accessTypes 
may vary between nodes (and probably statements too).

Your request implies, I think, that fields are not shared.  We might be 
able to do that, but likely not on more complex type fields.  But my 
mind is spinning, like enumerating each instance of a complex type.

Hopefully, this reexplains my case well enough.

John


On 4/7/21 2:22 AM, Don Brutzman wrote:
> On 4/6/2021 7:04 PM, John Carlson wrote:
>>
>> Good feedback!  Here's some feedback on the feedback.  Would it satisfy
>> you if we put appinfo from X3DUOM FieldType in an items' $comment?  If
>> that's okay, you can probably skip most of the rest of this message.
>
> Looking into JSON schema: yes just put an informational schema comment 
> within each field that lists type/accessType as found in X3DUOM (as 
> found in XML Schema, as found in X3D Specfication).
>
> Looking out of JSON schema: we want JSON Schema to allow "#comment" in 
> JSON files anywhere we put them.  We can consider modifying this 
> syntax if needed, since it is our creation to support fuller 
> interoperability/roundtrip-aility with other file encodings.
>
> Please don't agonize over type correspondences, they are really 
> simple.  If you want a correspondence table I will produce it.
>
> * https://www.web3d.org/x3d/tooltips/X3dTooltips.html#FieldTypesTable
>
> Overall goal restated: we want JSON schema to validate as might be 
> practical, and we don't want JSON schema to do what it can't do.
>
> Same for you please John: do what you can, and don't do what you 
> can't!  Your ability to check and test is impressive and much 
> appreciated.
>
> Some future good news: if we get everything working, and documented as 
> best we can, and even successfully specified via ISO: then future 
> maintenance will be everyone else's problem, we won't have to worry 
> about it!  8)
>
> I think that covers about everything in this message.  Simplify, 
> simplify...
>
>
>> There is potentially some possibility of parsing ContentModel for
>> ordering stuff in "head" and "X3D"--let me know if ContentModel is the
>> right place to look for order. I don't think that accessType is doable
>> due to potential shared fields (see below).
>>
>> On 4/6/21 7:07 PM, Don Brutzman wrote:
>>> thanks for review and progress
>>>
>>> On 4/6/2021 4:08 PM, John Carlson wrote:
>>>>
>>>>
>>>> Decision should be made whether array X3D type to array item X3D type
>>>> map should be in X3D JSON schema generator or X3DUOM.
>>>
>>> all types (array or singleton) already defined in X3DUOM so no
>>> decision needed.
>>>
>>> what question do you have?
>>
>>
>> For example, take FieldType SFVec3f in X3DUOM.    FieldType SFVec3f says
>> it's an array, but does not specify the X3D type of the items in the
>> array, except in appinfo (SFFloat), and I don't have a desire to parse
>> through that.   Perhaps we could add "itemType" to each FieldType in
>> X3DUOM which is an array--or is "itemType" too JSONic?  It's okay with
>> me if I put the proposed map in the JSON schema generator, I just figure
>> you would complain about hard-coded things that we're trying to make
>> explicit in X3DUOM, like the order of objects in head and X3D.
>>
>> If you've got the order expressed in X3DUOM, then we can do something
>> like this in JSON schema:
>> https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fstackoverflow.com%2Fquestions%2F31691247%2Fhow-can-i-define-the-sequence-of-properties-in-json-schema&data=04%7C01%7Cbrutzman%40nps.edu%7C7608645de739424bb1b408d8f969a088%7C6d936231a51740ea9199f7578963378e%7C0%7C0%7C637533579357994823%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=pPe3GF%2FopftsSkSRBgQFAR0oH3xBjDE4oThMn3YG%2FAs%3D&reserved=0 
>>
>> I am really surprised at how easy some stuff is in JSON Schema! As for
>> order, is that done in ContentModel in X3DUOM?  Just confirming before I
>> run off to program.  I am not familiar with how ContentModel works at
>> this time.   Do you have documentation on it?
>>
>>
>> What I feel we're trying to do is add "X3D" information to the X3D JSON
>> schema, so it would be good to remain "X3D-like" for item types, to
>> maintain historical information, and so we can avoid creating extra
>> files from X3DUOM for my JS programs--see fieldTypes.js and
>> mapToMethod.js in X3DJSONLD/src/main/node  (the JavaScript remains out
>> of date unless a full build is run).   Also it would be good to look at
>> mapToMethod2.js as these were the things I found missing from
>> mapToMethod.js (but I need to run through that with the newer X3DUOM,
>> sorry).
>>
>>>
>>>> As discussed in our meeting today, I am adding a $comment with X3D 
>>>> type
>>>> before "type": occurs in the X3D JSON schema.
>>>>
>>>> This will require more work than initially required, although the
>>>> changes so far have been fairly straightforward in the somewhat messy
>>>> code.
>>>>
>>>> For example:
>>>>
>>>>              "@url": {
>>>>                "pattern": "^(\\s|\\S)*$",
>>>> +              "$comment": "X3D type: MFString",
>>>>                "type": "array",
>>>>                "minItems": 1,
>>>>                "items": {
>>>>                  "format": "uri-reference",
>>>> +                "$comment": "X3D type: MFString",
>>>>                  "type": "string"
>>>>                }
>>>>              },
>>>>
>>>>
>>>> The X3D type for the items object, should be SFString one would think.
>>>> So I'll have to check to see if I'm in an array.
>>>
>>> Great.  No need to say "X3D type: "
>>
>> See suggestion on including appinfo in some $comment's.
>>
>>>
>>> I would add accessType as well please.
>>>
>>> example: "$comment": "MFString inputOutput",
>>
>> I'm a little bit fuzzy on adding an accessType, because shared field
>> definitions may have different accessTypes per node/statement. If we're
>> not sharing field definitions, then this would be OK.
>>
>> I like to write self-documenting code (note my lack of actual
>> documentation except for emails :().  Sometimes I do better than
>> others.   I was frankly puzzled when someone commented on the amount of
>> documentation in my repositories.  I was pretty shocked, and I wasn't
>> sure if they were kidding or not.
>>
>> Someday, I'm going to write an web crawler that reads all my emails and
>> constructs documentation :)  Or I'll just let a startup, OpenAI or
>> Google do it.
>>
>>>
>>>> Probably by looking for "items".  Then I'll look for MF in the field
>>>> type, and replace the first M with an S.
>>>>
>>>>
>>>> But here are added problems looking at arrays:
>>>>
>>>>
>>>>              "@bboxSize": {
>>>>                "pattern":
>>>> "^\\s*(([+-]?((0|[1-9][0-9]*)(\\.[0-9]*)?|\\.[0-9]+)([Ee][+-]?[0-9]+)?)\\s+){2}([+-]?((0|[1-9][0-9]*)(\\.[0-9]*)?|\\.[0-9]+)([Ee][+-]?[0-9]+)?)\\s*$", 
>>>>
>>>>
>>>> +              "$comment": "X3D type: SFVec3f",
>>>>                "type": "array",
>>>>                "minItems": 3,
>>>>                "maxItems": 3,
>>>>                "items": {
>>>>                  "default": -1,
>>>> +                "$comment": "X3D type: SFVec3f",
>>>>                  "type": "number"
>>>>                }
>>>>              },
>>>>
>>>>
>>>> In this case, the SFVec3f would be found, and a mapping between 
>>>> SFVec3f
>>>> to SFFloat for individual items should be made.
>>>
>>> I think your actual mapping is SFVec3f to Javascript "type": "number"
>>> with min/maxItems 3  which appears above.
>>
>>
>> Yes, we could leave off the $comment for native JavaScript.  I don't
>> actually know what the items of a SFVec3f are except that the appinfo in
>> X3DUOM says the numbers are SFFloat's.  That's the relationship I'm
>> trying to directly put in the X3DUOM so I don't have to build the map
>> myself.  I believe it will be more informative to declare the types of
>> the items of an array in a $comment in the schema (and in X3DUOM too).
>> If we're in JSON, we pretty much know it's a number, since there are no
>> quotes around it.
>>
>>>
>>>> So perhaps a simple map could be constructed which maps array type to
>>>> item type.
>>>>
>>>>
>>>> Does this sound pretty feasible, essentially a hard coded map from 
>>>> array
>>>> type to item type in the generator code?
>>>
>>> sounds trivially simple and workable
>>
>>
>> See above where I attempt a design for putting the map into X3DUOM, or
>> just copying appinfo into the $comment value for items.
>>
>>>
>>>
>>>> I will look for additional issues now. I'm not seeing much right away.
>>>> I know I ignore some of the "type" fields right now, particularly in
>>>> hard coded definitions.
>>>>
>>>> Note that I'm inserting documentation.  This shouldn't affect the
>>>> running of the generator.
>>>>
>>>> My main question is, where does the information reside? Should it 
>>>> be in
>>>> X3DUOM?
>>>
>>> (on loudspeaker) "I Say Again" all of the needed information is
>>> already in X3DUOM.
>>
>>
>> Yes, but perhaps not easily accessible? I don't want you to have to say
>> something twice, and I can certainly attempt to parse appinfo. I don't
>> particularly want you to have to parse appinfo either.  I can egrep
>> 'FieldType|appinfo' to collect information that is relevant from X3DUOM
>> to start--actually gathering the info by hand might be easier and then
>> you can verify the information I present and then put it into X3DUOM.
>> Then I'll start querying FieldType (or whatever you prefer) in the X3D
>> JSON schema generator. I currently only query for regex in FieldType, it
>> should be easy to add more features!  Hurrah!
>>
>>>
>>>> Does anyone mind if I start calling my generators "synthesizers"?  I'm
>>>> on a new kick!
>>>
>>> yes, please don't kick us with mixed terms...
>>
>>
>> Okay!
>>
>>>
>>>> Note that we are moving towards XSL for our primary JSON schema
>>>> synthesizer, lessening requirements on future maintainers. We're
>>>> currently using python to suss out patterns and requirements for
>>>> X3DUOM.   The python pretty much is impossible to maintain (even 
>>>> the guy
>>>> that wrote it admits that).  If you know any "younguns" who love XSL,
>>>> encourage them to join and contribute to X3D!
>>>
>>> when you have a good pattern in a good draft-07 schema, i will create
>>> an XSLT autogenerator that converts X3DUOM XML to build an X3D JSON
>>> schema.
>>
>>
>> Think about how to add JSON schemas for various profiles. That was one
>> thing I hadn't gotten to yet, and I know you added that to X3DUOM a
>> while ago.   Sorry for not keeping up-to-date.
>>
>>>
>>>> Building holistic schema synthesizers.
>>>>
>>>> My next job will be to build holistic spaceship synthesizers.
>>>
>>> please leave me behind
>>
>>
>> :)  maybe we'll take a saliva sample if we need an expert on XSL
>> programming.
>>
>> I think later JSON schema versions are adding stuff like "how to show on
>> a GUI" so it would be good to keep a handle on what they are adding to
>> the JSON schema when we want to upgrade.  Perhaps we could just put
>> appinfo in $comment and call it a day?
>>
>> It seems like there's still a lot of work surrounding X3D JSON. I enjoy
>> that everyone puts up with me, but I don't know of any customers for X3D
>> JSON!
>>
>> Respectfully (with a bit of tongue in check).
>>
>> John
>
> all the best, Don