[x3d-public] ... need x3d.py python support for Script CDATA; name sourceCode vice sourceText for clarity
Brutzman, Donald (Don) (CIV)
brutzman at nps.edu
Sat Jan 15 13:44:27 PST 2022
Thanks for all feedback, Holger and John.
Simple proposal for the moment: am working to demonstrate consistently unescaped script code within Script node. We achieved that capability with X3D JSON draft encoding, but didn't use it anywhere else.
While adding the capability to X3DUOM and Java (so far), it became obvious that the term "sourceText" is ambiguous since typically the entire model consists of source text (for XML, ClassicVRML, source code, etc.)
This name is not yet in any of our X3D specifications so we are free to change it... am suggesting "sourceCode" as a less ambiguous name, for containing unescaped source code within Script, ShaderProgram or ShaderPart node.
No further improvements to "sourceCode" yet heard - if that name is OK, then great. It seems clear when used in a sentence.
We can certainly carry forward code support for the original "sourceText" for a while, not trying to break experimental code/model implementation and evaluation, but am thinking that we don't want to perpetuate two names forever, since that just leads to confusion and inconsistency in years to come.
* Wikipedia: Don't repeat yourself (DRY principle)
* https://en.wikipedia.org/wiki/Don't_repeat_yourself
Hope this sounds sensible... thanks for continuing consideration.
I'll work on adding "sourceCode" capabilities to X3DPSAIL Python and X3D Ontology Turtle next, continuing with regression testing.
all the best, Don
--
Don Brutzman Naval Postgraduate School, Code USW/Br brutzman at nps.edu
Watkins 270, MOVES Institute, Monterey CA 93943-5000 USA +1.831.656.2149
X3D graphics, virtual worlds, Navy robotics https:// faculty.nps.edu/brutzman
From: x3d-public <x3d-public-bounces at web3d.org> On Behalf Of Holger Seelig
Sent: Saturday, January 15, 2022 1:28 PM
To: X3D <x3d-public at web3d.org>
Subject: Re: [x3d-public] Possible issue with X3dToPython.xslt, newly tested file from X3D4WA: need x3d.py python support for Script CDATA
A good way to implement changes in parser/generator is to output new style code, and parse new and old style code, ie. be backwardly compatible. It depends on how much changes in X3D JSON encoding. If it is only the change of #sourceText to #sourceCode then it is very easy to provide both variations. If it is more than this, a detailed list of changes would be very useful.
Best regards,
Holger
Am 15.01.2022 um 22:12 schrieb John Carlson <yottzumm at gmail.com<mailto:yottzumm at gmail.com>>:
In other words, let's fix known problems before creating new ones.
Thanks!
John
On Sat, Jan 15, 2022 at 3:06 PM John Carlson <yottzumm at gmail.com<mailto:yottzumm at gmail.com>> wrote:
It's higher priority for me to fix the X3dToJson.xslt so that Navigation.type produces a consistent type of data. At this time, I don't know if it produces a JSON string or a JSON array. I believe the JSON Schema says array. If we want it to be a string, okay, but can we change all other MFStrings to output JSON strings in X3dToJson.xslt?
Maybe I missed an email on this?
John
On Sat, Jan 15, 2022 at 2:52 PM John Carlson <yottzumm at gmail.com<mailto:yottzumm at gmail.com>> wrote:
It's just that people may already have JSON files working with X_ITE. We will have to go through regression testing etc. We don't have scripting with X3DOM working yet except on some of my published work. And even some of that is not working. AFAIK, all my scripts work ok in X_ITE. So I would encourage us to tread carefully when modifying JSONParser.js in X_ITE, and provide for a deprecation period.
I don't have to retest my shaders-they are in external files.
If we decide to force a change to X3D JSON Schema, I will probably slowly deprecate #sourceText in favor of sourceCode, but I doubt if all this will happen overnight. My surgery was postponed and I want to minimize impact on my wrists right now. That is, I will support sourceCode and sourceText in parallel. I will have to figure out how the new X3DUOM impacts all my python generators as well.
I'm guessing you don't actually have any Scripting working on those older products you mention, so it didn't really impact you until you wanted to do python+scripting in Jupyter.
I will look a bit into X3DUOM changes to see how much my other code is impacted, or can be enhanced.
There's also probably X3DJSONLD.java and X3DJSONLD.cpp that need to be changed.
I would encourage maintaining at least a synonym "#sourceText" for sourceCode.
We should probably create a plan for updating JSONParser.js in X_ITE.
I'm not really sure there will ever be a JSON encoding for X3D, but I'm glad X3dToJson.xslt is being maintained as I don't have a good replacement yet.
"someday" is today for me. What will it take to make X3D JSON a standard? We already have two working implementations, AFAIK.
John
On Sat, Jan 15, 2022 at 1:12 PM Brutzman, Donald (Don) (CIV) <brutzman at nps.edu<mailto:brutzman at nps.edu>> wrote:
I've got X3DUOM and Java working on my machine. Python, X3D Ontology and Tooltips next.
John, can we please change the JSON encoding to use "sourceCode" instead of sourceText for consistency? I don't think there is any need for synonyms or backwards compatibility, the encoding hasn't been formalized yet and (someday) JSON Schema will help people confirm correctness.
* X3D to JSON Stylesheet Converter: Data Types
* https://www.web3d.org/x3d/stylesheets/X3dToJson.html#DataTypes
Embedded source code for
Script<https://www.web3d.org/x3d/content/X3dTooltips.html#Script>, ShaderPart<https://www.web3d.org/x3d/content/X3dTooltips.html#ShaderPart> and ShaderProgram<https://www.web3d.org/x3d/content/X3dTooltips.html#ShaderProgram> nodes
CDATA (Character DATA) section
<[CDATA[ "world wild Web!" ]]>
"#sourceText" string array containing original code, possibly escaped
all the best, Don
--
Don Brutzman Naval Postgraduate School, Code USW/Br brutzman at nps.edu<mailto:brutzman at nps.edu>
Watkins 270, MOVES Institute, Monterey CA 93943-5000 USA +1.831.656.2149
X3D graphics, virtual worlds, Navy robotics https:// faculty.nps.edu/brutzman<http://faculty.nps.edu/brutzman>
From: John Carlson <yottzumm at gmail.com<mailto:yottzumm at gmail.com>>
Sent: Friday, January 14, 2022 5:25 AM
To: Brutzman, Donald (Don) (CIV) <brutzman at nps.edu<mailto:brutzman at nps.edu>>
Cc: X3D Graphics public mailing list <x3d-public at web3d.org<mailto:x3d-public at web3d.org>>
Subject: Re: Possible issue with X3dToPython.xslt, newly tested file from X3D4WA: need x3d.py python support for Script CDATA
I finally got a chance to think about this, and I think it will be good to support legacy files by providing a synonym for "#sourceText"-"sourceCode" as you say. If a particular binding can't provide for #sourceText, then sourceCode can be used as a fallback.
We can discuss changes to the JSON WD Draft at some point.
We will have to update X3D JSON schema as previously discussed implicitly.
John
On Tue, Jan 11, 2022 at 1:03 AM Brutzman, Donald (Don) (CIV) <brutzman at nps.edu<mailto:brutzman at nps.edu>> wrote:
1. Following some initial testing, am happy to report that this approach to embed source code in Script nodes seems consistently feasible for X3DUOM, Python, Java and X3D Ontology all at once.
It will take a little work to fully implement, but it should improve the expressive power (meaning "fix a hole") in each of our programming-language bindings.
------
2. In some sense embedding script code is already possible - you can put such a program in first string contained in a Script url list - but in practice that tends to be quite difficult because then all sorts of obfuscating escaping of special characters has to take place.
So having a separate place for source-code blocks is very helpful. The X3D Architecture recognizes this possibility and defines corresponding behaviors. Further details can be found in X3D XML Encoding.
* X3D4 Architecture, clause 19 Scripting component, 29.4.1 Script
* https://www.web3d.org/specifications/X3Dv4Draft/ISO-IEC19775-1v4-CD1/Part01/components/scripting.html#Script
* X3D4 Architecture, 9.2.3 Scripting language protocols
https://www.web3d.org/specifications/X3Dv4Draft/ISO-IEC19775-1v4-CD1/Part01/components/networking.html#ScriptingLanguageProtocols
* X3D XML Encoding, Clause 4 Concepts, 4.3.13 Encapsulating Script node code
https://www.web3d.org/documents/specifications/19776-1/V3.3/Part01/concepts.html#EncapsulatingScriptNodeCode
"If both a url field and a CDATA clause are encountered, the url field is processed first. Thus, the CDATA construct can also be considered equivalent to one additional value appended to the url MFString array. This ordering allows an online script code url to take priority over fallback default script code in the CDATA construct. This ordering also allows run-time updates if a viewer is connected to the network, if so desired by the originating author."
------
3. Next - naming. "CDATA" is pretty obscure as a field name, especially since that is an XML datatype (for Character Data). Thus am thinking "sourceCode" might be a better name... what do you think?
The ability to embed safe scripting code within any X3D Script is a powerful capability that also improves security possibilities for advanced X3D... it is easier to secure and safely transport an embedded script than it is to have multiple files handled safely.
John, it looks like we already did something just like this in the JSON encoding, we called it "sourceText", excerpt follows. (Excessive string quoting was necessary to pass JSON parsing rules for multiline text.)
----------------------------------------------------------------------------------------------------------------------------------------------
* https://X3dGraphics.com/examples/X3dForWebAuthors/Chapter14Prototypes/MaterialModulator.json<https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fx3dgraphics.com%2Fexamples%2FX3dForWebAuthors%2FChapter14Prototypes%2FMaterialModulator.json&data=04%7C01%7Cbrutzman%40nps.edu%7Ca21361bf5da14489edc908d9d86dfc84%7C6d936231a51740ea9199f7578963378e%7C0%7C0%7C637778789185932285%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=IOpAF1lGWKRYlN5OO8Sl%2FT1S7Ed7MxBT6BocnOHxq4U%3D&reserved=0>
{ "Script":
{
[....]
"#sourceText":[
"",
"",
"ecmascript:",
"function initialize ()",
"{",
" newColor = diffuseColor; // start with original color",
"}",
"function clockTrigger (timeValue)",
"{",
" if (!enabled) return;",
" red = newColor.r;",
" green = newColor.g;",
" blue = newColor.b;",
"",
" // note different modulation rates for each color component, % is modulus operator",
" newColor = new SFColor ((red + 0.02) % 1, (green + 0.03) % 1, (blue + 0.04) % 1);",
"\tif (enabled)",
"\t{",
"\t\tBrowser.print ('diffuseColor=(' + red +',' + green + ',' + blue + ') newColor=' + newColor.toString() + '\n');",
"\t}",
"}",
"function set_enabled (newValue)",
"{",
"\tenabled = newValue;",
"}",
"",
""
]
----------------------------------------------------------------------------------------------------------------------------------------------
Which in the .x3d XML CDATA simply looks like
<Script DEF='MaterialModulatorScript'>
<field accessType='inputOutput' name='enabled' type='SFBool'/>
<field accessType='inputOutput' name='diffuseColor' type='SFColor'/>
<field accessType='outputOnly' name='newColor' type='SFColor'/>
<field accessType='inputOnly' name='clockTrigger' type='SFTime'/>
<IS>
<connect nodeField='enabled' protoField='enabled'/>
<connect nodeField='diffuseColor' protoField='diffuseColor'/>
</IS>
<![CDATA[
ecmascript:
function initialize ()
{
newColor = diffuseColor; // start with original color
}
function clockTrigger (timeValue)
{
if (!enabled) return;
red = newColor.r;
green = newColor.g;
blue = newColor.b;
// note different modulation rates for each color component, % is modulus operator
newColor = new SFColor ((red + 0.02) % 1, (green + 0.03) % 1, (blue + 0.04) % 1);
if (enabled)
{
Browser.print ('diffuseColor=(' + red +',' + green + ',' + blue + ') newColor=' + newColor.toString() + '\n');
}
}
function set_enabled (newValue)
{
enabled = newValue;
}
]]>
</Script>
----------------------------------------------------------------------------------------------------------------------------------------------
So this possibility is a cool prospect, again thanks for flagging it. I'll work further on X3DUOM and Python, Java, OWL compatibility next.
Have fun with X3D! 8)
all the best, Don
--
Don Brutzman Naval Postgraduate School, Code USW/Br brutzman at nps.edu<mailto:brutzman at nps.edu>
Watkins 270, MOVES Institute, Monterey CA 93943-5000 USA +1.831.656.2149
X3D graphics, virtual worlds, navy robotics https:// faculty.nps.edu/brutzman<http://faculty.nps.edu/brutzman>
-----Original Message-----
From: Brutzman, Donald (Don) (CIV) <brutzman at nps.edu<mailto:brutzman at nps.edu>>
Sent: Monday, January 10, 2022 6:37 PM
To: John Carlson <yottzumm at gmail.com<mailto:yottzumm at gmail.com>>
Cc: X3D Graphics public mailing list <x3d-public at web3d.org<mailto:x3d-public at web3d.org>>; Brutzman, Donald (Don) (CIV) <brutzman at nps.edu<mailto:brutzman at nps.edu>>
Subject: RE: Possible issue with X3dToPython.xslt, newly tested file from X3D4WA: need x3d.py python support for Script CDATA
Thanks for the interesting trouble report John.
No changes needed to original X3D model, it passes all X3D Validator tests (though X3D Schematron provides a few warnings).
* https://X3dGraphics.com/examples/X3dForWebAuthors/Chapter14Prototypes/MaterialModulator.x3d<https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fx3dgraphics.com%2Fexamples%2FX3dForWebAuthors%2FChapter14Prototypes%2FMaterialModulator.x3d&data=04%7C01%7Cbrutzman%40nps.edu%7Ca21361bf5da14489edc908d9d86dfc84%7C6d936231a51740ea9199f7578963378e%7C0%7C0%7C637778789185932285%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=PcRk1T5cpkV1jnuSTR2%2FdvaYOc%2BALUhOr%2FvKGwVvfDA%3D&reserved=0>
* https://savage.nps.edu/X3dValidator
XMLSpy was stricter about type handling than our Java-based Saxon10 conversions for this example. XMLSpy diagnosed some problems that I fixed.
After also fixing the stylesheet name provided by a diagnostic, all X3dToPython.xslt changes were checked into subversion.
I now get the following conversion error:
=====================
create python:
C:\x3d-code\www.web3d.org<http://www.web3d.org/>\x3d\content\examples\X3dForWebAuthors/Chapter14Prototypes//MaterialModulator.x3d processing with X3dToPython stylesheet...
*** TODO x3d.py and X3dToPython.xslt need to handle embedded CDATA source code for Script C:\x3d-code\www.web3d.org<http://www.web3d.org/>\x3d\content\examples\X3dForWebAuthors/Chapter14Prototypes//MaterialModulator.py self-validation tests...
validate python:
File "C:\x3d-code\www.web3d.org<http://www.web3d.org/>\x3d\content\examples\X3dForWebAuthors\Chapter14Prototypes\MaterialModulator.py", line 74
*** TODO x3d.py and X3dToPython.xslt need to handle embedded CDATA source code for Script
^
SyntaxError: invalid syntax
Result: 1
===================================
So, embedded CDATA scripting code inside a Script node (and similar Shader nodes) remains an omission in our Python support.
To remedy this, we need to create a special field in the X3DPSAIL class Script. *What shall we call it?* Creating a Script member named CDATA seems unambiguous and eponymous with the XML encoding...
* X3D Tooltips, Script and Script url
https://www.web3d.org/x3d/tooltips/X3dTooltips.html#Script
https://www.web3d.org/x3d/tooltips/X3dTooltips.html#Script.url
* X3D Tooltips, XML Data Types
https://www.web3d.org/x3d/tooltips/X3dTooltips.html#XML
" CDATA is an XML term for Character Data. The base type for all XML attributes consists of string-based CDATA characters. CDATA is used throughout the X3D DOCTYPE definitions, which can only check for the presence of legal strings and thus are not able to validate numeric type information. XML Schema provides stricter validation based on data types."
* XML, 2.4 Character Data and Markup
https://www.w3.org/TR/REC-xml/#syntax<https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.w3.org%2FTR%2FREC-xml%2F%23syntax&data=04%7C01%7Cbrutzman%40nps.edu%7Ca21361bf5da14489edc908d9d86dfc84%7C6d936231a51740ea9199f7578963378e%7C0%7C0%7C637778789185932285%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=hURgR6DMZ4HAQdimBlphKzMDVhWzG5zrz9bOkVb9y6Y%3D&reserved=0>
all the best, Don
--
Don Brutzman Naval Postgraduate School, Code USW/Br brutzman at nps.edu<mailto:brutzman at nps.edu>
Watkins 270, MOVES Institute, Monterey CA 93943-5000 USA +1.831.656.2149
X3D graphics, virtual worlds, navy robotics https:// faculty.nps.edu/brutzman<http://faculty.nps.edu/brutzman>
-----Original Message-----
From: John Carlson <yottzumm at gmail.com<mailto:yottzumm at gmail.com>>
Sent: Monday, January 10, 2022 4:06 PM
To: Brutzman, Donald (Don) (CIV) <brutzman at nps.edu<mailto:brutzman at nps.edu>>; X3D Graphics public mailing list <x3d-public at web3d.org<mailto:x3d-public at web3d.org>>
Subject: Possible issue with X3dToPython.xslt, newly tested file from X3D4WA
X3D XML file attached. From X3DJSONLD's maven build log:
[INFO] --- xml-maven-plugin:1.0.2:transform (default) @ X3DJSONLD --- [INFO] Transforming file:
/home/coderextreme/X3DJSONLD/src/main/data/MaterialModulatorExamples.x3d
[INFO] Transforming file:
/home/coderextreme/X3DJSONLD/src/main/data/MaterialModulator.x3d
Type error at char 37 in xsl:value-of/@select on line 610 column 120 of
X3dToPython.xslt:
XPTY0004: A sequence of more than one item is not allowed as the first argument of
normalize-space() ("", "", ...)
at xsl:apply-templates
(file:/home/coderextreme/X3DJSONLD/src/main/lib/stylesheets/X3dToPython.xslt#606)
processing /X3D/Scene[1]/ProtoDeclare[1]/ProtoBody[1]/Script[1]
at xsl:apply-templates
(file:/home/coderextreme/X3DJSONLD/src/main/lib/stylesheets/X3dToPython.xslt#606)
processing /X3D/Scene[1]/ProtoDeclare[1]/ProtoBody[1]
at xsl:apply-templates
(file:/home/coderextreme/X3DJSONLD/src/main/lib/stylesheets/X3dToPython.xslt#606)
processing /X3D/Scene[1]/ProtoDeclare[1]
at xsl:apply-templates
(file:/home/coderextreme/X3DJSONLD/src/main/lib/stylesheets/X3dToPython.xslt#606)
processing /X3D/Scene[1]
at xsl:apply-templates
(file:/home/coderextreme/X3DJSONLD/src/main/lib/stylesheets/X3dToPython.xslt#123)
processing /X3D
[INFO]
------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO]
------------------------------------------------------------------------
[INFO] Total time: 2.840 s
[INFO] Finished at: 2022-01-10T17:59:22-06:00 [INFO]
------------------------------------------------------------------------
[ERROR] Failed to execute goal
org.codehaus.mojo:xml-maven-plugin:1.0.2:transform (default) on project
X3DJSONLD: Failed to transform input file
/home/coderextreme/X3DJSONLD/src/main/data/MaterialModulator.x3d: A sequence of more than one item is not allowed as the first argument of
normalize-space() ("", "", ...) -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1]
https://nam10.safelinks.protection.outlook.com/?url=http%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FMAVEN%2FMojoExecutionException&data=04%7C01%7Cbrutzman%40nps.edu%7C79b6d0212127425f31b408d9d4ab3773%7C6d936231a51740ea9199f7578963378e%7C0%7C0%7C637774654099751000%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=VYVojdsW4QsyjrkZGnqfPOPmUp7x5ftTGE3H0TzrTts%3D&reserved=0<https://nam10.safelinks.protection.outlook.com/?url=http%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FMAVEN%2FMojoExecutionException&data=04%7C01%7Cbrutzman%40nps.edu%7Ca21361bf5da14489edc908d9d86dfc84%7C6d936231a51740ea9199f7578963378e%7C0%7C0%7C637778789185932285%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=GEjKI%2BOCI%2FfcHC9D6D%2BZgZ4o4TF9GKL%2FJYEyyCHQ8fs%3D&reserved=0>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://web3d.org/pipermail/x3d-public_web3d.org/attachments/20220115/ff93ba60/attachment-0001.html>
More information about the x3d-public
mailing list