[x3d-public] VRML97 and ClassicVRML model validation from command line using view3dscene

Wed Aug 23 16:39:44 PDT 2023

Sounds good Michalis.  The  --enable-downloads switch worked great and noticeably improved the output logs, each of which is now updated online and in version control.

Corresponding detection of geovrml.org issues was particularly useful, separate correspondence posted.

Here is an example of backslash diagnostic, knowing line numbers will help (if possible, some tests have trouble with that).  I can probably figure these out, but thought that you would want consistent exception diagnostic outputs.

*	ClassicVRML validation: tovrmlx3d.exe C:\x3d-code\www.web3d.org\x3d\content\examples\X3dForWebAuthors\Chapter04ViewingNavigation/BindingOperations.x3dv --validate --enable-downloads
*	tovrmlx3d: Warning: X3D: Invalid X3D file: Invalid sequence in a string: "\n". Backslash must be followed by another backslash or double quote, for SFString and MFString (in X3D classic (VRML) encoding) and for MFString (in X3D XML encoding).
*	(21 more occurrences follow)
*	Offending text is within embedded javascript code and seems legal, for example:
*	  Browser.print ('\n===========\n time t0');

*	ClassicVRML validation: tovrmlx3d.exe C:\x3d-code\www.web3d.org\x3d\content\examples\X3dForWebAuthors\Chapter11LightingEnvironmentalEffects/BackgroundTimeOfDay.x3dv --validate --enable-downloads
*	tovrmlx3d: Warning: X3D: Invalid X3D file: Invalid sequence in a string: "\n". Backslash must be followed by another backslash or double quote, for SFString and MFString (in X3D classic (VRML) encoding) and for MFString (in X3D XML encoding).
*	(11 more occurrences follow)
*	Similar case.  Offending text is within embedded javascript code and seems legal, for example:
*	    Browser.print ('groundColorSunrise length=' + groundColorSunrise.length + ' '  + groundColorSunrise.toString() + '\n');

Here is an example that I first thought was an unrecognized node, but is actually an incorrect conversion that is a required missing field name.  I’ll fix the converter.

*	ClassicVRML validation: tovrmlx3d.exe C:\x3d-code\www.web3d.org\x3d\content\examples\X3dForWebAuthors\Chapter05AppearanceMaterialTextures/TwoSidedMaterialExample.x3dv --validate --enable-downloads
*	tovrmlx3d: Warning: VRML/X3D: Error when reading, will skip the rest of X3D file: Error at line 56 column 25: Invalid X3D node content (probably unknown or not allowed field, prototype or VRML 1.0-style children) inside "Appearance": got "TwoSidedMaterial"

Other entries that appear to be incorrectly failing “Invalid X3D node” content follow.  Long form provided here in case you want to test further.

*	ClassicVRML validation: tovrmlx3d.exe C:\x3d-code\www.web3d.org\x3d\content\examples\X3dForWebAuthors\Chapter01TechnicalOverview/EmptySceneCoreProfile.x3dv --validate --enable-downloads
*	tovrmlx3d: Warning: VRML/X3D: Error when reading, will skip the rest of X3D file: Error at line 40 column 14: Invalid X3D node content (probably unknown or not allowed field, prototype or VRML 1.0-style children) inside "WorldInfo": got "MetadataSet"

*	ClassicVRML validation: tovrmlx3d.exe C:\x3d-code\www.web3d.org\x3d\content\examples\X3dForWebAuthors\Chapter15Metadata/MetadataNodeExamplesX3D3.x3dv --validate --enable-downloads
*	tovrmlx3d: Warning: VRML/X3D: Node "MetadataString" is not allowed at the top level of the X3D file
*	tovrmlx3d: Warning: VRML/X3D: Error when reading, will skip the rest of X3D file: Error at line 47 column 6: Invalid X3D node content (probably unknown or not allowed field, prototype or VRML 1.0-style children) inside "WorldInfo": got keyword "DEF"

That’s it for now.  Your validation is very high quality!  Lots of other content errors for me to fix that were previously undetected… 8)

all the best, Don

-- 

Don Brutzman  Naval Postgraduate School, Code USW/Br        brutzman at nps.edu

Watkins 270,  MOVES Institute, Monterey CA 93943-5000 USA    +1.831.656.2149

X3D graphics, virtual worlds, navy robotics https://faculty.nps.edu/brutzman

-----Original Message-----
From: Michalis Kamburelis <michalis.kambi at gmail.com> 
Sent: Wednesday, August 23, 2023 7:09 AM
To: Brutzman, Donald (Don) (CIV) <brutzman at nps.edu>
Cc: X3D Public Mailing List (x3d-public at web3d.org) <x3d-public at web3d.org>
Subject: Re: VRML97 and ClassicVRML model validation from command line using view3dscene

Don,

1. As for some missing X3D 4.0 nodes: Yes, this can happen, as in the past I didn't carefully check whether all X3D 4.0 nodes have been added to CGE/view3dscene, and indeed I was fixing later some missing ones (like H-Anim nodes from X3D 4.0).

    If you can extract the list of missing X3D 4.0 nodes from above logs, I'll be grateful -- I didn't find relevant messages now from the logs you linked, but admittedly I only looked quickly.

2. To avoid the "Error EDownloadError when loading inline file from URL ...: Downloading network resources (from "http" or "https"

protocols) is not enabled. Set global EnableBlockingDownloads:=true to enable." pass the additional command-line option

    --enable-downloads

    to tovrmlx3d. Note that it will definitely increase the validation time. The tovrmlx3d will download the linked resources (Inline X3D content, X3D content referred by EXTERNPROTOs etc.) from the Internet at validation.

Regards,

Michalis

śr., 23 sie 2023 o 08:18 Brutzman, Donald (Don) (CIV) < <mailto:brutzman at nps.edu> brutzman at nps.edu> napisał(a):

> 

> Michalis, appreciate your adding this feature as part of tovrmlx3d executable.  I’ve integrated into the Ant build scripts and tested operation satisfactorily for all .wrl and .x3dv conversions in the X3D Examples Archives.

> 

> 

> 

> Example command-line invocation on windows, a little terser works out OK for these logs:

> 

> 

> 

> tovrmlx3d.exe 

> C:\x3d-code\www.web3d.org\x3d\content\examples\X3dForWebAuthors\Chapte

> r04ViewingNavigation/CollisionExample.x3dv --validate

> 

> 

> 

> Regression-test results online (and in version control to track progress) as follows.  Most look clean.

> 

> 

> 

>  <https://www.web3d.org/x3d/content/examples/build.validate.VRML.all.log> https://www.web3d.org/x3d/content/examples/build.validate.VRML.all.log

> .txt

> 

>  <https://www.web3d.org/x3d/content/examples/build.validate.x3dv.all.log> https://www.web3d.org/x3d/content/examples/build.validate.x3dv.all.log

> .txt

> 

> 

> 

>  <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fsour> https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fsour

> ceforge.net%2Fp%2Fx3d%2Fcode%2FHEAD%2Ftree%2Fwww.web3d.org%2Fx3d%2Fcon

> tent%2Fexamples%2Fbuild.validate.VRML.all.log.txt&data=05%7C01%7Cbrutz

> man%40nps.edu%7C2baece0d2d40458e15cb08dba3e28885%7C6d936231a51740ea919

> 9f7578963378e%7C0%7C0%7C638283965717273937%7CUnknown%7CTWFpbGZsb3d8eyJ

> WIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000

> %7C%7C%7C&sdata=q%2BSZdNZqtAMSJlRxmJai8uHirMFpE3N2czqe0LSgyh8%3D&reser

> ved=0

>  <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fsour> https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fsour

> ceforge.net%2Fp%2Fx3d%2Fcode%2FHEAD%2Ftree%2Fwww.web3d.org%2Fx3d%2Fcon

> tent%2Fexamples%2Fbuild.validate.x3dv.all.log.txt&data=05%7C01%7Cbrutz

> man%40nps.edu%7C2baece0d2d40458e15cb08dba3e28885%7C6d936231a51740ea919

> 9f7578963378e%7C0%7C0%7C638283965717273937%7CUnknown%7CTWFpbGZsb3d8eyJ

> WIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000

> %7C%7C%7C&sdata=Ng9yOUpVA392Jy9gjqoPqCF4vu106L2Zu3kC%2BtFwA8A%3D&reser

> ved=0

> 

> 

> 

> Execution is fast!  Execution took around a half hour for over 4000 examples, most of that time was likely writing test-result outputs to console.

> 

> 

> 

> I haven’t tried fixing any model errors or X3dToVrml97.xslt stylesheet conversion errors yet (e.g. true false vice TRUE FALSE etc.) – that will happen.

> 

> 

> 

> You might see some issues in there that relate to your side, saw a few 

> such as

> 

> new X3D4 nodes that might not yet be included in view3dscene, 

> Backslash and \n error reports didn’t include line numbers (might be 

> hard to isolate),

> 

> 

> 

> Saw this error message, but not sure how to implement setting the property via command line, please advise:

> 

> 

> 

> tovrmlx3d: Warning: X3D: Error EDownloadError when loading inline file from URL  <https://www.web3d.org/x3d/content/examples/Basic/HumanoidAnimation/NancyDiving.x3d> https://www.web3d.org/x3d/content/examples/Basic/HumanoidAnimation/NancyDiving.x3d: Downloading network resources (from "http" or "https"  protocols) is not enabled. Set global EnableBlockingDownloads:=true to enable.

> 

> 

> 

> Happy to now have these post-conversion tests available, enabling continued deliberate improvements in quality assurance (QA).  Again thanks for a tremendous capability, very helpful.

> 

> 

> 

> Have fun with VRML!  8)

> 

> 

> 

> all the best, Don

> 

> --

> 

> Don Brutzman  Naval Postgraduate School, Code USW/Br         <mailto:brutzman at nps.edu> brutzman at nps.edu

> 

> Watkins 270,  MOVES Institute, Monterey CA 93943-5000 USA    +1.831.656.2149

> 

> X3D graphics, virtual worlds, navy robotics 

>  <https://faculty.nps.edu/brutzman> https://faculty.nps.edu/brutzman

> 

> 

> 

> 

> 

> -----Original Message-----

> From: Michalis Kamburelis < <mailto:ichalis.kambi at gmail.com> ichalis.kambi at gmail.com>

> Sent: Monday, August 14, 2023 5:40 AM

> To: Brutzman, Donald (Don) (CIV) < <mailto:brutzman at nps.edu> brutzman at nps.edu>

> Cc: X3D Public Mailing List ( <mailto:x3d-public at web3d.org> x3d-public at web3d.org) 

> < <mailto:x3d-public at web3d.org> x3d-public at web3d.org>

> Subject: Re: VRML97 and ClassicVRML model validation from command line 

> using view3dscene

> 

> 

> 

> Don,

> 

> 

> 

> In short: I added now –validate option to tovrmlx3d 😊

> 

> 

> 

> Example usage on a valid model:

> 

> 

> 

> ```

> 

> $ ./tovrmlx3d ../demo-models/x3d/axis.x3d –validate $ echo $?

> 

> 0

> 

> ```

> 

> 

> 

> So empty output, and exit status 0, as expected from most tools. You can use this with Ant, Make or anything else.

> 

> 

> 

> Example usage on a model with problems:

> 

> 

> 

> ```

> 

> $ ./tovrmlx3d 

> ../demo-models/x3d/warnings/invalid_statement_omitted_1.x3dv

> 

> --validate

> 

> tovrmlx3d: Warning: VRML/X3D: Error when reading, will skip the rest of X3D file: Error at line 15 column 2: Expected node type or DEF or USE, got “}”

> 

> Exception “Exception”:

> 

> Validation failed (consult the warnings above), exiting with non-zero 

> status

> 

> 

> 

> $ echo $?

> 

> 1

> 

> ```

> 

> 

> 

> So you get output on stderr, and non-zero exit status.

> 

> 

> 

> Please give ~1 hour for Jenkins to process it, and this new version of 

> tovrmlx3d will be part of “snapshot" version on

> 

>  <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcastle-engine.io%2Fview3dscene.php&data=05%7C01%7Cbrutzman%40nps.edu%7C2baece0d2d40458e15cb08dba3e28885%7C6d936231a51740ea9199f7578963378e%7C0%7C0%7C638283965717273937%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=nD1LFbNHsqygeWo8hFkU1Sc6hb85nsiMljC5wLe9VWQ%3D&reserved=0> https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcastle-engine.io%2Fview3dscene.php&data=05%7C01%7Cbrutzman%40nps.edu%7C2baece0d2d40458e15cb08dba3e28885%7C6d936231a51740ea9199f7578963378e%7C0%7C0%7C638283965717273937%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=nD1LFbNHsqygeWo8hFkU1Sc6hb85nsiMljC5wLe9VWQ%3D&reserved=0 .

> 

> 

> 

> I used above test models from  <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fcastle-engine%2Fdemo-models&data=05%7C01%7Cbrutzman%40nps.edu%7C2baece0d2d40458e15cb08dba3e28885%7C6d936231a51740ea9199f7578963378e%7C0%7C0%7C638283965717273937%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=5vytRe7QjCwyp6twC57eXbSCA2L8FQroM3Is3WArmIE%3D&reserved=0> https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fcastle-engine%2Fdemo-models&data=05%7C01%7Cbrutzman%40nps.edu%7C2baece0d2d40458e15cb08dba3e28885%7C6d936231a51740ea9199f7578963378e%7C0%7C0%7C638283965717273937%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=5vytRe7QjCwyp6twC57eXbSCA2L8FQroM3Is3WArmIE%3D&reserved=0 .

> 

> 

> 

> Details:

> 

> 

> 

> So far, we did validation just by converting and piping output to 

> /dev/null, so "tovrmlx3d xxxx.x3d > /dev/null". But I agree this was

> 

> unoptimal:

> 

> 

> 

> - Explicit "--validate" means we don't needlessly waste time converting to /dev/null if you actually just want to validate. We also don't produce eventual warnings when writing, that would sometimes appear and duplicate warnings we already did when reading.

> 

> 

> 

> - In the future, maybe, explicit "--validate" may allow to run some additional tasks dedicated to validation (maybe because they are time-consuming and not necessary for just usual opening). We don't have such tasks right now, but it would be nice to have such option available.

> 

> 

> 

> - With explicit "--validate", any warning or error causes non-zero 

> exit status. Makes sense, since a warning (even if we can continue

> 

> working) still means validation failed.

> 

> 

> 

> So ... I added "--validate" option to tovrmlx3d.

> 

> 

> 

> Of note: A rename is on the horizon, for a few reasons (glTF export, consistency with other CGE tools). Most likely:

> 

> - tovrmlx3d -> caste-scene-process

> 

> - view3dscene -> castle-scene-view

> 

> I haven't made up my mind 100% yet, if rename happens I will of course 

> announce it prominently everywhere :)

> 

> 

> 

> Regards,

> 

> Michalis

> 

> 

> 

> niedz., 13 sie 2023 o 17:53 Brutzman, Donald (Don) (CIV) < <mailto:brutzman at nps.edu> brutzman at nps.edu> napisał(a):

> 

> >

> 

> > Hi Michalis.  Thanks for all of your immense, continuing work.

> 

> >

> 

> > I am hoping to add validation of converted .wrl and .x3dv files to the regression test suite for X3D Example Archives.  This will check the quality of output from the X3dToVrml.xslt stylesheets, and improve the quality assurance (QA) of all VRML models offered there – over 4000 in each case. In each case I’ll create and commit a build.log text file so that we can keep track of compliance and progress.

> 

> >

> 

> > X3D Resources, Examples: Scene Archives for X3D

> 

> >  <https://www.web3d.org/x3d/content/examples/X3dResources.html#Example> https://www.web3d.org/x3d/content/examples/X3dResources.html#Example

> > s

> 

> >

> 

> > To do so, am thinking that command-line invocation (CLI) of your view3dscene or tovrmlx3d executable tools can be accomplished through addition to the existing Ant build.xml scripts.  Am keen to proceed with that task.

> 

> >

> 

> > Am looking through the view3dscene CLI documentation, but am not exactly sure how to accomplish this.

> 

> >

> 

> > view3dscene, 4. Command-line options

> 

> >  <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fca> https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fca

> > stle-engine.io%2Fview3dscene.php%23section_command_line_options&data

> > =05%7C01%7Cbrutzman%40nps.edu%7C2baece0d2d40458e15cb08dba3e28885%7C6

> > d936231a51740ea9199f7578963378e%7C0%7C0%7C638283965717273937%7CUnkno

> > wn%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWw

> > iLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=Wxt6n6awK8i1MHeaKC1Wv8%2Fbdn%2

> > FsWIo4JGaCBfRFzkw%3D&reserved=0

> 

> >

> 

> > view3dscene, 4.2. Converting to X3D

> 

> >  <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fca> https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fca

> > stle-engine.io%2Fview3dscene.php%23section_converting&data=05%7C01%7

> > Cbrutzman%40nps.edu%7C2baece0d2d40458e15cb08dba3e28885%7C6d936231a51

> > 740ea9199f7578963378e%7C0%7C0%7C638283965717273937%7CUnknown%7CTWFpb

> > GZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6M

> > n0%3D%7C3000%7C%7C%7C&sdata=NZk2EMKFQcYMv11ajbUJrWXvp07dmDOhxr4QLSpz

> > nKE%3D&reserved=0

> 

> >

> 

> > Additional options needed, or not seen:

> 

> >

> 

> > Validation report only, plain text via console.  (Perhaps a 

> > -validate

> 

> > switch is a variation on your ---write switches.) Avoid creation of

> 

> > additional files, if possible.  (I might send each output to a 

> > single

> 

> > temp file, but all that file writing will likely slow down the

> 

> > already-long overall process.)

> 

> >

> 

> > Please advise if there is a way to invoke one of your tools to accomplish this.  CLI example will be helpful.  Thanks in advance for all guidance.

> 

> >

> 

> > Have fun with VRML!   8)

> 

> >

> 

> > all the best, Don

> 

> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://web3d.org/pipermail/x3d-public_web3d.org/attachments/20230823/1d7ce05e/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 5464 bytes
Desc: not available
URL: <http://web3d.org/pipermail/x3d-public_web3d.org/attachments/20230823/1d7ce05e/attachment-0001.p7s>