[x3d-public] VRML97 and ClassicVRML model validation from command line using tovrmlx3d

Wed Sep 27 16:15:00 PDT 2023

Hi Michalis, am finally getting back to this one.

Your tovrmlx3d.exe tool is super, and very fast.  Have followed your guidance here and integrated it in the quality assurance (QA) regression tests for X3D Examples Archives.  No response in log is good, meaning no error or warning.  Excerpt:

build.xml

ant -f C:\\x3d-code\\www.web3d.org\\x3d\\content\\examples\\HumanoidAnimation processScenes.X3dToX3dvClassicVrmlEncoding

processScenes.X3dToX3dvClassicVrmlEncoding:

C:\x3d-code\www.web3d.org\x3d\content\examples\HumanoidAnimation/Characters//HAnimModelFootLeft.x3d processing with X3dToX3dvClassicVrmlEncoding stylesheet

ClassicVRML validation: tovrmlx3d.exe Characters//HAnimModelFootLeft.x3dv --validate --enable-downloads

C:\x3d-code\www.web3d.org\x3d\content\examples\HumanoidAnimation/Characters//HAnimModelFootRight.x3d processing with X3dToX3dvClassicVrmlEncoding stylesheet

ClassicVRML validation: tovrmlx3d.exe Characters//HAnimModelFootRight.x3dv --validate --enable-downloads

C:\x3d-code\www.web3d.org\x3d\content\examples\HumanoidAnimation/Characters//HAnimModelHandLeft.x3d processing with X3dToX3dvClassicVrmlEncoding stylesheet

ClassicVRML validation: tovrmlx3d.exe Characters//HAnimModelHandLeft.x3dv --validate --enable-downloads

etc.

Next, have started debugging the X3dToX3dvClassicVrmlEncoding.xslt stylesheet.  Plenty of issues found  in first test sequence (perhaps 28 of 111 model conversions failed), will start fixing things.  Most look straightforward and recurring, relating to X3D 4.0 updates from X3D3 that were never added before.

*	https://sourceforge.net/p/x3d/code/HEAD/tree/www.web3d.org/x3d/stylesheets/X3dToX3dvClassicVrmlEncoding.xslt
*	which simply invokes
*	https://sourceforge.net/p/x3d/code/HEAD/tree/www.web3d.org/x3d/stylesheets/X3dToVrml97.xslt

Will post issue reports for tovrmlx3d.exe if any emerge.

*	https://github.com/castle-engine/view3dscene/issues

Initial regression-testing build log and archive:

*	https://www.web3d.org/x3d/content/examples/HumanoidAnimation/build.x3dv.log.txt

*	HumanoidAnimation (HAnim) X3D Examples Archive
*	https://www.web3d.org/x3d/content/examples/HumanoidAnimation

Eventually, after proceeding through all 8 X3D Example Archives, will next look at VRML97 .wrl files.  Should be pretty clean at that stage since it is the same stylesheet converter, which either notes unsupported X3D nodes or else inserts an excerpted VRML97 prototype when such backwards-compatibility support for VRML97 is possible.  Examples:

*	X3D Example Archives: Basic, development, Event Utility Prototypes
*	https://www.web3d.org/x3d/content/examples/Basic/development/EventUtilityPrototypesIndex.html
*	https://www.web3d.org/x3d/content/examples/Basic/development/EventUtilityPrototypes.wrl
*	… and other prototypes in that directory

Onward we go.  Having fun with ClassicVrml .x3dv!   8)

all the best, Don

-- 

Don Brutzman  Naval Postgraduate School, Code USW/Br        brutzman at nps.edu

Watkins 270,  MOVES Institute, Monterey CA 93943-5000 USA    +1.831.656.2149

X3D graphics, virtual worlds, navy robotics https://faculty.nps.edu/brutzman

-----Original Message-----
From: Michalis Kamburelis <michalis.kambi at gmail.com> 
Sent: Wednesday, August 23, 2023 7:09 AM
To: Brutzman, Donald (Don) (CIV) <brutzman at nps.edu>
Cc: X3D Public Mailing List (x3d-public at web3d.org) <x3d-public at web3d.org>
Subject: Re: VRML97 and ClassicVRML model validation from command line using view3dscene

Don,

1. As for some missing X3D 4.0 nodes: Yes, this can happen, as in the past I didn't carefully check whether all X3D 4.0 nodes have been added to CGE/view3dscene, and indeed I was fixing later some missing ones (like H-Anim nodes from X3D 4.0).

    If you can extract the list of missing X3D 4.0 nodes from above logs, I'll be grateful -- I didn't find relevant messages now from the logs you linked, but admittedly I only looked quickly.

2. To avoid the "Error EDownloadError when loading inline file from URL ...: Downloading network resources (from "http" or "https" protocols) is not enabled. Set global EnableBlockingDownloads:=true to enable." pass the additional command-line option

    --enable-downloads

    to tovrmlx3d. Note that it will definitely increase the validation time. The tovrmlx3d will download the linked resources (Inline X3D content, X3D content referred by EXTERNPROTOs etc.) from the Internet at validation.

Regards,

Michalis

śr., 23 sie 2023 o 08:18 Brutzman, Donald (Don) (CIV) < <mailto:brutzman at nps.edu> brutzman at nps.edu> napisał(a):

> 

> Michalis, appreciate your adding this feature as part of tovrmlx3d executable.  I’ve integrated into the Ant build scripts and tested operation satisfactorily for all .wrl and .x3dv conversions in the X3D Examples Archives.

> 

> Example command-line invocation on windows, a little terser works out OK for these logs:

> 

> tovrmlx3d.exe 

> C:\x3d-code\www.web3d.org\x3d\content\examples\X3dForWebAuthors\Chapte

> r04ViewingNavigation/CollisionExample.x3dv --validate

> 

> Regression-test results online (and in version control to track progress) as follows.  Most look clean.

> 

>  <https://www.web3d.org/x3d/content/examples/build.validate.VRML.all.log> https://www.web3d.org/x3d/content/examples/build.validate.VRML.all.log.txt

> 

>  <https://www.web3d.org/x3d/content/examples/build.validate.x3dv.all.log> https://www.web3d.org/x3d/content/examples/build.validate.x3dv.all.log.txt

> 

> 

> 

> […]

> 

> 

> 

> Execution is fast!  Execution took around a half hour for over 4000 examples, most of that time was likely writing test-result outputs to console.

> 

> 

> 

> I haven’t tried fixing any model errors or X3dToVrml97.xslt stylesheet conversion errors yet (e.g. true false vice TRUE FALSE etc.) – that will happen.

> 

> 

> 

> You might see some issues in there that relate to your side, saw a few 

> such as

> 

> new X3D4 nodes that might not yet be included in view3dscene, 

> Backslash and \n error reports didn’t include line numbers (might be 

> hard to isolate),

> 

> 

> 

> Saw this error message, but not sure how to implement setting the property via command line, please advise:

> 

> 

> 

> tovrmlx3d: Warning: X3D: Error EDownloadError when loading inline file from URL  <https://www.web3d.org/x3d/content/examples/Basic/HumanoidAnimation/NancyDiving.x3d> https://www.web3d.org/x3d/content/examples/Basic/HumanoidAnimation/NancyDiving.x3d: Downloading network resources (from "http" or "https"  protocols) is not enabled. Set global EnableBlockingDownloads:=true to enable.

> 

> 

> 

> Happy to now have these post-conversion tests available, enabling continued deliberate improvements in quality assurance (QA).  Again thanks for a tremendous capability, very helpful.

> 

> 

> 

> Have fun with VRML!  8)

> 

> 

> 

> all the best, Don

> 

> --

> 

> Don Brutzman  Naval Postgraduate School, Code USW/Br         <mailto:brutzman at nps.edu> brutzman at nps.edu

> 

> Watkins 270,  MOVES Institute, Monterey CA 93943-5000 USA    +1.831.656.2149

> 

> X3D graphics, virtual worlds, navy robotics 

>  <https://faculty.nps.edu/brutzman> https://faculty.nps.edu/brutzman

> 

> 

> 

> 

> 

> -----Original Message-----

> From: Michalis Kamburelis < <mailto:ichalis.kambi at gmail.com> ichalis.kambi at gmail.com>

> Sent: Monday, August 14, 2023 5:40 AM

> To: Brutzman, Donald (Don) (CIV) < <mailto:brutzman at nps.edu> brutzman at nps.edu>

> Cc: X3D Public Mailing List ( <mailto:x3d-public at web3d.org> x3d-public at web3d.org) 

> < <mailto:x3d-public at web3d.org> x3d-public at web3d.org>

> Subject: Re: VRML97 and ClassicVRML model validation from command line 

> using view3dscene

> 

> 

> 

> Don,

> 

> 

> 

> In short: I added now –validate option to tovrmlx3d 😊

> 

> 

> 

> Example usage on a valid model:

> 

> 

> 

> ```

> 

> $ ./tovrmlx3d ../demo-models/x3d/axis.x3d –validate $ echo $?

> 

> 0

> 

> ```

> 

> 

> 

> So empty output, and exit status 0, as expected from most tools. You can use this with Ant, Make or anything else.

> 

> 

> 

> Example usage on a model with problems:

> 

> 

> 

> ```

> 

> $ ./tovrmlx3d 

> ../demo-models/x3d/warnings/invalid_statement_omitted_1.x3dv

> 

> --validate

> 

> tovrmlx3d: Warning: VRML/X3D: Error when reading, will skip the rest of X3D file: Error at line 15 column 2: Expected node type or DEF or USE, got “}”

> 

> Exception “Exception”:

> 

> Validation failed (consult the warnings above), exiting with non-zero 

> status

> 

> 

> 

> $ echo $?

> 

> 1

> 

> ```

> 

> 

> 

> So you get output on stderr, and non-zero exit status.

> 

> 

> 

> Please give ~1 hour for Jenkins to process it, and this new version of 

> tovrmlx3d will be part of “snapshot" version on

> 

>  <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcastle-engine.io%2Fview3dscene.php&data=05%7C01%7Cbrutzman%40nps.edu%7C2baece0d2d40458e15cb08dba3e28885%7C6d936231a51740ea9199f7578963378e%7C0%7C0%7C638283965717273937%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=nD1LFbNHsqygeWo8hFkU1Sc6hb85nsiMljC5wLe9VWQ%3D&reserved=0> https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcastle-engine.io%2Fview3dscene.php&data=05%7C01%7Cbrutzman%40nps.edu%7C2baece0d2d40458e15cb08dba3e28885%7C6d936231a51740ea9199f7578963378e%7C0%7C0%7C638283965717273937%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=nD1LFbNHsqygeWo8hFkU1Sc6hb85nsiMljC5wLe9VWQ%3D&reserved=0 .

> 

> 

> 

> I used above test models from  <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fcastle-engine%2Fdemo-models&data=05%7C01%7Cbrutzman%40nps.edu%7C2baece0d2d40458e15cb08dba3e28885%7C6d936231a51740ea9199f7578963378e%7C0%7C0%7C638283965717273937%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=5vytRe7QjCwyp6twC57eXbSCA2L8FQroM3Is3WArmIE%3D&reserved=0> https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fcastle-engine%2Fdemo-models&data=05%7C01%7Cbrutzman%40nps.edu%7C2baece0d2d40458e15cb08dba3e28885%7C6d936231a51740ea9199f7578963378e%7C0%7C0%7C638283965717273937%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=5vytRe7QjCwyp6twC57eXbSCA2L8FQroM3Is3WArmIE%3D&reserved=0 .

> 

> 

> 

> Details:

> 

> 

> 

> So far, we did validation just by converting and piping output to 

> /dev/null, so "tovrmlx3d xxxx.x3d > /dev/null". But I agree this was

> 

> unoptimal:

> 

> 

> 

> - Explicit "--validate" means we don't needlessly waste time converting to /dev/null if you actually just want to validate. We also don't produce eventual warnings when writing, that would sometimes appear and duplicate warnings we already did when reading.

> 

> 

> 

> - In the future, maybe, explicit "--validate" may allow to run some additional tasks dedicated to validation (maybe because they are time-consuming and not necessary for just usual opening). We don't have such tasks right now, but it would be nice to have such option available.

> 

> 

> 

> - With explicit "--validate", any warning or error causes non-zero 

> exit status. Makes sense, since a warning (even if we can continue

> 

> working) still means validation failed.

> 

> 

> 

> So ... I added "--validate" option to tovrmlx3d.

> 

> 

> 

> Of note: A rename is on the horizon, for a few reasons (glTF export, consistency with other CGE tools). Most likely:

> 

> - tovrmlx3d -> caste-scene-process

> 

> - view3dscene -> castle-scene-view

> 

> I haven't made up my mind 100% yet, if rename happens I will of course 

> announce it prominently everywhere :)

> 

> 

> 

> Regards,

> 

> Michalis

> 

> 

> 

> niedz., 13 sie 2023 o 17:53 Brutzman, Donald (Don) (CIV) < <mailto:brutzman at nps.edu> brutzman at nps.edu> napisał(a):

> 

> >

> 

> > Hi Michalis.  Thanks for all of your immense, continuing work.

> 

> >

> 

> > I am hoping to add validation of converted .wrl and .x3dv files to the regression test suite for X3D Example Archives.  This will check the quality of output from the X3dToVrml.xslt stylesheets, and improve the quality assurance (QA) of all VRML models offered there – over 4000 in each case. In each case I’ll create and commit a build.log text file so that we can keep track of compliance and progress.

> 

> >

> 

> > X3D Resources, Examples: Scene Archives for X3D

> 

> >  <https://www.web3d.org/x3d/content/examples/X3dResources.html#Example> https://www.web3d.org/x3d/content/examples/X3dResources.html#Example

> > s

> 

> >

> 

> > To do so, am thinking that command-line invocation (CLI) of your view3dscene or tovrmlx3d executable tools can be accomplished through addition to the existing Ant build.xml scripts.  Am keen to proceed with that task.

> 

> >

> 

> > Am looking through the view3dscene CLI documentation, but am not exactly sure how to accomplish this.

> 

> >

> 

> > view3dscene, 4. Command-line options

> 

> >  <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fca> https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fca

> > stle-engine.io%2Fview3dscene.php%23section_command_line_options&data

> > =05%7C01%7Cbrutzman%40nps.edu%7C2baece0d2d40458e15cb08dba3e28885%7C6

> > d936231a51740ea9199f7578963378e%7C0%7C0%7C638283965717273937%7CUnkno

> > wn%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWw

> > iLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=Wxt6n6awK8i1MHeaKC1Wv8%2Fbdn%2

> > FsWIo4JGaCBfRFzkw%3D&reserved=0

> 

> >

> 

> > view3dscene, 4.2. Converting to X3D

> 

> >  <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fca> https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fca

> > stle-engine.io%2Fview3dscene.php%23section_converting&data=05%7C01%7

> > Cbrutzman%40nps.edu%7C2baece0d2d40458e15cb08dba3e28885%7C6d936231a51

> > 740ea9199f7578963378e%7C0%7C0%7C638283965717273937%7CUnknown%7CTWFpb

> > GZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6M

> > n0%3D%7C3000%7C%7C%7C&sdata=NZk2EMKFQcYMv11ajbUJrWXvp07dmDOhxr4QLSpz

> > nKE%3D&reserved=0

> 

> >

> 

> > Additional options needed, or not seen:

> 

> >

> 

> > Validation report only, plain text via console.  (Perhaps a 

> > -validate

> 

> > switch is a variation on your ---write switches.) Avoid creation of

> 

> > additional files, if possible.  (I might send each output to a 

> > single

> 

> > temp file, but all that file writing will likely slow down the

> 

> > already-long overall process.)

> 

> >

> 

> > Please advise if there is a way to invoke one of your tools to accomplish this.  CLI example will be helpful.  Thanks in advance for all guidance.

> 

> >

> 

> > Have fun with VRML!   8)

> 

> >

> 

> > all the best, Don

> 

> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://web3d.org/pipermail/x3d-public_web3d.org/attachments/20230927/b3376150/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 5464 bytes
Desc: not available
URL: <http://web3d.org/pipermail/x3d-public_web3d.org/attachments/20230927/b3376150/attachment-0001.p7s>