[x3d-public] Unexpected behaviour of X3D Validator; X3D regular expressions (regexes)

Don Brutzman brutzman at nps.edu
Fri Jun 1 05:59:40 PDT 2018 

Wow Vince and John, pretty amazing!  Thanks for sharing these insights.  Very important.

I have updated the X3D Regular Expressions page (long overdue) and placed the entries for SFVec3f and MFVec3f there.
	
	X3D Regular Expressions (regexes)
	http://www.web3d.org/specifications/X3dRegularExpressions.html#top
	
Review suggestions welcome, probably a little more layout refactoring and then will add entries for other X3D data types.

Original XML Schema regex announcement:

	[X3D-Public] new X3D quality assurance tests using regular expressions (regexes)
	http://www.web3d.org/pipermail/x3d-public_web3d.org/2012-March/001950.html

Frédéric, as before, getting a link to an example scene (however long) that triggers the problem will be quite helpful.

Memory-consumption issues for large values might be a further impetus to move X3D regexes for n-tuple numeric types to an alternate validation mechanism, switching X3D Schema to native XML Schema data types.  However one major difference in X3D is allowing commas as whitespace, so there are tradeoffs.  One variation is that X3D Schema only allows commas between n-tuples (e.g. since an SFVec3f including commas adds no benefit), which has proven to be quite helpful in debugging realllly long arrays.  Current design is acceptance of valid content, as strictly as possible, since false positives lead to people turning off validation.

Hopefully renewed work on X3D Regex page facilitates further checking of X3D regular expressions.  Good luck out there!  8)


On 5/31/2018 8:00 PM, John Carlson wrote:
> It would seem like the error is in xerces regular expressions, not x3d's schema.   You might look at the standard for encoding= to see if space is allowed.
> 
> John
> 
> 
> On Thu, May 31, 2018, 7:24 PM vmarchetti at kshell.com <mailto:vmarchetti at kshell.com> <vmarchetti at kshell.com <mailto:vmarchetti at kshell.com>> wrote:
> 
>     I have been looking some more at the XML Schema failure (or rather, failure to complete) in the shapley_small.x3d file. I have been running the Xerces Java-based validation tools locally so I can directly trap stderr and stdout
> 
> 
>     radium:shapley validation failure vmarchetti$ python -m xmlschema x3d-3.3.xsd shapley_small.x3d  # python just provides a wrapper around call to java
>     Exception in thread "main" java.lang.StackOverflowError
>     at org.apache.xerces.impl.xpath.regex.RegularExpression.matchString(Unknown Source)
>     ... (1020 repetition of this same line)
>     at org.apache.xerces.impl.xpath.regex.RegularExpression.matchString(Unknown Source)
>     at org.apache.xerces.impl.xpath.regex.RegularExpression.matchString(Unknown Source)
>     radium:shapley validation failure vmarchetti$
> 
>     Google tells me that that this is a well-known failure in using Regex libraries due to a regular expression that ends up taking exponential growing  resources;
>     see http://www.regular-expressions.info/catastrophic.html
> 
>     My conclusions
> 
>     -- Concur with Frederic that this issue should not impede his work, I am looking forward to seeing his results.
>     -- I will look into seeing if the regular expressions that are used in the XML-Schema tests for X3D can be made more robust against this problem, using the guidelines in http://www.regular-expressions.info/catastrophic.html
>     -- The DOCTYPE test:
>>     Performing DOCTYPE check...
>>     [Error] [X3dDoctypeChecker] failure: no valid XML declaration found in scene!
>>     <?xml version="1.0" encoding ="UTF-8"?>
>>     <!DOCTYPE X3D PUBLIC "ISO//Web3D//DTD X3D 3.3//EN" "http://www.web3d.org/specifications/x3d-3.3.dtd">
>>     usage: java X3dDoctypeChecker sceneName.x3d [-verbose | -setFinalDTD | -setTransitionalDTD]
>     Succeeds for me when I remove the blank space between attribute label 'encoding' and the = character, in the <?xml declaration.
> 
>     Vince Marchetti
> 
> 
> 
>>     On May 31, 2018, at 5:04 PM, Frédéric Vogt <frederic.vogt at alumni.anu.edu.au <mailto:frederic.vogt at alumni.anu.edu.au>> wrote:
>>
>>     Hi everyone,
>>
>>     Thank you for all the suggestions and prompt feedback. Here’s a status update:
>>
>>     - I’m pretty sure that the behavior with Safari 10.1.2 is related to the size of the X3D file. Anything below 10Mb works fine. Anything above doesn’t. This took me a while to figure out, because if I try to load a model <10Mb *after* trying to load a model >10Mb (and failing to do so), the error remains … and only disappears if I restart the browser. So the answer is: stay smaller than 10Mb if I want things to work ok in this older Safari version. A quick internet search also revealed that there indeed used to be some sort of a 10Mb limit for Safari, at some point.
>>
>>     - The fact that it works in a more recent Safari version is then probably an indication that this 10Mb limit has changed/been lifted - not quite sure just yet.
>>
>>     - As suggested, I installed X3D-edit, and tested the x3d file. The reported string errors popped up, plus a few warnings about missing/wrong “meta” fields, and the Collision entry. I fixed it all, and am now left with this:
>>
>>     --------- Commence validation checks for shapley_very_small.x3d ---------
>>
>>     Performing well-formed XML check...
>>     Checking file:/Users/fvogt/Projects/current/Misc/INAF/Shapley_3D/dev/shapley_very_small.x3d...
>>     Well-formed XML check: pass
>>
>>     Performing DOCTYPE check...
>>     [Error] [X3dDoctypeChecker] failure: no valid XML declaration found in scene!
>>     <?xml version="1.0" encoding ="UTF-8"?>
>>     <!DOCTYPE X3D PUBLIC "ISO//Web3D//DTD X3D 3.3//EN" "http://www.web3d.org/specifications/x3d-3.3.dtd">
>>     usage: java X3dDoctypeChecker sceneName.x3d [-verbose | -setFinalDTD | -setTransitionalDTD]
>>
>>     Performing DTD validation...
>>     Checking file:/Users/fvogt/Projects/current/Misc/INAF/Shapley_3D/dev/shapley_very_small.x3d...
>>     XML DTD validation: pass
>>
>>     Performing X3D schema validation...
>>     Checking file:/Users/fvogt/Projects/current/Misc/INAF/Shapley_3D/dev/shapley_very_small.x3d...
>>     Referenced entity at "nbres:/org/web3d/x3d/externals/schemas/x3d-3.3-Web3dExtensionsPublic.xsd".
>>     XML schema validation: pass
>>
>>     Performing X3D regular expression (regex) values check...
>>
>>     Performing X3dToClassicVrml.xslt conversion check...
>>
>>     Performing X3D Schematron check...
>>
>>     ———— Validation checks complete for shapley_very_small.x3d ---------
>>
>>     Not sure why the doctype still fails the verification. For reference, the “very_small” file is now <500kb, suggesting that this error is not related to the file size.
>>
>>     I’m ok to continue with my project now that I’ve figured out why Safari fails. But I’d still be keen to know why the validation fails. If anyone feels like playing around with the file, it is here: http://www.sc.eso.org/~fvogt/Shapley/shapley_very_small.x3d
>>
>>     Cheers,
>>     Frédéric
>>
>>
>>>     On May 31, 2018, at 17:27, Andreas Plesch <andreasplesch at gmail.com <mailto:andreasplesch at gmail.com>> wrote:
>>>
>>>     I can confirm the validator behaviour. Perhaps the 'small' x3d is
>>>     still too large to be processed although the no DTD version is
>>>     actually processed.
>>>
>>>     https://gist.githubusercontent.com/andreasplesch/973d94d6e88db471e9473b0af78a5a67/raw/8d8f0aa7b397973f42ee74f62f200a3849f92d5c/Shapley_small_tidied.x3d
>>>
>>>     is the result of the no dtd x3d after being processed by the
>>>     validator, plus the MFString fixes noticed by Leonard.
>>>
>>>     x3dom is less strict about quoting in text node string field. This is
>>>     why x3dom works for your nice ShaSS 3d page. No plugins needed.
>>>
>>>     Not sure what the Safari 10.1.2 issue could be. Sometimes you get
>>>     these no scene errors if the html/xml parser had problems, often with
>>>     self-closing tags which work only in xml. Did the inlined x3d used to
>>>     work in this version of Safari ? Other messages in the browser
>>>     console, or x3dom debug console (press 'd') ? Does the inline tutorial
>>>     example on x3dom.org <http://x3dom.org> work ?
>>>
>>>     -Andreas
>>>
>>>>     Date: Thu, 31 May 2018 11:19:44 -0300
>>>>     From: Fr?d?ric Vogt <frederic.vogt at alumni.anu.edu.au <mailto:frederic.vogt at alumni.anu.edu.au>>
>>>>     To: x3d-public at web3d.org <mailto:x3d-public at web3d.org>
>>>>     Subject: [x3d-public] Unexpected behaviour of X3D Validator
>>>>
>>>>     Dear X3D gurus,
>>>>
>>>>     I am once again trying to assemble an astro-X3D model using the mayavi python module, here: http://www.sc.eso.org/~fvogt/Shapley/shapley.html <http://www.sc.eso.org/~fvogt/Shapley/shapley.html>
>>>>
>>>>     I am on Mac OS X Yosemite (still?), and can load the model just fine with:
>>>>
>>>>     Firefox 60.0.1
>>>>     Chrome 67.0.3396.62
>>>>
>>>>     However, the model does *not* load with:
>>>>
>>>>     Safari 10.1.2 -> I get ?ERROR: no scene in resource? message in the development console right after it loads the underlying X3D file.
>>>>
>>>>     I have confirmed that under Mac OS X High Sierra, with Safari 11.1, it does works, though!
>>>>
>>>>     So I started worrying about possible formatting issues with my X3D file based on this old thread: https://sourceforge.net/p/x3dom/mailman/message/32684050/ <https://sourceforge.net/p/x3dom/mailman/message/32684050/>
>>>>     After all, I already know that the VTK X3D exporter is far from being perfect! http://fpavogt.github.io/x3d-pathway/dices.html <http://fpavogt.github.io/x3d-pathway/dices.html>
>>>>
>>>>     Which brings me to my question about the X3D Validator, which I tried to use to check for any bug in my model: https://savage.nps.edu/X3dValidator <https://savage.nps.edu/X3dValidator>
>>>>
>>>>     When I load a smaller version of the X3D model (I can?t load the actual model, likely too big), I get an error because of the lack of DTD info. But when I try to include
>>>>
>>>>     <!DOCTYPE X3D PUBLIC "ISO//Web3D//DTD X3D 3.3//EN" "http://www.web3d.org/specifications/x3d-3.3.dtd">
>>>>     <X3D profile='Immersive' version='3.3'
>>>>          xmlns:xsd='http://www.w3.org/2001/XMLSchema-instance'
>>>>          xsd:noNamespaceSchemaLocation =? http://www.web3d.org/specifications/x3d-3.3.xsd '>
>>>>
>>>>     then the X3D Validator returns me an empty page.
>>>>
>>>>     What am I doing wrong here ?
>>>>
>>>>     The file in question (if anyone is keen to try it out themselves) is: http://www.sc.eso.org/~fvogt/Shapley/shapley_small.x3d <http://www.sc.eso.org/~fvogt/Shapley/shapley_small.x3d>
>>>>
>>>>     Cheers,
>>>>     Fr?d?ric

all the best, Don
-- 
Don Brutzman  Naval Postgraduate School, Code USW/Br       brutzman at nps.edu
Watkins 270,  MOVES Institute, Monterey CA 93943-5000 USA   +1.831.656.2149
X3D graphics, virtual worlds, navy robotics http://faculty.nps.edu/brutzman

More information about the x3d-public mailing list