[x3d-public] Unexpected behaviour of X3D Validator; X3D regular expressions (regexes)
John Carlson
yottzumm at gmail.com
Fri Jun 1 09:38:04 PDT 2018
Another thing you might do is increase stack space in the JVM.
John
On Fri, Jun 1, 2018, 9:01 AM Don Brutzman <brutzman at nps.edu> wrote:
> Wow Vince and John, pretty amazing! Thanks for sharing these insights.
> Very important.
>
> I have updated the X3D Regular Expressions page (long overdue) and placed
> the entries for SFVec3f and MFVec3f there.
>
> X3D Regular Expressions (regexes)
> http://www.web3d.org/specifications/X3dRegularExpressions.html#top
>
> Review suggestions welcome, probably a little more layout refactoring and
> then will add entries for other X3D data types.
>
> Original XML Schema regex announcement:
>
> [X3D-Public] new X3D quality assurance tests using regular
> expressions (regexes)
>
> http://www.web3d.org/pipermail/x3d-public_web3d.org/2012-March/001950.html
>
> Frédéric, as before, getting a link to an example scene (however long)
> that triggers the problem will be quite helpful.
>
> Memory-consumption issues for large values might be a further impetus to
> move X3D regexes for n-tuple numeric types to an alternate validation
> mechanism, switching X3D Schema to native XML Schema data types. However
> one major difference in X3D is allowing commas as whitespace, so there are
> tradeoffs. One variation is that X3D Schema only allows commas between
> n-tuples (e.g. since an SFVec3f including commas adds no benefit), which
> has proven to be quite helpful in debugging realllly long arrays. Current
> design is acceptance of valid content, as strictly as possible, since false
> positives lead to people turning off validation.
>
> Hopefully renewed work on X3D Regex page facilitates further checking of
> X3D regular expressions. Good luck out there! 8)
>
>
> On 5/31/2018 8:00 PM, John Carlson wrote:
> > It would seem like the error is in xerces regular expressions, not x3d's
> schema. You might look at the standard for encoding= to see if space is
> allowed.
> >
> > John
> >
> >
> > On Thu, May 31, 2018, 7:24 PM vmarchetti at kshell.com <mailto:
> vmarchetti at kshell.com> <vmarchetti at kshell.com <mailto:
> vmarchetti at kshell.com>> wrote:
> >
> > I have been looking some more at the XML Schema failure (or rather,
> failure to complete) in the shapley_small.x3d file. I have been running the
> Xerces Java-based validation tools locally so I can directly trap stderr
> and stdout
> >
> >
> > radium:shapley validation failure vmarchetti$ python -m xmlschema
> x3d-3.3.xsd shapley_small.x3d # python just provides a wrapper around call
> to java
> > Exception in thread "main" java.lang.StackOverflowError
> > at
> org.apache.xerces.impl.xpath.regex.RegularExpression.matchString(Unknown
> Source)
> > ... (1020 repetition of this same line)
> > at
> org.apache.xerces.impl.xpath.regex.RegularExpression.matchString(Unknown
> Source)
> > at
> org.apache.xerces.impl.xpath.regex.RegularExpression.matchString(Unknown
> Source)
> > radium:shapley validation failure vmarchetti$
> >
> > Google tells me that that this is a well-known failure in using
> Regex libraries due to a regular expression that ends up taking exponential
> growing resources;
> > see http://www.regular-expressions.info/catastrophic.html
> >
> > My conclusions
> >
> > -- Concur with Frederic that this issue should not impede his work,
> I am looking forward to seeing his results.
> > -- I will look into seeing if the regular expressions that are used
> in the XML-Schema tests for X3D can be made more robust against this
> problem, using the guidelines in
> http://www.regular-expressions.info/catastrophic.html
> > -- The DOCTYPE test:
> >> Performing DOCTYPE check...
> >> [Error] [X3dDoctypeChecker] failure: no valid XML declaration found
> in scene!
> >> <?xml version="1.0" encoding ="UTF-8"?>
> >> <!DOCTYPE X3D PUBLIC "ISO//Web3D//DTD X3D 3.3//EN" "
> http://www.web3d.org/specifications/x3d-3.3.dtd">
> >> usage: java X3dDoctypeChecker sceneName.x3d [-verbose |
> -setFinalDTD | -setTransitionalDTD]
> > Succeeds for me when I remove the blank space between attribute
> label 'encoding' and the = character, in the <?xml declaration.
> >
> > Vince Marchetti
> >
> >
> >
> >> On May 31, 2018, at 5:04 PM, Frédéric Vogt <
> frederic.vogt at alumni.anu.edu.au <mailto:frederic.vogt at alumni.anu.edu.au>>
> wrote:
> >>
> >> Hi everyone,
> >>
> >> Thank you for all the suggestions and prompt feedback. Here’s a
> status update:
> >>
> >> - I’m pretty sure that the behavior with Safari 10.1.2 is related
> to the size of the X3D file. Anything below 10Mb works fine. Anything above
> doesn’t. This took me a while to figure out, because if I try to load a
> model <10Mb *after* trying to load a model >10Mb (and failing to do so),
> the error remains … and only disappears if I restart the browser. So the
> answer is: stay smaller than 10Mb if I want things to work ok in this older
> Safari version. A quick internet search also revealed that there indeed
> used to be some sort of a 10Mb limit for Safari, at some point.
> >>
> >> - The fact that it works in a more recent Safari version is then
> probably an indication that this 10Mb limit has changed/been lifted - not
> quite sure just yet.
> >>
> >> - As suggested, I installed X3D-edit, and tested the x3d file. The
> reported string errors popped up, plus a few warnings about missing/wrong
> “meta” fields, and the Collision entry. I fixed it all, and am now left
> with this:
> >>
> >> --------- Commence validation checks for shapley_very_small.x3d
> ---------
> >>
> >> Performing well-formed XML check...
> >> Checking
> file:/Users/fvogt/Projects/current/Misc/INAF/Shapley_3D/dev/shapley_very_small.x3d...
> >> Well-formed XML check: pass
> >>
> >> Performing DOCTYPE check...
> >> [Error] [X3dDoctypeChecker] failure: no valid XML declaration found
> in scene!
> >> <?xml version="1.0" encoding ="UTF-8"?>
> >> <!DOCTYPE X3D PUBLIC "ISO//Web3D//DTD X3D 3.3//EN" "
> http://www.web3d.org/specifications/x3d-3.3.dtd">
> >> usage: java X3dDoctypeChecker sceneName.x3d [-verbose |
> -setFinalDTD | -setTransitionalDTD]
> >>
> >> Performing DTD validation...
> >> Checking
> file:/Users/fvogt/Projects/current/Misc/INAF/Shapley_3D/dev/shapley_very_small.x3d...
> >> XML DTD validation: pass
> >>
> >> Performing X3D schema validation...
> >> Checking
> file:/Users/fvogt/Projects/current/Misc/INAF/Shapley_3D/dev/shapley_very_small.x3d...
> >> Referenced entity at
> "nbres:/org/web3d/x3d/externals/schemas/x3d-3.3-Web3dExtensionsPublic.xsd".
> >> XML schema validation: pass
> >>
> >> Performing X3D regular expression (regex) values check...
> >>
> >> Performing X3dToClassicVrml.xslt conversion check...
> >>
> >> Performing X3D Schematron check...
> >>
> >> ———— Validation checks complete for shapley_very_small.x3d ---------
> >>
> >> Not sure why the doctype still fails the verification. For
> reference, the “very_small” file is now <500kb, suggesting that this error
> is not related to the file size.
> >>
> >> I’m ok to continue with my project now that I’ve figured out why
> Safari fails. But I’d still be keen to know why the validation fails. If
> anyone feels like playing around with the file, it is here:
> http://www.sc.eso.org/~fvogt/Shapley/shapley_very_small.x3d
> >>
> >> Cheers,
> >> Frédéric
> >>
> >>
> >>> On May 31, 2018, at 17:27, Andreas Plesch <andreasplesch at gmail.com
> <mailto:andreasplesch at gmail.com>> wrote:
> >>>
> >>> I can confirm the validator behaviour. Perhaps the 'small' x3d is
> >>> still too large to be processed although the no DTD version is
> >>> actually processed.
> >>>
> >>>
> https://gist.githubusercontent.com/andreasplesch/973d94d6e88db471e9473b0af78a5a67/raw/8d8f0aa7b397973f42ee74f62f200a3849f92d5c/Shapley_small_tidied.x3d
> >>>
> >>> is the result of the no dtd x3d after being processed by the
> >>> validator, plus the MFString fixes noticed by Leonard.
> >>>
> >>> x3dom is less strict about quoting in text node string field. This
> is
> >>> why x3dom works for your nice ShaSS 3d page. No plugins needed.
> >>>
> >>> Not sure what the Safari 10.1.2 issue could be. Sometimes you get
> >>> these no scene errors if the html/xml parser had problems, often
> with
> >>> self-closing tags which work only in xml. Did the inlined x3d used
> to
> >>> work in this version of Safari ? Other messages in the browser
> >>> console, or x3dom debug console (press 'd') ? Does the inline
> tutorial
> >>> example on x3dom.org <http://x3dom.org> work ?
> >>>
> >>> -Andreas
> >>>
> >>>> Date: Thu, 31 May 2018 11:19:44 -0300
> >>>> From: Fr?d?ric Vogt <frederic.vogt at alumni.anu.edu.au <mailto:
> frederic.vogt at alumni.anu.edu.au>>
> >>>> To: x3d-public at web3d.org <mailto:x3d-public at web3d.org>
> >>>> Subject: [x3d-public] Unexpected behaviour of X3D Validator
> >>>>
> >>>> Dear X3D gurus,
> >>>>
> >>>> I am once again trying to assemble an astro-X3D model using the
> mayavi python module, here:
> http://www.sc.eso.org/~fvogt/Shapley/shapley.html <
> http://www.sc.eso.org/~fvogt/Shapley/shapley.html>
> >>>>
> >>>> I am on Mac OS X Yosemite (still?), and can load the model just
> fine with:
> >>>>
> >>>> Firefox 60.0.1
> >>>> Chrome 67.0.3396.62
> >>>>
> >>>> However, the model does *not* load with:
> >>>>
> >>>> Safari 10.1.2 -> I get ?ERROR: no scene in resource? message in
> the development console right after it loads the underlying X3D file.
> >>>>
> >>>> I have confirmed that under Mac OS X High Sierra, with Safari
> 11.1, it does works, though!
> >>>>
> >>>> So I started worrying about possible formatting issues with my
> X3D file based on this old thread:
> https://sourceforge.net/p/x3dom/mailman/message/32684050/ <
> https://sourceforge.net/p/x3dom/mailman/message/32684050/>
> >>>> After all, I already know that the VTK X3D exporter is far from
> being perfect! http://fpavogt.github.io/x3d-pathway/dices.html <
> http://fpavogt.github.io/x3d-pathway/dices.html>
> >>>>
> >>>> Which brings me to my question about the X3D Validator, which I
> tried to use to check for any bug in my model:
> https://savage.nps.edu/X3dValidator <https://savage.nps.edu/X3dValidator>
> >>>>
> >>>> When I load a smaller version of the X3D model (I can?t load the
> actual model, likely too big), I get an error because of the lack of DTD
> info. But when I try to include
> >>>>
> >>>> <!DOCTYPE X3D PUBLIC "ISO//Web3D//DTD X3D 3.3//EN" "
> http://www.web3d.org/specifications/x3d-3.3.dtd">
> >>>> <X3D profile='Immersive' version='3.3'
> >>>> xmlns:xsd='http://www.w3.org/2001/XMLSchema-instance'
> >>>> xsd:noNamespaceSchemaLocation =?
> http://www.web3d.org/specifications/x3d-3.3.xsd '>
> >>>>
> >>>> then the X3D Validator returns me an empty page.
> >>>>
> >>>> What am I doing wrong here ?
> >>>>
> >>>> The file in question (if anyone is keen to try it out themselves)
> is: http://www.sc.eso.org/~fvogt/Shapley/shapley_small.x3d <
> http://www.sc.eso.org/~fvogt/Shapley/shapley_small.x3d>
> >>>>
> >>>> Cheers,
> >>>> Fr?d?ric
>
> all the best, Don
> --
> Don Brutzman Naval Postgraduate School, Code USW/Br
> brutzman at nps.edu
> Watkins 270, MOVES Institute, Monterey CA 93943-5000 USA +1.831.656.2149
> X3D graphics, virtual worlds, navy robotics
> http://faculty.nps.edu/brutzman
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://web3d.org/pipermail/x3d-public_web3d.org/attachments/20180601/1e1becb7/attachment-0001.html>
More information about the x3d-public
mailing list