[x3d-public] X3D DOCTYPE problem; X3D Validator has workaround, necessary path forward confirmed; fixed, upgrade path proposed
John Carlson
yottzumm at gmail.com
Sun Sep 20 23:42:44 PDT 2020
Oh good, it sounds like you’ve got backwards compatibility in place so
people can migrate over time.
On Sun, Sep 20, 2020 at 2:17 PM Don Brutzman <brutzman at nps.edu> wrote:
> On 9/19/2020 9:11 PM, John Carlson wrote:
>
> >
>
> > Changing everything to https:// (2 fields per file, I think) sounds
> like a lot of work. I have only changed DOCTYPE in my examples, and I did
> it all by hand. I cannot imagine doing it on 3000 or more examples, unless
> you can do it with X3DTidy or similar.
>
>
>
> The model-update task isn't hard. I use Netbeans or UltraEdit for large
> global search/replace, then run full set of regression tests, then recommit
> updates to sourceforge. My old/second laptop stays busy...
>
>
>
> When this change occurs, it will also take some time to ensure that all of
> the related X3D4 tools (Stylesheets, schematron, X3dTidy, X3DJSAIL Java,
> X3DPSAIL Python etc.) and documentation get updated too.
>
>
>
> Thus won't make the change until we've had this kind of broader scrutiny,
> don't want to waste time on a sidetrack error. Carpenter's motto: "measure
> twice, cut once."
>
>
>
> > I suggest you create a small script that will run on Windows Terminal
> (WSL, Ubuntu), Linux Bash, and MacOSX zsh for all those who have ANY http:
> marked content. This is Huge, and may cause significant downtime for many
> people.
>
> >
>
> > I am surprised you are not shouting LOUDER!
>
> >
>
> > John
>
>
>
> Your concern is appreciated John. We ripple updates through the models
> tools and guidance, good things emerge.
>
>
>
> We currently expect zero downtime for anyone going forward because both
> http and https are supported for these assets, retrieving the same files.
>
>
>
> It was a blocker for several weeks, but several good lessons learned and
> potential improvements emerged.
>
>
>
> Onward we go!
>
>
>
>
>
> > On Sat, Sep 19, 2020 at 7:45 PM Don Brutzman <brutzman at nps.edu <mailto:
> brutzman at nps.edu>> wrote:
>
> >
>
> > Special thanks to Vince Marchetti who tracked down and fixed the
> unintended redirects from http to https that thwarted operation of
> regression testing and other validation tools.
>
> >
>
> > [1] X3D Specifications: Schema and DOCTYPE Validation
>
> > https://www.web3d.org/specifications
>
> >
>
> > Full regression testing in progress for all models in X3D Example
> Archives, with all tests so far are passing as expected.
>
> >
>
> > [2] X3D Resources, Examples: Scene Archives for X3D
>
> >
> https://www.web3d.org/x3d/content/examples/X3dResources.html#Examples
>
> >
>
> > Looking ahead: subject to X3D Working Group discussion and
> confirmation, I expect that all of our recommended "http" addresses for
> DOCTYPE and XML Schema should be upgraded to https.
>
> >
>
> > [3] X3D Scene Authoring Hints: Validation of X3D Scenes using DTD
> and XML Schema
>
> >
> https://www.web3d.org/x3d/content/examples/X3dSceneAuthoringHints.html#Validation
>
> >
>
> > Please advise if anyone has concerns with this approach.
>
> >
>
> > Considering pros and cons: additional security for X3D4 models is
> no problem. Backwards compatibility for X3D3 XML models seems fully
> workable since both http-served and https-served copies of these files
> remain available. Improved file security (avoiding any possibility of Man
> In The Middle attacks on our URIs) is good.
>
> >
>
> > [4] Wikipedia: Man-in-the-middle attack
>
> > https://en.wikipedia.org/wiki/Man-in-the-middle_attack
>
> >
>
> > Absent objections, I hope to accomplish this change from http to
> https for DOCTYPE and XML Schema to all published content this month.
>
> >
>
> >
>
> > On 9/13/2020 4:34 PM, Don Brutzman wrote:
>
> > > Again thanks for helpful diagnosis.
>
> > >
>
> > > I have been able to modify the X3D Validator so that if http://
> is used for XML DOCTYPE or XML Schema, they get replaced with https://
> instead before working on the full model for validation. Seems to work OK.
>
> > >
>
> > > [1] X3D Validator
>
> > > https://savage.nps.edu/X3dValidator/validate
>
> > > "The X3D Validator performs comprehensive Quality Assurance
> (QA) testing to ensure the validity of X3D3 and X3D4 models."
>
> > >
>
> > > Meanwhile despite work with XML Catalog and other techniques, am
> unable to get SAX-based Java tools to work with the redirect. This breaks
> the majority of our regression tests and conversion stylesheets.
>
> > >
>
> > > So one tool is fixed, and XML Spy was adept, but many others are
> now blocked from using X3D models.
>
> > >
>
> > > Therefore the immediate path forward is:
>
> > >
>
> > > a. Restore direct access to standardized-url DOCTYPE and schema.
> Fileset patterns are
>
> > >
>
> > > http://www.web3d.org/specifications/x3d-*.dtd
>
> > > http://www.web3d.org/specifications/x3d-*.xsd
>
> > >
>
> > > and the long-term path forward is:
>
> > >
>
> > > b. X3D Working Group consider specification change to all
> relevant X3D XML Encoding DOCTYPE and Schema references to support https://
> as well. Encouraging strong security seems like a good idea for all
> versions of X3D.
>
> > >
>
> > > Further trouble reports and feedback welcome. Again apologies
> for all lost development time, hopefully the server configuration for
> doctype and schema will be restored soon.
>
> > >
>
> > >
>
> > > On 9/12/2020 7:34 PM, vmarchetti at kshell.com <mailto:
> vmarchetti at kshell.com> wrote:
>
> > >>
>
> > >>
>
> > >> As observed at 10:20 PM EDT on 12 Sept 2020
>
> > >>
>
> > >> A HTTP request to
> https://www.web3d.org/specifications/x3d-3.3.dtd made with the curl tool
> (on Mac OS) returns the 142k DTD file, appears well formed
>
> > >>
>
> > >> The request to the http (unencrypted, discouraged by modern
> browsers) http://www.web3d.org/specifications/x3d-3.3.dtd returns a
> redirect HTTP response, with headers:
>
> > >>
>
> > >> HTTP/1.1 302 Found
>
> > >> Date: Sun, 13 Sep 2020 02:15:33 GMT
>
> > >> Server: Apache
>
> > >> X-Content-Type-Options: nosniff
>
> > >> Location: https://www.web3d.org/specifications/x3d-3.3.dtd
>
> > >> Cache-Control: max-age=1209600
>
> > >> Expires: Sun, 27 Sep 2020 02:15:33 GMT
>
> > >> Content-Length: 232
>
> > >> Content-Type: text/html; charset=iso-8859-1
>
> > >>
>
> > >>
>
> > >> and the body of the HTTP response is an HTML-markup version of
> this information, so not well formed DTD.
>
> > >>
>
> > >> It appears that some of the validation tools will follow the
> redirect and some will not.
>
> > >>
>
> > >> I do not know if the server configuration has been changed
> recently to redirect requests to URLS in
> http://www.web3d.org/specifications/* space.
>
> > >>
>
> > >> Request for our webpages i.e. http://www.web3d.org/ have been
> redirected for some time.
>
> > >>
>
> > >> Solutions to this problem would be
>
> > >>
>
> > >> -- allow server to directly server requests to
> http://www.web3d.org/specifications/* without redirection
>
> > >> -- update or configure tools to follow HTTP 302 redirections
>
> > >>
>
> > >> Vince Marchetti
>
> > >>
>
> > >>> On Sep 12, 2020, at 7:55 PM, Don Brutzman <brutzman at nps.edu
> <mailto:brutzman at nps.edu>> wrote:
>
> > >>>
>
> > >>> We noticed an emergent problem with X3D Validator about a week
> ago and, with difficulty, have been trying to sleuth what the root cause
> might be.
>
> > >>>
>
> > >>> After multiple diagnostic efforts, we now think that the
> problem is on the web3d.org <http://web3d.org> server, there is some
> issue within the DTD files themselves. Unfortunately there is also a
> problem with my account which currently prevents troubleshooting further
> and (hopefully) fixing it.
>
> > >>>
>
> > >>> This problem has blocked proper DTD validation, which might in
> turn block validation of numerous tools loading/validating X3D models.
> Have confirmed that some tools work (probably due to prior caching) and
> other tools fail. Example failure output:
>
> > >>>
>
> > >>>> check well-formedness
>
> > >>>> http://www.web3d.org/specifications/x3d-3.3.dtd:1:3: The
> markup declarations contained or pointed to by the document type
> declaration must be well-formed.
>
> > >>>> Could not validate document C:\x3d-code\www.web3d.org <
> http://www.web3d.org>\x3d\content\examples\X3dForWebAuthors\HelloWorld.x3d:
> org.xml.sax.SAXParseExceptionpublicId: ISO//Web3D//DTD X3D 3.3//EN;
> systemId: http://www.web3d.org/specifications/x3d-3.3.dtd; lineNumber: 1;
> columnNumber: 3; The markup declarations contained or pointed to by the
> document type declaration must be well-formed.
>
> > >>>> C:\x3d-code\www.web3d.org <http://www.web3d.org>\x3d\content\examples\X3dForWebAuthors\HelloWorld.x3d
> is not a valid XML document
>
> > >>>
>
> > >>> Meanwhile XML Spy can validate the same model successfully. It
> is both exceptionally strict and also added native X3D support a few months
> back. So this approach confirms that X3D models we are testing remain
> unchanged and correct.
>
> > >>>
>
> > >>> Web3D Communications Team and webmaster have been notified
> about the access problem preventing further troubleshooting.
>
> > >>>
>
> > >>> The DOCTYPE problem does not appear to prevent successful X_ITE
> and X3DOM display of X3D models.
>
> > >>>
>
> > >>> *
> https://x3dgraphics.com/examples//X3dForWebAuthors/Chapter02GeometryPrimitives/BoxExampleX_ITE.html
>
> > >>> *
> https://x3dgraphics.com/examples//X3dForWebAuthors/Chapter02GeometryPrimitives/BoxExampleX3dom.xhtml
>
> > >>>
>
> > >>> Once this is also sorted out, will post regarding resumption of
> capability. Apologies for any difficulties this problem causes tool
> developers and scene modelers.
>
> > >>>
>
> > >>> all the best, Don
>
> > >>> --
>
> > >>> Don Brutzman Naval Postgraduate School, Code USW/Br
> brutzman at nps.edu <mailto:brutzman at nps.edu>
>
> > >>> Watkins 270, MOVES Institute, Monterey CA 93943-5000 USA
> +1.831.656.2149
>
> > >>> X3D graphics, virtual worlds, navy robotics
> http://faculty.nps.edu/brutzman
>
> > >>>
>
> > >>> _______________________________________________
>
> > >>> communications mailing list
>
> > >>> communications at web3d.org <mailto:communications at web3d.org>
>
> > >>> http://web3d.org/mailman/listinfo/communications_web3d.org
>
> > >>
>
> > >
>
> > > all the best, Don
>
> >
>
> > all the best, Don
>
> > --
>
> > Don Brutzman Naval Postgraduate School, Code USW/Br
> brutzman at nps.edu <mailto:brutzman at nps.edu>
>
> > Watkins 270, MOVES Institute, Monterey CA 93943-5000 USA
> +1.831.656.2149
>
> > X3D graphics, virtual worlds, navy robotics
> http://faculty.nps.edu/brutzman
>
> >
>
> > _______________________________________________
>
> > x3d-public mailing list
>
> > x3d-public at web3d.org <mailto:x3d-public at web3d.org>
>
> > http://web3d.org/mailman/listinfo/x3d-public_web3d.org
>
> >
>
>
>
> all the best, Don
>
> --
>
> Don Brutzman Naval Postgraduate School, Code USW/Br
> brutzman at nps.edu
>
> Watkins 270, MOVES Institute, Monterey CA 93943-5000 USA +1.831.656.2149
>
> X3D graphics, virtual worlds, navy robotics
> http://faculty.nps.edu/brutzman
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://web3d.org/pipermail/x3d-public_web3d.org/attachments/20200921/a665f507/attachment-0001.html>
More information about the x3d-public
mailing list