[x3d-public] X3D DOCTYPE problem; X3D Validator has workaround, necessary path forward confirmed; fixed, upgrade path proposed
Don Brutzman
brutzman at nps.edu
Sun Sep 20 12:17:52 PDT 2020
On 9/19/2020 9:11 PM, John Carlson wrote:
>
> Changing everything to https:// (2 fields per file, I think) sounds like a lot of work. I have only changed DOCTYPE in my examples, and I did it all by hand. I cannot imagine doing it on 3000 or more examples, unless you can do it with X3DTidy or similar.
The model-update task isn't hard. I use Netbeans or UltraEdit for large global search/replace, then run full set of regression tests, then recommit updates to sourceforge. My old/second laptop stays busy...
When this change occurs, it will also take some time to ensure that all of the related X3D4 tools (Stylesheets, schematron, X3dTidy, X3DJSAIL Java, X3DPSAIL Python etc.) and documentation get updated too.
Thus won't make the change until we've had this kind of broader scrutiny, don't want to waste time on a sidetrack error. Carpenter's motto: "measure twice, cut once."
> I suggest you create a small script that will run on Windows Terminal (WSL, Ubuntu), Linux Bash, and MacOSX zsh for all those who have ANY http: marked content. This is Huge, and may cause significant downtime for many people.
>
> I am surprised you are not shouting LOUDER!
>
> John
Your concern is appreciated John. We ripple updates through the models tools and guidance, good things emerge.
We currently expect zero downtime for anyone going forward because both http and https are supported for these assets, retrieving the same files.
It was a blocker for several weeks, but several good lessons learned and potential improvements emerged.
Onward we go!
> On Sat, Sep 19, 2020 at 7:45 PM Don Brutzman <brutzman at nps.edu <mailto:brutzman at nps.edu>> wrote:
>
> Special thanks to Vince Marchetti who tracked down and fixed the unintended redirects from http to https that thwarted operation of regression testing and other validation tools.
>
> [1] X3D Specifications: Schema and DOCTYPE Validation
> https://www.web3d.org/specifications
>
> Full regression testing in progress for all models in X3D Example Archives, with all tests so far are passing as expected.
>
> [2] X3D Resources, Examples: Scene Archives for X3D
> https://www.web3d.org/x3d/content/examples/X3dResources.html#Examples
>
> Looking ahead: subject to X3D Working Group discussion and confirmation, I expect that all of our recommended "http" addresses for DOCTYPE and XML Schema should be upgraded to https.
>
> [3] X3D Scene Authoring Hints: Validation of X3D Scenes using DTD and XML Schema
> https://www.web3d.org/x3d/content/examples/X3dSceneAuthoringHints.html#Validation
>
> Please advise if anyone has concerns with this approach.
>
> Considering pros and cons: additional security for X3D4 models is no problem. Backwards compatibility for X3D3 XML models seems fully workable since both http-served and https-served copies of these files remain available. Improved file security (avoiding any possibility of Man In The Middle attacks on our URIs) is good.
>
> [4] Wikipedia: Man-in-the-middle attack
> https://en.wikipedia.org/wiki/Man-in-the-middle_attack
>
> Absent objections, I hope to accomplish this change from http to https for DOCTYPE and XML Schema to all published content this month.
>
>
> On 9/13/2020 4:34 PM, Don Brutzman wrote:
> > Again thanks for helpful diagnosis.
> >
> > I have been able to modify the X3D Validator so that if http:// is used for XML DOCTYPE or XML Schema, they get replaced with https:// instead before working on the full model for validation. Seems to work OK.
> >
> > [1] X3D Validator
> > https://savage.nps.edu/X3dValidator/validate
> > "The X3D Validator performs comprehensive Quality Assurance (QA) testing to ensure the validity of X3D3 and X3D4 models."
> >
> > Meanwhile despite work with XML Catalog and other techniques, am unable to get SAX-based Java tools to work with the redirect. This breaks the majority of our regression tests and conversion stylesheets.
> >
> > So one tool is fixed, and XML Spy was adept, but many others are now blocked from using X3D models.
> >
> > Therefore the immediate path forward is:
> >
> > a. Restore direct access to standardized-url DOCTYPE and schema. Fileset patterns are
> >
> > http://www.web3d.org/specifications/x3d-*.dtd
> > http://www.web3d.org/specifications/x3d-*.xsd
> >
> > and the long-term path forward is:
> >
> > b. X3D Working Group consider specification change to all relevant X3D XML Encoding DOCTYPE and Schema references to support https:// as well. Encouraging strong security seems like a good idea for all versions of X3D.
> >
> > Further trouble reports and feedback welcome. Again apologies for all lost development time, hopefully the server configuration for doctype and schema will be restored soon.
> >
> >
> > On 9/12/2020 7:34 PM, vmarchetti at kshell.com <mailto:vmarchetti at kshell.com> wrote:
> >>
> >>
> >> As observed at 10:20 PM EDT on 12 Sept 2020
> >>
> >> A HTTP request to https://www.web3d.org/specifications/x3d-3.3.dtd made with the curl tool (on Mac OS) returns the 142k DTD file, appears well formed
> >>
> >> The request to the http (unencrypted, discouraged by modern browsers) http://www.web3d.org/specifications/x3d-3.3.dtd returns a redirect HTTP response, with headers:
> >>
> >> HTTP/1.1 302 Found
> >> Date: Sun, 13 Sep 2020 02:15:33 GMT
> >> Server: Apache
> >> X-Content-Type-Options: nosniff
> >> Location: https://www.web3d.org/specifications/x3d-3.3.dtd
> >> Cache-Control: max-age=1209600
> >> Expires: Sun, 27 Sep 2020 02:15:33 GMT
> >> Content-Length: 232
> >> Content-Type: text/html; charset=iso-8859-1
> >>
> >>
> >> and the body of the HTTP response is an HTML-markup version of this information, so not well formed DTD.
> >>
> >> It appears that some of the validation tools will follow the redirect and some will not.
> >>
> >> I do not know if the server configuration has been changed recently to redirect requests to URLS in http://www.web3d.org/specifications/* space.
> >>
> >> Request for our webpages i.e. http://www.web3d.org/ have been redirected for some time.
> >>
> >> Solutions to this problem would be
> >>
> >> -- allow server to directly server requests to http://www.web3d.org/specifications/* without redirection
> >> -- update or configure tools to follow HTTP 302 redirections
> >>
> >> Vince Marchetti
> >>
> >>> On Sep 12, 2020, at 7:55 PM, Don Brutzman <brutzman at nps.edu <mailto:brutzman at nps.edu>> wrote:
> >>>
> >>> We noticed an emergent problem with X3D Validator about a week ago and, with difficulty, have been trying to sleuth what the root cause might be.
> >>>
> >>> After multiple diagnostic efforts, we now think that the problem is on the web3d.org <http://web3d.org> server, there is some issue within the DTD files themselves. Unfortunately there is also a problem with my account which currently prevents troubleshooting further and (hopefully) fixing it.
> >>>
> >>> This problem has blocked proper DTD validation, which might in turn block validation of numerous tools loading/validating X3D models. Have confirmed that some tools work (probably due to prior caching) and other tools fail. Example failure output:
> >>>
> >>>> check well-formedness
> >>>> http://www.web3d.org/specifications/x3d-3.3.dtd:1:3: The markup declarations contained or pointed to by the document type declaration must be well-formed.
> >>>> Could not validate document C:\x3d-code\www.web3d.org <http://www.web3d.org>\x3d\content\examples\X3dForWebAuthors\HelloWorld.x3d: org.xml.sax.SAXParseExceptionpublicId: ISO//Web3D//DTD X3D 3.3//EN; systemId: http://www.web3d.org/specifications/x3d-3.3.dtd; lineNumber: 1; columnNumber: 3; The markup declarations contained or pointed to by the document type declaration must be well-formed.
> >>>> C:\x3d-code\www.web3d.org <http://www.web3d.org>\x3d\content\examples\X3dForWebAuthors\HelloWorld.x3d is not a valid XML document
> >>>
> >>> Meanwhile XML Spy can validate the same model successfully. It is both exceptionally strict and also added native X3D support a few months back. So this approach confirms that X3D models we are testing remain unchanged and correct.
> >>>
> >>> Web3D Communications Team and webmaster have been notified about the access problem preventing further troubleshooting.
> >>>
> >>> The DOCTYPE problem does not appear to prevent successful X_ITE and X3DOM display of X3D models.
> >>>
> >>> * https://x3dgraphics.com/examples//X3dForWebAuthors/Chapter02GeometryPrimitives/BoxExampleX_ITE.html
> >>> * https://x3dgraphics.com/examples//X3dForWebAuthors/Chapter02GeometryPrimitives/BoxExampleX3dom.xhtml
> >>>
> >>> Once this is also sorted out, will post regarding resumption of capability. Apologies for any difficulties this problem causes tool developers and scene modelers.
> >>>
> >>> all the best, Don
> >>> --
> >>> Don Brutzman Naval Postgraduate School, Code USW/Br brutzman at nps.edu <mailto:brutzman at nps.edu>
> >>> Watkins 270, MOVES Institute, Monterey CA 93943-5000 USA +1.831.656.2149
> >>> X3D graphics, virtual worlds, navy robotics http://faculty.nps.edu/brutzman
> >>>
> >>> _______________________________________________
> >>> communications mailing list
> >>> communications at web3d.org <mailto:communications at web3d.org>
> >>> http://web3d.org/mailman/listinfo/communications_web3d.org
> >>
> >
> > all the best, Don
>
> all the best, Don
> --
> Don Brutzman Naval Postgraduate School, Code USW/Br brutzman at nps.edu <mailto:brutzman at nps.edu>
> Watkins 270, MOVES Institute, Monterey CA 93943-5000 USA +1.831.656.2149
> X3D graphics, virtual worlds, navy robotics http://faculty.nps.edu/brutzman
>
> _______________________________________________
> x3d-public mailing list
> x3d-public at web3d.org <mailto:x3d-public at web3d.org>
> http://web3d.org/mailman/listinfo/x3d-public_web3d.org
>
all the best, Don
--
Don Brutzman Naval Postgraduate School, Code USW/Br brutzman at nps.edu
Watkins 270, MOVES Institute, Monterey CA 93943-5000 USA +1.831.656.2149
X3D graphics, virtual worlds, navy robotics http://faculty.nps.edu/brutzman
More information about the x3d-public
mailing list