[x3d-public] X3D DOCTYPE problem; X3D Validator has workaround, necessary path forward confirmed; fixed, upgrade path proposed

Sat Sep 19 17:44:03 PDT 2020

Special thanks to Vince Marchetti who tracked down and fixed the unintended redirects from http to https that thwarted operation of regression testing and other validation tools.

[1] X3D Specifications: Schema and DOCTYPE Validation
     https://www.web3d.org/specifications

Full regression testing in progress for all models in X3D Example Archives, with all tests so far are passing as expected.

[2] X3D Resources, Examples: Scene Archives for X3D
     https://www.web3d.org/x3d/content/examples/X3dResources.html#Examples

Looking ahead: subject to X3D Working Group discussion and confirmation, I expect that all of our recommended "http" addresses for DOCTYPE and XML Schema should be upgraded to https.

[3] X3D Scene Authoring Hints: Validation of X3D Scenes using DTD and XML Schema
     https://www.web3d.org/x3d/content/examples/X3dSceneAuthoringHints.html#Validation

Please advise if anyone has concerns with this approach.

Considering pros and cons:  additional security for X3D4 models is no problem.  Backwards compatibility for X3D3 XML models seems fully workable since both http-served and https-served copies of these files remain available.  Improved file security (avoiding any possibility of Man In The Middle attacks on our URIs) is good.

[4] Wikipedia: Man-in-the-middle attack
     https://en.wikipedia.org/wiki/Man-in-the-middle_attack

Absent objections, I hope to accomplish this change from http to https for DOCTYPE and XML Schema to all published content this month.

On 9/13/2020 4:34 PM, Don Brutzman wrote:
> Again thanks for helpful diagnosis.
> 
> I have been able to modify the X3D Validator so that if http:// is used for XML DOCTYPE or XML Schema, they get replaced with https:// instead before working on the full model for validation.  Seems to work OK.
> 
> [1] X3D Validator
>      https://savage.nps.edu/X3dValidator/validate
>      "The X3D Validator performs comprehensive Quality Assurance (QA) testing to ensure the validity of X3D3 and X3D4 models."
> 
> Meanwhile despite work with XML Catalog and other techniques, am unable to get SAX-based Java tools to work with the redirect.  This breaks the majority of our regression tests and conversion stylesheets.
> 
> So one tool is fixed, and XML Spy was adept, but many others are now blocked from using X3D models.
> 
> Therefore the immediate path forward is:
> 
> a. Restore direct access to standardized-url DOCTYPE and schema.  Fileset patterns are
> 
>      http://www.web3d.org/specifications/x3d-*.dtd
>      http://www.web3d.org/specifications/x3d-*.xsd
> 
> and the long-term path forward is:
> 
> b. X3D Working Group consider specification change to all relevant X3D XML Encoding DOCTYPE and Schema references to support https:// as well.  Encouraging strong security seems like a good idea for all versions of X3D.
> 
> Further trouble reports and feedback welcome.  Again apologies for all lost development time, hopefully the server configuration for doctype and schema will be restored soon.
> 
> 
> On 9/12/2020 7:34 PM, vmarchetti at kshell.com wrote:
>>
>>
>> As observed at 10:20 PM EDT on 12 Sept 2020
>>
>> A HTTP request to https://www.web3d.org/specifications/x3d-3.3.dtd made with the curl tool (on Mac OS) returns the 142k DTD file, appears well formed
>>
>> The request to the http (unencrypted, discouraged by modern browsers) http://www.web3d.org/specifications/x3d-3.3.dtd returns a redirect HTTP response, with headers:
>>
>> HTTP/1.1 302 Found
>> Date: Sun, 13 Sep 2020 02:15:33 GMT
>> Server: Apache
>> X-Content-Type-Options: nosniff
>> Location: https://www.web3d.org/specifications/x3d-3.3.dtd
>> Cache-Control: max-age=1209600
>> Expires: Sun, 27 Sep 2020 02:15:33 GMT
>> Content-Length: 232
>> Content-Type: text/html; charset=iso-8859-1
>>
>>
>> and the body of the HTTP response is an HTML-markup version of this information, so not well formed DTD.
>>
>> It appears that some of the validation tools will follow the redirect and some will not.
>>
>> I do not know if the server configuration has been changed recently to redirect requests to URLS  in http://www.web3d.org/specifications/* space.
>>
>> Request for our webpages i.e. http://www.web3d.org/ have been redirected for some time.
>>
>> Solutions to this problem would be
>>
>> -- allow server to directly server requests to http://www.web3d.org/specifications/* without redirection
>> -- update or configure tools to follow HTTP 302 redirections
>>
>> Vince Marchetti
>>
>>> On Sep 12, 2020, at 7:55 PM, Don Brutzman <brutzman at nps.edu> wrote:
>>>
>>> We noticed an emergent problem with X3D Validator about a week ago and, with difficulty, have been trying to sleuth what the root cause might be.
>>>
>>> After multiple diagnostic efforts, we now think that the problem is on the web3d.org server, there is some issue within the DTD files themselves.  Unfortunately there is also a problem with my account which currently prevents troubleshooting further and (hopefully) fixing it.
>>>
>>> This problem has blocked proper DTD validation, which might in turn block validation of numerous tools loading/validating X3D models.  Have confirmed that some tools work (probably due to prior caching) and other tools fail. Example failure output:
>>>
>>>> check well-formedness
>>>> http://www.web3d.org/specifications/x3d-3.3.dtd:1:3: The markup declarations contained or pointed to by the document type declaration must be well-formed.
>>>> Could not validate document C:\x3d-code\www.web3d.org\x3d\content\examples\X3dForWebAuthors\HelloWorld.x3d: org.xml.sax.SAXParseExceptionpublicId: ISO//Web3D//DTD X3D 3.3//EN; systemId: http://www.web3d.org/specifications/x3d-3.3.dtd; lineNumber: 1; columnNumber: 3; The markup declarations contained or pointed to by the document type declaration must be well-formed.
>>>> C:\x3d-code\www.web3d.org\x3d\content\examples\X3dForWebAuthors\HelloWorld.x3d is not a valid XML document
>>>
>>> Meanwhile XML Spy can validate the same model successfully.  It is both exceptionally strict and also added native X3D support a few months back.  So this approach confirms that X3D models we are testing remain unchanged and correct.
>>>
>>> Web3D Communications Team and webmaster have been notified about the access problem preventing further troubleshooting.
>>>
>>> The DOCTYPE problem does not appear to prevent successful X_ITE and X3DOM display of X3D models.
>>>
>>> * https://x3dgraphics.com/examples//X3dForWebAuthors/Chapter02GeometryPrimitives/BoxExampleX_ITE.html
>>> * https://x3dgraphics.com/examples//X3dForWebAuthors/Chapter02GeometryPrimitives/BoxExampleX3dom.xhtml
>>>
>>> Once this is also sorted out, will post regarding resumption of capability.  Apologies for any difficulties this problem causes tool developers and scene modelers.
>>>
>>> all the best, Don
>>> -- 
>>> Don Brutzman  Naval Postgraduate School, Code USW/Br       brutzman at nps.edu
>>> Watkins 270,  MOVES Institute, Monterey CA 93943-5000 USA   +1.831.656.2149
>>> X3D graphics, virtual worlds, navy robotics http://faculty.nps.edu/brutzman
>>>
>>> _______________________________________________
>>> communications mailing list
>>> communications at web3d.org
>>> http://web3d.org/mailman/listinfo/communications_web3d.org
>>
> 
> all the best, Don

all the best, Don
-- 
Don Brutzman  Naval Postgraduate School, Code USW/Br       brutzman at nps.edu
Watkins 270,  MOVES Institute, Monterey CA 93943-5000 USA   +1.831.656.2149
X3D graphics, virtual worlds, navy robotics http://faculty.nps.edu/brutzman