[x3d-public] jupyter notebook; X3DOM syntax for including X3D within HTML

Brutzman, Donald (Don) (CIV) brutzman at nps.edu
Sun Jun 2 21:18:24 PDT 2019

Thanks for in-depth study analysis and implementation efforts Andreas (and Holger).

Your getting proper capitalization to be accepted is important since that was the major munging going on that made content into invalid XML.

Avoiding singleton self-closing tags such as <Material/> by using <Material></Material> is something we can live with since the resulting X3D content is still valid XML.  Also I believe that it is the same DOM.

Wondering if there is a path to getting X3D handled similarly to SVG or MathML (perhaps as a Foreign Element) so that singleton elements are allowed and self-closing flags are defined?

It is curious that HTML syntax allows closing elements to be omitted, but apparently doesn't handle a singleton element as equivalent to an open element that is not followed by a closing element.  It seems like a flaw or omission in HTML5 if XHTML content can't be read as HTML.

(Also wondering about a special case, if the X3D <meta/> tag would seem to work according to the HTML parsing rules since it matches the HTML5 meta element.)

If no path is available, then this might be feedback that we provide to W3C HTML5 working group, since it seems like HTML parsers are being over-assertive.  The note about SGML seems to be overshadowing XML.  Perhaps we should comment in the forum you indicate below.

Even simpler HTML5 spec strategy would be to propose that X3D be treated as a Foreign Element similar to SVG and MathML.

On 6/2/2019 12:52 PM, Andreas Plesch wrote:
> For anybody who is interest in how HTML parsing is defined, here are
> pointers to the relevant portions in the HTML5 spec.:
> The optional tags section here is normative:
> https://www.w3.org/TR/html5/syntax.html#optional-start-and-end-tags
> I can find it right now but HTML treats the /> sequence by ignoring
> the / and expecting no end tag: so <input /> is equivalent to <input>
> https://www.w3.org/TR/html5/syntax.html#parsing-html-documents
> - points out that HTML became separated from XML and has its own rules
> - during parsing there is a stack of open elements:
> https://www.w3.org/TR/html5/syntax.html#the-stack-of-open-elements
> Elements are grouped into special, formatting and ordinary. SVG is
> special, X3D elements are ordinary (anything else).
> - SVG and MathML are treated explicitly throughout
> There are adjustment tables for case in SVG attributes.
> - HTML have a self-closing flag which is checked during parsing.
> The algorithm explicitly acknowledges a set self-closing flag to be ok
> for these elements:
> https://www.w3.org/TR/html5/syntax.html#the-in-head-insertion-mode :
> start tag whose tag name is one of: "base", "basefont", "bgsound", "link"
> start tag whose tag name is "meta"
> https://www.w3.org/TR/html5/syntax.html#the-after-head-insertion-mode
> start tag whose tag name is one of: "area", "br", "embed", "img", "wbr"
> start tag whose tag name is "input"
> start tag whose tag name is one of: "param", "source", "track"
> start tag whose tag name is "hr"
> start tag whose tag name is "math"
> start tag whose tag name is "svg"
> https://www.w3.org/TR/html5/syntax.html#the-in-column-group-insertion-mode
> start tag whose tag name is "col"
> https://www.w3.org/TR/html5/syntax.html#the-in-frameset-insertion-mode
> start tag whose tag name is "frame"
> https://www.w3.org/TR/html5/syntax.html#the-rules-for-parsing-tokens-in-foreign-content
> Any start tag other than a start tag whose tag name is one of: "b",
> "big", "blockquote", "body", "br", "center", "code", "dd", "div",
> "dl", "dt", "em", "embed", "h1", "h2", "h3", "h4", "h5", "h6", "head",
> "hr", "i", "img", "li", "listing", "meta", "nobr", "ol", "p", "pre",
> "ruby", "s", "small", "span", "strong", "strike", "sub", "sup",
> "table", "tt", "u", "ul", "var" or A start tag whose tag name is
> "font", if the token has any attributes named "color", "face", or
> "size"
> Take home, I think, is that HTML parsing only allows singletons in
> special cases.
> -Andreas
> On Sun, Jun 2, 2019 at 1:47 PM Andreas Plesch <andreasplesch at gmail.com> wrote:
>> Hi Don,
>> let me see what I can respond to right away.
>> On Sun, Jun 2, 2019, 12:23 AM Brutzman, Donald (Don) (CIV)
>> <brutzman at nps.edu wrote:
>>> Thanks for continuing scrutiny, this issue is quite important for X3Dv4 specification efforts.
>>> I think an earlier version of X_ITE used regular X3D elements without problem - not sure, that was some time ago.  Hard to tell now since the online .html examples all seem to use <X3DCanvas src="somesuch"><!-- fallback --></X3DCanvas>
>>> I spot-checked a few of the examples, am seeing same X3DCanvas pattern at
>>> http://create3000.de/x_ite/getting-started/#x3d-examples
>> X_ITE went back to using a X3DCanvas element because the X3D tag is
>> already taken by the X3D document and because it conveys that it is
>> derived from the HTML5 canvas element, and can be styled in the same
>> way.
>>> Perhaps X3DOM is avoiding direct inclusion and parsing of .x3d source for same reason?
>> No, it uses the browser parsing mechanism because that allows external
>> access to the scene as part of the DOM.
>>> Looks like you've worked on this in X_ITE already Andreas
>>> * http://create3000.de/x_ite/getting-started/#xhtml-dom-integration
>>> * http://create3000.de/x_ite/xhtml-dom-integration/
>>> * http://create3000.de/x_ite/xhtml-dom-integration/#example
>>> * http://media.create3000.de/create3000/dom-integration/dom.integration.xhtml
>>> excerpt:
>>>          <body>
>>>                  <X3DCanvas class="browser" timings="false">
>>>                          <img class="fallback" src="XHTML.png"/>
>>>                          <X3D xmlns="http://www.web3d.org/specifications/x3d-namespace" id="spheres">
>>>                                  <Scene DEF='scene'>
>>> Wondering if you had tried a .html version of that preceding file, and what any difficulties might have been?
>> Yes, Holger and me added a HTML compatibility layer to X_ITE. Let me
>> find an example:
>> https://andreasplesch.github.io/x_ite_dom/tests/html5/html5.html
>> The compatibility layer only addresses capitalization, but not self
>> closing tags. So html with self-closing X3D tags also does not work in
>> the X_ITE dom integration.
>>> Wondering why you have a DEF on the <Scene> ?  If applying HTML events, presumably it could be an id="scene" reference instead.
>> Yeah, I do not know. I do not think it is necessary.
>>> SVG recommendation allows singleton elements in fragments within HTML, also seems to sometimes use namespaces and sometimes not.  Also SVG appears to have at least 2 lowerCamelCase elements, clipPath and foreignObject, plus a large number of lowerCamelCase attributes - so not every element/attribute has to be all lower case.
>>>          Scalable Vector Graphics (SVG) 2
>>>          W3C Candidate Recommendation 04 October 2018
>>>          Chapter 5: Document Structure
>>>          https://www.w3.org/TR/SVG2/struct.html
>> I think for HTML lowercase is not required, only a defacto standard in
>> the markup. The DOM standard element.nodeName property always
>> normalizes the element name to UPPERCASE.
>> Singletons are ok for certain elements, but not for the X3D elements
>> since they are Unknown.
>>> Also wondering if you folks think that the <X3DCanvas src=""/> pattern from X_ITE is something we should repeat for X3DOM.  Seems like a good fallback that supports multiple X3D encodings.
>> If there is a way to preserve backward compatibility that may be
>> doable. I think it should already be possible in x3dom to wrap the X3D
>> element in a noop X3DCanvas element.
>> But it will be harder to preserve the styling options. In x3dom to the
>> idea was to follow the SVG element, in x_ite the idea is to follow
>> more the old plugin <object> element, with SAI access.
>>> I recommend that we try to pursue every possibility that might allow inclusion of .x3d (XML valid) models "as is" within the HTML page.  Certainly the fact that they work when relabeled as an XHTML page shows that it is possible.  SVG also provides corrolary evidence that it is possible.  If we force a special conversion of all X3D models used in HTML, then
>>> - that has a huge impact on the number of just-slightly-different scenes that get created,
>>> - it greatly complicates validation, and
>>> - it also is a huge negative when trying to edit/test/apply improvements to a model when the deployed version has unnecessarily different syntax from the original version.
>> The issue is that 'relabeling' may not the correct term here. xhtml is
>> a different encoding and is treated very differently by browsers.
>> Inclusion of XML within HTML is only possible with workarounds, eg.
>> wrapping in a script tag, or from a file. The proper way to include
>> XML is to use xhtml. Since the goal is to use html, we would have to
>> consider non-XML for X3D.
>> SVG is special. Let's see:
>> https://www.w3.org/TR/html5/syntax.html#foreign-elements
>> In html, svg namespace elements are Foreign Elements.
>> And Foreign Elements can have self-closing tags:
>> https://www.w3.org/TR/html5/syntax.html#start-tags , item 6).
>> When or if X3D elements are also designated Foreign Elements in HTML,
>> they could become self-closing. But until then, they are Unknown
>> Elements in HTML5 and need a start and end tag.
>> Oh, this is actually still being discussed in the context of custom
>> elements for web components:
>> https://github.com/w3c/webcomponents/issues/624
>> The summary of this discussion is while there is a demand for
>> self-closing tags for custom elements, the changes to the html parsing
>> algorithm are considered too invasive for the benefit.
>> In a way, an x3d html encoding would be parallel to the x3d json or
>> java encoding. It just should be simpler to define based on the XML
>> encoding. Starting with something like XML but case-insensitive and
>> with explicit closing tags rather than self-closing tags.
>>> As far as jupyter goes, since that notebook supports python and a wide variety of embedded languages I expect that we will be able to align satisfactorily as this work proceeds - you've made great progress already.
>> Yes, with the goal in mind in mind to make the python encoding easy to
>> use for jupyter. Perhaps a separate conda/pip python package which
>> depends on a future x3dpsail and provides .toStringX3Dhtml(),
>> toFileX3Dhtml
>> -Andreas
>>> On 5/31/2019 5:56 PM, Andreas Plesch wrote:
>>>> Well, if it was xhtml we were talking about you would be correct.
>>>> But, it is html without the x which is expected by the HTML renderer
>>>> in Jupyter. There is no XHTML renderer since the Jupyter notebook
>>>> itself is a HTML document.
>>>> And html allows singleton tags only for certain tags, at least in how
>>>> all browser parse the markup, and probably also as defined in the
>>>> HTML5 spec. These certain tags may include all known HTML elements
>>>> which do not use their text content. Unfortunately, the X3D tags are
>>>> considered Unknown Elements by html and they seem to require the
>>>> explicit end tag.
>>>> x3dom does not use any HTML parsing libraries but completely relies on
>>>> the web browser to do the parsing. And the browsers get confused,
>>>> because they do not understand the singleton x3d tags in a html
>>>> document.
>>>> Alas, if the xml serializing library in X3DJSAIL does not have an
>>>> option to generate full tags (they would be still legal XML, no?), we
>>>> would need to settle for xhtml IFrames which can also be included in a
>>>> Jupyter notebook (but not tested yet), or, as another option, wrap the
>>>> xml X3D into an Inline since x3dom always uses the browser XML parser
>>>> for Inlines. Would x3d.toStringX3DInline(filename) be considered too
>>>> extravagant ? It would generate an xml X3D document string with a
>>>> single Inline node, and also the content X3D in the file filename.
>>>> x3d.toFileX3DInline(x3dFileName, inlineFileName) would generate two
>>>> files.
>>>> Perhaps Python has an option to serialize out the full tags, perhaps
>>>> as HTMLFragments. Then a x3d.toStringFullTags() could be included
>>>> outside of X3DJSAIL, and with X3DPSAIL.
>>>> To consider X_ITE and x3dom together it would be first necessary get
>>>> X_ITE to work at all with Jupyter locally generated content. It may be
>>>> possible since X_ITE works with remote URLs in Jupyter.
>>>> -Andreas
>>>> ---on the phone---
>>>> On Fri, May 31, 2019, 5:14 PM <x3d-public-request at web3d.org wrote:
>>>>> Send x3d-public mailing list submissions to
>>>>>           x3d-public at web3d.org
>>>>> To subscribe or unsubscribe via the World Wide Web, visit
>>>>>           http://web3d.org/mailman/listinfo/x3d-public_web3d.org
>>>>> or, via email, send a message with subject or body 'help' to
>>>>>           x3d-public-request at web3d.org
>>>>> You can reach the person managing the list at
>>>>>           x3d-public-owner at web3d.org
>>>>> When replying, please edit your Subject line so it is more specific
>>>>> than "Re: Contents of x3d-public digest..."
>>>>> Today's Topics:
>>>>>      1. Re: jupyter notebook; X3DOM syntax for including X3D within
>>>>>         HTML/XHTML (Brutzman, Donald (Don) (CIV))
>>>>> ----------------------------------------------------------------------
>>>>> Message: 1
>>>>> Date: Fri, 31 May 2019 10:14:05 +0000
>>>>> From: "Brutzman, Donald (Don) (CIV)" <brutzman at nps.edu>
>>>>> To: "x3d-public at web3d.org" <x3d-public at web3d.org>
>>>>> Subject: Re: [x3d-public] jupyter notebook; X3DOM syntax for including
>>>>>           X3D within HTML/XHTML
>>>>> Message-ID: <b6c49fa9-6afc-34fe-ee57-59f0ad5fc99d at nps.edu>
>>>>> Content-Type: text/plain; charset="utf-8"
>>>>> thanks for great progress.
>>>>> On 5/28/2019 12:12 PM, Andreas Plesch wrote:
>>>>>> - x3dom has a problem with generated X3D outside of an Inline because
>>>>>> the serialized XML output uses self-closing tags such as <Material />
>>>>>> which is not HTML compatible.
>>>>> HTML allows singleton elements, i.e. self-closing tags.
>>>>> Similarly, element CamelCaseElementNames are legal, so we should not have to rename <Material/> to <material></material> simply because of similarity to another language's naming patterns.
>>>>> i think root of this problem likely lies in how X3DOM uses the accompanying DOM libraries - not sure of those details.
>>>>> Note that I got *all* of the X_ITE and X3DOM examples to work with HTML by assigning scenes to .html and .xhtml respectively.
>>>>> http://www.web3d.org/x3d/content/examples/X3dResources.html#Examples
>>>>> http://x3dgraphics.com/examples/X3dForWebAuthors/Chapter01TechnicalOverview/HelloWorldIndex.html
>>>>> http://x3dgraphics.com/examples/X3dForWebAuthors/Chapter01TechnicalOverview/HelloWorldX3dom.xhtml
>>>>> http://x3dgraphics.com/examples/X3dForWebAuthors/Chapter01TechnicalOverview/HelloWorldX_ITE.html
>>>>>> So the suggestion would be to provide a method, say
>>>>>> x3d.toStringX3DClosingTags(), which generates explicit closing tags:
>>>>>> <Material></Material>
>>>>> I don't think it is good to propagate a side-effect fix (which encourages mysterious improper practices and hides path towards fixes).
>>>>> Rather it is more desirable to isolate and fix the actual problem.  If someone can figure out why X3DOM runs differently when included within .xhtml and .html pages, that may reveal whatever idiosyncrasy is actually occurring.
>>>>> Finally we want to match the same patterns in X_ITE and X3DOM so that choice of player is independent of source-model syntax when using X3D version 4.  So this is a worthy mismatch to figure out and fix.  Hopefully we can get closer to sorting things out fully.
>>>>> - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
>>>>> Meanwhile: found and fixed several errors in online X3DOM .xhtml examples:
>>>>> - corrected url to match changed release address:
>>>>>      https://x3dom.org/release/x3dom.css
>>>>>      https://x3dom.org/release/x3dom-full.js
>>>>> - fixed bug in X3dToX3dom.xslt code logic in order to include function toggleFullscreen ()
>>>>> - using XHTML Transitional (vice Strict) DTD in order for anchor target attribute to pass validation
>>>>>      https://stackoverflow.com/questions/4666523/xhtml-strict-1-0-target-blank-not-valid
>>>>> X3dToX3dom.xslt stylesheet correction checked in, X3DOM example pages updated and uploaded.  Having fun with X3DOM!  8)
>>>>> all the best, Don
>>>>> --
>>>>> Don Brutzman  Naval Postgraduate School, Code USW/Br       brutzman at nps.edu
>>>>> Watkins 270,  MOVES Institute, Monterey CA 93943-5000 USA   +1.831.656.2149
>>>>> X3D graphics, virtual worlds, navy robotics http://faculty.nps.edu/brutzman
>>>>> ------------------------------
>>>>> Subject: Digest Footer
>>>>> _______________________________________________
>>>>> x3d-public mailing list
>>>>> x3d-public at web3d.org
>>>>> http://web3d.org/mailman/listinfo/x3d-public_web3d.org
>>>>> ------------------------------
>>>>> End of x3d-public Digest, Vol 122, Issue 137
>>>>> ********************************************
>>>> _______________________________________________
>>>> x3d-public mailing list
>>>> x3d-public at web3d.org
>>>> http://web3d.org/mailman/listinfo/x3d-public_web3d.org
>>> all the best, Don
>>> --
>>> Don Brutzman  Naval Postgraduate School, Code USW/Br       brutzman at nps.edu
>>> Watkins 270,  MOVES Institute, Monterey CA 93943-5000 USA   +1.831.656.2149
>>> X3D graphics, virtual worlds, navy robotics http://faculty.nps.edu/brutzman

all the best, Don
Don Brutzman  Naval Postgraduate School, Code USW/Br       brutzman at nps.edu
Watkins 270,  MOVES Institute, Monterey CA 93943-5000 USA   +1.831.656.2149
X3D graphics, virtual worlds, navy robotics http://faculty.nps.edu/brutzman

More information about the x3d-public mailing list