[X3D-Public] Fwd: Re: [X3D] X3D HTML5 meeting discussions: Declarative 3D interest group at W3C

[Philipp Slusallek]

Hi Joe, all,

Am 31.12.2010 06:11, schrieb Joe D Williams:
>> Can you elaborate what you mean by this statement? To make this a
>> constructive discussion: What is it that you are missing in XML3D or
>> X3DOM?
> 
> I don't know. I will look deeper. I saw the shows from html5 on the web.
> Great stuff. At this point I think I appreciate the X3DOM stuff a little
> more because it looks to retain more of the inevitable complexity needed
> and so far better recognizes the need to embed a high perfomance
> rendering system in the html5 browser. The high performance is needed
> for behavior stuff and high level scriptless interactions. So, for the
> DOM or however the event graph and the rendering graph or whatever, it
> doesn't actually need to be a DOM, it just needs to act like one from
> the outside.

That is exactly what XML3D does as well.

> Fortunately this is easy, except I recall that X3D uses a 'graph' (DAG)
> instead of a tree due to certain advantages that may not even be
> required for simple stuff. For example if you don't use DEF or PROTO
> then you get a natural opening for CSS because the styled item is truely
> everywhere in the tree instead of just once on the graph.

This is one of the biggest issues with X3DOM and any mapping from X3D to
HTML (from my perspective) X3D inherently has a DAG model that shows up
in subtle and not so subtle areas. This is foreign to HTML and adding a
DAG under the hood making it look like a tree from the outside causes
funny things to happen, if you make changes to one of the "copies" of a
multiple referenced objects. Note that you run into CSS issues as well
this way, which may be as difficult as those in SVG.

In XML3D we have avoided the need for a graph in the first place. Every
object has a clear location in the tree, but you may refer to an object
somewhere else in the tree through a URL fragment ("#id"). Currently, we
only allow references to geometry (leafs in the tree, not to internal
nodes with entire subtrees) for simplicity reason. This means you may
have to replicate these internal nodes but still share the nodes that
really maintain the most data. However, for good reasons we may want
eventually to allow references to entire subtrees. This needs to be
discussed.

> So I mainly want to keep track of what is happening and contribute where
> possible
> 
>> Here are some possible answers, for what you may have meant:
>>
>> -- I have not said that we will use the DOM to optimize the rendering or
>> anything like this. In XML3D we have a fully configurable and optimized
>> rendering engine behind the 3D-part of the DOM that can drive a
>> rasterization as well as various ray tracing renderers.
> 
> This sounds really neat and is all nice features. To me, it should be
> readable and expose teh author to necessary complexity as gently as
> possible. I would like to get some reference points by adapting some old
> and current stuff. As I said, so far the X3DOM picked up some examples
> real easy.
> I deeply appreciate all progress on these items of interest.

As I mentioned there is a reasonable good X3D to XML3D converter on
XML3D.org. Note that the version there currently only supports realtime
ray tracing and so is not hardware accelerated. This will change in the
next version though.

>> -- One thing that might have confused you: In the past many people have
>> mixed up a scene graph with a graph that accelerates rendering.
> 
> not really, but I think the item of interest with respect to html5 is
> what we can see and do in the DOM. Whatever is under the covers, it
> should be very straighforward to expose every content and presentation
> detail in a DOM. That, i think is the simple part. The hard part is
> connectiong al the peces and hierarchies. For hifi repeatable realism,
> it gets complicated fast.

It would be good to know examples of what you consider hard here. Do you
have any examples that we can refer to?

>> However,
>> conceptually they are very different but obviously share the same data.
> 
> The DOM has a fairly well defined api, just stick to it and fix where
> necessary and everything will be OK.
> 
>> The first is a way to organize your scene data from an application point
>> of view, while the other organizes the same data for optimal rendering
>> performance. The two organizations typically have very little in common.
>> Note, that the rendering organization typically differs dramatically
>> even between different rendering strategies (e.g. rasterization/ray
>> tracing).
> 
> All that is fine I just didn't get the empahsis on modelling behaviors
> of interesting UI objects..

UI is a great topic in general and probably one of the hardest. If we
are in the HTML ecosystems and get exposed to billions of users,
providing a intuitive and consistent UI is key. We cannot have every 3D
window have a completely different UI and so must standardize on some of
the basic stuff. Having said this, it also seems clear that we need to
be able to adapt the UI depending on the data or the application. This
area is hard to get a grip on and is not helped by the many different
devices that different users will use on the same scene. The scale of
variation will be enormous on the Web.

We have thought about using sensor nodes in XML3D but have decided not
to do so, as we are not confident enough that they are the right
approach (and they do have many issues still as well). In XML3D you can
get events on any 3D as you can on 2D object (onmouseover, onclick, etc)
but you have to handle this in JS directly. Its more low-level but also
way more flexible.

Way this can develop in are something something like sensors (they are
pretty much tied to routes though that I am not  convinced of yet or
more likely at this time "standardized" UI libraries that handles these
events (something like Jquery on the HTML side).

>> -- What do you mean by automation?
> 
> Well, like having builtin sensors and timers and interpolators and
> generally, behavior details, and a reliable event system and easy
> authoring because you can just use the browser and a text editor or a
> full-fledged authoring system, and being able to lift great gobs of
> existing content into my little html5 show.

OK, I have presented my take on sensors and routes already.

Interpolators are a very different beast. It turns out that indeed you
want the ability to operate on large pieces of data such as
interpolators, morphing, skinning, skeletal animations but also some
image processing for AR and other purposes.

In XML3D we have indeed developed exactly such a system which also uses
a mechanism that acts somewhat like routes. There are essentially data
containers (<data> elements) that are collections of arrays (similar to
Vertex Arrays in OpenGL. They can be contained in mesh elements directly
to be rendered but can also be referenced through a URL fragment and so
be reused (see above). As data elements can also reference each other
and augment or override certain arrays, you can easily have a single
geometry data set with different colors or texture coordinates.

Now the key part of XFlow is that we also have data elements that can
refer to a script node and implement any transformation on the input
data they refer to (other data elements) and provide as "output" new
vertex arrays. We currently do the processing on the CPU and only have
predefined "scripts" (using a URI notation), but this is designed to
work with our new shading language system such that you can write you
own scripts to modify the data. Eventually, we will even be able to
optimize the scripts over multiple of these scriptable nodes and pus
some of these nodes also into the geometry shader pipeline of OpenGL and
such. But this is still work in progress and not in the distributed
versions yet.

>> There is a decent API that deals with
>> the DOM (including AJAX to download them and many other nice Web
>> technologies).
> 
> Great, I have seen some of that and it works and can work everywhere. In
> fact WebGL is pushing the limits of XHR and JSON. So, of course we have
> access to allthat, given the encoding makes sense.
> 
>> We have extended this to better support access in native
>> 3D data types (like WebGL). Javascript is also a pretty decent
>> programming language that gets a lot of attention in terms of
>> performance speedup and such. All of this we get for free when working
>> in the same ecosystem.
> 
> Right, looking at stuff in pure WebGL here is showing lots of agreement
> in all the browsers, so this first pass at doc3D only needs to do the
> basic features. That makes part of the opportunity easier and gives us
> some basic features to build a live DOM that understands the kind of
> data expected by the underlying process and is able to make it live.
> 
>>
>> The DOM just happens to be used by many times more people world wide on
>> a daily basis that all programmers that have ever dealt with a 3D scene
>> graph combined. So even if it has flaws, at least it works for a heck
>> lot of people and is very well tested :-).
> 
> Interesting. But I think it sounds like you have not examined the X3D
> SAI interfaces. Really, it is just like the DOM if you want it to be
> that way. To the user it can look like a DOM, just that a child can have
> may parents and that there is also a hierarchy of contexts.

The API is different enough that Web people have to learn it anew. This
argument actually cut even better the other way: Why not replace the SAI
with the Web DOM API that millions of people use on a daily basis if the
two are so similar anyway?

>> -- In case you are referring to Routes, I am not sure I agree with you.
>> From my point of view, routes help solving the simple cases but easily
>> get into your way if you want to do more complex stuff. But if you
>> really need them, they can be easily simulated using other techniques.
>> And maybe there are convincing arguments that we should indeed add them.
> 
> That is fine, the SX3D SAi also gives event notifation and attachemts
> and that, if yo need them. With little imagination, it gets important to
> be able to get information around the scene. Route just gives a
> convenient way to send strongly typed data from mode to node, and is
> easily readble/settable from the outsde.

I definitely see the things one can do with the synchronized event
system and data propagation model of X3D. We just do not have this
available in HTML. So we could try to add it (but we would have to do
this across all of (2D-)HTML, which will be a real challenge). Or we can
see if we can live without it or create a different model. We are
currently doing without it in XML3D to see how far we can push this (but
we have the data references and transformations of XFlow). But I can
definitely see that we end up needing something like this eventually.

> Thanks, Philipp for your work on this and to everyone else that is
> interested.
> I hope you can use this list to expllain the details and show some
> examples.

the best references that we can give are all available on xml3d.org
including the spec, the papers, and the implementations in modified
browsers and using JS/WebGL. These are alpha versions so far, but good
enough to get your hand dirty and tell us what you do not like. Please
try out the X3D converter and fix it (or tell us what to fix). XML3D is
close enough to X3D that we should be able to capture a lot of the
typical functionality of X3D scenes.


	Philipp
> Joe
>>
>>
>> Am 30.12.2010 21:58, schrieb Joe D Williams:
>>>> And funny enough the DOM is exactly what is needed: a nice declarative
>>>> scene graph
>>>
>>> for a certain level of funny and a certain set of what is needed, yes.
>>> The hard part and where the contribution is made for authors is
>>> automating that DOM.
>>>
>>> Best Regards,
>>> Joe
>>>
>>
>