[X3D-Public] Fwd: Re: [X3D] X3D HTML5 meeting discussions:Declarative 3D interest group at W3C

[Philipp Slusallek]

Hi Chris,

I fully support your approach to world domination :-). We are focusing
on the declarative part now and fully support your first two steps (CSS
and WebGL).

The issue even with shader arguments in CSS is that CSS does not allow
arbitrary name/values pairs with unknown data types in CSS. It is
strictly a fixed function model, where all parameters and their types
must be known in advance at spec time. The spec even explicitly states
that additional parameters are not evaluated. I am not entirely sure
what the reasons behind this explicit behavior are and how difficult it
would be to change this to a more open model that we would need for
arbitrary shader arguments. Would be interesting to pursue, though.
Particularly, if you think about the ablit to apply such shaders also to
non-3D objects in HTML (such as the background, or maybe even really
rounded boxes around buttons and such that have nice Phong highlights
and such). A "real" 3D UI even when still used on a 2D canvas :-).

Your CSS interpolators are fully supported in XML3D already and they
work great. We even drive them through SMIL once in a while.

To fully support CSS styling with XML3D (or anything else) on top of
WebGL, would require fixing some JS API issues. Particularly getting
callbacks for some changes of the CSS data, so we can update the
rendering graph. Getting this into WebKit could be a great first step
for fast adoption as it could be made to work well with any browser that
supports WebGL and these fixes). It would be great if you would help
push these changes into browsers (they are biting anyone that seriously
wants to monitor CSS via JS). We can easily give you a full list of
things that would need to be added/changed (these are separate from the
more fundamental arbitrary CSS attributes issues mentioned above).


The other processing that you need is essentially arbitrary geometry
processing from morphing, skinning, and skeletal animations to freeform
deformations, (adaptive) subdivision surfaces, displacements, and more
esoteric stuff. I do not see how you can put this into CSS at all as
even the set of inputs is unknown (e.g. for keyframe animation).

Using the typed array API of the data elements, anyone is free to do all
this in JS, but short of a data-parallel extension to JS (for which
there is some early work), I do not see this as fast enough, as this is
at the inner loop of interaction and very performance critical (there is
a reason this stuff is in the OpenGL pipeline and on the GPU now, where
NV and AMD are pushing this as the main competitive battle in their
latest chips).

So, some form of procedural CUDA/OpenCL/whatever shaders, or a more
declarative approach using processing graphs (XFlow) that abstracts from
the lower-level languages seems needed here to get really interactive
and highly dynamic scenes going. Note, that you are still free to use JS
directly at any point and XML3D works nicely without XFlow either. Its
an accelerator that I believe is really important.

Any once you have that the step to apply this also to image processing
for AR style applications using video or images coming in through the
DeviceAPI seems really intriguing.


I am not so sure what you mean by the paragraph about "perceived way to
do 3D the right way". Which part did you not like or thought would be
hard to get W3C support for? XFlow? -- for this reason, we carefully
designed it as a separate module independent of XML3D (even though I
believe its important for dynamics -- see above).

If you refer to the ability to sprinkle 3D objects anywhere within HTML
page (like with you 3D transformation applied to a plane), we have been
discussing that back and forth. However, we had a hard time coming up
with intuitive semantics for this. How would you specify which objects
get rendered via which camera (there may be multiple views that you want
to show of either the same or different scenes), etc. The point is that
your 2D stuff comes with a well defined 2D position, that you apply a 3D
"warp" to.

General 3D objects do not have that reference to the 2D plane to start
with. You could attach a camera to each and every 3D (or even 2D?)
object via CSS (interesting idea, really!) but then any object would
only be visible in one view (but we could use multiple instantiations to
get around this). Its probably the best option around, if you want to go
there. We did not see the need to do this as it would also complicate
the mapping to a scene graph -- but it would be doable and would be even
more "Web-like" and "CSS-ish" (except for SVG, which follow the same
model that XML3D does right now with its "svg" tag).

Another option could be to attach the scene graph to the canvas instead
of a separate xml3d tag, which would be a no-brainer (we just wanted to
have better control for our experiments for now).

If you were referring to anything else, please explain.


	Philipp

Am 03.01.2011 02:19, schrieb Chris Marrin:
> 
> On Jan 2, 2011, at 3:19 AM, Philipp Slusallek <slusallek at cs.uni-saarland.de> wrote:
> 
>>
>> Shaders, lighting and such as separately described and attached to
>> geometry via CSS, and so on.
> 
> It does seem conceptually compelling to put shaders into CSS, but I don't see that happening anytime soon. It's too domain specific. But parameters to the shaders (uniforms and attribs) certainly belong there. 
> 
>>
>> I do believe in declarative interpolators for data intensive processing
>> (a la XFlow, see other email) as there is much more than just rigid body
>> animations that we need. Particularly if we are interested in getting to
>> live-like virtual characters on the Web. We need careful ways to modify
>> (interpolate, skin, displacement map, light map, whatever) the data
>> sets. While this could be done in JS (and the nice interface to typed
>> arrays allows this to be done in XML3D). However, I believe that a
>> declarative way to specify these operations (similar to a shader network
>> in many animation tools) is a better and more intuitive way to specify
>> these operations (and can be implemented way more efficiently and
>> portably that way). 
> 
> CSS does have interpolators in CSS Animations. It includes customizable easing functions. Probably the only practical CSS properties to interpolate are transform and color, but that gets you a lot of the interpolators in X3D. So the question becomes what do you do about the rest?
> 
>     a) Don't support them and use JS when need be?
>     b) Add new properties to CSS to support them?
>     c) Represent them as elements?
> 
> There are different answers for different types and usage patterns. That's what I mean by separating the issues. It's not "how do we support inrpolators?", but rather "how do we support these useful usage patterns, while still staying within the web model?". 
> 
>  As much as you might like to warp the current web HTML/DOM model to match what you perceive as the "right" way to do 3D, I don't see that happening any time soon. I tried swimming up that stream for many years with little success but with a lot of experience about what NOT to do. When my fellow team members and I here at Apple added CSS Transforms, Transitions and Animations, we were able to get 3D into the browser, not as a scene graph of 3D models but as "planes in space". No monkeys or globes, but 3D that could be applied to many web design scenarios. 
> 
> That was step 1. Step 2 was making it possible to represent 3D models in the simplest way possible - WebGL. Declarative 3D is step 3. This is all my personal model of world domination, of course. But I believe that the step to declarative 3D has to be another baby step or it will fail. 
>