[X3D-Public] Fwd: Re: [X3D] X3D HTML5 meeting discussions: Declarative 3D interest group at W3C

[John Carlson]

I would hope any declarative approach would allow for addition, querying, update and deletion of any node.

John
On Dec 27, 2010, at 9:05 PM, Philipp Slusallek wrote:

> Dear Chris, all,
> 
> First of all, my best wishes for the holidays season!
> 
> Thanks for excellent comments that you have raised in your emails. Let
> me comment on them in more detail but also from a much broader
> perspective here as I believe it is important to start with a clear
> outline where we want to go right from the start. Sorry that this email
> got quite long, but I hope its worth it.
> 
> Let me start with a few words on my own background: I have worked mostly
> in realistic rendering and we have started a lot of the realtime ray
> tracing work over the past 10 years (as a visiting professor I was
> involved in the initial NV Optix activities and many of my Ex-PhD
> students are working in the respective groups at Intel and NV). Besides
> my professorship at Saarland University, I am director of a research
> group at the German Research Center for Artificial Intelligence (DFKI)
> and I am heading the newly founded Intel Visual Computing Institute
> together with a colleague.
> 
> Across these organizations we are investing considerably into
> declarative 3D. Our main motivation for declarative 3D is to
> "democratize 3D" (if you want to call it like this). For much too long
> 3D has been locked into circles of experts that had to learn weird
> "graphics things" before even getting the most basic things to work.
> This is fine as long as all you want to do is games, where you can have
> experts working for months and years on game engines for very specific
> types of games, and on the other side content experts who develop very
> specific content exactly for this specific type of engine (I am somewhat
> simplifying things here -- but not too much).
> 
> While this has fed us graphics people for many years, it has excluded
> many if not most other valid uses of 3D. Its just amazingly hard for a
> small company to add any use of 3D to their product without
> having to hire a team of graphics experts. Note that WebGL will not
> change this by itself, since it just repackages what we have
> already got for more then 20 years! It "just" makes delivery easier.
> Wile this is important, its not enough.
> 
> While some simply believe that there is nothing we can do about
> this, we strongly disagree. Streaming video technology was considered
> much too difficult for most users until YouTube made it really simple to
> upload your own clips and get a few lines of HTML in return to copy to
> your Web page and -- viola -- even you can create, share, and experience
> videos. And this after the technology to do video processing on a PC was
> already available for about a decade (MPEG, MMX, etc.).
> 
> Similarly, all the technology to do 3D on the Web is already available
> even on mobile devices. "All it takes" is to develop an easy way to put
> this technology to use for everyone. This, in particular, means that
> anyone who is able to create a Web page should be able to embed
> interactive 3D into it as well and make 2D and 3D work together as one
> piece. So whatever we create must empower the millions of Web developers
> out there.
> 
> People argue that content creation is much more difficult for 3D and
> they are right -- up to a point. Creating original new content is
> hard for most people -- but then the computer vision people are making
> great progress to create 3D content out of photographs, videos, or 3D
> scanners. Also unlike video, we can re-use 3D content, which will likely
> create a network effect: The more 3D content there is the easier it will
> be to create novel scenes. And finally, with a possible new world-wide
> market for 3D content and apps, I am sure there will be people who will
> find interesting ways to deal with any remaining problems.
> 
> While WebGL makes a lot of this *possible* in the browser, it does not
> make it more *accessible* since its still just OpenGL with all its
> issues. So we need some layer on top of this -- a scene graph. And funny
> enough the DOM is exactly what is needed: a nice declarative scene graph
> -- in particular one that millions of Web developer already know
> inside-out. So making the DOM a great declarative 3D scene graph should
> be our main goal here. As we have shown this already works quite well
> today, but there are still a number of shortcomings and nice-to-haves
> that we need to address in the process and you mention some of them
> (e.g. types access to the data, not having to store text versions of the
> data, the ability of extending the DOM in JS (a la XBL) etc).
> 
> 
> The key aspect of a scene graph is that it must provide a higher level
> of abstraction than the underlying API, otherwise there is not much
> point to creating it in the first place. Take lighting as an example:
> OpenGL does not know about lighting (any more), its all just part of a
> shader. However, this is something that a user will not (and should not
> need to) understand. A user will think of lighting as a separate concept
> via placing light sources with certain characteristics in a scene. Now
> the key part of any scene graph implementation is to *map this higher
> level abstraction* to something that gets executed in hardware eventually.
> 
> My point here is that we cannot just use GLSL as a shading language for
> a declarative approach, as its operating at the wrong level of
> abstraction. We need a layer in between that merges the lighting
> information from the scene graph with the light reflection
> characteristics part of a GLSL (or some other) shader. And this is just
> one example of the mismatch that we will have to deal with.
> Unfortunately, this makes is more difficult -- but certainly not
> impossible -- to provide a "crack-through" for authors.
> 
> Now since we have to introduce a mapping stage anyway, we can actually
> use this to our advantage by creating this abstraction such that we can
> map not only to WebGL but also to other rendering layers, including
> native OpenGL 4.x or even ray tracing. This provides many advantages as
> it opens a whole new space for innovation between the low-level
> rendering API and the high-level scene representation. But it also
> complicates things a bit more.
> 
> Luckily, there is now some pretty nice and solid technology that can
> help us here: Our AnySL prototype system allows even for using RenderMan
> shaders as input and spit out GLSL code at the end. Now there are still
> some limitations for this rasterization backend that we are addressing,
> but we can already translate shaders to several other rendering
> implementations.
> 
> I guess my point here is that while I agree with you that we should base
> our work on WebGL, we should not make this the only target and take
> advantage of the fact that we are introducing a higher level of
> abstraction.
> 
> 
> But let me get back to your email. I full-heartedly agree with your call
> for an open mind. While X3D has taught us many lessons that are worth
> keeping in mind, 3D on the Web is just at its beginning and will likely
> be so much bigger than whatever we had before. So we need to make the
> best effort to fix any known issues and get it right. While it had
> sometimes been hard to argue the case for this within the Web3D
> consortium with the same arguments you raised here (the LA Petree-Cafe
> and other occasions come to mind :-) I see increasing openness to these
> ideas now and I am happy to have them and their experience on board.
> 
> In particular, many of the solutions that Johannes with X3DOM came up
> with are pretty close to XML3D and we are working together with him to
> hopefully resolve remaining issues as part of a large Germany industry
> project both groups are involved in (Spitzencluster "Software for the
> Digital Enterprise"). This include most of the aspects you raised (using
> existing Web technology including tags such as <img>, etc., JS, CSS. I
> believe we are 100% d'accord on this.
> 
> Also the feedback after our presentation of XML3D and X3DOM at the W3C
> TPAC conference in Lyon in November has been very clear: The W3C really
> wants to see 3D as an integral part of HTML and other Web technology. In
> our charter we also already outlined the main areas where we will have
> to interact with other W3C groups including areas like CSS, SVG, and
> others. It seems that a lot of the new stuff we will have to add  such
> as programmability through shaders is something that can add to many
> other parts of the Web as well (e.g. their use age for fill style in SVG
> or simply the background).
> 
> I also fully agree with your worries about the size of the spec (e.g. in
> X3D). The key from my perspective is to build a small, orthogonal, and
> extensible spec. Pretty much like OpenGL at the beginning and after the
> latest cleanup -- but for a declarative approach. We should only specify
> aspects that we cannot deal with in other ways or which would otherwise
> be too inefficient. We must also keep anything we specify as orthogonal
> to each other and to the rest of the Web as possible. Ideally, each
> aspect should be able to stand on its own (geometry, shading, lighting,
> viewing, etc) and be accessible such that Web developers can change it
> as much as they may need to. We should never embed any policy issues
> into the spec but leave them to the application developer. However, we
> should provide the necessary tools and make it as easy as possible to
> them to implement those policies, though.
> 
> We currently have two native implementations of XML3D in both Firefox
> and Webkit. The interfaces for both are generated automatically from the
> spec but their implementation within the browsers is still rather
> different due to their internal differences (and due to the fact that
> unlike WebGL we have a much wider exposure via the DOM, CSS, JS, and
> other areas in the browser.
> 
> 
> I hope you get a chance to have a detailed look at the XML3D spec (on
> xml3d.org) as it already contains many of your suggestions, including
> the use of CSS for appearance modeling, transformations, and such --
> just as you suggested in your last email. My hope is that we can start
> our discussions with a version of this spec and argue very concretely
> about what is or is not important for declarative web. We plan to drive
> this discussion with a set of concrete and near term use cases so we do
> not go down the route of building some dream castle in the air.
> 
> Finally, let me say that I highly welcome your comments and insights and
> that I hope you find enough time to contribute to the XG. Getting Apple
> to support it as a W3C member would be highly welcome as well.
> 
> 
> Best regards,
> 
> 	Philipp
> 
> Am 21.12.2010 18:56, schrieb Chris Marrin:
>> 
>> On Dec 21, 2010, at 7:08 AM, Kristian Sons wrote:
>> 
>>> Dear all,
>>> 
>>> in preparation for our meeting today we worked on a draft for the
>>> "Declarative 3D" XG charter. We can use it as basis for the discussions
>>> today.
>> 
>> 
>> Hi,
>> 
>> For those of you who don't know me, I'm currently the editor of the WebGL spec and its main implementor on WebKit/Safari. I was involved with VRML from the beginning, but withdrew from the X3D efforts when our goals diverged.
>> ==========
>> 
>> Before I start, let me give the executive summary (since this thing has gotten so long):
>> 	- It's great to see this effort get underway
>> 	- A "declarative 3D language" should avoid SVG's pitfalls and stay small and focused.
>> 	- Use other web technologies (including WebGL) so you don't have to solve every problem.
>> 	- Make your nodes work well with the rest of HTML, including the DOM and CSS.
>> 	- Especially make sure your node set integrates well with WebGL to allow for extensions and modification of the rendering result.
>> 	- Consider prototyping on top of WebGL as a way to test features and integration with the page.
>> 
>> ==========
>> I applaud your effort to start an XG. I hope you go into the effort with an open mind and don't instantly attempt to ratify some form of the X3D spec. I have long felt that the main flaw of X3D is the fact that it stands alone. It has its own parser, its own scripting, its own image and media handling, it own event model and its own DOM. The event model in particular is completely foreign to the HTML model. It is also too big, even in its most modest forms. Look at SVG. It has taken many years for it to get even a toehold in modern browsers and even those implementations (including the one in WebKit) are inefficient and slow.'
>> 
>> I don't have insight to any browser other than those based on WebKit, but I can say that one of the main reasons for our implementation not being more performant is that the spec is too big. Implementing all those features on all supported platforms is extremely difficult. If you look at WebGL in contrast, there are now 7 implementations, all interoperable and performant, on 3 platforms (Windows, Mac, Linux). The Google Body example:
>> 
>> 	http://bodybrowser.googlelabs.com/body.html
>> 
>> is rendering 1.5 million triangles per second and runs at frame rate on all 7 WebGL capable browsers. This is not bragging, I mention it because it took us less than 2 years to get to where we are. I believe the reasons for that are:
>> 
>> 1) Our images are loaded via <img>, <video> or <canvas> elements. We hope in the future to be able to use rendered HTML as an image source as well.
>> 
>> 2) We use the browser's JavaScript engine, so we get to take advantage of performance improvements currently being done by all the browsers. And since we're integrated, we can make improvements to the native interface between JavaScript and WebGL to further improve performance.
>> 
>> 3) WebGL is created via the Canvas element, so we don't need to worry about compositing, layout or any other element level features.
>> 
>> 4) WebGL is based on OpenGL ES 2.0. We didn't have to solve any of the hard 3D rendering problems, we just had to adapt an existing engine.
>> 
>> 5) WebGL has wide acceptance among the browser vendors. Chrome, Firefox and Safari all support it, and an Opera version is in the works.
>> 
>> That last point has been incredibly helpful. Since Chrome and Safari are based on the same codebase, when you fix a bug for one, you fix it for all. The Google team has fixed a huge number of bugs that have benefited Safari. And Google also implemented ANGLE as a free an open toolkit which allows WebGL to run on D3D. The Firefox team is using ANGLE now and soon WebKit will be available on Windows because of that great effort.
>> 
>> 
>> 	I want to put a caveat on all this. WebGL is in unreleased versions of all the browsers. I believe 
>> 	Chrome and Firefox will have a version in they released products soon, but it is still only in 
>> 	nightly builds of WebKit for Mac. There are stirrings about having it on mobile platforms but no 
>> 	implementations yet. Mozilla showed an experimental version on a Nokia 900. I'm not sure if 
>> 	that has deployed, or even if it will deploy. I don't know about progress on Android and I can't 
>> 	talk about any Apple plans. But the spec is clearly aimed at being implementable on mobile 
>> 	devices.
>> 
>> 
>> The biggest challenge in WebGL was adapting the OpenGL model, which is completely unsafe and left all of the resource management to the application developer, to a Web model where the browser must never do anything unsafe, insecure, or leak resources. We also tuned the API to make it possible to implement on D3D in addition to OpenGL.
>> 
>> WebGL is almost ready to be ratified by Khronos. We have the above mentioned implementations, with more on the way. We also have a conformance suite that is growing every day. And many bugs have been fixed, as more content is written to challenge the implementations, so WebGL is getting very stable.
>> 
>> I'm saying all this so hopefully you don't go down the path of trying to solve too many problems. One of the motivators of this message was because there are 2 references to WebGL in your document. In the first you stated how WebGL had deficiencies which motivated your XG, and in the other you state that APIs like WebGL are out of scope for your XG. This troubled me. New web technologies build on the existing specifications and implementations. You could base your work on WebGL and you would reduce your workload dramatically.
>> 
>> For instance, assuming you integrate shader technology into your proposal, what shader language will you use? In the past this has been undefined, so if someone wanted to run on OpenGL and D3D, they would need to write shaders in both GLSL and HLSL and possibly more than one variant to match the hardware being used. WebGL has solved that problem in a way that guarantees good cross platform fidelity and implementation feasibility. We use GLSL ES 1.0 with constraints listed in the spec so we translate to desktop GLSL (various versions) and HLSL using ANGLE. This restricts the shader capabilities to those in GLSL ES, but it ensures wide deployment, all the way down to mobile devices. We have an extension mechanism so we can add features from that base (texture_float is an early extension which has been implemented and tested by Google).
>> 
>> Another issue is how to handle binary data such as compressed textures, coordinates and animation keyframes. This is another area we knew early on would be an issue, so we created Typed Arrays (https://cvs.khronos.org/svn/repos/registry/trunk/public/webgl/doc/spec/TypedArray-spec.html) to deal with the problem. The next issue that came up was how to load binary data. Today's HTML browsers aren't very good at handling data types other than string and to convert the data to text would be very inefficient. So we've made a proposal to add the ability to load binary data using XMLHttpRequest. The data is loaded directly into a Typed Array so there is no conversion step getting in the way. The data can then be loaded into a VBO without any copying having been done.
>> 
>> I hope you can find all this existing work helpful to you. I agree that a declarative form is needed for 3D on the Web. I just hope it integrates well with the rest of the web technologies, including WebGL. For instance, wouldn't it be great if an author could write an extension node in JavaScript and WebGL. You could also allow authors to drop down into a WebGL API to make low-level adjustments to the state or to the buffers loaded in the GPU.
>> 
>> This brings me to my last point. Johannes and the others at Fraunhofer have done an incredible job of producing an X3D implementation on top of WebGL. So it is clearly possible to represent a complex declarative form on top of a JavaScript API. I know they experienced some pitfalls and deficiencies along the way. You can (or we all can) learn from the issues they found. To get around these issues your XG could avoid that element or feature of an element if possible. Or you could raise the issue with other W3C groups so browsers can be changed in the future to better suite your needs.
>> 
>> One area where WebGL is deficient is in its inability to use CSS directly. Even if you make a node set like X3DOM, being able to use a CSS color property on one of those nodes is very difficult. It's not hard to read the color and apply that to a 3D primitive. But being notified that the color had changed, either by programmatically changing the style, or by animating the color with CSS Animation, is impossible. A huge advantage of a native node set is that such notification is easy and would provide great integration with the CSS style system of today's browsers.
>> 
>> So my first hope is that you can use WebGL as a way to prototype design ideas and avoid having to tackle the difficult problems that have already been solved by other technologies. My second hope is that you make your node set small and well integrated with the HTML DOM and with CSS to make it an attractive addition to web browsers.
>> 
>> -----
>> ~Chris
>> chris at marrin.com
>> 
>> 
>> _______________________________________________
>> X3D-Public mailing list
>> X3D-Public at web3d.org
>> http://web3d.org/mailman/listinfo/x3d-public_web3d.org
> 
> 
> _______________________________________________
> X3D-Public mailing list
> X3D-Public at web3d.org
> http://web3d.org/mailman/listinfo/x3d-public_web3d.org