Augmented Reality Roadmap for X3D

[Editors' note. We have had a lot of discussions to describe a comprehensive strategy for adding Augmented Reality (AR) capabilities in X3D. Here is a snapshot of many ideas proposed so far. This is being worked into a position paper. Feedback and improvements are welcome.]

Motivation: Value of X3D Standard for AR

Augmented Reality (AR) and Mixed Reality (MR) operate across a many devices, ranging from small hand-held mobile phones to laptops, desktop PCS and full-blown walk-around CAVES. There is always a mix of both 2D imagery and 3D graphics, plus a mix of the real world and the virtual world. Many people are especially interested in lightweight AR that they can use as part of the GPS-aware mobile devices.

X3D is a mature standard with many implementations and many converters. It collects numerous proven technologies together in a workable way, providing a simple and coherent way for authors to create 3D models and VR applications. Current X3D efforts include identification of critical lightweight components suitable for mobile devices. X3D can save a lot of time for developers looking to design new AR applications.

It is sometimes tempting to think of a mobile device as simply a rotatable 2D display situated within the 3D real world. However 2D displays by themselves can be pretty limiting and oversimplified. The X3D group is thinking in terms of the larger 2D + 3D space that authors and users want. Demonstrations are also showing that mobile X3D can fit well with HTML5 and DOM improvements. This all "plays well" when moving from simple mobile applications to bigger models and spaces.

Because there are multiple implementations and extensions for AR that have already been demonstrated for X3D, it is clear that we can build an AR component for X3D. It will have lightweight capabilities suitable for mobile devices, plus additional compatible capabilities that work with larger 3D scenes and spaces. We will take advantage of X3D extensibility to define component levels and profiles that capture these palettes as easily expanded extensions.

Now is the time for serious AR developers to consider these essential challenges. By joining and commenting, you can help influence and ensure that these efforts reach the broadest possible success. Our current work is focused on harmonizing proven capabilities into best practices for AR and 3D graphics, implementable by multiple X3D viewers and usable by content authors.

X3D Strategy for AR

Lots of discussions at Web3D Conference, SIGGRAPH and ISO meeting continue to improve and refine this strategy. What do other participants think?

• Multiple AR nodes have been proposed and implemented for X3D by Fraunhofer team, Gerry's team and others
• Current technical work includes harmonizing these proposals for best fitting AR capabilities into X3D scenes
• This work will be formally written up as the Augmented Reality (AR) Component for the X3D Specification
• Establishing common implementations and examples is important to demonstrate successful interoperable capabilities for the spec
• Complementary work in HTML5/DOM, geospatial GpsSensor, and palettes for interactive scenes might be composed as a new Mobile Profile for X3D
• Heavyweight advanced capabilities for mixed reality, CAVEs etc. kept separate from Mobile but available in an AR Profile
• Continued collaboration and reaching out to W3C Augmented Reality Community Group, newly formed OGC ARML, Christine Perry's AR workshops and other arenas of interest
• ISO Standards Committee (SC24) which administers X3D review as an International Standard has established a new Working Group for Augmented and Mixed Reality
• Common goal remains maximum interoperability with all Web standards

Interoperability with Web Standards and Other Working Groups

• Open Geopatial Consortium (OGC) ARML group
• Christine Perry AR workshop series
• W3C Geolocation working group (need correct name, link)
• ISO JTC-1 SC-24 standards committee forming AR/MR Working Group 9

As we achieve clarity on our larger strategy for AR in X3D, it will help a lot as we reach out for common ground with other standards groups.

We especially want to encourage new participants to join this important effort.

Position Paper: AR Roadmap for X3D

Paper Due: September 23rd - To be presented at the AR Standards Group Meeting in Basel,SW - October 24-26, 2011

1. AR/MR: The next wave of 3D interactive immersive experience

2. What's propelling X3D/AR forward

Augmented Reality (AR) and Mixed Reality (MR) operate across many display devices, ranging from small hand-held mobile phones to tablets, laptops, desktops, and full-blown walk-around CAVE spaces. There is always a combination of both 2D imagery and 3D graphics, plus a mix of the real world and virtual worlds. Many people are especially interested in lightweight AR that they can use as part of the GPS-aware mobile devices.

Extensible 3D graphics(X3D) a roylaty free and ISO certified Standard developed by the Web3D Consortium (www.web3d.org) that originated from VRML, now in XML already has capabilities to support AR and MR applications. X3D is a mature standard with many implementations and many converters. It collects numerous proven technologies together in a workable way, providing a simple and coherent way for authors to create 3D models and VR applications. Current X3D efforts include identification of critical lightweight components suitable for mobile devices. X3D can save a lot of time for developers looking to design new AR applications.

It is sometimes tempting to think of a mobile device as simply a rotatable 2D display situated within the 3D real world. However 2D displays by themselves can be pretty limiting and oversimplified. The X3D group is thinking in terms of the larger 2D + 3D space that authors and users want. Demonstrations are also showing that mobile X3D can fit well with HTML5 and DOM improvements. This all "plays well" when moving from simple mobile applications to bigger models and spaces.

Because there are multiple implementations and extensions for AR that have already been demonstrated for X3D, it is clear that we can build an AR component for X3D. It will have lightweight capabilities suitable for mobile devices, plus additional compatible capabilities that work with larger 3D scenes and spaces. We will take advantage of X3D extensibility to define component levels and profiles that capture these palettes as easily expanded extensions.

Now is the time for serious AR developers to consider these essential challenges. Our current work is focused on harmonizing proven capabilities into best practices for AR and 3D graphics, implementable by multiple X3D viewers and usable by content authors. Collaborative work is happening to extend X3D to make AR/MR capable. Numerous innovative AR activities are ongoing by many researchers, companies and study groups. Web3D is engaged is all these efforts. Coordination and collaboration between numerous developers and standards organizations is necessary for AR to eventually become a seamless part of the Web infrastructure.

3. Extending X3D for AR/MR applications

• Requirements of X3D to be AR/MR capable
  • Adding real world view
    • Live video (Camera on the user's computer)
      • Camera sensor (on the browser device)
      • Local live video stream background and texture
    • Merging real and virtual image correctly
      • Camera calibration
      • Standard Viewpoint Nodes
      • Viewpoint node for MR visualization needs
      • Occlusion
        • Correct occlusions and Augmented Virtuality
      • Shadow
      • Reflection & Refraction
    • Live movie texture
  • Registration
    • Static -Relationship between real and virtual spaces
    • Dynamic -Tracking user's viewpoint
  • Real-time Interactivity
    • Tracking (users & other real world objects)
      • Sensor based Tracking
        • Ultrasonic, Electromagnetic, Mechanical, Optical motion capture
      • Computer Vision based Tracking
        • ARToolkit (HITLab), ARTag (Canadian NRC), BazAR (EPFL)[ARToolkit, HITLab]

[BazAR,

4. X3D-Interactive and X3D-Mobile profile = X3D-AR Profile

• X3D and X3D-Earth Working Groups will begin to address X3D Mobile profile Fall 2011
  • Support for all existing and appropriate mobile sensors
    • Geo-location & Tracking
    • Cameras (front & back)
    • Accelerometers
    • Orientation
    • Environment (temperature, pressure, humidity)
    • Touch (screen, buttons)
  • Small footprint and efficient operation
    • Reduce geometric modeling options
    • Reduce high-quality rendering options
    • Reduce lighting options
  • Heavyweight advanced capabilities for mixed reality, CAVEs etc. kept separate from Mobile but available in an AR Profile

• X3D Interactive Profile Supports

• X3D and X3dom

The traditional way to embed a X3D augmented reality scene into a webpage uses a plug-in that needs to be installed by the user. The plug-in approach has always been an issue for web developers and users. Several efforts have been initiated by Web3D on how to integrate X3D with HTML. Our goal is to bring 3D to Web developers by fully integrating 3D content into HTML documents. Today Web pages exist as large collections of hierarchical elements that are rendered to the page; all content appears in the Document Object Model (DOM). Several tools are available to allow programmers to manipulate the DOM, in particular tools and frameworks based on JavaScript. Placing 3D graphics in the DOM, allows these JavaScript tools to access and manipulate 3D graphics. Web3D and Fraunhofer IGD‘s recent efforts – X3DOM (www.x3dom.org), integrating X3D with HTML and the DOM, have proven this capability. X3DOM uses three basic approaches to display an X3D scene inside an HTML page without a plug-in.

• • External reference - HTML page includes an object element tag that refers to an .x3d scene, implemented via an X3D plug-in. Data might be passed within the page using DOM events.
  • X3D as XML in HTML - HTML page directly includes X3D source, likely with an XML namespace prefix, presumably implemented via an X3D plug-in or the browser itself. Again data might be exchanged within the page using DOM events.
  • API access - HTML page includes some form of canvas (or maybe Canvas3D) element that allows programmatic access to the page, so that X3D Scene Access Interface (SAI) might draw a bitmap.

The recently formed “Declarative 3D” W3C Community Group's efforts to define requirements to add interactive high-level declarative 3D objects through the HTML DOM is using X3DOM as one of its test models. This effort has strategic importance for Web3D and indeed for all 3D graphics, since X3D is an interchange format for a wide variety of models. We intend to establish a solid foundation for X3D to properly support 3D graphics for the native Web page enhancing AR capabilities and experience.

5. Why X3D is a viable and proven standard platform for implementing AR/MR

The Web3D consortium is a long term standardization group that has been producing and maintaining it's standard X3D for ISO for more than 10 years. Today, the Web3D Consortium is utilizing its broad-based industry support to develop the X3D specification, for communicating 3D on the web, between applications and across distributed networks and web services. Through the well-coordinated efforts with the ISO, OGC and W3C, the Web3D Consortium is maintaining and extending its standardization activities.

X3D is a mature standard with many examples, many players and many converters. Multiple implementations and extensions for AR have already been demonstrated in X3D. These efforts have been successful because X3D collects numerous proven technologies together in a workable way, providing a simple and coherent path for authors to create 3D models and VR applications. X3D’s extensibility provides the advantage to define component levels and profiles as extensions, allowing applications to not have to implement the entire specification. It has also been demonstrated that mobile X3D can fit well with HTML5 and DOM improvements. This all "plays well" when moving from simple mobile applications to bigger models and spaces. Current Web3D Consortium AR working group activities are focused on harmonizing proven capabilities into best practices for AR and 3D graphics, implementable by multiple X3D viewers and usable by content authors. This Working Group is thinking in terms of the larger 2D + 3D space that authors and users want. We will harmonize lightweight AR components within an easily implementable X3D Mobile Profile. Heavyweight requirements for MR and CAVEs can be compatibly defined. X3D is becoming a viable and reliable standard for AR developers looking to save time when designing new AR applications.

6. Interoperability with other Web and AR Standards

Lots of discussions at Web3D Conference, SIGGRAPH and ISO meeting continue to improve and refine our strategy on interoperability with other standards. Current technical work includes harmonizing these proposals for best fitting AR capabilities into X3D scenes. This work will be formally written up as the Augmented Reality (AR) Component for the X3D Specification. Establishing common implementations and examples is important to demonstrate successful interoperable capabilities for the spec. Continued collaboration and reaching out to W3C Augmented Reality Community Group, newly formed OGC ARML, AR Standards group and others is essential. ISO Standards Committee (SC24) which administers X3D review as an International Standard has established a new Working Group for Augmented and Mixed Reality. Our common goal remains maximum interoperability with all Web standards.

7. How to come to Fruition

TODO and Next Steps

• short description of X3D and an AR component that we want to achieve
• short description of the the VR/AR/MR/Reality continuum
• screen-capture pictures showing key capabilities already demonstrated
• Further discussion, details and comments to finish this document in September