[x3d-public] X3D VR

[Mitchell Williams]

I’m developing GearVR’s X3D implementation.

Our implementation is an X3D parser that builds GearVR data structures.  You won’t find the X3D data structures such as <Appearance> or <Shape> in GearVR.  We only  support X3D data types such as SFColor, SFVec3f, etc to enable scripting.  At times, additional methods were added to these data structures [For example SFBool can set the Boolean by an integer 0 = false, or ‘false’ as a text string] to support our developers.


I expect to implement the Render-to-Texture node based on X3Dom.  GearVR does implement Render-to-Texture functionality.


A few issues of X3D don’t apply to virtual reality, for example setting Viewpoint’s orientation.  Generally in VR, we want to fix the orientation in the direction one is looking.  The <Anchor> can link to a web page – not clear how we would implement that in VR.

X3D may consider implementing some of the gyroscopic sensors in VR headsets.  Also, be prepared for various I/O controllers such as Wands, or Oculus hand controllers.

Mitch

From: x3d-public [mailto:x3d-public-bounces at web3d.org] On Behalf Of Andreas Plesch
Sent: Tuesday, December 20, 2016 9:51 PM
To: X3D Graphics public mailing list <x3d-public at web3d.org>
Subject: [x3d-public] X3D VR


Hi

Working with x3dom and WebVR I am exploring what additions to X3D would be necessary or useful to effectively use current VR hardware such as the Rift, Vive, Gear, or Cardboard.

The proposed RenderedTexture node is currently used in x3dom with special stereo modes to generate left and right views from a single viewpoint in a single GL context.

A Layer for each left and right view could also be used with two coordinated but separate viewpoints. This would be Cobweb's pattern.

Since the HMD and its own runtime know best the internal optical characteristics of the display, the preferred interface at least in WebVR are view and projection matrixes directly usable by consumers for 3d graphics generation, and not position, orientation and fov. This leads to a new requirement for X3D to accept these raw matrixes as input.

Since x3dom uses RenderedTexture for VR, I added there additional special stereo modes which directly receive these matrixes from the HMD to be used for rendering at each frame. This in effect accomplishes a first, head and body based, level of navigation. In my first tests (on github) it works well and should be robust across devices. This approach does require some standard API to the HMD such as WebVR.

Another possibility is to add view and projection matrix fields input fields to viewpoints which can be continually updated. One could convolve the view matrix with position/orientation fields, or optionally completely ignore them.

One then could use external SAI to keep updating or perhaps introduce an environment ProjectionSensor. It would relay these matrix events from an HMD runtime to then be routed to the viewpoints.

A second level of navigation is accomplished with handheld controllers. Until some standard gestures evolve, it will be necessary to expect custom per scene navigation. X3d should have a way to sense buttons and axes velocities of such controllers so these are then available to manipulate the view in some way. InstantReality has a general IOSensor to that effect. On the web the GamePad API is a standard which would need to be used.

Summary: RenderedTexture with stereo modes, matrix fields for viewpoints, a ProjectionSensor, and a ControllerSensor would all be candidates for x3d VR support.

Thanks for reading, any thoughts welcome,

Andreas
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://web3d.org/pipermail/x3d-public_web3d.org/attachments/20161221/6c6b8d06/attachment.html>