[x3d-public] New volume rendering extension proposals - more work needed
Brutzman, Donald (Don) (CIV)
brutzman at nps.edu
Mon Jul 15 17:54:28 PDT 2019
Dick and I tried to integrate this prose into the draft specification today, but we are struggling to handle some of the following questions.
We first took a close look at Texture3D Component as a cross check, but do agree that ImageTextureAtlas node belongs in Texturing component.
a. Does there need to information in the 18.2 Concepts section? If so what would it be called, Texture Atlas perhaps? Usually clauses in the Concepts section describe general principles of purpose, structure and use, especially for concepts applicable to more than one node.
b. If the ImageTextureAtlas is a set of closely related image slices stacked together as shown, it is not clear how texture coordinates apply.
c. Do the image slices have to be related, or can they be completely independent, thus using an atlas as a convenient data structure?
d. For MPRVolumeStyle, planes do not seem well defined.
e. Images and example scenes would help.
We originally thought we could adjust the draft spec prose but sorry, the gaps in our understanding are too great to overcome today.
Suggest we meet to further discuss, perhaps a separate teleconference? Best case scenario would be if preceding questions were answered by email and then we discussed face-to-face during Web3D 2019 in Los Angeles, perhaps during the 26 JUL working-group meeting period.
Thanks in advance for further work to help with these important additions.
v/r Don and Dick
On 6/12/2019 2:18 AM, Ander Arbelaiz wrote:
> Dear X3D WG members,
>
> I would like to propose a series of extensions/additions to the X3D Volume Rendering Component.
>
> These extensions were first presented on the Web3D2017 conference.
> Community-driven Extensions to the X3D Volume Rendering Component <https://www.researchgate.net/publication/317122169_Community-driven_extensions_to_the_X3D_volume_rendering_component>(full PDF copy available)
>
> On this paper the following additions were proposed:
>
> 1. New Proposed Nodes:
> 1.1 ImageTextureAtlas node
> 1.2 MPRVolumeStyle node
>
> 2. New Proposed Additions:
> 2.1 quality attribute to target device computation capabilities
> 2.2 allowViewpointInside attribute for inside exploration of volume data
> 2.3 sceneDepth attribute for hybrid rendering
> 2.4 cap attribute for Clipping
>
> So far, from those proposals I have already contribute in the X3DOM library with the following:
>
> *1.1 ImageTextureAtlas node
> *
> *1.2 MPRVolumeStyle*
> *2.2 allowViewpointInside attribute for inside exploration of volume data*
>
> Therefore, I think these additions (1,1, 1.2, 2.2) can be discussed for inclusion in future revisions of the X3D standard.
>
> I will prepare some public live examples as soon as I can, and will post them in the mailing list.
>
> I understand a more formal spec definition will be required. @Don Brutzman <mailto:brutzman at nps.edu> @Nicholas Polys <mailto:npolys at vt.edu>, any help and guidance on this regard will be appreciated. When required I could present/discuss these extensions on any next X3D WG meeting.
>
> An initial description of these extensions is at the end of this post.
>
> BR,
>
>
>
> 1.1 ImageTextureAtlas node
>
> Volumetric data, specially those obtained from a MRI or a CT scan, can be seen as a set of 2D image slices in an array. This data is usually stored in a 3D Texture. Unfortunately, in the web context, WebGL 1.0 does not support 3D texture data types. X3DTexture3DNode derived nodes can not be implemented with this API. An alternative structure is required for compatibility with the web.
>
> We propose a new ImageTextureAtlas node which allows to represent the 3D volume data by composing all the 2D slices into a single 2D texture. Instead of adding a Z dimension to the texture, all 2D slices are arranged in a matrix configuration into one image.
>
> Listing below shows the basic X3D definition of the proposed node.
>
> ImageTextureAtlas : X3DTexture2DNode
>
> SFNode [in,out] metadata NULL [X3DMetadataObject]
>
> SFBool [in,out] repeatS TRUE
>
> SFBool [in,out] repeatT TRUE
>
> SFNode [in,out] textureProperties NULL [TextureProperties]
>
> MFString [in,out] url [] [URI]
>
> SFInt32 [in,out] numberOfSlices 0 [0,∞)
>
> SFInt32 [in,out] slicesOverX 0 [0,∞)
>
> SFInt32 [in,out] slicesOverY 0 [0,∞)
>
> SFBool [in,out] hideChildren TRUE
>
> SFString [in,out] channels "R"
>
> SFString [in,out] sortOrder "NONE" ["ROW", "CHANNEL", “NONE”]
>
> }
>
>
> The following attributes are inherited by X3DTexture2DNode: metadata, repeatS, repeatT, textureProperties and url.
>
> The numberOfSlices attribute indicates the number of slices or the dimension in Z axis. SlicesOverX attribute indicates the number of slices along X axis or columns, while the SlicesOverY indicates the number of slices along Y axis or number of rows in the matrix configuration. This values must be provided by the user, since they cannot be deduced by the input image.
>
> The channels field defines in which color channel of the texture volume data is being stored. The default behavior is to store the volume data in the Red color channel by specifying the "R" value. When, a larger volume is required to be converted into an ImageTextureAtlas additional channels can be specified with "R", "G", "B", "A" characters. For instance, to store up to three times more data. A "RGBA" value in the channel field will indicate that all color channels of the texture are being used to store the data.
>
> Once multiple color channels are defined, the order in which 2D slices are tiled into the atlas must be set in the sortOrder field. The default behavior is to tile the 2D slices that represent the Z axis of the volume data in "ROW” (row-major) order. For each slice of the ImageTextureAtlas the next slice in the Z axis is the contiguous slice in the row of the matrix of slices. When the sortOrder is set to "CHANNEL" the next slice of the atlas in the Z axis direction is stored in the the contiguous color channel.
>
> A 2D image representing an ImageTextureAtlas of the MRI ventricles (Volvis)dataset (color inverted and contrast enhanced) as a set of 2D slices tiled into a matrix configuration
>
>
> Not only to represent raw volume data, the ImageTextureAtlas can also be used for gradient volume data and segmented volume data.
>
> Surface normals are specified to be provided as an additional 3D texture. Using the same approach as before, we can create an ImageTextureAtlas using the color channels of the 2D texture to store the vector information of gradient data. We encode the gradient vector for each pixel in the atlas in the RGB color channels {R: X, G: Y, B: Z}.
>
> Listing below shows an example declaration of the gradient data.
>
> <ImageTextureAtlas containerField="surfaceNormals" url="gradient.png" slicesOverX="8" slicesOverY="8" channels=”RGB” sortOrger=”NONE”> </ImageTextureAtlas>
>
>
> The containerField and URL attributes are the only modifications required for the ImageTextureAtlas declaration. The containerField attribute is used to target the volume data voxels field in a X3DVolumeDataNode or gradient data surfaceNormals field in a X3DVolumeStyleNode. This also requires, for the X3DVolumeStyleNode type nodes with a surfaceNormals attribute, to accept a X3DTexture2DNode as an input argument.
>
> The following Listing definition shows how X3DVolumeData derived nodes, such as the VolumeData, should allow a 2D texture input to accept the ImageTextureAtlas as a parameter.
>
> VolumeData : X3DVolumeDataNode {
>
> SFVec3f [in,out] dimensions 1 1 1 (0,∞)
>
> SFNode [in,out] metadata NULL [X3DMetadataObject]
>
> SFNode [in,out] renderStyle NULL [X3DVolumeRenderStyleNode]
>
> SFNode [in,out] voxels NULL [X3DTexture3DNode, X3DTexture2DNode]
>
> SFVec3f [] bboxCenter 0 0 0 (-∞,∞)
>
> SFVec3f [] bboxSize -1 -1 -1 [0,∞) or -1 -1 -1
>
> }
>
>
> The SegmentedVolumeData node allows the user to discern regions of the volume and apply different rendering styles to each one. The segmented region must be labeled per voxel.
>
> Textures provide a mechanism to input data to the GPU, updating directly the visualization. In the Web platform we can take advantage of the 2D canvas API to create or modify images that can be copied to the GPU. This is of special interest to reflect dynamic changes in the data and, for instance, to perform the construction of the atlas in the browser.
>
> Listing below shows an ImageTextureAtlas node declaration with a canvas. The hideChildren attribute hides the atlas image visibility from the user, but makes it available to be modified with JavaScript.
>
> <ImageTextureAtlas containerField="voxels" url="" slicesOverX="8" slicesOverY="8" hideChildren="true">
>
> <canvas id="v" style="width:2048px; height:2048px;">
>
> </canvas>
>
> </ImageTextureAtlas>
>
>
> 1.2 MPRVolumeStyle node
>
> The Multi-Planar Reconstruction (MPR) is a wide spread rendering technique. Essentially, it enables the user to define arbitrary planes slicing through the volume data. The rendering algorithm resamples the volume data to reconstruct the desired plane. Usually, the following planes are defined: Axial, Sagital, Coronal and Oblique.
>
> The following Listing shows the proposed X3D MPR volume rendering style definition.
>
> MPRVolumeStyle : X3DVolumeRenderStyleNode {
>
> SFBool [in,out] enabled TRUE
>
> SFNode [in,out] metadata NULL [X3DMetadataObject]
>
> SFNode [in,out] transferFunction NULL [X3DTexture2DNode, X3DTexture3DNode]
>
> SFBool [in,out] forceOpaque TRUE
>
> MFNode [in,out] planes NULL [MPRPlane]
>
> }
>
>
> An additional interesting enhancement to MPR is the ability to add a transfer function (TF) in order to illustrate or filter regions in the reconstructed planes. It is used to correlate the visual output with obtained results. The proposed MPR node defines a /transferFunction /attribute with the same functionality as the one already defined in the OpacityMapVolumeStyle. This node also defines a planes attribute to allow the user to declare arbitrary planes.
>
> The following Listing presents an arbitrary plane definition for the MPRVolumeStyle node. The /normal /attribute defines the normal vector of the plane, while the /position /attribute establishes the position of the plane from the origin of the volume in the normal direction.
>
> MPRPlane : X3DNode {
>
> SFBool [in,out] enabled TRUE
>
> SFNode [in,out] metadata NULL [X3DMetadataObject]
>
> SFVec3 [in,out] normal 0 0 1
>
> SFFloat [in,out] position 0.0 [0,1]
>
> }
>
>
> 2.2 allowViewpointInside attribute for inside exploration
>
> Allowing to place the viewers virtual camera inside the volume provides a new perspective to analyze the data. Some regions of the volume data can be occluded even after filtering the data with a transfer function (TF). We suggest to define a new attribute to enable or disable this behavior for performance purposes.
>
> The following Listing shows the definition with the proposed attribute.
>
> X3DVolumeDataNode : X3DChildNode, X3DBoundedObject
>
> SFVec3f [in,out] dimensions 1 1 1 (0,∞)
>
> SFNode [in,out] metadata NULL [X3DMetadataObject]
>
> SFVec3f [] bboxCenter 0 0 0 (-∞,∞)
>
> SFVec3f [] bboxSize -1 -1 -1 [0,∞) or -1 -1 -1
>
> SFBool [in,out] allowViewPointInside TRUE
>
> }
>
>
> Our proposal is to declare a new boolean attribute allowViewPointInside to all nodes inherited by the X3DVolumeDataNode. By default we suggest to allow the inspection of the volume data and if the user does not require to do so, it can be explicitly disabled. This feature allows to avoid additional computation.
>
> --
> <http://www.vicomtech.org>
> Ander Arbelaiz Aranzasti
> Researcher | Investigador
>
> aarbelaiz at vicomtech.org <mailto:aarbelaiz at vicomtech.org>
> +[34] 943 30 92 30Industry and Advanced Manufacturing | Industria y Fabricación Avanzada
>
> <https://www.linkedin.com/company/vicomtech> <https://www.youtube.com/user/VICOMTech> <https://twitter.com/@Vicomtech>
> member of: <http://www.graphicsmedia.net/>
> Legal Notice - Privacy policy <http://www.vicomtech.org/en/proteccion-datos>
all the best, Don
--
Don Brutzman Naval Postgraduate School, Code USW/Br brutzman at nps.edu
Watkins 270, MOVES Institute, Monterey CA 93943-5000 USA +1.831.656.2149
X3D graphics, virtual worlds, navy robotics http://faculty.nps.edu/brutzman
More information about the x3d-public
mailing list