X3D Binary Compression Capabilities and Plans

Motivation

Lots of work has already been accomplished using the X3D Efficient Binary Encoding (EBE) standard.

X3D has numerous coherent approaches already available that meet author requirements for a general Web-based 3D transmission format.

We are combining Efficient XML Interchange (EXI) data compression with Fraunhofer streaming and geometry compression. This new encoding will create dramatic new capabilities that provide full functional compatibility with existing X3D encodings in XML, ClassicVRML and the previous binary format.

We are now ready to demonstrate and standardize multiple interoperable improvements for a final standard.

Strategy and Milestones, 2013-2017

• Annual. We reviewed goals and developmental capabilities at the Web3D Conferences and SIGGRAPH in 2013, 2014 and (soon) 2015.
• 2013. We accomplished our strategic goal to define X3D Compressed Binary Encoding (CBE) requirements and planned all steps.
• 2014. We received multiple contributions for geometric compression and progressive streaming for X3D.
• 2014. Efficient XML Interchange (EXI) is a fully approved W3C Recommendation with multiple implementations (both commercial and open source).
• 2015. Decision: retain existing Compressed Binary Encoding (.x3db) for model stability, add Efficient Binary Encoding (.x3de) for improved capabilities.
• 2015. All needed components are in hand. Shape Resource Container (SRC) efforts ongoing.
• 2016. Continued participation in EXI Working Group. Need to review Call for Contributions, implementations and specification efforts.
• 2017. SRC is now ready to continue work. Consider comparing BLAST to SRC.
• Target completion? With sufficient member contributions, we may be able to finish by the end of the year.

Call For Contributions

X3D solutions currently support a wide range of author requirements. Further improvements and standards-based partnerships are possible for achieving broader industry interoperability.

The X3D Compressed Binary Encoding Call For Contributions to help accomplish these achievable goals has been released. We are seeking next-generation improvements that further advance the technical capabilities of the X3D Graphics International Standard.

• Press release summary

Prior work is essential, useful and relevant.

• The first-generation X3D Compressed Binary Encoding Request For Proposals (RFP) (currently unavailable) from August 2003 illustrates this steady evolution.
• The first-generation process successfully created the current X3D Compressed Binary Encoding (CBE) International Standard.

All submitters must meet certain requirements prior to consideration.

• The Intellectual Property Rights (IPR) protections for X3D specification.
• Patented technologies can be considered, but only when eventual use will be royalty free for X3D use (if eventually accepted).
• Submitters can restrict access to patented submissions during member-only working group review, if desired.

Existing Compression Usage for X3D and VRML97

A solid foundation exists for continued progress.

• Approved ISO standard Compressed Binary Encoding (CBE) for X3D.
  • Based on ISO standard Fast Infoset (FI) for XML compression.
• Optional, alternative gzip compression and MIME Type definitions for X3D.
  • XML encoding (.x3dz/.x3d.gz), ClassicVRML encoding (.x3dvz/.x3dv.gz) and Compressed Binary encoding (.x3db.gz) file extensions.
• Optional, alternative gzip compression for VRML97.
  • Original compression technique of applying gzip to .wrl compressed VRML97 files was called .wrz.
  • This emerged as a common practice when gzip was originally used. No formal specification of .wrz or corresponding mime type was produced.
  • Occasionally authors might also gzip .wrl files while retaining the .wrl file extension.

Multiple X3D compressed binary encoding (CBE) codebases exist for producing .x3db scenes under the existing standard. However, conformance interoperability between different implementations is not well tested.

Web3D will not formally advance a next-generation X3D binary compression format as a candidate standard until current interoperability has been satisfactorily demonstrated for the existing standard.

Polygon Reduction and Geometric Compression

Formal specifications:

• The X3D Compressed Binary Encoding defines extensible, repeatable compression and decompression algorithms for data-flow production chains. Multiple type-specific field encoders and geometric compression algorithms can be used in concert, but no specific geometric compression algorithms are referenced or required.
• X3D Canonicalization (C14N) provides standardized formatting so that digital signatures are not thwarted by whitespace variations.

Best practices and examples:

• CAD Distillation Filter (CDF) technique allowing successive refinement of large X3D scenes into tighter X3D scenes.
• A highly effective exemplar algorithm for Coding Polygon Meshes as Compressable ASCII is demonstrated in the Experimental Binary Compression examples.
• Multiple CAD Distillation Filter (CDF) algorithms implemented in Xj3D and X3D-Edit as open-source.
• The Chisel VRML Optimisation Tool has an excellent set of data-reduction tools for VRML that are worth repeating for X3D.
• Multiple other conversion and translation tools available with supporting capabilities.
• Following submission of candidate technologies, we will build a table of 3D graphics compression benchmark results, providing a full comparison of existing tools (compression ratio, computational complexity, memory usage, etc.)
• MPEG4 capabilities are welcome, if IPR prerequisites are met. Discussion to date has indicated that Royalty Free (RF) solutions are not available.
• Related efforts reported in academic papers and Web3D conferences need to be reviewed and reflected here

Data-Centric Binary Encodings

The X3D Compressed Binary Encoding (CBE) uses the ISO Fast Infoset (FI) standard to compress XML information, which is how the CBE maintains equivalent expressive power with other X3D scene encodings.

• Plan to add a further-improved X3D Compressed Binary Encoding using now-approved W3C Recommendation for Efficient XML Interchange (EXI).
• Web3D contributed to EXI working group and XML Binary Characterization working group efforts.
  • 3D Model Compression, Serialization and Transmission use-case requirements are defined and met by EXI.
• Design includes compatibility with CDF techniques, XML Encryption, and XML Digital Signature for author authentication.
  • Relevant example scenes maintained as part of X3D Basic Examples Archive - Security.
• These capabilities meets most needs of digital authors for digital rights management (DRM).
• Review Open Geospatial Consortium (OGC) requirements to determine whether additional optimizations might improve the already-excellent coverage of geospatial datasets
  • Also explore the possibility that additional compression algorithms may show better efficiency for geospatial datasets
• Following submission of candidate technologies, we will build a table of data compression benchmark results, providing a full comparison of existing tools (compression ratio, computational complexity, memory usage, etc.)

Network Streaming

• Multiple capabilities are already available in X3D for flexible network transmission.
  • Anchor, Inline, LOD, LoadSensor, Script and Prototype nodes support successive retrieval of content once initial model is displayed.
• Progressive-mesh geometric streaming technologies hold definite interest. Mesh-compression algorithms that can be applied to existing X3D polygonal data will be the most valuable.
• Need to demonstrate whether http/https and local-file url retrieval are sufficient for a network protocol for use cases of interest.
  • Other network protocols (Web sockets, P2P channels, etc.) might be possible or needed, but only if security restrictions and implementation/deployment can be handled satisfactorily.
• Javascript Object Notation (JSON) might be suitable for simple streaming of X3D scene-graph data via Script node or external HTML page, which can be useful for progressive mesh and other incremental network-update approaches.
  • Support in clients and servers is becoming widely available.
  • JSON might have other uses as well, see Khronos work on Collada2JSON and glTF. This is an active area of work.

Resource Bundling

• Mechanisms for bundling multiple files (e.g. X3D scene, Inlined subscenes, image files, audio and video files, etc.) into a single archive file will be considered.
• Similar bundling capabilities are emerging for uncompressed X3D scenes embedded within HTML pages.
• Benefits support improved deployability and archivability of X3D content
• Technical issues include .gzip versus .zip, bundling of metadata (demonstrated by Java .jar and .war formats), etc.
• The FreeWRL project is implementing and exploring possible approaches to this challenge

X3D Implementations and Benchmark Testing

The following tools implement the X3D Compressed Binary Encoding (CBE) standard.

• Two independent open-source implementations available.
  • C++ codebase: XIOT X3D Input Output Tool library.
  • Java codebase: Xj3D.
• Ongoing status is maintained on Web3D-member wiki pages:
  • Player support for X3D components
  • Tool support for X3D components
• Several thousand X3D Examples are available in .x3db form, encoded by Xj3D.

Existing assets can be improved to establish a comprehensive X3D compression benchmark suite.

• Such a suite can facilitate testing of new contributed capabilities.
• We plan to automate conversions and comparisons, using Xj3D XIOT and other submissions, for cross-check interoperability and validation testing.
• It will be easy to automate such a suite using our many example scenes and the already-existing nightly build (continuous integration) processes.
• This will automate the production of results summarized in the compression performance tables constructed as part of the call for technologies.
• Primary metrics are file size, conversion speed and memory requirements. Visual inspection of result quality will also be provided.

Looking Ahead

Work in Progress.

• Web3D's X3D and CAD Working Groups each have member commitments to pursue this continued innovative work during 2013-2014.
• We are always keen to consider common, sharable technical strategies with other industry standards. Web3D Consortium has been waiting for a response from The Khronos Group and MPEG standards SC 29 committee since a joint meeting at SIGGRAPH conference August 2012. To date, formal MPEG4 communications have indicated that a royalty-free solution is not possible. Khronos work appears likely but royalty requirements and design compatiblity remains unclear.

Meetings.

• The first weekly X3D Working Group meeting of each month is dedicated to an X3D Futures discussion, including X3D Binary Compression Capabilities and Plans.
• The Web3D 2013 conference included an open meeting on 3D Transmission Formats, to be held Wednesday 19 June 2013 in San Sebastian Spain.
  • Responses to the X3D Binary Compression Call for Contributions will be reviewed and candidate technologies will be discussed.
  • Inquiries and presentation requests can be sent to the X3D Working Group Cochairs.
• We further plan to meet publicly at the annual SIGGRAPH Conferences.

Adding example Use Cases will help evaluate the value of each contribution.

• Technical requirements are important for performing comparative measurements, but sometimes they don't tell the whole story.
• Also defining work-flow authoring requirements for compressed X3D scenes is helpful, providing useful criteria for evaluating technical capabilities. Use cases enable us to determine whether each possible improvement effectively meets a declared need.
• X3D authors are asked to help us document common workflow practices. We will extend the requirements to build an updated set of use cases to help evaluation. The emerging next-generation standard is being designed to provide the best possible value to Web authors, users, and software developers.
• Once the basic framework updated, we will ask each working group to identify special case examples for demonstrating value, avoiding problems, or further optimization.

Follow-on work

• Provide an additional X3D MIME Type submission to IANA
• Update references in all other X3D specifications

Contributions Received

• Mesh encodings for X3DOM - Recent Advances Fraunhofer provided a comprehensive list of proposed improvements on an X3D Working Group teleconference, 12 November 2013
  • Further detail available at | X3DOM Progressively Ordered Primitives (POP)]