[X3D-Public] [FreeWRL-develop] .x3z and minizip; bundling of resources for X3D binary

[doug sanden]

>
> Interesting progress on the freewrl developers list by Doug Sanden regarding the bundling of resources for a single .x3d file (such as images, audio, etc.) in a single zipped file.
>
> This type of approach, if a common best practice emerges, might do a lot for deployability and archivability of fully-complete X3D content. So we need to add it to the X3D Binary Compression effort.
>
> This thread is copied here with permission... Doug please feel free to tell everyone more about how this is working out. Thanks for your efforts.
>

.x3z model/x3d+zip and zlib/contrib/minizip
What:
I copied the idea from GoogleEarth which has a .kmz file format which is a .zip with a doc.kml in the root of the .zip, and images in /images and collada in /models. The .zip format is produced by programs like winzip, 7zip and zlib's /contrib/minizip zip and unzip, which can contain multiple files and folders in the archive. Minizip has API functions that can be called from a program.

How:
I do the same for web3d scenes. I'm currently using .x3z and model/x3d+zip for extension and mimetype, and zlib's /contrib/minizip functions in freewrl.2 version due out in October.
The key idea is to always name the starting scene doc.x3d and always put it in the root of the .zip. By root that means there are no extra folder names. 
So a player/tool can unzip to a temp folder and queue doc.x3d for parsing.

For Scene authors: if you are touching up an existing scene, you need to do what I call 'grooming the urls': as you re-arrange the scene and it's resources, you change the urls in the .x3d file to reflect any change to the relative paths. If you have an Anchor closure - another scene file points back to the starting scene- change that to doc.x3d
.x3z
- doc.x3d
- /images
- /???
I didn't refine or specify exactly the folder structure. The critical thing is doc.x3d in the root, and that your urls make sense. But probably something like this would make sense:
.x3z
- doc.x3d
- /images
- /models
- /audio
- /video
I developed a handy online web service for bundling existing x3d scenes with an http address:
http://www.cloudvr.net/x3z.html 
x doesn't work with .gzipped or binary scene files
x only uncompressed .x3d
* does the url grooming and scene folder flattening for /images (but puts .x3d in root, not /models)
* can do the bundling while you wait for download to begin

For Tool Developers:
When cleaning up the temp folder and subfolder -on exit, or scene change- don't forget to skip the ".." folder when looping over subfolders for deletion. Otherwise it could recurse up to the disk drive root, deleting everything on the disk. For that reason it might be helpful to limit the folder depth to 2 levels by convention, as an extra safety measure.
zlib source code has a subfolder for contributions /contrib/minizip. I extracted a simple subset of the files from there for use as library calls in freewrl. If you download the freewrl CVS code you'll see a freewrl/freex3d/src/minizip folder with the subset. 


Motivation: 
there's a trend away from webbrowser plugins starting on 
mobile devices and some hints/rumors NPAPI plugins will be deprecated in
 Firefox and Chrome. So instead of a plugin being passed an http:// url, and 
pulling resources one by one, a web browser may offer to DownloadFile 
and SaveAs. Then you would open it as a local file with your web3d tool.
 That works well for scenes with no url resources. If it has urls with 
relative paths, the program that opens the local file will have no idea 
of the original base url, and won't be able to pull/download relative resources.
By bundling all resources into a single BLOB (binary large object) archive file, the DownloadFile>SaveAs scenario works.

For web3d specification committee:
I didn't think too deep or hard when coming up with .x3z and model/x3d+zip, so if there's something better I'm all ears. I'm not an expert in .zip but I think there's different compression schemes, some of which are proprietary. I didn't check but assume the zlib/contrib/minizip stays with non-proprietary processes, and I'm already using it, so it seems like a good benchmark/reference when in doubt.
However I'm not planning any IANA registration as a private or freewrl mimetype because it would degrade into a vanity or vendor prefix on the MIME and cause proliferation of equivalents. So I would be very pleased to have web3d.org make an application for this or equivalent mime and extension so all web3d tool and scene authors can benefit from a shared format.

Thanks,
Doug Sanden

> On 9/4/2013 7:42 AM, Brutzman, Donald (Don) (CIV) wrote:
>> Wondering, what are resources? Is that a "mac thing" ?
>>
>> Of relevance: there already exists an .x3db binary encoding, and an option to use gzip.
>>
>> Of further interest: we have a call for contributions for compressing X3D further, perhaps you want to contribute further.
>>
>> http://www.web3d.org/realtime-3d/working-groups/x3d/compressed-binary-encoding-activity
>>
>> http://www.web3d.org/realtime-3d/news/3d-graphics-compress-call-contributions
>>
>> http://www.web3d.org/realtime-3d/working-groups/x3d/compressed-binary/x3d-compressed-binary-encoding-call-contributions
>>
>> http://web3d.org/wiki/index.php/X3D_Binary_Compression_Capabilities_and_Plans
>>
>> Also: we put the finishing touches on a mime type update for X3D during last week's working group teleconference and submitted it to IANA. Feedback also welcome on that.
>>
>> http://web3d.org/wiki/index.php/X3D_MIME-Type
>
> p.s. Leonard and I have received and responded to initial IANA response on mime type update during last week's Wednesday teleconference for X3D working group, that is moving right along too.
>
>> Recommendation: unless this is only a freeWrl-internal capability you are considering, you might want to bring this discussion to the x3d-public list.
>>
>> Thanks for your efforts.
>>
>>
>> -----Original Message-----
>> From: doug sanden [mailto:highaspirations at hotmail.com]
>> Sent: Monday, September 02, 2013 3:09 PM
>> To: freewrl-develop at lists.sourceforge.net
>> Subject: [FreeWRL-develop] .x3z and minizip
>>
>> With some mods -not checked in yet- I could load a .x3z file -.zip of a doc.x3d and its resources- with freewrl.
>>
>> -Doug
>> more..
>> .x3z is just .zip. Reason for a different suffix: so it can be registered as a MIMEtype handler and directed to the appropriate application ie. freewrl -similar to how GoogleEarth handles .kmz which is a .zip file with a doc.kml and its resources.
>>
>> minizip - zlib source code comes with a /contrib folder and /contrib/minizip is a handful of .c files which can archive/unarchive files from a .zip file format:
>> ioapi.c/.h, zip.c/.h, unzip.c/.h (there are others, not needed) It relies on and works well with zlib.
>>
>> I needed 170 lines of user code to use the unzip functions. Basically I handle resource type resm_x3z and unzip it all to a temporary folder, and resitem_enqueue(tempfolder/doc.x3d).
>> The rest of freewrl works normally.
>>
>> touchups needed:
>> - doesn't delete the temp folder yet
>> - works with local files .x3z, but not yet with http .x3z