[x3d-public] X3D features missing, but desired, by a game engine

[doug sanden]

So summary:
modulation (result := material.rgb * texture.rgb)  (vs current replace) in web3d v4+ specs for v4+ scene files would make it easier to implement properly/consistently across browsers and platforms, and harmonize with other 3D and texture tools.
Sounds good to me.
-Doug

________________________________________
From: Michalis Kamburelis <michalis.kambi at gmail.com>
Sent: April 14, 2017 4:41 PM
To: doug sanden
Cc: x3d-public at web3d.org
Subject: Re: [x3d-public] X3D features missing, but desired, by a game engine

2017-04-14 23:24 GMT+02:00 doug sanden <highaspirations at hotmail.com>:
>> In any case, neither of these are valid arguments anymore, I think.
>> *Everyone* right now is by default multiplying the "texture * material
>> color" (Blender default material, Unity3d default material, Collada
>> default material, Spine... every format I can think of), otherwise it
>> seems like a wasted opportunity. And this multiplication has a
>> zero-cost on GPU, as far as my practice shows:)
>>
>> So, I would like to update the X3D to also do this by default. It also
>> makes the spec more consistent (making multi-tetxuring more in line
>> with single-texturing; the multi-tetxuring nodes always by default
>> multiply, for both grayscale and RGB).
>>
>
> Michalis,
> Sounds good.  f(web3d file version)?

This is a change that breaks compatibility (even if the existing
browsers are already widely inconsistent in implementing the current
specification-mandated behavior, as my test "9.
material_color_mixed_with_texture_color" on
https://castle-engine.sourceforge.io/x3d_multi_texturing.php shows).

So, yes, I would propose to make the change dependent on declared X3D
version, i.e.

if X3D.major_version >= 4 then
  result := material.rgb * texture.rgb
else
  result := <do whatever you were doing previously, to not surprise
your own users>;

>> > Detecting grayscale image after image library has loaded the image in a generic way:
>> > recent freewrl runs a cpu intensive test for R==G==B when loading image files, in separate loading thread
>> > - starts with first pixel, runs till it finds a difference or end of image
>> > - if no difference, sets number of color channels to 1 (else 3) (plus alpha). Then in above code
>> >                         channels = getImageChannelCountFromTTI(node->appearance);
>> > returns channel count for influencing use in shader
>>
>> I am doing it in Castle Game Engine too, to more efficiently deliver
>> images to GPU (grayscale image, without alpha, is 4x smaller than RGBA
>> image after all :).
>>
>> But I don't think that such detection should be used to decide whether
>> the texture is "Intensity" or "RGB" texture (with respect to the
>> http://www.web3d.org/documents/specifications/19775-1/V3.3/Part01/components/lighting.html#t-Litcolourandalpha
>> ). This makes it impossible for texture author to mark the texture as
>> RGB in GIMP/Photoshop (thus forcing the "multiply" behavior in X3D, by
>> storing the image using RGB).
>>
>
> Image header > Ouch - assuming freewrl can ever conform to web3d v4.0, it would need work:
> 3 of freewrl's supported platforms use image loader libs that load generically - gdiplus image loader (win32), WIC image loader (uwp) and Imlib (linux). All I get is RGBA, rows x cols. So I don't see the header details.
> For a few other platforms freewrl is using separate jpeg, png, gif libs.

Castle Game Engine is in a similar boat. We load our images using
various APIs, and the image header is not always (easily) available.
But I think that "guessing" the image type by analyzing the content is
also not good (e.g. it makes it impossible to mark a clear white
texture as RGB in GIMP/Photoshop).

The advantage of my proposed change for X3D browser creators is that
it would no longer matter. You no longer need to be concerned whether
the texture is grayscale or RGB.

- If you want, just load everything as RGB.
- If you want, load as grayscale (single intensity per pixel) only
images declared as grayscale in their header.
- If you want, analyze the image contents, and auto-detect grayscale
as a "special case" of RGB.

The end result (for user) of all the approaches will look the same.

>
> Q. what are the relative pros and cons of putting an additional field on ImageTexture node to say how it should be used / how many channels should be used ./ how many channels to interpret it as having? For example all the other renderedImaged etc how would they work either way?
> Would that help with renderedTexture somehow?

RenderedTexture has a "dimensions" field that specifies (among other
things) the number of color components (1-4). So, in this case, the
user explicitly declares whether it's RGB or grayscale.

Adding an additional field to the ImageTexture would solve the problem
partially:

- Browser implementors have an easier job: instead of looking at the
texture header, we look at the ImageTexture field.

- But X3D authors remain in a poor situation: unlike all the other
asset creation and interchange tools/formats, X3D would still treat
RGB textures in a different way (not doing component-wise
"material.rgb * texture.rgb").

  Collada, Spine, Blender... all behave differently from X3D, and if
you convert from them to X3D, the resulting material interacts
differently with the texture. (Unless you use an X3D browser like
view3dscene that always multiplies "material * texture", knowingly
breaking the specification.)

So, I would opt for fixing this completely:)

>
> Or any other options besides image headers? Does someone have a sniffer function for getting the details without needing an image lib? How hard would that be?

The way I see it: Yes, you can read only an image header, and analyze
it. It's easy ... for 1 image format:) Do it for 10 image formats, and
you're effectively reimplementing a significant part of your
image-loading library:)

You're also wasting resources -- you open each texture file twice. If
the data came from a stream (like a socket), you need to be able to
rewind it (so, you probably save it to a temporary location).

All this additional work, and you still have only solved the 1st part
of the problem ("how to detect texture grayscale / RGB"), while X3D
users have a problematic behavior:)

Best regards,
Michalis