[x3d-public] [Semantics] FW: Spatiotemporal queries, creating objects and relationships onthefly. Creating meaning and semantics.

[John Carlson]

Sent from Mail for Windows 10

From: John Carlson
Sent: Monday, June 10, 2019 3:06 PM
To: semantics at web3d.org; aono at tut.jp
Subject: RE: Spatiotemporal queries, creating objects and relationships onthefly. Creating meaning and semantics.

Interesting!

https://www.aaai.org/ocs/index.php/AAAI/AAAI18/paper/download/17059/16287

John

Sent from Mail for Windows 10

From: John Carlson
Sent: Monday, June 10, 2019 1:39 PM
To: semantics at web3d.org; aono at tut.jp
Subject: Spatiotemporal queries, creating objects and relationships on thefly. Creating meaning and semantics.

In addition to Geo stuff, a lot of stuff has been done with spatiotemporal.   My guess is our main work will be classification of various 3D shapes or transforms into semantics, and vica versa.   We now have an artificial neural network problem problem: create Shape2Meaning and Meaning2Shape.  Dr. Aono wants to do Speech2Shape.   I think we should do some Mechanical Turk and help him do labelling.   Thus we can do something just like Shape2Vec to convert Shapes to Meanings:  https://dl.acm.org/citation.cfm?id=2980253 and thus Meaning to Shapes if we run it in reverse.  Our own Dr. Masaki Aono is referenced in that paper!

https://web.engr.oregonstate.edu/~erwig/st/ (spatiotemporal queries)
https://arxiv.org/abs/1803.08495 (Text2Shape)
http://openaccess.thecvf.com/content_cvpr_2018/papers/Kanezaki_RotationNet_Joint_Object_CVPR_2018_paper.pdf (categorizing 3D shapes)
Are we doing SHRDLU all over again?  http://hci.stanford.edu/~winograd/shrdlu/

We are also looking for relationships between 3D shapes, correct?  https://towardsdatascience.com/whats-new-in-deep-learning-research-neural-networks-that-detect-relationships-between-objects-4758e07b7e64 (deepmind).

So I think we can probably buy a trained neural network or neural network model to train.

So we have ways to convert models to meaning, possibly available to us.   Should we reach out and ask for a neural network model?

So where does it get us in the generation and validation of 3D content.   We have a way of converting meaning to shapes.  Creating brand new non-existent relationships between objects perhaps not shown yet.   May need to run CNNs (convolutional neural networks) on videos.   Thus we need a GAN (generative adversarial network) which progressively creates the relationship between shapes.   A generator which creates the relationship, and a discriminator which says whether the relationship has been made or not.  I think this may be a unique use of neural networks, but do your own searches.  Please report!

I am VERY interested in creating objects and relationships (aka meaning) for 3D worlds with whatever technology is best.  Right now, Nvidia has a way to pick a color which is a category, and it will create a 3D picture from the shape and type of the color https://www.youtube.com/watch?v=p5U4NgVGAwg but the user has to create the object.  I will have to add neural (GAN) generators in addition to hyper generators, stochastic generators, chaotic generators and quantum generators.

See https://github.com/MustafaMustafa/cosmoGAN for a GAN which creates guess what?  Virtual universes! Cosmologically speaking, I think unfortunately.  (or is it related to CosmoCode?). https://phys.org/news/2019-05-cosmogan-neural-network-dark.html

I am willing to collaborate on a GAN that creates realistic objects (of many types) out of categories of nodes in X3D or the semantic web, and create relationships (of many types) between those realistic objects.  In particular I am interested in GANs which create RDF, RDFS, OWL, SWRL, SPARQL, and finally, games (similar to EGGG, Ludi and the game description language, GDL).

How do you create meaning and semantics?  Do you really know?  That should be in the charter of the working group.  If the person can’t answer the question, they are not invited to the group.

I ask myself.  How did I create this email, and what message am I trying to get across.

Who does the strawman, the computer or the human?  GAN discriminators still require some realism to get better.

John 



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://web3d.org/pipermail/x3d-public_web3d.org/attachments/20190613/b5e8615d/attachment-0001.html>