[x3d-public] Programming by Demonstration. How LLNL's Programming By Demonstration Project (UT or TWB/TE) did/did not create interactive programs

Sun Jan 3 16:48:07 PST 2021

Christoph,

I think it is necessary that we first specify what it is we wish to define in a 3D application network protocol. I see discussions relating to different layers, which can make it a little confusing. 

Also, I feel that JavaScript or ECMAScript is probably welcome on the client side, since it works so well with X3D, so I am not sure I follow your comments in that regard, especially that avatars are generally scripted, even when played in classical players.

Perusing your recently published paper suggests that you are thinking of a different underlying network protocol than tcp/ip (i.e. SCTP over UDP), though I did not have time to look into it in details, being out of town for the new year. I will say that I am not necessarily against it, having considered SCTP myself for server-to-server communications, keeping in mind that this is a completely different layer of operation and that our purpose is X3D.

Furthermore, looking at the BS Contact example you published, I see that many fields and events are very similar to what is in use with X3Daemon, which is not surprising since they are essentially doing the same things.

Perhaps a good start would be to compile a list of what has been done in the past with the various systems, and see where they coincide in terms of functionality and nomenclature. For example things like VRTP and DIS are of interest, and could give us a good basis to build upon. GL

________________________________________________________

* * * Interactive Multimedia - Internet Management * * *

  * *  Virtual Reality -- Application Programming  * *

    *   3D Net Productions  3dnetproductions.com   *

From: x3d-public [mailto:x3d-public-bounces at web3d.org] On Behalf Of Christoph Valentin
Sent: Sunday, January 3, 2021 5:27 PM
To: John Carlson
Cc: x3d-public at web3d.org
Subject: Re: [x3d-public] Programming by Demonstration. How LLNL's Programming By Demonstration Project (UT or TWB/TE) did/did not create interactive programs

Hi John.

This is actually a great collection of pros and cons. Thank you. I am feeling free to copy to x3d-public.

I used the term "real" programming language to indicate I do NOT mean JavaScript.

It's not only about pre-compiling, but I also mean more efficient debugging by strict type enforcement.

kr
Christoph

-- 
Diese Nachricht wurde von meinem Android Mobiltelefon mit GMX Mail gesendet.

Am 03.01.21, 21:12 schrieb John Carlson <yottzumm at gmail.com>:

I know this is kind of long.  I’m not requiring you to read this.  I’m just trying to record some history of a misstep, and potential place of where to step next.

Also, it’s kind of a brain splat. Sorry for lack of higher organization.

I’m not sure what a *real* programming language is.   One that fast machine code can be derivable from?   A professional programming language?  One that is standardized?  One that fully implements lambda calculus?

Here are some things that might not be considered “fake” programming languages, either due to that they are either interpreted, or they have interpreted/compiled byte code, or are transpiled:  Java, C#, Python, TypeScript, JavaScript, Lisp, Scheme, HyperTalk, and AppleScript.  Also note that C, C++ are interpretable, so that’s not really a criteria for *real*.   See below for better criteria.

Perhaps we should consider the “programming environment” (UT) our team built at LLNL

1)      The “Abstract Syntax Tree” (AST) was serialized/loaded/saved/loaded to/from C++ and a flat-file database.

2)      The AST tree was executable, provided play() method for each operation.

3)      The AST was reversible, provided an unplay() method for each operation.

4)      We supported Loops and Conditional Logic.

5)      We supported reversible debugging and breakpoints in our authoring tool.

There’s a thing I call a “macro recorder,” which I would consider *not* a programming language.  In other words, all the things you do at the computer might be done in a “programming language”—maybe in the old days.   Perhaps what should be called a real programming language should be “evokes the halting problem.”

Perhaps there are things like “classic” or “legacy” programming languages.   I think those terms fit what is called “real” better.

Another thing that might be considered a *real* program is one that does not receive human generated events (touch, mouse, keyboard, etc.).  This would fit back-end snobbery better.  So what’s the middle ground between *real* programming language and a “macro recorder”?

This middle ground appears to be a “scripting language.”  So JavaScript and Bash might be considered scripting languages. But not macro recorders.   Both receive human generated events. Both run in batch mode.

So how do you classify our UT project?  We had an engine/interpreter did not accept user input for the most part. Our I/O library was mainly files in batch mode.  We had a user interface that was like a macro recorder of sorts, but there were not many human “events” to speak of in our programming language, just the user interface in our construction mode/workbench.  I would classify our project as a “scripting language” authoring tool, even if we did “script” C++ objects, but we had this totally back-end aspect without human interruption.

In other words, it seems like you’re going down the non-interactive route with a *real* programming language.  Apparently “interaction” is something only for “junior devs” last I heard.

I encourage you to look at Node.js (after January 6th) with VSCode.   Node.js is somewhere between a real programming language and a scripting language.

I am *for* keeping the AST around in easily parseable form.   Aka JSON, HTML, XML and “class grammars”—see Cameron Browne’s paper.

So how might one add “Event Handlers” or Dispatch table in a non-textual programming language like our UT project, stepping away from the typical graph of play() operations?   Do we need a Generate Event (desktop) Object? The UT project did contain a way to enter text on the engine’s console, but I don’t think it was used in production.  We had a way to insert current date and current time.  Perhaps at the highest level, one would have a table of Event to Recorder (process) or Procedure (thread).   Or it would seem simple enough to add event predicates to a Branch (left mouse == ‘clicked’), which might provide scope of for the events, bubble up, bubble down, etc.  The Distributed User Interface (DUI) had a submit button for the form, similar to HTML, otherwise, events were kept local to the widget, I think.

I think that perhaps that we separated the user interface from the engine was a critical one (yes, I realize that I was *for* this, following the MVC pattern).  An engine was more suitable for EDI/X12 translation, alas.   Now I understand we were creating a batch based translation system, not an interactive one.  We could have been translating interactions.  See “Model Transformation by Demonstration”—Sun, Gray from U of A.

How do we advance UT and MTBD beyond either batch mode (UT) or interactive mode (MTBD) of programs? How do we advance MTBD past the “Macro recorder” stage? How do we do scripting by example?

So perhaps there is not a classification of “real programming languages,” but rather a snobbery of what are called “real programs.” Perhaps “real programs” are ones that do not interact with the user.  But really, are you using a “real program” to read this email, write programs, etc.?

How does one interact with strings, numbers, dates, booleans, dates, documents/forms/files, predicates, windows, controls, threads, and processes as models in MTBD.

A follow-on project implemented a forms based distributed user interface (DUI).  I did try to create an InterViews and Web version of DUI with mild failure. 

I think a good approach would be to take https://ludii.games and study how they create games (for 3, below).   How might one create a user interface for creating ludii game description language?

Or just jump on the Lively Next/4/Kernel project.

My main goal would be to create games, I think, so I will step away from X3D and return to my multiuser programming of games.

Trala!

John

Sent from Mail <https://go.microsoft.com/fwlink/?LinkId=550986>  for Windows 10

From: Christoph <mailto:christoph.valentin at gmx.at>  Valentin
Sent: Sunday, January 3, 2021 5:43 AM
To: X3D Graphics public mailing <mailto:x3d-public at web3d.org>  list
Subject: Re: [x3d-public] higher level networking and potential for animated semantics.

Gina Lauren,

Citing

[...] 

One last note. While HTML5 has its place, I see it more as a stepping stone into lower lever applications. My opinion is that HTML5 can be a great way to help people get their feet wet, but to truly bring forward the power of 3D/VR into business apps, we need to be able to tap directly into modern graphics card APIs. Gina Lauren

[...]

I think the truth is manifold. X3D itself comes in many flavours.

1) WebGL based (poor performance, but will be supported by Browser Vendors, good for tinkering),

2) classical players written in *real* programming languages: better for professional usage.

3) If you need outstanding performance and development costs do not count (due to high number of sellings), then the classical game engine or even self-written game engine will fit your business plan

Nevertheless, the required 3D network protocol could be used by ALL of them, also the server software can be ONE. If the protocol is specified, then it can be used by any API, even concurrently in the same multiuser session.

We just need a volunteer, who implements and defines the protocol in his/her server (best 2 concurrent volunteers)

All the best

Christoph

Gesendet: Samstag, 02. Januar 2021 um 18:54 Uhr
Von: "John Carlson" <yottzumm at gmail.com>
An: "Christoph Valentin" <christoph.valentin at gmx.at>
Cc: "Don Brutzman" <brutzman at nps.edu>, "X3D Graphics public mailing list" <x3d-public at web3d.org>
Betreff: Re: [x3d-public] higher level networking and potential for animated semantics.

Yes, sounds like a Granovetter diagram.

John

On Sat, Jan 2, 2021 at 11:29 AM Christoph Valentin <christoph.valentin at gmx.at> wrote:

FIRST you receive, THEN you give.

Every mother knows that.
--
Diese Nachricht wurde von meinem Android Mobiltelefon mit GMX Mail gesendet.

Am 02.01.21, 17:56 schrieb John Carlson <yottzumm at gmail.com>:

I’ve got some ideas based on the 5 love languages:

[ Chapman ]

Give/Receive Acts (services)

Give/Receive Time

Give/Receive Gifts (including space, money, food)

Give/Receive Affirmatives (praise, worship)

Give/Receive Touch       

Also consider Distribution, Aggregation

Sent from Mail <https://go.microsoft.com/fwlink/?LinkId=550986>  for Windows 10

From: John Carlson <mailto:yottzumm at gmail.com> 
Sent: Friday, January 1, 2021 11:11 PM
To: Christoph Valentin <mailto:christoph.valentin at gmx.at> 
Cc: Don Brutzman <mailto:brutzman at nps.edu> ; X3D Graphics public mailing list <mailto:x3d-public at web3d.org> 
Subject: RE: Re: [x3d-public] higher level networking and potential for animated semantics.

The 5 love languages [ed. Note: Protocols]--Gary Chapman).

Here’s the kind of human protocol I was talking about.   Enjoy working out the protocol, Christoph!

John

Sent from Mail <https://go.microsoft.com/fwlink/?LinkId=550986>  for Windows 10

From: Christoph Valentin <mailto:christoph.valentin at gmx.at> 
Sent: Monday, December 28, 2020 7:00 PM
Cc: John Carlson <mailto:yottzumm at gmail.com> ; Don Brutzman <mailto:brutzman at nps.edu> ; X3D Graphics public mailing list <mailto:x3d-public at web3d.org> 
Subject: Aw: Re: [x3d-public] higher level networking and potential for animated semantics.

Hi John, Gina Lauren, Don

Please confirm you have received this e-mail. I was encountering connectivity issues with my free-mail provider gmx.de lately.

@John

Thanks for your contribution at the S&P-ARK WIKI. I am considering to merge it with the position issue and to move them both together to the digits WIKI (yet to be created :-) ).

@All - regarding the MU protocol story

May I officially suggest to use RTP and SCTP as transport protocols and to define an own "application layer protocol", as we brainstormed together in spring 2019 and I am currently going to describe in more detail?

I am going to be finished with the draft description by next weekend, but be AWARE: this is all phantasy, nothing has been implemented for this idea.

Comments are welcome already now.

https://github.com/christoph-v/spark/blob/master/ALP/01-ALP-over-RTP(BCP).pdf

Kind regards,

Christoph

P.S.: since my employer is not interested in 3D graphics at all, I can guarantee to be unbiased ;-)

Gesendet: Samstag, 26. Dezember 2020 um 13:45 Uhr
Von: "GL" <info at 3dnetproductions.com>
An: "'John Carlson'" <yottzumm at gmail.com>, "'Don Brutzman'" <brutzman at nps.edu>
Cc: "'X3D Graphics public mailing list'" <x3d-public at web3d.org>
Betreff: Re: [x3d-public] higher level networking and potential for animated semantics.

John, Thank you for referencing some of my work. VR business has been a near life time goal of mine. Like you, I have realized many years ago that I couldn't do it alone. I was, however, very fortunate to have a group of X3D developers to share my ambitions. That culminated in the current state of 'Office Towers' which is still, unfortunately, not anywhere near the goals we had originally set out (circa 2008).

It is no secret that I have supported open standards for 3D on the web since the very beginning, 3D being a passion I had even before the web became a thing, back in the days of BBSs and the 'Remote Imaging Protocol' language for vector graphics over ASCII. I could also speak of several of our team members' impressive background.

But even with such a good team and all the drive we had to really make a difference, it wasn't enough. We needed an even bigger team that would have included more people in accounting, graphics design, marketing, sales, etc. with the ability to devote significant time to the project. In other words, I underestimated the scope of the project while over-estimating the staying power of people working from the goodness of their heart.

Perhaps we should've tried to seek funding, but seeing how other seemingly well funded endeavors failed almost consistently, this was never a decision I was prepared to make, and quite possibly the single most important reason why Office Towers remains up to this day.

However, we are now in a different time. I believe the ideas you brought forward are good if they can be integrated into sustainable business models and backed by reasonable injections of venture capital. Otherwise, enterprises with such attributes will always have the upper hand in the marketplace and continue to make it difficult for others to achieve a meaningful presence. 

One thing I have learned for myself is that 'timing is everything'. Maybe now is the time for open standards deployment on a mass scale. Like many, I too long for standard 3D network protocols, and I like that you seem to be suggesting something in the lines of 'interaction protocols'. 

One last note. While HTML5 has its place, I see it more as a stepping stone into lower lever applications. My opinion is that HTML5 can be a great way to help people get their feet wet, but to truly bring forward the power of 3D/VR into business apps, we need to be able to tap directly into modern graphics card APIs. Gina Lauren

________________________________________________________

* * * Interactive Multimedia - Internet Management * * *

  * *  Virtual Reality -- Application Programming  * *

    *   3D Net Productions  3dnetproductions.com   *

From: x3d-public [mailto:x3d-public-bounces at web3d.org] On Behalf Of John Carlson
Sent: Wednesday, December 23, 2020 9:09 PM
To: Don Brutzman
Cc: X3D Graphics public mailing list
Subject: [x3d-public] higher level networking and potential for animated semantics.

Long semi-rant.   Take aways:   Let’s animate semantics.   Let’s create a “12”-level networking stack (would JINI be appropriate?).  Let’s create apps and networks that morph to counteract malware.

I do think that X3D sits at what I call “The application level” of networking, and it has a strong position there.   I don’t know any other product with such a strong 3D presence.   Second life and apps selling “3D chat” seem to have faded.  NVIDIA’s certainly pushing towards 3D transfer near the application level.   As we have seen with other applications on the web, these application level transfers should be visible to the developer for debugging and verification.

I can see a merging of remote work and 3D.

There are several types of work, research, design, development, integration, quality assurance, deployment, and maintenance.   Something like officetowers.com might provide insights into how we might integrate 3D into our software work.   I am very sorry that I fumbled many times in creating an environment suitable for this.   Thank goodness I learned the lesson that I couldn’t do this all by myself.

I think we need to step up from the application level of networking and think about more than client-server programming.   We need higher level protocols, perhaps human patterns or protocols for building human networks and systems that can survive COVID-19, 20, 21, ...  we can’t, for the sake of the survival of our species, give up space exploration.   But perhaps we need to explore more mystical paths, yet also provide a charitable path as well (mixed?).

The HTML5 environment has progressed far enough along I think, to support “3D enterprises” or “3D commerce.”  Obviously everyone is trying to see how 3D fits into remote work.   At the same time, I think we should consider how virtual worlds affect our belief systems and governments.   Has anyone shown a simulation of belief?   How important is honesty?  Semantics is only the first step.   What if we could animate semantics?   This would likely be a very big project.   Perhaps we could make it a mobile distributed project.

What lessons were there in Open Wonderland?

I think Google’s “generic” approach is appropriate for the public.   I do not think we should make all organizations generic lest they fall prey to virus or cyberattack.

Imagine if we still had the stovepipe systems of the 60s, 70s, and 80s.   Anyone remember “Cuckoo’s Egg”?

Is anyone selling virtual estate besides Second Life?

Does the Internet need an “off” switch?

Don’t mind me.   I live among the corn and beans.

Don’t mind me, someone wrote metamindwriter on my mind today.   A new word!   I feel like Neale Walsch.   I’m not a writer, I’m a mathematician!

I do thing we need a verification process and fingerprinting for cyberattack tools.

I’m thinking we need applications that can morph yet maintain functionality—take a page out of malware playbook.

John

_______________________________________________ x3d-public mailing list x3d-public at web3d.org http://web3d.org/mailman/listinfo/x3d-public_web3d.org

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://web3d.org/pipermail/x3d-public_web3d.org/attachments/20210103/a5b5216b/attachment-0001.html>