[X3D-Ecosystem] OSC mocap support in X3D Browsers

[John Carlson]

We’ll probably need the specifics for your device. We will need protocol
dumps, probably via wireshark.  AFAIK, transmission is over UDP/IP, but I’m
wondering about the physical layer.  If it already displays in a browser,
great,. But I don’t know if it supports the UDP version in the browser,
specifically QUIC.  I don’t know if UDP is available via websockets.  It
isn’t.  That means we need an OSC server that talks both TCP or HTTP/S
(hopefully QUIC) and UDP (for OSC).  This doesn’t appear to be better than
DIS.  DIS uses UDP or a web gateway.  I don’t really know if DIS handles
HAnim.

So I the best approach is to set up MediaPipe with OSC and start
experimenting. Any webcam should do, and not everyone has a dedicated mocap
device: Once we get familiar with OSC and MediaPipe, then we can try to
integrate your specific device:

Google AI:

*MediaPipe does not have built-in Open Sound Control (OSC) functionality*,
but the two technologies are frequently used together through custom
integrations to create real-time, gesture-based control systems for
multimedia applications.
How They Work Together
Developers commonly use custom scripts (often in Python or JavaScript) to
bridge the gap between MediaPipe's machine learning outputs and the OSC
protocol:

   1. *MediaPipe Processes Data*: MediaPipe is used to perform real-time
   computer vision tasks like hand, face, or pose tracking, extracting
   landmark coordinates or gesture classifications from a camera feed.
   2.
      - *Script Converts Data*: A custom script then reads these outputs
      from MediaPipe and formats them into OSC messages (address patterns and
      arguments).
   - *OSC Transmits Data*: The OSC messages are sent over a network (even
      on the same computer) to other applications that can receive OSC
data, such
      as Max/MSP, PureData, Ableton Live <https://www.ableton.com/>, or
      TouchDesigner.
   - *External App Interprets and Acts*: The receiving application
      interprets the OSC messages to control various functions like sound
      synthesis, audio effects, video manipulation, or interactive graphics.


Common Implementations and Resources
Several open-source projects and tutorials provide ready-made solutions for
integrating MediaPipe and OSC:

   - *mediapipe2osc*: An open-source project that uses Google's MediaPipe
   for 3D skeletal tracking and streams the data points via OSC.
   - *cansik/mediapipe-osc*: A GitHub <https://github.com/>repository with
   various MediaPipe examples (pose, hand, face) that stream their detections
   over OSC.
   - *Handmate*: A browser-based controller that uses MediaPipe for hand
   tracking and outputs Web Audio, MIDI, or OSC messages.
   - *Max/MSP Integration*: The jweb-mediapipe library allows Max/MSP users
   to directly leverage MediaPipe's landmark detection within the Max
   environment, which inherently supports OSC communication.
   - *Python Libraries*: Projects often use the python-osc library in their
   Python scripts to send gesture data to other applications.

These integrations allow for the creation of sophisticated, low-latency,
gesture-controlled interactive systems without requiring specialized
hardware beyond a standard webcam.




On Wed, Nov 19, 2025 at 1:51 PM Bergstrom, Aaron <aaron.bergstrom at und.edu>
wrote:

> https://ccrma.stanford.edu/groups/osc/index.html
>
>
>
> *From:* John Carlson <yottzumm at gmail.com>
> *Sent:* Wednesday, November 19, 2025 1:29 PM
> *To:* X3D Ecosystem public discussion <x3d-ecosystem at web3d.org>
> *Cc:* Bergstrom, Aaron <aaron.bergstrom at und.edu>
> *Subject:* Re: [X3D-Ecosystem] OSC mocap support in X3D Browsers
>
>
>
> While I have a mocap protocol of my own design, I don’t have the OSC mocap
> protocol.  I still need to apply IK and calibration after my protocol
> reaches the browser to determine rotations.  I did/do have viewers that
> take joint locations.
>
>
>
> I have 2 viewers:
>
>
>
> https://github.com/coderextreme/lircom/tree/master/src/main/java/impact
>
>
>
> (sorry, Java, no X3D and no rotations).
>
>
>
> If you have a pointer to OSC protocol or API, I will see what i can do.  I
> am interested in time-series graphs (particularly protocols) and sign
> language.
>
>
>
> Here’s my mocap clients:
>
>
>
> https://github.com/coderextreme/deafblind/
>
> (Python. Yay!)
>
>
>
> Here’s my other viewer (X_ITE):
>
>
>
>
> https://github.com/coderextreme/JSONvirse/blob/main/public/x3d/JinLOA4LiveMocap.x3d
>
>
>
> Here’s my JavaScript for .x3d
>
> So integrating OSC is very interesting, and perhaps I could help?  I am
> mostly available on zoom after 10am or so.
>
>
>
>
> https://github.com/coderextreme/JSONvirse/blob/main/public/javascripts/JinAvatar.js
>
>
>
> It might be interesting to hook up my python renderer to mocap:
>
>
>
> https://github.com/coderextreme/x3danari
>
>
>
>
>
> John
>
>
>
> On Wed, Nov 19, 2025 at 1:03 PM Bergstrom, Aaron via X3D-Ecosystem <
> x3d-ecosystem at web3d.org> wrote:
>
> Do any of the X3D Browsers support the OSC mocap protocol, or is anyone
> working on a project that is using both X3D and OSC?
>
>
>
> I just purchased a Movin Tracin device for the DREAM Lab, and as of v2 of
> Movin Studio, it now supports OSC applications.
>
> https://www.movin3d.com/tracin
>
>
>
> I would just like to see if I could get it working with one of the X3D
> Browsers.
>
>
>
> Aaron
>
> --
> X3D-Ecosystem mailing list
> X3D-Ecosystem at web3d.org
> http://web3d.org/mailman/listinfo/x3d-ecosystem_web3d.org
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://web3d.org/pipermail/x3d-ecosystem_web3d.org/attachments/20251119/792fc8db/attachment-0001.html>