[X3D-Ecosystem] OSC mocap support in X3D Browsers

Wed Nov 19 13:10:46 PST 2025

https://github.com/laurencecliffe/mediapipe2osc (JavaScript)

https://github.com/laurencecliffe/mediapipe2osc (not sure about portability)

AFAIK, OSC appears to be a binary protocol?

On Wed, Nov 19, 2025 at 2:58 PM John Carlson <yottzumm at gmail.com> wrote:

> My approach would be to use my existing TCP socket.io server, and add UDP
> services to make it an OSC server your device can talk to. Then we use the
> server to gateway OSC requests into my existing X_ITE app which uses a
> socket.io client.  Alternatively, we have to write an OSC client or
> server in Castle or FreeWRL.  I don’t know if Sunrize/Electron supports
> UDP.  It doesn’t support DIS, last I checked.
>
> I’m pretty sure OSC supports joint rotations, so that’s a bonus!
>
> John
>
> On Wed, Nov 19, 2025 at 2:41 PM John Carlson <yottzumm at gmail.com> wrote:
>
>> We’ll probably need the specifics for your device. We will need protocol
>> dumps, probably via wireshark.  AFAIK, transmission is over UDP/IP, but I’m
>> wondering about the physical layer.  If it already displays in a browser,
>> great,. But I don’t know if it supports the UDP version in the browser,
>> specifically QUIC.  I don’t know if UDP is available via websockets.  It
>> isn’t.  That means we need an OSC server that talks both TCP or HTTP/S
>> (hopefully QUIC) and UDP (for OSC).  This doesn’t appear to be better than
>> DIS.  DIS uses UDP or a web gateway.  I don’t really know if DIS handles
>> HAnim.
>>
>> So I the best approach is to set up MediaPipe with OSC and start
>> experimenting. Any webcam should do, and not everyone has a dedicated mocap
>> device: Once we get familiar with OSC and MediaPipe, then we can try to
>> integrate your specific device:
>>
>> Google AI:
>>
>> *MediaPipe does not have built-in Open Sound Control (OSC) functionality*,
>> but the two technologies are frequently used together through custom
>> integrations to create real-time, gesture-based control systems for
>> multimedia applications.
>> How They Work Together
>> Developers commonly use custom scripts (often in Python or JavaScript) to
>> bridge the gap between MediaPipe's machine learning outputs and the OSC
>> protocol:
>>
>>    1. *MediaPipe Processes Data*: MediaPipe is used to perform real-time
>>    computer vision tasks like hand, face, or pose tracking, extracting
>>    landmark coordinates or gesture classifications from a camera feed.
>>    2.
>>       - *Script Converts Data*: A custom script then reads these outputs
>>       from MediaPipe and formats them into OSC messages (address patterns and
>>       arguments).
>>    - *OSC Transmits Data*: The OSC messages are sent over a network
>>       (even on the same computer) to other applications that can receive OSC
>>       data, such as Max/MSP, PureData, Ableton Live
>>       <https://www.ableton.com/>, or TouchDesigner.
>>    - *External App Interprets and Acts*: The receiving application
>>       interprets the OSC messages to control various functions like sound
>>       synthesis, audio effects, video manipulation, or interactive graphics.
>>
>>
>> Common Implementations and Resources
>> Several open-source projects and tutorials provide ready-made solutions
>> for integrating MediaPipe and OSC:
>>
>>    - *mediapipe2osc*: An open-source project that uses Google's
>>    MediaPipe for 3D skeletal tracking and streams the data points via OSC.
>>    - *cansik/mediapipe-osc*: A GitHub <https://github.com/>repository
>>    with various MediaPipe examples (pose, hand, face) that stream their
>>    detections over OSC.
>>    - *Handmate*: A browser-based controller that uses MediaPipe for hand
>>    tracking and outputs Web Audio, MIDI, or OSC messages.
>>    - *Max/MSP Integration*: The jweb-mediapipe library allows Max/MSP
>>    users to directly leverage MediaPipe's landmark detection within the Max
>>    environment, which inherently supports OSC communication.
>>    - *Python Libraries*: Projects often use the python-osc library in
>>    their Python scripts to send gesture data to other applications.
>>
>> These integrations allow for the creation of sophisticated, low-latency,
>> gesture-controlled interactive systems without requiring specialized
>> hardware beyond a standard webcam.
>>
>>
>>
>>
>> On Wed, Nov 19, 2025 at 1:51 PM Bergstrom, Aaron <aaron.bergstrom at und.edu>
>> wrote:
>>
>>> https://ccrma.stanford.edu/groups/osc/index.html
>>>
>>>
>>>
>>> *From:* John Carlson <yottzumm at gmail.com>
>>> *Sent:* Wednesday, November 19, 2025 1:29 PM
>>> *To:* X3D Ecosystem public discussion <x3d-ecosystem at web3d.org>
>>> *Cc:* Bergstrom, Aaron <aaron.bergstrom at und.edu>
>>> *Subject:* Re: [X3D-Ecosystem] OSC mocap support in X3D Browsers
>>>
>>>
>>>
>>> While I have a mocap protocol of my own design, I don’t have the OSC
>>> mocap protocol.  I still need to apply IK and calibration after my protocol
>>> reaches the browser to determine rotations.  I did/do have viewers that
>>> take joint locations.
>>>
>>>
>>>
>>> I have 2 viewers:
>>>
>>>
>>>
>>> https://github.com/coderextreme/lircom/tree/master/src/main/java/impact
>>>
>>>
>>>
>>> (sorry, Java, no X3D and no rotations).
>>>
>>>
>>>
>>> If you have a pointer to OSC protocol or API, I will see what i can do.
>>> I am interested in time-series graphs (particularly protocols) and sign
>>> language.
>>>
>>>
>>>
>>> Here’s my mocap clients:
>>>
>>>
>>>
>>> https://github.com/coderextreme/deafblind/
>>>
>>> (Python. Yay!)
>>>
>>>
>>>
>>> Here’s my other viewer (X_ITE):
>>>
>>>
>>>
>>>
>>> https://github.com/coderextreme/JSONvirse/blob/main/public/x3d/JinLOA4LiveMocap.x3d
>>>
>>>
>>>
>>> Here’s my JavaScript for .x3d
>>>
>>> So integrating OSC is very interesting, and perhaps I could help?  I am
>>> mostly available on zoom after 10am or so.
>>>
>>>
>>>
>>>
>>> https://github.com/coderextreme/JSONvirse/blob/main/public/javascripts/JinAvatar.js
>>>
>>>
>>>
>>> It might be interesting to hook up my python renderer to mocap:
>>>
>>>
>>>
>>> https://github.com/coderextreme/x3danari
>>>
>>>
>>>
>>>
>>>
>>> John
>>>
>>>
>>>
>>> On Wed, Nov 19, 2025 at 1:03 PM Bergstrom, Aaron via X3D-Ecosystem <
>>> x3d-ecosystem at web3d.org> wrote:
>>>
>>> Do any of the X3D Browsers support the OSC mocap protocol, or is anyone
>>> working on a project that is using both X3D and OSC?
>>>
>>>
>>>
>>> I just purchased a Movin Tracin device for the DREAM Lab, and as of v2
>>> of Movin Studio, it now supports OSC applications.
>>>
>>> https://www.movin3d.com/tracin
>>>
>>>
>>>
>>> I would just like to see if I could get it working with one of the X3D
>>> Browsers.
>>>
>>>
>>>
>>> Aaron
>>>
>>> --
>>> X3D-Ecosystem mailing list
>>> X3D-Ecosystem at web3d.org
>>> http://web3d.org/mailman/listinfo/x3d-ecosystem_web3d.org
>>>
>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://web3d.org/pipermail/x3d-ecosystem_web3d.org/attachments/20251119/1da2d3d5/attachment-0001.html>