<div><div dir="auto"><a href="https://github.com/laurencecliffe/mediapipe2osc">https://github.com/laurencecliffe/mediapipe2osc</a> (JavaScript)</div><div dir="auto"><br></div><div dir="auto"><div dir="auto"><a href="https://github.com/laurencecliffe/mediapipe2osc">https://github.com/laurencecliffe/mediapipe2osc</a> (not sure about portability)</div><div dir="auto"><br></div><div dir="auto">AFAIK, OSC appears to be a binary protocol?</div></div><br></div><div><br><div class="gmail_quote gmail_quote_container"><div dir="ltr" class="gmail_attr">On Wed, Nov 19, 2025 at 2:58 PM John Carlson <<a href="mailto:yottzumm@gmail.com">yottzumm@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;padding-left:1ex;border-left-color:rgb(204,204,204)"><div dir="auto">My approach would be to use my existing TCP <a href="http://socket.io" target="_blank">socket.io</a> server, and add UDP services to make it an OSC server your device can talk to. Then we use the server to gateway OSC requests into my existing X_ITE app which uses a <a href="http://socket.io" target="_blank">socket.io</a> client.  Alternatively, we have to write an OSC client or server in Castle or FreeWRL.  I don’t know if Sunrize/Electron supports UDP.  It doesn’t support DIS, last I checked.</div><div dir="auto"><br></div><div dir="auto">I’m pretty sure OSC supports joint rotations, so that’s a bonus!</div><div dir="auto"><br></div><div dir="auto">John </div><div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Nov 19, 2025 at 2:41 PM John Carlson <<a href="mailto:yottzumm@gmail.com" target="_blank">yottzumm@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;padding-left:1ex;border-left-color:rgb(204,204,204)"><div dir="auto">We’ll probably need the specifics for your device. We will need protocol dumps, probably via wireshark.  AFAIK, transmission is over UDP/IP, but I’m wondering about the physical layer.  If it already displays in a browser, great,. But I don’t know if it supports the UDP version in the browser, specifically QUIC.  I don’t know if UDP is available via websockets.  It isn’t.  That means we need an OSC server that talks both TCP or HTTP/S (hopefully QUIC) and UDP (for OSC).  This doesn’t appear to be better than DIS.  DIS uses UDP or a web gateway.  I don’t really know if DIS handles HAnim.</div><div dir="auto"><br></div><div dir="auto">So I the best approach is to set up MediaPipe with OSC and start experimenting. Any webcam should do, and not everyone has a dedicated mocap device: Once we get familiar with OSC and MediaPipe, then we can try to integrate your specific device:</div><div dir="auto"><br></div><div dir="auto">Google AI:</div><div dir="auto"><br></div><div dir="auto"><div><div style="font-family:"google sans",roboto,arial,sans-serif;font-size:21.6px;margin:0px 0px 20px;font-weight:400;font-style:normal;letter-spacing:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none;color:rgb(10,10,10)"><strong style="font-weight:bolder;font-family:"google sans",roboto,arial,sans-serif"><div style="font-family:"google sans",roboto,arial,sans-serif">MediaPipe does not have built-in Open Sound Control (OSC) functionality</div></strong>, but the two technologies are frequently used together through custom integrations to create real-time, gesture-based control systems for multimedia applications.<span style="white-space:nowrap;font-family:"google sans",roboto,arial,sans-serif"> <button style="margin:0px 6px 0px 0px;border:medium;border-radius:10px;height:20px;padding:0px;width:28px;outline:0px"><span style="display:inline-block"></span></button></span></div><div style="font-family:"google sans",roboto,arial,sans-serif;font-size:21.6px;font-style:normal;font-weight:400;letter-spacing:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none;color:rgb(10,10,10)"></div><div style="font-family:"google sans",roboto,arial,sans-serif;font-weight:600;margin:20px 0px 10px;font-style:normal;letter-spacing:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none;color:rgb(10,10,10)">How They Work Together<span style="white-space:nowrap;font-family:"google sans",roboto,arial,sans-serif"><button style="margin:0px 6px 0px 0px;border:medium;border-radius:10px;height:20px;padding:0px;width:28px;outline:0px"><span style="display:inline-block"></span></button></span></div><div style="font-family:"google sans",roboto,arial,sans-serif;font-size:21.6px;margin:10px 0px 20px;font-weight:400;font-style:normal;letter-spacing:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none;color:rgb(10,10,10)">Developers commonly use custom scripts (often in Python or JavaScript) to bridge the gap between MediaPipe's machine learning outputs and the OSC protocol:<span style="white-space:nowrap;font-family:"google sans",roboto,arial,sans-serif"> <button style="margin:0px 6px 0px 0px;border:medium;border-radius:10px;height:20px;padding:0px;width:28px;outline:0px"><span style="display:inline-block"></span></button></span></div><ol style="margin:10px 0px 20px;padding:0px;font-family:"google sans",roboto,arial,sans-serif;font-size:21.6px;font-style:normal;font-weight:400;letter-spacing:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none;color:rgb(10,10,10)"><li style="margin:0px 0px 12px;padding:0px;list-style:decimal;font-family:"google sans",roboto,arial,sans-serif"><strong style="font-weight:bolder;font-family:"google sans",roboto,arial,sans-serif">MediaPipe Processes Data</strong>: MediaPipe is used to perform real-time computer vision tasks like hand, face, or pose tracking, extracting landmark coordinates or gesture classifications from a camera feed.</li><li style="font-family:"google sans",roboto,arial,sans-serif"><ul style="font-family:"google sans",roboto,arial,sans-serif"><li style="margin:0px 0px 12px;padding:0px;list-style:decimal;font-family:"google sans",roboto,arial,sans-serif"><strong style="font-weight:bolder;font-family:"google sans",roboto,arial,sans-serif">Script Converts Data</strong>: A custom script then reads these outputs from MediaPipe and formats them into OSC messages (address patterns and arguments).</li></ul><ul style="font-family:"google sans",roboto,arial,sans-serif"><li style="margin:0px 0px 12px;padding:0px;list-style:decimal;font-family:"google sans",roboto,arial,sans-serif"><strong style="font-weight:bolder;font-family:"google sans",roboto,arial,sans-serif">OSC Transmits Data</strong>: The OSC messages are sent over a network (even on the same computer) to other applications that can receive OSC data, such as Max/MSP, PureData, <a href="https://www.ableton.com/" style="text-decoration:underline;font-weight:500;outline:0px;font-family:"google sans",roboto,arial,sans-serif" target="_blank">Ableton Live</a>, or TouchDesigner.</li></ul><ul style="font-family:"google sans",roboto,arial,sans-serif"><li style="margin:0px 0px 12px;padding:0px;list-style:decimal;font-family:"google sans",roboto,arial,sans-serif"><strong style="font-weight:bolder;font-family:"google sans",roboto,arial,sans-serif">External App Interprets and Acts</strong>: The receiving application interprets the OSC messages to control various functions like sound synthesis, audio effects, video manipulation, or interactive graphics.<span style="white-space:nowrap;font-family:"google sans",roboto,arial,sans-serif"> <button style="margin:0px 6px 0px 0px;border:medium;border-radius:10px;height:20px;padding:0px;width:28px;outline:0px"><span style="display:inline-block"></span></button></span></li></ul></li></ol><span style="font-family:"google sans",roboto,arial,sans-serif;font-size:21.6px;font-style:normal;font-weight:400;letter-spacing:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none;color:rgb(10,10,10)"></span><span style="font-family:"google sans",roboto,arial,sans-serif;font-size:21.6px;font-style:normal;font-weight:400;letter-spacing:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none;color:rgb(10,10,10)"></span><span style="font-family:"google sans",roboto,arial,sans-serif;font-size:21.6px;font-style:normal;font-weight:400;letter-spacing:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none;color:rgb(10,10,10)"></span><span style="font-family:"google sans",roboto,arial,sans-serif;font-size:21.6px;font-style:normal;font-weight:400;letter-spacing:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none;color:rgb(10,10,10)"></span><span style="font-family:"google sans",roboto,arial,sans-serif;font-size:21.6px;font-style:normal;font-weight:400;letter-spacing:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none;color:rgb(10,10,10)"></span><span style="font-family:"google sans",roboto,arial,sans-serif;font-size:21.6px;font-style:normal;font-weight:400;letter-spacing:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none;color:rgb(10,10,10)"><div style="font-family:"google sans",roboto,arial,sans-serif"></div></span><span style="font-family:"google sans",roboto,arial,sans-serif;font-size:21.6px;font-style:normal;font-weight:400;letter-spacing:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none;color:rgb(10,10,10)"><div style="font-family:"google sans",roboto,arial,sans-serif;font-weight:600;margin:20px 0px 10px">Common Implementations and Resources<span style="white-space:nowrap;font-family:"google sans",roboto,arial,sans-serif"><button style="margin:0px 6px 0px 0px;border:medium;border-radius:10px;height:20px;padding:0px;width:28px;outline:0px"><span style="display:inline-block"></span></button></span></div></span><span style="font-family:"google sans",roboto,arial,sans-serif;font-size:21.6px;font-style:normal;font-weight:400;letter-spacing:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none;color:rgb(10,10,10)"><div style="font-family:"google sans",roboto,arial,sans-serif;margin:10px 0px 20px;font-weight:400">Several open-source projects and tutorials provide ready-made solutions for integrating MediaPipe and OSC:<span style="white-space:nowrap;font-family:"google sans",roboto,arial,sans-serif"><button style="margin:0px 6px 0px 0px;border:medium;border-radius:10px;height:20px;padding:0px;width:28px;outline:0px"><span style="display:inline-block"></span></button></span></div></span><span style="font-family:"google sans",roboto,arial,sans-serif;font-size:21.6px;font-style:normal;font-weight:400;letter-spacing:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none;color:rgb(10,10,10)"><ul style="margin:10px 0px 20px;padding:0px;font-family:"google sans",roboto,arial,sans-serif"><li style="margin:0px 0px 12px;padding:0px;list-style:outside;font-family:"google sans",roboto,arial,sans-serif"><strong style="font-weight:bolder;font-family:"google sans",roboto,arial,sans-serif"><code dir="ltr" style="font-size:14px;line-height:22px;border:1px solid;border-radius:4px;padding:2px 4px;font-family:monospace">mediapipe2osc</code></strong>: An open-source project that uses Google's MediaPipe for 3D skeletal tracking and streams the data points via OSC.</li><li style="margin:0px 0px 12px;padding:0px;list-style:outside;font-family:"google sans",roboto,arial,sans-serif"><strong style="font-weight:bolder;font-family:"google sans",roboto,arial,sans-serif"><code dir="ltr" style="font-size:14px;line-height:22px;border:1px solid;border-radius:4px;padding:2px 4px;font-family:monospace">cansik/mediapipe-osc</code></strong>: A <a href="https://github.com/" style="text-decoration:underline;font-weight:500;outline:0px;font-family:"google sans",roboto,arial,sans-serif" target="_blank">GitHub</a>repository with various MediaPipe examples (pose, hand, face) that stream their detections over OSC.</li><li style="margin:0px 0px 12px;padding:0px;list-style:outside;font-family:"google sans",roboto,arial,sans-serif"><strong style="font-weight:bolder;font-family:"google sans",roboto,arial,sans-serif">Handmate</strong>: A browser-based controller that uses MediaPipe for hand tracking and outputs Web Audio, MIDI, or OSC messages.</li><li style="margin:0px 0px 12px;padding:0px;list-style:outside;font-family:"google sans",roboto,arial,sans-serif"><strong style="font-weight:bolder;font-family:"google sans",roboto,arial,sans-serif">Max/MSP Integration</strong>: The <code dir="ltr" style="font-size:14px;line-height:22px;border:1px solid;border-radius:4px;padding:2px 4px;font-family:monospace">jweb-mediapipe</code> library allows Max/MSP users to directly leverage MediaPipe's landmark detection within the Max environment, which inherently supports OSC communication.</li><li style="margin:0px 0px 12px;padding:0px;list-style:outside;font-family:"google sans",roboto,arial,sans-serif"><strong style="font-weight:bolder;font-family:"google sans",roboto,arial,sans-serif">Python Libraries</strong>: Projects often use the <code dir="ltr" style="font-size:14px;line-height:22px;border:1px solid;border-radius:4px;padding:2px 4px;font-family:monospace">python-osc</code> library in their Python scripts to send gesture data to other applications.<span style="white-space:nowrap;font-family:"google sans",roboto,arial,sans-serif"> <button style="margin:0px 6px 0px 0px;border:medium;border-radius:10px;height:20px;padding:0px;width:28px;outline:0px"><span style="display:inline-block"></span></button></span></li></ul></span><span style="font-family:"google sans",roboto,arial,sans-serif;font-size:21.6px;font-style:normal;font-weight:400;letter-spacing:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none;color:rgb(10,10,10)"></span><span style="font-family:"google sans",roboto,arial,sans-serif;font-size:21.6px;font-style:normal;font-weight:400;letter-spacing:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none;color:rgb(10,10,10)"></span><span style="font-family:"google sans",roboto,arial,sans-serif;font-size:21.6px;font-style:normal;font-weight:400;letter-spacing:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none;color:rgb(10,10,10)"></span><span style="font-family:"google sans",roboto,arial,sans-serif;font-size:21.6px;font-style:normal;font-weight:400;letter-spacing:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none;color:rgb(10,10,10)"></span><span style="font-family:"google sans",roboto,arial,sans-serif;font-size:21.6px;font-style:normal;font-weight:400;letter-spacing:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none;color:rgb(10,10,10)"></span><span style="font-family:"google sans",roboto,arial,sans-serif;font-size:21.6px;font-style:normal;font-weight:400;letter-spacing:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none;color:rgb(10,10,10)"></span><span style="font-family:"google sans",roboto,arial,sans-serif;font-size:21.6px;font-style:normal;font-weight:400;letter-spacing:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none;color:rgb(10,10,10)"><div style="font-family:"google sans",roboto,arial,sans-serif;margin:10px 0px 20px;font-weight:400">These integrations allow for the creation of sophisticated, low-latency, gesture-controlled interactive systems without requiring specialized hardware beyond a standard webcam.<span style="white-space:nowrap;font-family:"google sans",roboto,arial,sans-serif"> </span></div></span></div><br></div><div dir="auto"><br></div><div dir="auto"><br></div><div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Nov 19, 2025 at 1:51 PM Bergstrom, Aaron <<a href="mailto:aaron.bergstrom@und.edu" target="_blank">aaron.bergstrom@und.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;padding-left:1ex;border-left-color:rgb(204,204,204)">





<div lang="EN-US" link="blue" vlink="purple">
<div>
<p class="MsoNormal"><a href="https://ccrma.stanford.edu/groups/osc/index.html" target="_blank">https://ccrma.stanford.edu/groups/osc/index.html</a><u></u><u></u></p></div></div><div lang="EN-US" link="blue" vlink="purple"><div>
<p class="MsoNormal"><u></u> <u></u></p>
<div style="border-width:1pt medium medium;border-style:solid none none;padding:3pt 0in 0in;border-color:rgb(225,225,225) currentcolor currentcolor">
<p class="MsoNormal"><b>From:</b> John Carlson <<a href="mailto:yottzumm@gmail.com" target="_blank">yottzumm@gmail.com</a>> <br>
<b>Sent:</b> Wednesday, November 19, 2025 1:29 PM<br>
<b>To:</b> X3D Ecosystem public discussion <<a href="mailto:x3d-ecosystem@web3d.org" target="_blank">x3d-ecosystem@web3d.org</a>><br>
<b>Cc:</b> Bergstrom, Aaron <<a href="mailto:aaron.bergstrom@und.edu" target="_blank">aaron.bergstrom@und.edu</a>><br>
<b>Subject:</b> Re: [X3D-Ecosystem] OSC mocap support in X3D Browsers<u></u><u></u></p>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal">While I have a mocap protocol of my own design, I don’t have the OSC mocap protocol.  I still need to apply IK and calibration after my protocol reaches the browser to determine rotations.  I did/do have viewers that take joint locations.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">I have 2 viewers:<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<div>
<p class="MsoNormal"><a href="https://github.com/coderextreme/lircom/tree/master/src/main/java/impact" target="_blank">https://github.com/coderextreme/lircom/tree/master/src/main/java/impact</a><u></u><u></u></p>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">(sorry, Java, no X3D and no rotations).<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">If you have a pointer to OSC protocol or API, I will see what i can do.  I am interested in time-series graphs (particularly protocols) and sign language.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Here’s my mocap clients:<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<div>
<p class="MsoNormal"><a href="https://github.com/coderextreme/deafblind/" target="_blank">https://github.com/coderextreme/deafblind/</a><u></u><u></u></p>
</div>
<p class="MsoNormal">(Python. Yay!)<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Here’s my other viewer (X_ITE):<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<div>
<p class="MsoNormal"><a href="https://github.com/coderextreme/JSONvirse/blob/main/public/x3d/JinLOA4LiveMocap.x3d" target="_blank">https://github.com/coderextreme/JSONvirse/blob/main/public/x3d/JinLOA4LiveMocap.x3d</a><u></u><u></u></p>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Here’s my JavaScript for .x3d<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">So integrating OSC is very interesting, and perhaps I could help?  I am mostly available on zoom after 10am or so.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<div>
<p class="MsoNormal"><a href="https://github.com/coderextreme/JSONvirse/blob/main/public/javascripts/JinAvatar.js" target="_blank">https://github.com/coderextreme/JSONvirse/blob/main/public/javascripts/JinAvatar.js</a><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">It might be interesting to hook up my python renderer to mocap:<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<div>
<p class="MsoNormal"><a href="https://github.com/coderextreme/x3danari" target="_blank">https://github.com/coderextreme/x3danari</a><u></u><u></u></p>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">John<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div>
<p class="MsoNormal">On Wed, Nov 19, 2025 at 1:03 PM Bergstrom, Aaron via X3D-Ecosystem <<a href="mailto:x3d-ecosystem@web3d.org" target="_blank">x3d-ecosystem@web3d.org</a>> wrote:<u></u><u></u></p>
</div>
<blockquote style="border-width:medium medium medium 1pt;border-style:none none none solid;padding:0in 0in 0in 6pt;margin-left:4.8pt;margin-right:0in;border-color:currentcolor currentcolor currentcolor rgb(204,204,204)">
<div>
<div>
<p class="MsoNormal">Do any of the X3D Browsers support the OSC mocap protocol, or is anyone working on a project that is using both X3D and OSC?<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">I just purchased a Movin Tracin device for the DREAM Lab, and as of v2 of Movin Studio, it now supports OSC applications.<u></u><u></u></p>
<p class="MsoNormal"><a href="https://www.movin3d.com/tracin" target="_blank">https://www.movin3d.com/tracin</a><u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">I would just like to see if I could get it working with one of the X3D Browsers.<u></u><u></u></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">Aaron<u></u><u></u></p>
</div>
</div>
<p class="MsoNormal">-- <br>
X3D-Ecosystem mailing list<br>
<a href="mailto:X3D-Ecosystem@web3d.org" target="_blank">X3D-Ecosystem@web3d.org</a><br>
<a href="http://web3d.org/mailman/listinfo/x3d-ecosystem_web3d.org" target="_blank">http://web3d.org/mailman/listinfo/x3d-ecosystem_web3d.org</a><u></u><u></u></p>
</blockquote>
</div>
</div>
</div>
</div>

</blockquote></div></div>
</blockquote></div></div>
</blockquote></div></div>