
MuJoCo on the web looks like a way to simulate robots.  While taking a broad brush to their web-based work, it looks like they are looking at the physics of robots, which isn’t a horrible thing, it just looks like the robots don’t have any clothes or soft-body physics involved.   I have seen trampolines and ponchos in their non-web work.  It looks like some groups are leveraging Three.JS for rendering, and WASM for simulation, much like I use Ammo.js for simulation and Three.Js for rendering.  So I think if we want to do something with MuJoCo, we should separate out the physics engine from the rest of the project.<div dir="auto"><br></div><div dir="auto">I have little experience interfacing with JavaScript and WASM and mainly depends on demo work by others, so I might be able to construct a demo with an X3D-based encoding and MuJoCo physics engine using AI, essentially stripping out Ammo.Js, but I’m going to need to know how to include MuJoCo in a website, that’s not clear yet.   So here goes Google AI.  I’ll be testing this later tonight.</div><div dir="auto"><br></div><div dir="auto">1. Command setup:</div><div dir="auto"><br></div><div dir="auto">=========</div><div dir="auto"><div style="font-size:inherit">npm init -y<br style="font-size:inherit">npm install three mujoco<br style="font-size:inherit">npm install -D vite</div><br></div><div dir="auto"><br></div><div dir="auto">Package.json</div><div dir="auto">==========</div><div dir="auto"><div style="font-size:inherit">{<br style="font-size:inherit">  "name": "mujoco-poncho",<br style="font-size:inherit">  "version": "1.0.0",<br style="font-size:inherit">  "type": "module",<br style="font-size:inherit">  "scripts": {<br style="font-size:inherit">    "dev": "vite",<br style="font-size:inherit">    "build": "vite build"<br style="font-size:inherit">  },<br style="font-size:inherit">  "dependencies": {<br style="font-size:inherit">    "mujoco": "^3.1.2",<br style="font-size:inherit">    "three": "^0.160.0"<br style="font-size:inherit">  },<br style="font-size:inherit">  "devDependencies": {<br style="font-size:inherit">    "vite": "^5.0.0"<br style="font-size:inherit">  }<br style="font-size:inherit">}</div><br></div><div dir="auto"><br></div><div dir="auto"><br></div><div dir="auto">Index.html</div><div dir="auto">============</div><div dir="auto"><br></div><div dir="auto"><div style="font-size:inherit"><!DOCTYPE html><br style="font-size:inherit"><html lang="en"><br style="font-size:inherit"><head><br style="font-size:inherit">    <meta charset="UTF-8"><br style="font-size:inherit">    <meta name="viewport" content="width=device-width, initial-scale=1.0"><br style="font-size:inherit">    <title>MuJoCo Poncho Drape</title><br style="font-size:inherit">    <style><br style="font-size:inherit">        body { margin: 0; overflow: hidden; background: #222; }<br style="font-size:inherit">        #loading { color: white; position: absolute; top: 20px; left: 20px; font-family: sans-serif; }<br style="font-size:inherit">    </style><br style="font-size:inherit"></head><br style="font-size:inherit"><body><br style="font-size:inherit">    <div id="loading">Loading MuJoCo WASM...</div><br style="font-size:inherit">    <script type="module" src="/main.js"></script><br style="font-size:inherit"></body><br style="font-size:inherit"></html></div><div style="font-size:inherit" dir="auto"><br></div><div style="font-size:inherit" dir="auto"><br></div><div style="font-size:inherit" dir="auto">Scene.xml</div><div style="font-size:inherit" dir="auto">============</div><div style="font-size:inherit" dir="auto"><br></div><div style="font-size:inherit"><mujoco model="poncho_drape"><br style="font-size:inherit">    <compiler angle="degree"/><br style="font-size:inherit">    <option timestep="0.005" gravity="0 0 -9.81"/><br style="font-size:inherit"><br style="font-size:inherit">    <worldbody><br style="font-size:inherit">        <!-- Lighting & Floor --><br style="font-size:inherit">        <light directional="true" diffuse="0.8 0.8 0.8" pos="0 0 5" dir="0 0 -1"/><br style="font-size:inherit">        <geom name="floor" type="plane" size="5 5 0.1" rgba="0.2 0.3 0.4 1"/><br style="font-size:inherit"><br style="font-size:inherit">        <!-- Static Humanoid Body (Welded to the world so it stands still) --><br style="font-size:inherit">        <body name="humanoid" pos="0 0 1.0"><br style="font-size:inherit">            <!-- Torso --><br style="font-size:inherit">            <geom name="torso" type="capsule" size="0.15 0.25" rgba="0.8 0.6 0.4 1"/><br style="font-size:inherit">            <!-- Head --><br style="font-size:inherit">            <geom name="head" type="sphere" pos="0 0 0.45" size="0.12" rgba="0.8 0.6 0.4 1"/><br style="font-size:inherit">            <!-- Arms --><br style="font-size:inherit">            <geom name="left_arm" type="capsule" pos="-0.25 0 0.1" size="0.05 0.25" euler="0 -45 0" rgba="0.8 0.6 0.4 1"/><br style="font-size:inherit">            <geom name="right_arm" type="capsule" pos="0.25 0 0.1" size="0.05 0.25" euler="0 45 0" rgba="0.8 0.6 0.4 1"/><br style="font-size:inherit">            <!-- Legs --><br style="font-size:inherit">            <geom name="left_leg" type="capsule" pos="-0.1 0 -0.5" size="0.06 0.35" rgba="0.8 0.6 0.4 1"/><br style="font-size:inherit">            <geom name="right_leg" type="capsule" pos="0.1 0 -0.5" size="0.06 0.35" rgba="0.8 0.6 0.4 1"/><br style="font-size:inherit">        </body><br style="font-size:inherit"><br style="font-size:inherit">        <!-- The Poncho (Cloth Grid) --><br style="font-size:inherit">        <!-- Placed slightly above the head, falls under gravity --><br style="font-size:inherit">        <composite type="cloth" count="21 21 1" spacing="0.04" offset="0 0 1.8"><br style="font-size:inherit">            <geom size="0.015" rgba="0.2 0.7 0.3 1" friction="1 0.005 0.0001"/><br style="font-size:inherit">        </composite><br style="font-size:inherit">    </worldbody><br style="font-size:inherit"></mujoco></div><br></div><div dir="auto"><br></div><div dir="auto">Main.js</div><div dir="auto">==========</div><div dir="auto"><div style="font-size:inherit">import * as THREE from 'three';<br style="font-size:inherit">import { OrbitControls } from 'three/examples/jsm/controls/OrbitControls.js';<br style="font-size:inherit">import load_mujoco from 'mujoco';<br style="font-size:inherit"><br style="font-size:inherit">async function init() {<br style="font-size:inherit">    // ==========================================<br style="font-size:inherit">    // 1. Three.js Setup<br style="font-size:inherit">    // ==========================================<br style="font-size:inherit">    const scene = new THREE.Scene();<br style="font-size:inherit">    const camera = new THREE.PerspectiveCamera(45, window.innerWidth / window.innerHeight, 0.1, 100);<br style="font-size:inherit">    camera.position.set(3, 3, 3);<br style="font-size:inherit"><br style="font-size:inherit">    const renderer = new THREE.WebGLRenderer({ antialias: true });<br style="font-size:inherit">    renderer.setSize(window.innerWidth, window.innerHeight);<br style="font-size:inherit">    renderer.setPixelRatio(window.devicePixelRatio);<br style="font-size:inherit">    document.body.appendChild(renderer.domElement);<br style="font-size:inherit"><br style="font-size:inherit">    const controls = new OrbitControls(camera, renderer.domElement);<br style="font-size:inherit">    controls.target.set(0, 1, 0);<br style="font-size:inherit"><br style="font-size:inherit">    const ambientLight = new THREE.AmbientLight(0xffffff, 0.4);<br style="font-size:inherit">    scene.add(ambientLight);<br style="font-size:inherit">    const dirLight = new THREE.DirectionalLight(0xffffff, 0.8);<br style="font-size:inherit">    dirLight.position.set(5, 10, 5);<br style="font-size:inherit">    scene.add(dirLight);<br style="font-size:inherit"><br style="font-size:inherit">    // Coordinate Fix: MuJoCo is Z-up, Three.js is Y-up.<br style="font-size:inherit">    // By placing all meshes in a rotated group, we avoid complex math per-mesh.<br style="font-size:inherit">    const mujocoGroup = new THREE.Group();<br style="font-size:inherit">    mujocoGroup.rotation.x = -Math.PI / 2;<br style="font-size:inherit">    scene.add(mujocoGroup);<br style="font-size:inherit"><br style="font-size:inherit">    // ==========================================<br style="font-size:inherit">    // 2. MuJoCo & VFS Setup<br style="font-size:inherit">    // ==========================================<br style="font-size:inherit">    const mujoco = await load_mujoco();<br style="font-size:inherit">    <br style="font-size:inherit">    // Create a virtual file system so the C++ engine can read the XML<br style="font-size:inherit">    mujoco.FS.mkdir('/working');<br style="font-size:inherit">    mujoco.FS.mount(mujoco.MEMFS, { root: '.' }, '/working');<br style="font-size:inherit"><br style="font-size:inherit">    // Fetch our XML and write it to WASM memory<br style="font-size:inherit">    const xmlContent = await fetch('/scene.xml').then(res => res.text());<br style="font-size:inherit">    mujoco.FS.writeFile('/working/scene.xml', xmlContent);<br style="font-size:inherit"><br style="font-size:inherit">    // Initialize physics states<br style="font-size:inherit">    const model = mujoco.MjModel.loadFromXML('/working/scene.xml');<br style="font-size:inherit">    const data = new mujoco.MjData(model);<br style="font-size:inherit"><br style="font-size:inherit">    // Remove loading text<br style="font-size:inherit">    document.getElementById('loading').style.display = 'none';<br style="font-size:inherit"><br style="font-size:inherit">    // ==========================================<br style="font-size:inherit">    // 3. Mapping MuJoCo Geoms to Three.js<br style="font-size:inherit">    // ==========================================<br style="font-size:inherit">    const meshes =[];<br style="font-size:inherit"><br style="font-size:inherit">    for (let i = 0; i < model.ngeom; i++) {<br style="font-size:inherit">        const type = model.geom_type[i];<br style="font-size:inherit">        <br style="font-size:inherit">        // MuJoCo flat arrays: size is index * 3, rgba is index * 4<br style="font-size:inherit">        const size = [<br style="font-size:inherit">            model.geom_size[i * 3 + 0],<br style="font-size:inherit">            model.geom_size[i * 3 + 1],<br style="font-size:inherit">            model.geom_size[i * 3 + 2]<br style="font-size:inherit">        ];<br style="font-size:inherit">        const rgba =[<br style="font-size:inherit">            model.geom_rgba[i * 4 + 0],<br style="font-size:inherit">            model.geom_rgba[i * 4 + 1],<br style="font-size:inherit">            model.geom_rgba[i * 4 + 2],<br style="font-size:inherit">            model.geom_rgba[i * 4 + 3]<br style="font-size:inherit">        ];<br style="font-size:inherit"><br style="font-size:inherit">        let geometry;<br style="font-size:inherit">        // 0: Plane, 2: Sphere, 3: Capsule, 6: Box<br style="font-size:inherit">        if (type === 0) {<br style="font-size:inherit">            geometry = new THREE.PlaneGeometry(size[0] * 2, size[1] * 2);<br style="font-size:inherit">        } else if (type === 2) {<br style="font-size:inherit">            geometry = new THREE.SphereGeometry(size[0], 16, 16);<br style="font-size:inherit">        } else if (type === 3) {<br style="font-size:inherit">            // MuJoCo capsule sizes: [radius, half-length]<br style="font-size:inherit">            geometry = new THREE.CapsuleGeometry(size[0], size[1] * 2, 8, 16);<br style="font-size:inherit">            // CRITICAL: Three.js aligns capsules on Y, MuJoCo aligns them on Z.<br style="font-size:inherit">            geometry.rotateX(Math.PI / 2); <br style="font-size:inherit">        } else if (type === 6) {<br style="font-size:inherit">            geometry = new THREE.BoxGeometry(size[0] * 2, size[1] * 2, size[2] * 2);<br style="font-size:inherit">        } else {<br style="font-size:inherit">            geometry = new THREE.SphereGeometry(0.01); // Fallback<br style="font-size:inherit">        }<br style="font-size:inherit"><br style="font-size:inherit">        const material = new THREE.MeshStandardMaterial({<br style="font-size:inherit">            color: new THREE.Color(rgba[0], rgba[1], rgba[2]),<br style="font-size:inherit">            opacity: rgba[3],<br style="font-size:inherit">            transparent: rgba[3] < 1.0,<br style="font-size:inherit">            side: THREE.DoubleSide<br style="font-size:inherit">        });<br style="font-size:inherit"><br style="font-size:inherit">        const mesh = new THREE.Mesh(geometry, material);<br style="font-size:inherit">        // We will manually overwrite the transform matrix from MuJoCo each frame<br style="font-size:inherit">        mesh.matrixAutoUpdate = false; <br style="font-size:inherit">        <br style="font-size:inherit">        mujocoGroup.add(mesh);<br style="font-size:inherit">        meshes.push(mesh);<br style="font-size:inherit">    }<br style="font-size:inherit"><br style="font-size:inherit">    // ==========================================<br style="font-size:inherit">    // 4. The Animation / Simulation Loop<br style="font-size:inherit">    // ==========================================<br style="font-size:inherit">    const tempMatrix = new THREE.Matrix4();<br style="font-size:inherit">    <br style="font-size:inherit">    // Calculate how many physics steps to run per frame to match real time<br style="font-size:inherit">    // Default framerate is ~60fps (0.016s). MuJoCo timestep is 0.005s.<br style="font-size:inherit">    const stepsPerFrame = Math.ceil(0.016 / model.opt.timestep);<br style="font-size:inherit"><br style="font-size:inherit">    function animate() {<br style="font-size:inherit">        requestAnimationFrame(animate);<br style="font-size:inherit"><br style="font-size:inherit">        // Step physics simulation<br style="font-size:inherit">        for (let i = 0; i < stepsPerFrame; i++) {<br style="font-size:inherit">            mujoco.mj_step(model, data);<br style="font-size:inherit">        }<br style="font-size:inherit"><br style="font-size:inherit">        // Sync transforms<br style="font-size:inherit">        for (let i = 0; i < model.ngeom; i++) {<br style="font-size:inherit">            // Read 3D position<br style="font-size:inherit">            const px = data.geom_xpos[i * 3 + 0];<br style="font-size:inherit">            const py = data.geom_xpos[i * 3 + 1];<br style="font-size:inherit">            const pz = data.geom_xpos[i * 3 + 2];<br style="font-size:inherit"><br style="font-size:inherit">            // Read 3x3 rotation matrix<br style="font-size:inherit">            const m0 = data.geom_xmat[i * 9 + 0]; const m1 = data.geom_xmat[i * 9 + 1]; const m2 = data.geom_xmat[i * 9 + 2];<br style="font-size:inherit">            const m3 = data.geom_xmat[i * 9 + 3]; const m4 = data.geom_xmat[i * 9 + 4]; const m5 = data.geom_xmat[i * 9 + 5];<br style="font-size:inherit">            const m6 = data.geom_xmat[i * 9 + 6]; const m7 = data.geom_xmat[i * 9 + 7]; const m8 = data.geom_xmat[i * 9 + 8];<br style="font-size:inherit"><br style="font-size:inherit">            // Load into Three.js 4x4 Matrix (Row-major order)<br style="font-size:inherit">            tempMatrix.set(<br style="font-size:inherit">                m0, m1, m2, px,<br style="font-size:inherit">                m3, m4, m5, py,<br style="font-size:inherit">                m6, m7, m8, pz,<br style="font-size:inherit">                0,  0,  0,  1<br style="font-size:inherit">            );<br style="font-size:inherit"><br style="font-size:inherit">            meshes[i].matrix.copy(tempMatrix);<br style="font-size:inherit">        }<br style="font-size:inherit"><br style="font-size:inherit">        controls.update();<br style="font-size:inherit">        renderer.render(scene, camera);<br style="font-size:inherit">    }<br style="font-size:inherit"><br style="font-size:inherit">    animate();<br style="font-size:inherit"><br style="font-size:inherit">    // Handle Window Resize<br style="font-size:inherit">    window.addEventListener('resize', () => {<br style="font-size:inherit">        camera.aspect = window.innerWidth / window.innerHeight;<br style="font-size:inherit">        camera.updateProjectionMatrix();<br style="font-size:inherit">        renderer.setSize(window.innerWidth, window.innerHeight);<br style="font-size:inherit">    });<br style="font-size:inherit">}<br style="font-size:inherit"><br style="font-size:inherit">init();</div><br></div><div dir="auto"><br></div><div dir="auto"><br></div><div dir="auto"><div style="font-size:inherit" dir="auto"><span style="font-size:inherit;font-style:normal;font-weight:400;letter-spacing:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none;float:none;display:inline!important;background-color:rgba(0,0,0,0);border-color:rgb(0,0,0);color:rgb(0,0,0)">Command to run</span></div></div><div dir="auto">========</div><div dir="auto"><br></div><div dir="auto"><br></div><div dir="auto"><div style="font-size:inherit">npm run dev</div><br></div>