I taught a six-legged robot to walk, on my own
There is a golden robot with six legs. Nobody showed it how to walk. It worked that out by falling over, again and again. Now it runs in the browser on real-time physics, and when you throw a steel ball at it, it staggers and then walks on. This post is about how I built it.
The demo is here. Nothing to install, it opens right in the browser.
https://www.coolspeed.org/gold-spider/
What it is
I named it Gold Spider. It has six legs and a golden body, and that is the whole reason.
I didn't draw a single frame of its walk. A neural network learned it through reinforcement learning. When you open the demo page, the MuJoCo physics engine computes the body 480 times a second, and in between, the network decides 30 times a second what angle each of the 24 joints should aim for. The server only hands out static files. All of the computation runs in the visitor's browser.
While it walks over a terrain of unevenly stacked blocks, you can throw a 171 kg steel ball or a 79 kg wooden crate at it. It staggers, braces with its legs, and goes back to walking the way it was headed. I didn't make that reaction either. Physics and learning did, which is why it takes the same ball thrown at the same spot a little differently each time.
Why I built it
It started with Embark Studios. In 2021 Embark published a video of a robot learning to walk in a physics simulation through reinforcement learning, and at NDC this year they talked quite candidly about the years of trial and error behind the robots of ARC Raiders. The whole time I watched, I kept thinking the same thing. Could one person do this?
I work alone, and there is no animator sitting next to me. If you want a character to get hit, slip and regain its balance differently every time, hand-keyed animation hits a wall quickly. I wanted to see for myself how far one person could get with only what has been said in public.
To be clear up front, I have no connection to Embark. I never used Embark's code or data. I only borrowed ideas from their public article and talk.
The network is a driver, not an animator
The structure is not far from what Embark described in their talk. The game gives only a goal: go this way, at this speed. The network looks at the body's current pose and the shape of the ground under its feet, and picks a target angle for every joint. The motor on each joint then pushes toward that angle, and the physics engine decides how the body actually moves. The network looks at the body again and picks the next angles. This repeats without end.
So what the network learns is closer to a habit than to a motion. It is like learning to drive. Staggering when the steel ball hits, or placing a foot again at the edge of a block: nobody put those in. They come out of this loop by themselves. Training means letting it practice that habit in a great many virtual worlds at once.
How it came together
Roughly in this order, keeping to the moments when the direction changed.
- I decided to use MuJoCo for training and MuJoCo's official WebAssembly build in the browser. The same physics has to run on both sides for the learned walk to look the same on screen.
- At first I put my effort into the wrong thing. More on that below.
- I rebuilt the body, and for the first time a trained policy walked on flat ground.
- I changed the yardstick to "quality you can see on screen" and moved on to block terrain. Progress picked up sharply from here.
- At block edges the feet visibly sank a little into the ground, so I raised the physics rate to 480 steps a second.
- A policy trained with reinforcement learning crossed the block terrain in the browser at real speed.
- I shot the intro film. It is the one at the top of this page.
- I released the demo. After that I kept training to fix the robot getting stuck in front of obstacles.
- I found out why the physics occasionally blew up during training, cut the first load, and put a CDN in front.
What I learned
These are the things I tried and confirmed myself.
Trust the video, not the reward score
A handful of reward terms is enough. Is it following the target speed, is it upright, is it facing where it wants to go, is it wasting effort, is anything other than its feet touching the ground. Embark said they mixed about ten terms of this kind.
The trouble is that a rising score doesn't guarantee a better walk. Policies that score well but drag their feet, leave one leg idle, or drift along as if floating turn up all the time. So I recorded every candidate at real speed and watched it. Tweaking rewards never ends. I decided in advance how many tries one hypothesis would get, and when it didn't work, I moved on to another cause without looking back.
Half of the walk comes from the physics settings
How stiff the joints are, how quickly they settle, how contact is computed, how finely physics is stepped. These values change the feel of the walk completely. Put the same network in with different settings and it looks like a different robot. Embark said they tuned motor stiffness and damping for years, so it wasn't just me.
In my case I could see the feet digging slightly into the ground at block edges. At first I thought refining the collision shapes would fix it. It didn't. The right move was to keep the collision shapes simple and shorten the time step. In exchange, every time I changed the physics I threw away the existing policies and test results, then trained and measured again. Once the physics changes, it is no longer the same robot.
Mix accidents into training, and test separately
During training I changed the terrain every time, shook up the starting position and heading, pushed the robot mid-stride, and put crates in its way. The point is to keep it from getting comfortable with one situation. Once I let it see crates as part of the ground under its feet and trained with this mix, the rate of getting stuck around obstacles fell from about 40% to about 10%.
Evaluation happens on terrain never used in training, with different random seeds. And 0% stuck isn't the goal. Some layouts are physically impossible to pass, and if you chase a single number you end up with a policy that is only good at that test. When the gain shrinks to about the size of the noise between test runs, I stop.
Start with simple eyes
The robot sees the ground under its feet as a height map: rays cast from above measure the terrain height. Embark did the same for years, and said they later switched to casting rays from the robot's body after running into trouble where something is overhead, such as tunnels or the space under a bridge. For walking on open block terrain a height map was enough. I didn't make the eyes more complicated before I needed to.
Three physics have to agree
Training runs many worlds at once on a fast version of the physics. But the same robot really runs in three places: the fast version where it trains, a more precise one where it is checked carefully before being adopted, and the browser where it is actually shown. If the three don't give the same answer, a robot that walked well in training stumbles on screen. So candidates had to pass the precise check, and the browser build went up only after I compared it step by step against that reference simulation and confirmed the results were practically identical.
On the training side the physics occasionally blew up, meaning the numbers suddenly went bad. I captured the state just before a blow-up and replayed it with a more precise computation, and it was fine. The culprit was a setting that made friction be computed very strictly, which didn't get along with the fast but less precise computation used for training. On top of that, depending on how training treated episodes that ended this way, a run that had been going well could collapse entirely. Fast physics hides traps like this. The habit of replaying on a different physics was the quickest way to the cause.
Set the bar first, and don't move it afterwards
Before starting a training run I wrote down the hypothesis, what would change, the budget, and the pass criteria. When it finished I kept the result as it was, failure or not. There are more than 440 of those research notes by now.
The most recent run cut stuck cases by 9.8% on one test and 20% on the other, but the bar was "more than 10% on both", so I didn't adopt it. It missed by 0.2 points, and honestly that stung. But once you start adjusting the bar after seeing the result, you can't trust any number from then on.
It isn't done until it runs in the browser
Finished training is worth nothing if nobody can see it. The goal was to run the physics engine (WebAssembly), the neural network (ONNX Runtime Web) and the graphics (three.js) together in the browser and still hold real speed. On a slow device it doesn't skip steps. It simply runs slower, because skipping would change the physics.
The last thing I worked on was the first load. I removed large files that weren't used, which took the first download from 16.6 MB to 11.6 MB, made the files download all at once from the start, and put a CDN in front. A first visit that swung between 5 and 26 seconds from one measurement to the next came down to about 3 seconds. A return visit loads in a second or two.
What didn't go well
The one that hurts most is that I spent the early part on precision nobody can see. I held on to proving even tiny contact errors, and the walking policy arrived much later than it should have. Things only sped up once I changed the yardstick to "quality you can see on screen". It was the most expensive lesson of the project.
The robot still gets stuck in dips in the terrain now and then. In the demo, if it covers less than 0.3 m in 6 seconds, the page turns its heading for it, and if that doesn't help, clears nearby objects away or gives it a light push, no stronger than the pushes it met in training. The screen says it is helping while this happens. A policy that gets itself out is still in training.
The walk isn't fully natural either. Embark said they taught gait style by using motions made by animators as examples (AMP). I had no example motions to use, so I chose handling rough terrain first.
And it can't see places with something overhead. That is the limit of a height map. The day it has to walk through a tunnel, the eyes are the first thing to change.
Try it yourself
The demo lives here. https://www.coolspeed.org/gold-spider/
A few things worth trying.
- Press Throw steel ball and a 171 kg steel ball flies in.
- Turn on Contacts to see red dots where the feet touch the ground.
- Turn off Auto demo and stack crates in its path. Does it go around, or get stuck?
- There is an English / Korean switch at the top right.
Any recent browser with WebAssembly and WebGL will do, and it opens on phones too.
Closing
Following alone a story Embark laid out over several years, I found far more tedious work than bright ideas: matching physics settings, writing down experiments, getting three physics to give the same answer, making the page load fast in a browser. Even so, every time I watch the robot take a ball, stagger and carry on its way, I still smile.
This is my own re-implementation inspired by Embark Studios' ML robots, and it is not affiliated with Embark. The public sources I used are Embark's 2021 article “Transforming animation with machine learning” and the NDC26 talk “Machine Learning Implementation – The Case of Embark Games” (Martin Singh-Blom).