What does RoboDojo really simulate?

Before blaming the sim-to-real gap, read the simulator. We did, for one recent robot benchmark. At almost every step, it simulates something other than what it says.

Rendered side by side: the same bottle, grasp and arm motion. As shipped, the bottle tips 78 degrees in the hand until its neck rests on the table; after our fix, it stays level.
The same bottle, grasp and arm motion, rendered in the benchmark's simulator. As shipped, the bottle tips in the hand until its neck rests on the table. After our fix, it stays level.
Some of what we found in RoboDojo
A bottle dropped into the bin flies back out
Its physics engine can throw a dropped bottle back out of the bin, higher than it was let go.
A mallet passes over a xylophone without touching it
As shipped, full marks on a xylophone the mallet never touches.
A robot pushes bottles off the table into the bin
As shipped, shoving bottles into the bin, picking none up, earns up to 40 points.
TL;DR

We followed one robot benchmark from the real world to its score, one link at a time.

People who train language models learned "garbage in, garbage out" long ago, and much of their progress since has come from reading their data. Robot learning leans on simulators in the same way: robots are trained and scored in simulation, because real ones are slow, expensive and break. When a policy that works in simulation fails on a real robot, people blame the "sim-to-real gap" and reach for a cleverer training method.

We asked a simpler question first: does the simulator do what everyone assumes? To find out, we read one closely: RoboDojo, a recent benchmark of 54 tasks for a two-armed robot, built on NVIDIA's Isaac Sim.

A bottle stickier than glue

One task gives its bottles a friction coefficient of 10,000. Real materials sit around 1 or below; at 10,000, a bottle would stick to anything it touched, as if glued. Surely, we thought, someone had cranked it up to make up for a weak grip.

Then we read the scene the simulator actually loads. The 10,000 is there. For almost every other object, though, the friction written in the task's settings is never even read. And it gets stranger.

We asked the physics engine itself what friction it was using. The answer was 0.5. For every object, in every task. Why? The benchmark attaches each material a moment after the engine has loaded the object, and the engine never looks again. The 10,000 was never too high. It was never there.

The bottles' friction, as the task's settings write it, as the engine used it, and after our fix.

Four places a simulator can lie

If the most visible number in the task never arrived, what else had not? To answer that, and to know when we were done, we followed one number from the real world to the leaderboard. It passes through four links, and each one can break on its own.

Anything wrong with an episode has to enter at one of these links. We check delivery first, because judging a value the engine never sees is wasted effort, and the score last, because a score computed on the wrong physics says little either way. Every test runs through the benchmark's own code, the path a policy takes, and we wrote down what we expected before each one, so that a test could surprise us.

Delivery · ask the engine

What the engine never received

Does the description reach the engine intact? Not always. Fill a block with lead and it gets heavier and harder to spin. In RoboDojo, however, only the first happens: one task's T-block is 3.5× heavier than another's, yet both spin up alike. The reason is that the benchmark sets the weight only after the engine has worked out how hard the block is to spin.

as shipped both turn alike after our fix the heavy one turns 3.5× less light 3.5× heavier light 3.5× heavier time
Two T-blocks, one 3.5× heavier, get the same push at one end. As shipped they turn exactly alike; after our fix, the weight reaches the engine and the heavy one turns 3.5× less. Worked out from the engine's own numbers.
Values · compare with the real thing

Values no real thing has

Suppose the values do arrive. Could they describe anything in the mortal world? Often not. We hold them to a loose standard, plausible rather than exact, and even so, some are not even close.

as shipped after our fixes real materials water steel osmium toy cube toy cube, as shipped: 22,400 kg/m³ 22,400 kg/m³ toy cube, after our fixes: 720 kg/m³ 720 wood T-block T-block, as shipped: 9,722 kg/m³ 9,700 T-block, after our fixes: 1,060 kg/m³ 1,060 plastic coin coin, as shipped: 65,205 kg/m³ 65,000 coin, a plausible value: steel, 7,820 kg/m³7,800steel
How dense things are. As shipped, a toy cube is as dense as osmium, the densest metal; after the fix it is wood. A coin whose weight was never set defaults to half a kilogram, nearly three times denser than osmium; a steel coin would be about 7,800.

Five of the arm's six joints may turn 573° each way, more than one and a half turns, where the real arm's joints stop a little past half a turn at most. The benchmark's own demonstrations even use the extra room: about half of them move some joint further than the real arm can. The garbage is already inside the dataset.

how far the wrist may bend 573° 1.6 turns the real wrist about a quarter turn each way the shipped simulator 1.6 turns each way furthest a demonstration bends demonstrations that pass the real arm's limits 1,699 of 3,500
How far the wrist may bend each way: the real arm about a quarter turn (grey), the shipped simulator 1.6 turns (amber). The dots are the share of released demonstrations that move some joint further than the real arm can: about half.
Computation · rerun it differently

What the engine computed wrong

Suppose the values are sensible. Does the engine compute what physics says? To check, we rerun each scene in ways that should not change the answer: smaller time steps, another processor, several copies at once. It does not always pass. Smaller steps, for instance, should only make it more accurate. Yet with a stabilising shortcut the benchmark switches on, a pushed bottle slides further the finer the steps get.

as shipped after our fix 0 10 20 30 40 mm 4 ms step 1 ms step 4 ms step 1 ms step what friction allows 28 mm 37 mm 320 ms after the push · slowed 10× 4 ms is the benchmark's step; 1 ms should only be more exact
A bottle pushed at 0.5 m/s slides to a stop, measured at every step of the engine and slowed 10×. As shipped, it creeps on past what friction allows, and further at the finer time step; after our fix it stops where friction says.

Worse, energy can appear from nowhere.

A steel bottle dropped into the bin hits a light bottle, flies back out higher than it was released, and lands on the floor beside the bin
A steel bottle is let go over the bin, hits a light bottle inside, and lands on the floor. Slowed five times. This run used our corrected settings; under the shipped ones, dropped bottles also bounce back faster than physics allows.
let go falls into the bin, reaching 3.0 m/s let go one 4 ms step at first contact it is already inside the light bottle, which is pinned against the bin floor as the step begins: down at 3.0 m/s as it ends: up at 3.5 m/s, faster than it fell out of the bin higher than it was let go +14 cm height of the steel bottle 0 0.5 1 m bin rim let go hits the light bottle higher than it was let go lands on the floor 0 0.5 1 s
What we measured: at first contact the steel bottle is already inside the light one, which is pinned against the bin floor. Within that one time step it turns around and heads up faster than it fell.
Scoring · try to pass without doing the task

What the score counts

Does the score count what the instruction asks? To find out, we read each check beside its instruction and try to pass the way a lazy student would, then confirm that an honest solution still passes. Often the lazy way passes too.

One task asks the robot to strike every key of a xylophone. Yet the check only asks that the mallet pass close above each key, so hovering without ever touching a bar scores full marks.

A robot carries a mallet over the eight xylophone keys without touching them; the strike counter climbs to 8 while the contact count stays at 0
The task credits all eight strikes. The engine records no contact at all.

In the bottle task, the robot can even skip the picking up altogether.

A robot pushes bottles off the table into the bin without picking any up
Shoving bottles off the table into the bin, picking none up, earns up to 40 points under the shipped check.
scored by the check as shipped scored by our fixed check Push the bottles off the table into the bin, picking none up our fixed check: 0 0 check as shipped: 40 40 Quit the bottle task at once, without doing anything our fixed check: 0 0 check as shipped: 100 Wave the mallet over the xylophone keys without touching any scored by our fixed check: 00 check as shipped: 100 0 50 100 score
Cheap ways to pass without doing the task, scored out of 100 by the check as shipped and by our fixed check.

Even the number of tests can mislead. The tic-tac-toe task is scored over 150 games, yet all 150 put the board and pieces in exactly the same place. Only the opponent's moves differ, and there are just 12 opponents.

150 scored games, all in the same scene a new opponent: only 12 in all the same opponent again
Each small grid is one scored game, all in the same scene. Only 12 different opponents play them, one per row.

The robot's hand

Each example so far sits at a single link. In practice, problems rarely stay put: fixing one link often exposes the next. To show how that goes, we follow one part of the robot through the whole chain: the gripper, the robot's two-fingered hand.

Our first test drove the fingers directly and squeezed as hard as a real gripper can. The bottles held. However, through the benchmark's own controls, the path a policy uses, they slipped. Why? A limit in the controller caps the squeeze near the force of holding a 2.5 kg bag of sugar, however hard the policy asks, and the squeeze even starts over with every new command, 25 times a second. Worse, the demonstrations that policies learn from record this capped squeeze, not what the operator asked for. Another piece of garbage in the dataset.

So we copied the real robot's controller. Instead of gripping, the fingers buzzed, and the bottles twisted free. What was going wrong? Our fingers were too light. The controller expects fingers made heavy by the motor that drives them, about as heavy as a packed suitcase, but ours weighed about as much as a bar of soap. So we gave the fingers the motor's weight, and the grip steadied.

as shipped: capped near 25 N, drops at every command 25 N 80 N the real robot's controller, light fingers: buzzes a bar of soap 25 N 80 N with the motor's weight on the fingers: steady a packed suitcase 25 N 80 N 0 40 80 120 160 200 ms slowed 20× sideways movement ×60
How hard the fingers squeeze a held bottle over a fifth of a second, slowed 20×. As shipped: capped near 25 N and falling back at every command. The real controller on light fingers: a buzz at every step. With the motor's weight: steady.
Rendered side by side: the same bottle gripped near its base as the hand rises and holds. As shipped it tips until its neck rests on the table; with the real controller on light fingers it keeps turning; with the motor's weight on the fingers it stays level.
The same 0.5 kg bottle as the hand rises 15 cm and holds for a second, slowed 2×. As shipped, it tips until its neck rests on the table. With the real controller on light fingers, it keeps turning. With the motor's weight on the fingers, it stays level.

Does fixing it matter?

With every fix in place, we reran our hand-written solutions on 20 test layouts.

as shipped after our fixes Putting bottles in the bin succeeds 20 succeeds, as shipped: 20 of 20 20 succeeds, after our fixes: 20 of 20 drops a bottle on the way 3 drops a bottle on the way, as shipped: 3 of 20 1 drops a bottle on the way, after our fixes: 1 of 20 Swapping two T-blocks succeeds with the original route 20 succeeds with the original route, as shipped: 20 of 20 13 succeeds with the original route, after our fixes: 13 of 20 succeeds with a route that keeps within the real wrist's reach 20 succeeds with a route that keeps within the real wrist's reach, as shipped: 20 of 20 20 succeeds with a route that keeps within the real wrist's reach, after our fixes: 20 of 20 each dot is one of 20 test layouts
Hand-written solutions on 20 test layouts, under the shipped and the corrected physics. The second block-swapping route was written after the first one failed.

The bottle solution succeeds every time either way, and once the grip is fixed it drops fewer bottles on the way. The block-swapping solution, however, fails 7 times in 20: each failure is a route that asks the wrist to roll further than the real robot can. A route that stays within the real wrist's range succeeds every time, so the task is still solvable.

Before you blame the sim-to-real gap

  1. Ask the engine what it is using. A settings file is a wish.
  2. Compare values with the real thing. Plausible is enough; impossible is common.
  3. Rerun it differently. The answer should not change, and nothing should gain energy from nowhere.
  4. Try to pass without doing the task, and watch the episodes you score.
  5. Look at what your demonstrations actually recorded before you train on them.

Garbage in, garbage out holds for simulators as much as for text. The cure is the same: read the data.

Limitations