Before blaming the sim-to-real gap, read the simulator. We did, for one recent robot benchmark. At almost every step, it simulates something other than what it says.
Bo Peng, Wenhao Chai and Zhifei Li, with a lot of agent help · November 2026
The same bottle, grasp and arm motion, rendered in the benchmark's simulator. As shipped, the bottle tips in the hand until its neck rests on the table. After our fix, it stays level.
Some of what we found in RoboDojo
Its physics engine can throw a dropped bottle back out of the bin, higher than it was let go.
As shipped, full marks on a xylophone the mallet never touches.As shipped, shoving bottles into the bin, picking none up, earns up to 40 points.
TL;DR
We followed one robot benchmark from the real world to its score, one link at a time.
No friction anyone wrote reaches the physics engine. Every object runs at the default, 0.5.
Toy cubes as dense as osmium, and an arm that bends far further than the real one.
The physics engine can throw a dropped bottle back out of the bin, higher than it was let go.
Checks that pass without the task being done.
With the physics corrected, a hand-written solution that relied on an impossible wrist roll fails 7 times in 20; one that respects the real arm succeeds every time.
People who train language models learned "garbage in, garbage out" long ago, and much of their progress since has come from reading their data. Robot learning leans on simulators in the same way: robots are trained and scored in simulation, because real ones are slow, expensive and break. When a policy that works in simulation fails on a real robot, people blame the "sim-to-real gap" and reach for a cleverer training method.
We asked a simpler question first: does the simulator do what everyone assumes? To find out, we read one closely: RoboDojo, a recent benchmark of 54 tasks for a two-armed robot, built on NVIDIA's Isaac Sim.
Inventing a better sim-to-real training algorithm
Checking that the simulator’s friction reaches the physics
A bottle stickier than glue
One task gives its bottles a friction coefficient of 10,000. Real materials sit around 1 or below; at 10,000, a bottle would stick to anything it touched, as if glued. Surely, we thought, someone had cranked it up to make up for a weak grip.
Then we read the scene the simulator actually loads. The 10,000 is there. For almost every other object, though, the friction written in the task's settings is never even read. And it gets stranger.
We asked the physics engine itself what friction it was using. The answer was 0.5. For every object, in every task. Why? The benchmark attaches each material a moment after the engine has loaded the object, and the engine never looks again. The 10,000 was never too high. It was never there.
Written in the settings10,000stickier than glue
Used by the engine, as shipped0.5the default, for every object in every task
Used after our fix0.3a plausible value for plastic
The bottles' friction, as the task's settings write it, as the engine used it, and after our fix.
Four places a simulator can lie
If the most visible number in the task never arrived, what else had not? To answer that, and to know when we were done, we followed one number from the real world to the leaderboard. It passes through four links, and each one can break on its own.
The real worlda plastic bottle, a table, a robot arm
ValuesIs the description true of the world? Compare it with real objects.
The descriptionthe settings and 3D models someone wrote
DeliveryDid it arrive intact? Ask the engine what it is using.
What the engine receives
ComputationIs the motion what physics says? Rerun it in ways that should not matter.
The simulated motion
ScoringDoes the score count what was asked? Try to pass without doing the task.
The score on the leaderboard
Anything wrong with an episode has to enter at one of these links. We check delivery first, because judging a value the engine never sees is wasted effort, and the score last, because a score computed on the wrong physics says little either way. Every test runs through the benchmark's own code, the path a policy takes, and we wrote down what we expected before each one, so that a test could surprise us.
Delivery · ask the engine
What the engine never received
Does the description reach the engine intact? Not always. Fill a block with lead and it gets heavier and harder to spin. In RoboDojo, however, only the first happens: one task's T-block is 3.5× heavier than another's, yet both spin up alike. The reason is that the benchmark sets the weight only after the engine has worked out how hard the block is to spin.
Two T-blocks, one 3.5× heavier, get the same push at one end. As shipped they turn exactly alike; after our fix, the weight reaches the engine and the heavy one turns 3.5× less. Worked out from the engine's own numbers.
Values · compare with the real thing
Values no real thing has
Suppose the values do arrive. Could they describe anything in the mortal world? Often not. We hold them to a loose standard, plausible rather than exact, and even so, some are not even close.
How dense things are. As shipped, a toy cube is as dense as osmium, the densest metal; after the fix it is wood. A coin whose weight was never set defaults to half a kilogram, nearly three times denser than osmium; a steel coin would be about 7,800.
Five of the arm's six joints may turn 573° each way, more than one and a half turns, where the real arm's joints stop a little past half a turn at most. The benchmark's own demonstrations even use the extra room: about half of them move some joint further than the real arm can. The garbage is already inside the dataset.
How far the wrist may bend each way: the real arm about a quarter turn (grey), the shipped simulator 1.6 turns (amber). The dots are the share of released demonstrations that move some joint further than the real arm can: about half.
Computation · rerun it differently
What the engine computed wrong
Suppose the values are sensible. Does the engine compute what physics says? To check, we rerun each scene in ways that should not change the answer: smaller time steps, another processor, several copies at once. It does not always pass. Smaller steps, for instance, should only make it more accurate. Yet with a stabilising shortcut the benchmark switches on, a pushed bottle slides further the finer the steps get.
A bottle pushed at 0.5 m/s slides to a stop, measured at every step of the engine and slowed 10×. As shipped, it creeps on past what friction allows, and further at the finer time step; after our fix it stops where friction says.
Worse, energy can appear from nowhere.
A steel bottle is let go over the bin, hits a light bottle inside, and lands on the floor. Slowed five times. This run used our corrected settings; under the shipped ones, dropped bottles also bounce back faster than physics allows.What we measured: at first contact the steel bottle is already inside the light one, which is pinned against the bin floor. Within that one time step it turns around and heads up faster than it fell.
Scoring · try to pass without doing the task
What the score counts
Does the score count what the instruction asks? To find out, we read each check beside its instruction and try to pass the way a lazy student would, then confirm that an honest solution still passes. Often the lazy way passes too.
One task asks the robot to strike every key of a xylophone. Yet the check only asks that the mallet pass close above each key, so hovering without ever touching a bar scores full marks.
The task credits all eight strikes. The engine records no contact at all.
In the bottle task, the robot can even skip the picking up altogether.
Shoving bottles off the table into the bin, picking none up, earns up to 40 points under the shipped check.Cheap ways to pass without doing the task, scored out of 100 by the check as shipped and by our fixed check.
Even the number of tests can mislead. The tic-tac-toe task is scored over 150 games, yet all 150 put the board and pieces in exactly the same place. Only the opponent's moves differ, and there are just 12 opponents.
Each small grid is one scored game, all in the same scene. Only 12 different opponents play them, one per row.
The robot's hand
Each example so far sits at a single link. In practice, problems rarely stay put: fixing one link often exposes the next. To show how that goes, we follow one part of the robot through the whole chain: the gripper, the robot's two-fingered hand.
Our first test drove the fingers directly and squeezed as hard as a real gripper can. The bottles held. However, through the benchmark's own controls, the path a policy uses, they slipped. Why? A limit in the controller caps the squeeze near the force of holding a 2.5 kg bag of sugar, however hard the policy asks, and the squeeze even starts over with every new command, 25 times a second. Worse, the demonstrations that policies learn from record this capped squeeze, not what the operator asked for. Another piece of garbage in the dataset.
So we copied the real robot's controller. Instead of gripping, the fingers buzzed, and the bottles twisted free. What was going wrong? Our fingers were too light. The controller expects fingers made heavy by the motor that drives them, about as heavy as a packed suitcase, but ours weighed about as much as a bar of soap. So we gave the fingers the motor's weight, and the grip steadied.
How hard the fingers squeeze a held bottle over a fifth of a second, slowed 20×. As shipped: capped near 25 N and falling back at every command. The real controller on light fingers: a buzz at every step. With the motor's weight: steady.The same 0.5 kg bottle as the hand rises 15 cm and holds for a second, slowed 2×. As shipped, it tips until its neck rests on the table. With the real controller on light fingers, it keeps turning. With the motor's weight on the fingers, it stays level.
Does fixing it matter?
With every fix in place, we reran our hand-written solutions on 20 test layouts.
Hand-written solutions on 20 test layouts, under the shipped and the corrected physics. The second block-swapping route was written after the first one failed.
The bottle solution succeeds every time either way, and once the grip is fixed it drops fewer bottles on the way. The block-swapping solution, however, fails 7 times in 20: each failure is a route that asks the wrist to roll further than the real robot can. A route that stays within the real wrist's range succeeds every time, so the task is still solvable.
Before you blame the sim-to-real gap
Ask the engine what it is using. A settings file is a wish.
Compare values with the real thing. Plausible is enough; impossible is common.
Rerun it differently. The answer should not change, and nothing should gain energy from nowhere.
Try to pass without doing the task, and watch the episodes you score.
Look at what your demonstrations actually recorded before you train on them.
Garbage in, garbage out holds for simulators as much as for text. The cure is the same: read the data.
Limitations
We studied three tasks in depth: bottles into a bin, cups over blocks, and swapping T-blocks. The other 51 got lighter checks.
We judged values against spec sheets and real objects, not measurements of the actual robot.
The comparisons above use 20 test layouts each, so small effects would not show.
Our comparisons use hand-written solutions, not trained policies.