Gym-Anything
for Robotics

Scaling Physical Simulations with Digital Agents

Pranjal Aggarwal · Sean Welleck
Carnegie Mellon University

Coding agents build realistic robotics simulations of real physical work. RoboWorld has 25 of them, from welding and drilling to pouring wine and folding laundry, and even the best frontier models fail every test task.

📄 Paper 💻 Code 🧭 Explore 🎬 RoboWorld 📋 Tasks 🏆 Leaderboard 📈 Results

25 simulations spanning manufacturing, construction, food service, households, laboratories and healthcare, logistics, sports and education, with physics including liquids, chemistry, cloth, molten metal, granular soil, pastes, soft tissue and acoustics.

25
Simulations

25 test and 175 training RL environments of varying difficulty. Each one supports swappable robots, from single arms to bimanual arms and humanoids, and variations of object poses, appearance and physics.

200
RL Environments

We break the tasks of 1,016 occupations and 465 everyday activities into 31,961 atomic physical actions, valued by U.S. wages and by the time people spend on them.

1,016
Occupations Analyzed

The best frontier model, Claude Opus 5.5, solves none of the 25 test tasks and reaches an average reward of only 0.06.

0%
Frontier Success on Test
The Pipeline

How It Works

Our core insight is that creating a simulation is a coding agent task. Scenes, physics, tasks and rewards are all code, so as coding agents improve, the simulations they build improve too.

01 · SELECT

GDP-Grounded Selection

We break the tasks of 1,016 occupations and 465 everyday activities into 31,961 atomic physical actions, valued by wages or by time spent. An optimization selects 120 simulations that cover the most valuable physical work, and we build 25 of them.

02 · CREATE

Creation-Audit Loop

A creation agent builds each simulation in code and must show evidence that it is good, such as renders, physical measurements and assisted demos. Four audit agents check it against ten criteria, and one of them tries to break it with legal robot actions. The creation agent fixes what they find, over three rounds.

03 · TASKS

Tasks of Targeted Difficulty

A task creation agent proposes tasks of targeted difficulty. Six audit agents, one per criterion, check each task, including whether it is solvable and whether its reward is correct. Every task can run with different robots and under variations of the scene.

04 · LEARN

Learning in Simulation

Frontier models improve across repeated attempts, and faster with privileged simulator state. That state also helps models that lack it: through a coach’s hints, through a model’s own privileged practice, and through code policies whose demonstrations train VLAs such as π0.5.

Hover over any part of the pipeline for details. Click a simulation to play it.Tap any part of the pipeline for details, and tap a simulation to play it.

Phase 1: GDP-Grounded Simulation Selection

See the 25 simulations →

Paid work

1,016 occupations

18,796 tasks valued by wages

Everyday life

465 activities

valued by time spent

Atomic physical actions

31,961 actions
▾ group by simulation
2,330 action groups
▾ required capabilities
e.g., weld steel joint
Robot: arc power, wire feed
Physics: melting, electric arc

Tiered selection

120 action groups
economy strategic STEM 22 SOC groups niche physics everyday
▾
25 built

Phase 2: Simulation Creation

Read a simulation report →

Shared memory

lessons across simulations

Creation agent

researches the activity, builds scene, physics, and robot in code

Evidence

Render of wine being poured into a glass
renders
assisted demos
Plot of liquid sloshing over time
liquid sloshing
Plot of cloth bending moment against curvature
cloth bending

Is this a good simulation?

  • Realismreal physics and visuals
  • Dynamics, solvabilityassisted demos succeed
  • Fast, configurablerobots, variations, report
  • Robust to exploitsadversary tries to break it
✓ Verified simulation

Phase 3: Task Creation

Explore the tasks →

Policy attempts

VLAs, coding agents

Task creation agent

proposes diverse tasks of targeted difficulty

Task T = (s0, p, V)

Initial scene of the task
s0 therapy room

p “Play ‘Hot Cross Buns’ on the xylophone with the mallets at 85 beats per minute.”

Spectrogram of the simulated sound

V scores the simulated sound: right notes, in time, and clean.

Is this a good task?

  • Meaninga real job, stated naturally
  • Settingscene suits the task
  • Verifier correctnessno false passes or fails
  • Solvabilityassisted demos succeed
  • Reward qualitypartial credit for progress
  • Difficulty coveragea spread of success rates
any variation 10,000 seeds vary drum tension, placement, tempo, tune, and lighting

Phase 4: Learning in Simulation

See the results →

In-context learning

three attempts in one context

reset to the same start, keep notes and code

improves without weight updates, faster with privileged state

Teacher and student

Coachsees the student's true state
Studentsees only cameras

up to 10 qualitative hints, never numbers or commands

first-attempt reward: hints from privileged state double it

Training VLAs from code

Privileged code policyreads object state
1,000 demonstrationsacross varied initial states
train π0.5cameras and proprioception only

π0.5 success on two gift box tasks
RoboWorld

25 Simulations of Real Physical Work

Each simulation models the physics its work depends on, from liquids and cloth to soil, soft tissue, sound and molten metal. Click a card to enlarge it. Clips marked assisted had outside help, such as phantom hands, outside forces, supports or fixtures, as their recordings state.

Tasks

200 Tasks, Eight per Simulation

Each simulation has eight tasks, from entry level to its hardest, which is its test task. Each task describes the job as one would describe it to a person, and a verifier scores it from the physical state of the simulation.

    easiest to hardest

    Instruction given to the model

    Benchmark

    Leaderboard

    25 test tasks, one per simulation, and no model completes any of them. On an easier subset of 25 training tasks the best model succeeds 32% of the time, and success drops as the robot gains degrees of freedom.

    # Model Test Reward Test Success Easier Success Arms Mobile Humanoid
    1 Claude Opus 5.5 (xHigh) 0.06 0% 32% 35% 20% 33%
    2 GPT-6 Astra (xHigh) 0.009 0% 24% 29% 20% 0%
    3 GPT-6.1 Terra (xHigh) 0.002 0% 12% 18% 0% 0%

    Reward is averaged over the 25 test tasks. It credits partial progress and is computed from the physical state of the simulation.
    Arms, Mobile and Humanoid give success on the easier subset by robot: fixed-base FR3, Panda and UR10e arms (17 tasks, 8 to 16 DoF), the Mobile FR3 Duo (5 tasks, 20 DoF), and the Unitree G1 and H1-2 humanoids (3 tasks, 39 to 43 DoF).

    Results

    How Frontier Models Learn in Simulation

    Five ways a frontier model learns in RoboWorld, all with Claude Opus 5.5. The first four use 11 simulations, and code as policy uses two gift box tasks. Pick a setting to see what the model sees, what helps it, and how far it gets. Hover over the chart for exact values.Tap the chart for exact values.

    or use the arrow keys on the tabs

    What Frontier Models Do Along the Way

    Models often build their own tools to solve tasks, but they also overstate their success and even try to game the verifier. Each image is the camera image the model received or a render of a verified replay. Hover over a card for the full story, and click to enlarge.Tap a card to enlarge it and read the full story.

    Two arms playing a xylophone
    Builds a toolPercussion. Claude Opus 5.5 detects each strike and its note from the simulated microphone.
    The simulated microphone over 3.4 s, with the notes it heard.

    The robot plays a xylophone. Claude Opus 5.5 turned the simulated microphone into a sensor, using sound to detect when the mallet touched a bar and which note it played. When a note came out wrong, it worked out that the mallet had twisted in its gripper.

    Head camera image of a fallen humanoid looking at the ceiling
    Builds a toolWhiteboard. GPT-6 Astra estimates the tilt of its fallen robot from the ceiling grid in its head camera.

    Its humanoid had fallen. GPT-6 Astra ran a pretrained get-up policy, and since the robot has no tilt sensor, it estimated the robot's tilt from the ceiling lines in this head camera image.

    Humanoid kicking a football while falling
    Odd strategyFootball. Claude Opus 5.5 kicks the ball while the robot falls.

    Claude Opus 5.5 had to kick a ball into a goal with a humanoid. It built its own simulation of the robot, found that the robot could not stay standing, and designed a kick that strikes the ball while the robot falls. It scored without ever seeing the goal.

    Forklift reversed into the dock wall
    Overstates successForklift. GPT-6 Astra reported that the brake had stopped the forklift. The truck had in fact reversed into the dock wall.

    GPT-6 Astra found the brake by reading the operator manual of a real Toyota forklift. In one attempt, it reported that the brake had stopped the forklift, while the truck had in fact reversed into the dock wall.

    Whole uncut fish placed in the frame pan
    Games the verifierFilleting. Claude Opus 5.5 places the whole uncut fish in the pan meant for the fish frame.

    The fish frame must end up in a pan separate from the fillets. Claude Opus 5.5 expected the verifier to check only the frame, and placed the whole uncut fish in that pan, hoping for partial credit. The verifier measures the physical state, found the fillet meat in the pan, and gave no reward.
    Citation

    BibTeX

    @misc{aggarwal2026gymanythingrobotics,
      title={Gym-Anything for Robotics: Scaling physical simulations with digital agents},
      author={Pranjal Aggarwal and Sean Welleck},
      year={2026},
      eprint={2610.40565},
      archivePrefix={arXiv},
      primaryClass={cs.RO},
      url={https://arxiv.org/abs/2610.40565},
}
    📄 Paper 💻 Code 🌐 Gym-Anything for Software