Scaling Real-Time Environments for Vision-Language Agents
Video games and physical devices keep changing while an agent thinks. Gym-Anything-RT builds real-time minigames to measure this, and controls their clock so the same task can be played live, slowed down, or paused.
Oracle solutions of minigames in Gym-Anything-RT, at difficulty 5 with full interaction.
Frontier models take several seconds for every input, and the minigame keeps moving in the meantime. The model replays are sped up, and their clock shows the real elapsed time.
All runs use difficulty 5 with full interaction in the live setting, on the same seed. Orange rings mark where the model clicked after seeing each screenshot.
100 real-time minigames at the highest difficulty. Every model does far better on the non-real-time counterparts of the same minigames, and every model is slower than a human.
| # | Model | Real-time | Non-real-time | Time / attempt | Cost / attempt |
|---|---|---|---|---|---|
| – | Human* | 45.0% | – | 2.2 min | – |
| 1 | GPT-6 Astra (xHigh) | 9.0% | 66.0% | 7.2 min | $4.50 |
| 2 | GPT-6 Astra (Low) | 8.0% | 58.0% | 4.3 min | $2.76 |
| 3 | Claude Opus 5.5 (xHigh) | 5.0% | 57.0% | 15.8 min | $6.40 |
| 4 | GPT-6 Luna (xHigh) | 1.0% | 17.0% | 6.4 min | $0.07 |
| 5 | Qwen3.8-27B | 1.0% | 2.0% | 63.0 min | $0.64 |
| 6 | Qwen3.5-9B | 0.0% | 1.0% | 75.9 min | $0.62 |
Pass rates at difficulty 5 with full interaction in the live setting. Time and cost are averages per real-time attempt.
*One of the authors, on 20 randomly sampled real-time tasks, seeing each minigame for the first time.
Frontier models fail mainly because they are too slow. Giving them more time recovers most of the gap, and letting them write code that acts on their behalf recovers part of it.
Pass rates on the 100 real-time minigames and their non-real-time counterparts.
Slowing the environment raises Astra's pass rate on the same real-time tasks from 9% to 57%, and pausing raises it to 75%.
When allowed to run code, Astra writes scripts that perceive and act between its own responses, raising its real-time pass rate from 9% to 31%.
Coding agents build each minigame as a program, and our evaluation stack controls its clock.
A creator agent turns a seed puzzle into a real-time minigame program with an oracle solution that shows the task is solvable. Independent audit agents test it and send problems back until it passes.
Our evaluation stack controls every clock in the minigame, so the same task can run at its natural speed, slowed down to give the agent more time, or paused between observation and action.
Each minigame has a non-real-time counterpart, five difficulty levels, and two interaction modes. Review agents label the capabilities each configuration requires.
The minigame runs at its natural speed while the model reasons.
Every clock in the minigame runs slower, while actions and screenshots stay real time.
The minigame advances through a short observation window, then freezes until the model acts.
@misc{aggarwal2026gymanythingrt,
title={Gym-Anything-RT: Scaling Real-Time Environments for Vision-Language Agents},
author={Pranjal Aggarwal and Sean Welleck},
year={2026},
eprint={2610.37567},
archivePrefix={arXiv},
primaryClass={cs.LG},
url={https://arxiv.org/abs/2610.37567},
}