Can an agent understand what it sees, choose a useful action and respond while the game is still moving?
Intelligence, viewed through play
A game gives an agent something concrete to do. It has to understand a scene, choose an action and deal with the consequences. The feedback can be immediate: a coin collected, an obstacle avoided, a target missed. That makes games an interesting place to investigate perception, control and planning together.
The project motto is: “Intelligence is the ability to adapt to new environments.” — We test this. The current benchmark measures bounded visual gameplay; it does not turn a game score into a complete measure of intelligence.
A clock that counts the whole response
GeneralGameBench gives an agent rendered pixels and a set of allowed controls. A fresh challenge ties each reply to its observation, while the referee owns the game score and the timing. Agents cannot simply report their own reward or their own reaction time.
The leaderboard limit is suite p95 below 200 milliseconds. Exactly 200 milliseconds fails. We pool all recorded response times and use linear interpolation to calculate p95. The timer includes image retrieval, encoding, transport, inference and validation. Game updates between decisions stay outside the timer. Slower agents can appear in the model exhibition. The game waits between actions. These tests do not establish continuous real-time control.
What the first season actually shows
The first published snapshot contains 436 episodes and 26,712 decisions: 400 runs from four local baseline policies, plus 36 short exhibition episodes. That initial suite used two original 2D games and eight ViZDoom scenarios. Those eight scenarios belong to one Doom game family.
In that first snapshot, Pixel Tracker led the local baseline suite. Actual GPT-6 Astra and Claude Opus 5 calls also played the exhibition tasks, but both exceeded the original 100-millisecond limit. That historical rule required every response to pass. Their calls included fresh client startup and network time on each decision. With only two seeds and an eight-decision horizon, that exhibition is an integration demonstration rather than a reliable model comparison.
All current results are provisional and locally measured. The official leaderboard remains empty until independently administered, isolated evaluation and signed evidence are available. Recorded frames, action traces and replay checks make the current work inspectable without pretending that a local run is independently certified.
Expanding the suite on one Mac
The expanded implementation adds 23 research tasks: all sixteen Procgen games, Crafter, MiniWorld, Pistonball, Breakout, Airstriker, NetHack and MiniHack. That brings the tested suite to 33 scenarios across 26 catalog cards. Each new task must repeat the same rendered frames and score under the same seed and actions before it is admitted.
The expanded snapshot records 1,089 episodes and 14,394 decisions. Seventeen configured models played: seven OpenAI, three Anthropic and seven local vision models, alongside four built-in reference policies. The shared task set, prompt, seed and eight-decision horizon make the setup inspectable. Invalid actions remain visible as errors and zero-score episodes. This short demonstration cannot establish a robust intelligence ranking, and hosted versus persistent local transports carry different latency overheads.
Version 0.3 adds ten validated tasks: Unity VisualFoodCollector, Google Research Football Academy, SuperTuxKart Lighthouse, Luanti ChopTree, SuperTux, Dungeon Crawl Stone Soup, OpenTTD, Mindustry, Cataclysm: DDA and Warzone 2100. Their controls, native scores and exact camera replays were tested on this Mac. Nine run in local Linux ARM64 containers; Unity uses the pinned Mac example. Published engine patches synchronize native ticks, input and rendering. The catalog now has 43 runnable scenarios across 36 validated cards. The 5 October model refresh evaluates all 43 scenarios in a separate cohort. The previous 33-scenario results remain available, including the local vision models, and are not mixed into the new aggregate.
Nine catalog cards remain unfinished. Native 0 A.D., StarCraft II and Veloren prototypes run here but still fail exact visual replay or task admission. Six other game families need usable local installations as well as adapter work. The StarCraft II beacon route repeats its native score, but animation pixels differ; that is why it is not counted as a verified task. A preview image is not evidence that an integration is complete.
Rerunning the local references headlessly
The 5 October baseline refresh runs Idle, Random, Pixel React and Pixel Tracker across all 43 admitted scenarios. Three fixed seeds and a 24-decision horizon give 516 episodes. All game rendering stays off screen, and scored episodes run one at a time before the replay audit begins. No OpenAI or Anthropic calls are needed for these coded reference policies.
All 516 episodes passed checks against their recorded screenshots, accepted actions and scores. The 21 timeouts remain as zero-score episodes. The result archive and website data download contain the fresh baselines. The public page keeps Model exhibition and Official ranked, as requested later. These measurements come from a shared Mac without independent certification. All four baseline policies pass the new suite p95 limit below 200 milliseconds. Their scores and original recordings stay unchanged.
What the leaderboard score means
Each task maps its native reward into a fixed range from zero to one. We average the declared seeds within each scenario, average all scenarios equally, then multiply by 100. Scenario scores of 80, 40 and 0 therefore give a suite score of 40/100. Aborted episodes count as zero; incomplete suites receive no aggregate score.
The anchors are task-specific, not human performance or an IQ scale. A game family with several scenarios contributes more weight than a family with one. The latency limit is separate from the score. The complete suite p95 must be below 200 milliseconds. Slow responses, errors and aborted episodes stay in the evidence. A latency pass does not grant an official rank.
The hosted-model refresh keeps the short one-seed, eight-decision settings and adds the new environments. This is a reproducible integration demonstration. Many tasks barely start in that time, so it does not establish which model is generally more intelligent. Longer runs and more held-out seeds are needed for a stronger comparison.
Published on Hugging Face
The public leaderboard also has a Hugging Face Space and a separate results Dataset. Each publication retains exact seeds, task and model revisions, scores, latency and trust labels. The Space links to the matching Dataset revision.
Separate computers run the games. The Space displays the current model results and an empty official track. It does not execute submitted code or store model-provider credentials. Baseline results and earlier tests remain in the GitHub archive.
See the games before reading the scores
The environment catalog now has an image and a short description for every card. Selected environments also have short clips that play on demand. Cards identify recorded benchmark frames and credited project or publisher previews separately; a preview is not itself evaluation evidence.
The catalog reaches beyond the initial suite, from procedural 2D games and open-source worlds to Minecraft, StarCraft II, Factorio and other commercial titles. These are integration candidates, not a claim of existing support. Each card keeps that distinction visible and links to its source.
Bring an agent, keep the evidence
The participant interface is deliberately small: read an image and allowed actions, then return the observation's challenge and an action. Reference policies and a runnable example are included, so a first local evaluation does not require a model API account. Supported benchmark runs render games off-screen and save their images and actions for replay without opening game windows on the desktop.
The next work is about stronger experiments: longer tasks, more seeds, modern 3D adapters, continuous capture and an independently isolated competition service. The public repository contains the code, methodology and roadmap; the leaderboard on this site is the place to explore the measured snapshot and the growing catalog.
