Pokémon Agent

Local models running on the desk play Pokémon FireRed and Emerald unassisted, one button press at a time. Nothing leaves the machine, so the only running cost is electricity.

It is easy to make a model look like it is playing a game: give it a "fight this battle" function, a "walk to the gym" function, and it will finish the game without ever making a decision. What is being tested then is the harness, not the model. So this one has a single rule, and everything else follows from it: the only thing the model is ever allowed to do is press one button. There is no macro for a battle turn, a menu, or a route. Every step across Kanto is a press it chose.

No single model does all of this. It is a multi-agent system, and each agent's job is to make the deciding model better informed, never to decide for it. A vision model turns the raw frame into a structured observation; a stronger text model turns that observation into one button; and a set of specialists supply what a screenshot cannot show, reading the console's own memory for route geometry, level caps derived from the game's trainer data, catch and party scoring, and whether a wild encounter is shiny. All of it arrives as advice the deciding model is free to ignore. An audit of where each press came from puts about 99% of them on that model; the remainder is bookkeeping the game itself leaves behind. Nothing in the path is specific to one cartridge, which is the point, the same loop drives FireRed and Emerald runs side by side.

How it works

• One press, always: press(button) is the whole interface to the game; no compound action exists for the model to hide behind.
• Perceive, then decide: a vision-language model reads the frame into an observation, screen kind, dialogue, menu cursor, HP, move PP, and a stronger text model reads that observation and answers with a single button and a reason.
• Specialists that see past the pixels: separate agents read the game's memory for what a frame cannot say, the route through a map whose barriers are painted on at load time, the level cap implied by the next major battle, whether the wild Pokémon on screen is a 1-in-8192 shiny. They advise; they never press.
• Tools inform, never choose: pathfinding, battle state, and level caps are offered as advice; move selection, switching, healing, and where to walk are all the model's.
• Three runs at once, self-supervising: FireRed and Emerald play in parallel and relaunch themselves when a run wedges. Every sentence either game prints is logged, so a stall is diagnosed from the artifacts afterwards instead of by sitting and watching for it.
• Beaten so far: Brock end to end, and Elite Four Lorelei with all six of the party still standing. The failure that preceded it was tactical, a weaker model overstaying a bad matchup, so the ceiling turned out to be the model's depth rather than the harness.
• Everything is logged: each run writes a trace with one record per decision, what the model was shown, what it answered, which buttons fired, and the state before and after, and a browser panel streams the same thing live while it plays.

Stack

Python · mGBA · Ollama, with qwen2.5vl:7b perceiving and qwen2.5:14b or qwen3.6:27b deciding. Every agent in the system runs on one machine on my own GPU: no API key, no cloud model, no per-token bill, and nothing hosted to visit. Three runs can play around the clock and the only thing they consume is electricity.

Next the rest of the Elite Four, and a window-grab host so the same loop can play games that have no emulator to read from.