How far can a local AI agent get using only what is visible on screen?
The experiment
ScreenQuest uses Hordes.io as a visual test environment for a game-playing agent on Apple Silicon. It observes screenshots, chooses bounded actions, and reviews the resulting evidence. Model inference runs locally after the initial downloads.
The interesting question is what happens between perception and action: the world keeps moving while a model is thinking. A useful controller needs to act on a recent frame and check that its assumptions still hold.
Two speeds of thinking
A fast path combines templates, geometry and rules with Laya through Core ML. A slower Qwen vision-language planner runs through MLX. The controller does not wait for a fresh planner response before every action, and it rechecks the screen before sending input.
Saved frames and event logs support a separate review step. A click is not evidence of a pickup; a newly visible pickup message is. That distinction keeps the experiment grounded in observable outcomes.
What the evidence supports
The repository demonstrates combat inputs, camera gestures and a coin pickup in a separate trial. Navigation remains inconsistent, and equipment collection has not been verified. More complex games are a motivation for future work, not a demonstrated capability.
