Published on

Teaching an AI to Play Games by Watching YouTube

Authors
Written by Ghostpen � How it works
This post was AI-generated from development pipeline artifacts and reviewed by me before publishing.

Introduction

I wanted to build an AI that could learn to play games by watching other people play them. Not by training a neural network on millions of frames -- by actually understanding what is happening on screen, the way a human would learn from watching a YouTube video.

AutoAgent is the result. (The repo is currently private but may be made public in the future.) It watches gaming VODs (video-on-demand recordings), extracts game knowledge using vision APIs, compiles that knowledge into deterministic behavior scripts, and then executes those scripts in real-time using computer vision. The LLM writes the bot. Deterministic code runs it.

The Three-Phase Architecture

The system has three phases, and they run at very different speeds:

Watch (Offline)

The Watch phase is where learning happens. It takes a VOD URL, extracts frames at configurable intervals, and sends them to a vision API (Claude Haiku, GPT-4o-mini, or Gemini Flash via llm-router) for analysis.

The vision model looks at pairs of consecutive frames and infers what happened between them -- what changed on screen, what action the player took, what the game state was before and after. These observations go into a SQLite knowledge base as situational knowledge:

Frame 1: Player at town, health 100%, inventory open
Frame 2: Player at town, health 100%, inventory closed, new item equipped
Inferred action: Equipped item from inventory

For a typical 30-minute VOD, this means analyzing hundreds of frame pairs. That is why llm-router's batch processing matters -- submitting all pairs to Anthropic's Batch API at 50% cost instead of real-time pricing.

The Watch phase also runs pure computer-vision discovery passes that cost nothing -- no LLM calls. These discover HUD regions (health bars, minimaps, resource counters) and navigation patterns (tile dimensions, movement mechanics) directly from pixel analysis. The results feed into the Play phase's runtime perception.

Compile (Offline)

The Compile phase takes the situational knowledge from Watch and turns it into behavior scripts -- deterministic condition-action rules:

{
  "condition": "health_pct < 30 AND has_potion",
  "action": "use_health_potion",
  "priority": 90
}

These scripts are generated by a generation-tier LLM (Claude Sonnet). The key constraint is that conditions can only reference state fields that the runtime perception system can actually detect. If the CV system cannot read "health_pct" from the screen, the script cannot use it as a condition.

Play (Real-Time)

The Play phase runs in real-time with no LLM calls. The loop is:

  1. Screen capture via mss (~5ms)
  2. State detection via CV/OCR (~20ms) -- template matching, pixel checks, region reading
  3. Script lookup -- find the highest-priority matching script (~1ms)
  4. Execute -- send keyboard/mouse input

The entire loop runs at ~30ms per cycle. No API latency, no token costs, no network dependency. The LLM did its work offline during Watch and Compile. Play is pure deterministic execution.

If the agent encounters a situation no script covers, it flags it for the Improve phase rather than making an LLM call during gameplay.

The Dashboard

AutoAgent has a Next.js dashboard for managing the pipeline. You can add streamers, queue VODs for analysis, monitor job progress, and see cost breakdowns per provider. The recent model selector feature lets you choose which vision provider to use and toggle batch mode per job, with dynamic cost estimates.

The dashboard also shows provider health status -- colored dots indicating whether each LLM provider's circuit breaker is healthy, unhealthy, or recovering. This was part of the failover feature that ensures the Watch phase continues working when a provider goes down.

The cost calculator gives you an estimate before running a Watch session -- how many frames, how many API calls, and what it will cost at current provider pricing:

AutoAgent Cost Calculator

The Ecosystem

AutoAgent does not stand alone. It depends on two shared libraries:

  • llm-router -- handles all LLM API calls, provider selection, rate limiting, batching, and failover
  • frame-intelligence -- shared frame extraction and perceptual hashing for video analysis

The whole development workflow runs through EcoOrchestra, which enforces the persona pipeline across all repos.

Current State

As of March 2026, the Watch and Compile phases are functional. The Play phase is in development. The dashboard handles streamer management, VOD queuing, job monitoring, cost estimation, and provider health visualization. The next priority is wiring a live stream monitoring tool into the ecosystem via llm-router's HTTP API.

Conclusion

The core bet of AutoAgent is that LLMs are good at understanding game state from screenshots but too slow and expensive for real-time gameplay. So let them do the understanding offline, compile that understanding into fast deterministic rules, and run those rules at 30ms per cycle with zero API costs.