LGeneral-bench

A benchmark for long-horizon sequential decision-making: LLMs play the turn-based strategy game LGeneral against the built-in CPU, over a REST API, and are scored on a graded victory scale.

A game in progress: the Poland scenario, played move-by-move over the REST API.

Most LLM benchmarks are single-shot (answer a question) or short-horizon (one coding task). This one asks a model to play a whole game — 10–20 turns under fog of war — where every move depends on the last: spatial reasoning on a hex grid, resource management (fuel, ammo, prestige), risk assessment from combat odds, adapting to an opponent, and the discipline of driving a stateful API without losing track of what changed.

It is resistant to training-data contamination (LGeneral is niche; the arena map is hand-authored), cheap and reproducible (local engine, ~8 min/game, no cloud dependency), and it already discriminates: a strong agent playing Poland-as-attacker loses to the CPU, while the same agent wins the symmetric arena — so there is real dynamic range.

---

How it works

   ┌─────────────┐   HTTP (schema:3)    ┌──────────────────────────┐
   │  pi + LLM   │ ───── GET /state ──▶ │  LGeneral engine          │
   │  (an agent) │ ◀──── deltas ─────── │  hosts the game + a       │
   │             │ ───── POST /action ─▶│  libmicrohttpd REST server│
   └─────────────┘                      │  opponent = built-in CPU  │
         ▲                              └──────────────────────────┘
         │ prompt = frozen API docs               │ /healthz.result
         │                                         ▼
   ┌─────┴───────────────────────────────────────────────────────┐
   │  bench/  — runner owns the engine process, watches liveness   │
   │  via state_version, classifies the outcome, scores it,        │
   │  appends one JSONL line, renders the leaderboard.             │
   └───────────────────────────────────────────────────────────────┘

The engine itself hosts an HTTP server (a third player-control mode, --control remote). An external agent GETs the game state and POSTs actions. Units are addressed by hex coordinate, and every unit comes with its inline legal moves and attacks (with expected-loss estimates), so the model reasons about strategy, not about re-deriving the rules. See lgeneral/README.remote-player.md for the full API contract.

The benchmark harness (bench/) wraps pi (an agentic CLI) as the fixed agent and swaps only the model under test, so it scores the combination model + fixed prompt + fixed harness, identically for everyone.

Quick start

1. Build the engine (needs SDL 1.2 and libmicrohttpd):

cd lgeneral
./configure --disable-install        # on macOS/Homebrew, brew-install sdl12-compat + libmicrohttpd first
make                                  # ⇒ lgeneral/src/lgeneral

2. Get the game data (required — see Game data & licensing):

# download pg-data.tar.gz from https://sourceforge.net/projects/lgeneral/files/lgeneral-data/
tar xzf pg-data.tar.gz -C /tmp
cd lgeneral/lgc-pg && ./lgc-pg -s /tmp/pg-data -d ../src   # converts into lgeneral/src/

3. Smoke the whole pipeline with zero LLM cost (a dummy agent that just passes its turns):

MODELS="dummy/dummy" AGENT=dummy N_ARENA=1 N_POLAND=1 bench/run-matrix.sh
cat bench/LEADERBOARD.md

4. Run a real model (via your pi install, keys already configured):

MODELS="deepseek/deepseek-v4-flash" N_ARENA=10 N_POLAND=5 bench/run-matrix.sh  # full protocol: 70 games
# (bare `bench/run-matrix.sh` defaults to a quick N_ARENA=2 N_POLAND=1 → 14 games)
cat bench/LEADERBOARD.md

Full protocol, scoring tables, and pinned versions are in bench/README.md.

Play against the LLM yourself

The benchmark is headless (LLM vs the built-in CPU), but the same runner does double duty: set OPP=human and you take the keyboard in the SDL window while the LLM commands the other side — on any scenario, using the exact same frozen prompt the benchmark scores. You play the side opposite SIDE; there are no timeouts and the session isn't scored.

# You play axis (attacker) on Poland; the LLM defends as allies:
MODEL=deepseek/deepseek-v4-flash SCENARIO=pg/Poland SIDE=allies OPP=human \
  bench/run-one.sh

# Flip it — you defend the arena while the LLM attacks:
MODEL=deepseek/deepseek-v4-flash SCENARIO=pg/Benchmark SIDE=axis OPP=human \
  bench/run-one.sh

While the LLM is on the move the game window ignores input (that side is not yours to play), so the runner prints whose turn it is and how long the model has been thinking. Two flags make the wait pleasant: VERBOSE=1 streams the model's reasoning to your terminal, and THINKING=low (or off) makes it move faster.

The engine keeps running after the game ends so you can read the result from GET /healthz. Full control-mode and API details: lgeneral/README.remote-player.md.

The difficulty ladder (protocol v2)

Each track is played from both sides (mirrored) for fairness; harder tracks carry more weight in the overall score.

Level	Scenario	Terrain	Weight
L1	pg/Benchmark	open plain, 13×9, 4 units/side	1.0
L2	pg/Benchmark2	central river + one ford, forested towns	1.5
L3	pg/Benchmark3	larger 17×11, 6 units/side, 2 towns/side, weather	2.0
Poland	pg/Poland	the real PG scenario — attack & defense	2.5 / 2.0

The three arenas are hand-authored and point-symmetric (both sides get identical mirrored forces and terrain), so any score gap is skill, not map luck:

In-game views (you command axis — black cross; the CPU holds allies — red/white). L2 and L3 add a central river — ford it under fire or detour around the open ends — and forests that shield each objective town.

Scoring

Per game, outcomes are graded, not binary — a model that loses 4:1 in a bloodbath scores above one that gets annihilated, and variance hurts less.

• Arena (layered): destroy-all 1.0 · more cities 0.8 · more units 0.6 · draw 0.5 · lost-by-units 0.4 · lost-by-cities 0.2 · annihilated 0.0.
• Poland: attack — major 1.0 / minor 0.8 / defeat 0.0; defense (inverted) — held 1.0 / narrow loss 0.4 / crushed 0.0.

Per track the score is shrunk toward a 0.5 prior ((Σpts + 4·0.5)/(N+4)), so a lucky game can't spike it. Overall is the difficulty-weighted mean over all tracks, where an unplayed track scores 0 — coverage is part of the score, so you can't rank high on two easy games or farm the easy arena while skipping the hard ones. A second column, Played, averages only the tracks played so far: how well the model plays, independent of how much of the ladder it has covered. Non-finishes where the model responded but never acted (dnf_exit/stalled/timeout) score 0 and show as a DNF-rate; failures where the model couldn't respond — provider api_error (429/quota/auth/5xx) or engine error — are excluded from scoring and re-run, so an exhausted token quota never counts as a loss. The board also reports per-level score ±95% CI, coverage, exchange ratio, turns-to-win, and seconds-per-turn.

Repository layout

Path	What
lgeneral/	The LGeneral engine (C, SDL 1.2, GPL) — a Panzer General reimplementation, v1.4.4, extended here with the PLAYER_CTRL_REMOTE REST server (lgeneral/src/remote_player.c).
lgeneral/README.remote-player.md	The agent-facing REST API contract + a "lessons learned" tactics appendix.
bench/	The benchmark: run-matrix.sh / run-one.sh (runner), score.py (graded scorer, unit-tested), dummy-agent.sh (LLM-free stand-in), prompt.md + frozen api-docs.md (the fixed agent input).
lgeditor/	A separate SDL2 map/scenario editor (unrelated to the benchmark).

Status

Protocol v1. The engine's REST API is covered by two test suites (lgeneral/tests/rest-api-v3-tests.sh, 36 checks; …-llm-tests.sh, 9 checks), and the scorer by bench/tests/test_score.sh (16 checks). The pipeline has been validated end-to-end with the dummy agent and with real LLM games.

Deferred to later: illegal-action-rate instrumentation, LLM-vs-LLM head-to-head (the API already supports two remote players), and a hosted leaderboard.

Game data & licensing

The benchmark needs the original Panzer General (DOS) data — unit, terrain and nation databases plus the Poland scenario. It is not included in this repository and never will be: that data is copyrighted by SSI (now Ubisoft) and carries no free license. The LGeneral project redistributes it as pg-data.tar.gz "in the spirit of abandonware" — download it from there yourself and convert it with the bundled lgc-pg (Quick start, step 2). The converter's output (lgeneral/src/scenarios/pg/, maps/pg/, units/, …) is a derivative of that data and is gitignored for the same reason.

What is in this repository is original work: the hand-authored symmetric arena (scenarios/pg/Benchmark + maps/pg/benchmark) contains no SSI content — it references unit prototypes by numeric id only and is unplayable without the user-supplied PG data, like any game mod.

Credits & license

LGeneral is by Michael Speck and contributors, released under the GNU GPL (see lgeneral/COPYING); this fork inherits that license. The REST remote-player layer and the benchmark harness are additions on top of the upstream engine.