CharadeBench

CharadeBench is a benchmark for testing spatial reasoning in language models. The local runner asks one model to generate structured 2D stick-figure animation data, obfuscates that data, asks a second model to guess the action, and uses a third model to adjudicate whether open-vocabulary guesses are close enough.

Requirements

• Python 3.11 or newer.
• No Python package install is required for the mock runner.
• OpenRouter API key only if you want to run real models.

The website in www/ uses Node and Next.js, but the benchmark runner itself is Python.

Run A Local Smoke Test

Use the mock provider first. It runs offline and verifies the full benchmark flow:

python bench/scripts/run_single_task.py --provider mock --seed 1

Run a small batch:

python bench/scripts/run_benchmark.py --provider mock --task-count 5 --seed 1

Run the default 50-action local batch:

python bench/scripts/run_benchmark.py --provider mock --task-count 50 --seed 1

The runner prints live progress by default. Add --quiet to suppress logs.

Run With OpenRouter

Set your key locally:

export OPENROUTER_API_KEY=...

Then run one task:

python bench/scripts/run_single_task.py \
  --provider openrouter \
  --llm1-model openai/gpt-4.1-mini \
  --llm2-model openai/gpt-4.1-mini \
  --llm3-model openai/gpt-4.1-mini \
  --seed 1

Run a batch:

python bench/scripts/run_benchmark.py \
  --provider openrouter \
  --llm1-model openai/gpt-4.1-mini \
  --llm2-model openai/gpt-4.1-mini \
  --llm3-model openai/gpt-4.1-mini \
  --task-count 50 \
  --seed 1

API keys are read from environment variables and are not written into artifacts.

Useful Options

• --task-count N: number of random tasks to run.
• --attempt-limit N: number of LLM 2 guesses per task, default 25.
• --llm1-repair-attempts N: correction attempts when LLM 1 returns invalid animation JSON, default 2.
• --no-llm3-adjudication: disable LLM 3 semantic adjudication.
• --no-render: skip preview and viewer generation.
• --output-dir PATH: write run artifacts somewhere other than runs/.

Validate And View Outputs

Validate a saved animation:

python bench/scripts/validate_run.py runs/<run-id>/tasks/task-0001/animation.json

Generate a browser viewer for a saved animation:

python bench/scripts/view_animation.py runs/<run-id>/tasks/task-0001/animation.json

Open it directly:

python bench/scripts/view_animation.py runs/<run-id>/tasks/task-0001/animation.json --open

Each benchmark task also writes viewer.html automatically unless --no-render is used.

Artifact Layout

Runs are written to runs/ by default:

runs/
  <timestamp>_<run-id>/
    config.json
    summary.json
    tasks/
      task-0001/
        action.json
        llm1_prompt.txt
        llm1_response.json
        animation.raw.txt
        animation.json
        validation.json
        frames.json
        preview.svg
        viewer.html
        obfuscated.txt
        llm2_prompt_initial.txt
        guesses.jsonl
        llm3_adjudication_prompt_01.txt
        llm3_adjudication_response_01.json
        score.json

If LLM 1 repair is needed, the task directory also includes llm1_repair_prompt_XX.txt, llm1_repair_response_XX.json, and animation.repair_XX.raw.txt.

More Docs

• Benchmark runner details: bench/README.md
• Default config: bench/configs/default.json
• Action set: bench/data/actions.json