SIA Eval Harness

A read-only compiler for SIA self-improvement runs. One command turns runs/run_* into a receipt that separates harness gains, weight gains, and overfit.

The solution

Problem

SIA improves both code (harness) and weights in one loop, but after a run you only get scattered files — results.json, improvement.md, agent diffs. Nothing answers: where did the score gain actually come from?

Solution

sia-eval compile walks a finished run folder and writes a run receipt — one JSON + markdown file with harness Δ, weights Δ, leak check, overfit residue, and artifact hashes. No re-run. No fork of SIA.

What it separates

Harness gain — code/prompt changes (target_agent.py)
Weight gain — RL tuning (train.py)
Overfit — task-specific residue + gold-label leaks

Test the solution

Click the button — runs the real compiler on a bundled demo run. Works in-browser, no install.

terminal

Press the button above to compile…

Receipt output

Run compile above to generate receipt

Waiting

—

Gen	Focus	Metric Δ	Harness Δ	Leak	Residue
No receipt yet — click Run compile

// receipt appears here after compile

The product

Open-source Python CLI — github.com/Abhishek21g/sia-eval-harness

$ git clone https://github.com/Abhishek21g/sia-eval-harness.git

$ pip install -e . && sia-eval compile demo/runs/run_1

Related OSS work

#52 — spaceship evaluator (opened)
#51 — transfer evidence (opened)
#36 — leak fix (approved)
#41 — Windows paths (approved)

Get the repo Portfolio