Semantic TraceGraph

What this is

Coding agents produce long transcripts. Scoring those traces with a single success bit collapses three different questions: what the agent claimed, what the trajectory actually shows, and which claims have mechanical proof (a test, a patch, a reproduction).

Semantic TraceGraph turns a transcript into a small evidence-grounded accomplishment graph. Nodes are coarse, non-overlapping claims (localization, diagnosis, reproduction, implementation, verification, handoff). Edges record causal structure. A separate judge scores each node as claimed, evidenced, or verified without rewriting the graph.

How it works

Both the decomposer and the judge are tool-using LLM loops. The model sees a trace summary and can inspect events through a read-only Python sandbox (trace.event, trace.grep, coverage of uncited spans, and so on). Invalid graphs and verdicts are rejected so the agent can retry inside a step budget.

Evaluation treats graph progress as more than node count: semantic duplicates are merged, invalidation and regressions are penalized, and results are grouped by task ID so several model trajectories for the same issue are not treated as independent samples.

Status

This is an early prototype with a working ingest, decompose, verify, and evaluate pipeline on SWE-Router-style coding traces. Pilot numbers so far are a pipeline smoke test, not evidence for the research claim. Human agreement, execution-level checks, and a handoff continuation experiment are still ahead of us.

Gold patches and hidden tests stay evaluation-only. A successful command is evidence that something ran, not that the semantic claim is correct.

Using the prototype

Clone the repository, install with pip install -e '.[dev]', and set TRACEGRAPH_API_KEY. The CLI entry point is tracegraph (audit, sample, decompose, verify, evaluate). Details are in the README.