CULPRIT MODEL-STACK FORENSICS Open case

The debugging agent for multi-model systems

Reconstruct the failure.
Put the culprit on record.

CULPRIT does not ask a model to read logs and guess. It re-runs the system, changes one component at a time, and follows the evidence from symptom to component, checkpoint, and training data.

Investigate incident 004 Review the evidence
Decisive moment
Frame 7 · 0.7s
Proven actor
perception.detector
Boundary
ckpt-3 → ckpt-4
CAM/WRIST-01 MONO · 10 FPS
TC 00:00:00:700
040506070809
10/10 OUTCOME FLIPS

01 / THE CASE

A wrong output is a symptom.
A cause survives intervention.

Robots and agent systems are model stacks. A downstream planner may look wrong because perception gave it the wrong world. CULPRIT separates sequence from cause by replaying the same run with one output replaced by an oracle reference.

01

Logs show order

A trace can show which call came first. It cannot prove which output made the final outcome fail.

OBSERVATION
02

Replay tests cause

Substitute the detector reference, re-run planner and controller, and observe whether the task outcome changes.

INTERVENTION
03

Descent explains why

Bisect the implicated model history, confirm rollback, then compare the exact training manifests at the boundary.

ATTRIBUTION

02 / EXHIBIT A

One fixture.
Three evidenced verdicts.

The shipped CPU reference harness creates twelve frames, one low-light failure, five detector checkpoints, and a known training-manifest change. Every number below is regenerated by the Python implementation.

A

Responsible component

perception.
detector

DETECTOR100%
PLANNER0%
CONTROL0%

Oracle substitution flips the task outcome across 10/10 deterministic seeds.

B

Regressing transition

ckpt-3 →
ckpt-4

12345

Two binary-search evaluations, 38 failing probes, eight controls, rollback confirmed.

C

Training-data change

DATA_
COMPOSITION

CKPT-34.1%
CKPT-40.7%
REGRESSION SET81.6%

The depleted low-light slice matches the regression-set composition.

Evidence standard: correlation proposes a hypothesis. A deterministic outcome flip supports component attribution. A bounded pass-to-fail transition plus rollback supports checkpoint attribution. Manifest direction matching the regression slice supports a tier-2 data-composition finding.

03 / METHOD

A controlled descent, not a generated explanation.

  1. Normalize the run

    Agent traces and decoded MCAP-style envelopes become one actor-keyed timeline. The original evidence remains untouched.

    INGEST

  2. Find the decisive moment

    A logarithmic replay search finds frame 7, the earliest substitution horizon that recovers the outcome.

    BISECT TIME

  3. Intervene component by component

    Ground-truth detector output recovers the grasp. Planner and controller substitutions do not.

    REPLAY × 10

  4. Bisect model history

    The probe set bounds the regression between ckpt-3 and ckpt-4 and confirms that rollback holds.

    2 EVALUATIONS

  5. Audit the changed evidence

    Content-addressed manifests expose the low-light drop; the changed slice matches 31 of 38 regressions.

    TIER 2
ENTER THE RECONSTRUCTION ROOM Run the intervention yourself

04 / TWO HONEST ENDINGS

Evidence can name the culprit.
It can also refuse.

WHEN AN OUTCOME FLIPS

ATTRIBUTED

Name the component and decisive frame. Unlock checkpoint bisection. Descend into the manifest only after rollback is confirmed.

  • Intervention recorded
  • Ruled-out actors retained
  • Confidence tied to repeatability
WHEN NO TEST DISCRIMINATES

UNATTRIBUTED

Stop the descent. Preserve missing references and failed hypotheses. Export an unresolved finding instead of guessing.

  • No synthetic confidence
  • Downstream claims remain locked
  • Next evidence request is explicit

05 / UNDER THE FLOOR

A real reference harness beneath the interface.

The workbench is a browser-side reconstruction of artifacts emitted by the dependency-free Python engine. Installed users can execute that engine through the CLI or a loopback HTTP service with health/readiness probes, SQLite lineage, finding hashes, and per-run JSON/HTML evidence bundles.

Inspect the repository
JOURNEY 0 · KEYLESS · CPU ONLY $ git clone https://github.com/Aneesh-Pothuru/culprit
$ cd culprit && make demo
$ culprit serve # installed live reference API

06 / ABOUT THE CLAIM

What CULPRIT is—
and what v0.2 is not.

Implemented

  • Deterministic component → checkpoint → data descent
  • Normalized agent and decoded MCAP-style JSON traces
  • Multi-seed counterfactuals and honest abstention
  • Checkpoint probes, controls, rollback, and slice audit
  • Installable CLI + local HTTP API with durable evidence

Not claimed

  • No raw binary MCAP or ROS 2 runtime in the keyless path
  • No Who&When or external benchmark measurement
  • No real trained PyTorch components in the toy stack
  • No tier-3 TRAK/TracIn or automatic repair
CASE // READY

Don’t read the postmortem.
Re-run the cause.

Open the investigation workbench