Data engineer

Your golden path is raw capture → canonical RRD episode → catalog lineage → immutable selection → lazy training read or export.

The path

Start from an exact task

Carry one TaskContractRef into capture and keep it on every derived artifact. Digest mismatch or unknown identity fails closed.

Preserve the source

Raw robot data is immutable and is never deleted. Processing derives new artifacts; it does not rewrite the evidence that produced them.

Register the complete episode

RRD is the governed episode form across real and simulated data. Calibration, spatial context, action provenance, and lineage travel together.

Freeze membership

Search, semantic retrieval, and atlas selection are exploratory. Snapshot the exact dataset or explicit selection before training so membership is reproducible.

Project at the edge

Catalog reads remain lazy Arrow/DataFusion plans. NumPy, Torch, LeRobot, or customer exports are terminal projections, not parallel sources of truth.

Give this to your agent

Trace dataset <name> from raw source through canonical RRD registration to its latest immutable
training selection. Cite task refs, episode refs, provenance, transforms, and readiness.
Do not mutate or export. Identify missing lineage and the smallest fail-closed repair.

Review checklist

One source owns each independent fact; derived indexes and exports can be rebuilt.
Raw paths are outside every delete, cleanup, and migration action.
Every episode records the exact embodiment and action interface that produced it.
Corpus membership is content-addressed before compute begins.
Training never materializes more than its terminal batch boundary requires.
Errors identify the failed contract part; no empty string, default, or stale cache hides it.

Use agents for evidence archaeology

Ask an agent to follow one episode across receipts, outboxes, catalog rows, and tests. This is often faster and safer than starting from storage topology.