Transcript Lake

Documentation / Understand

Architecture

Runtime adapters interpret native transcript stores; shared readers handle appended lines and whole stores, then canonicalize, mask, and publish events. Tama telemetry enters through its separate segment reader.

Data flow#

text
vendor stores / Tama closed segments (read-only)
                |
          runtime adapter
                | RawEvent (unmasked, in memory)
                v
       canonicalizer + masker + caps
                |
        CanonicalEvent (masked)
          /          |          \
 daily NDJSON   Oko projection   cursor checkpoint
(authoritative) (rebuildable)    (resume state)
          |
 compiled DuckDB views
     /        |       \
read CLI   Parquet   operator labels
           mirror    (separate store)

Components#

Adapter layer#

src/adapters/ implements the runtime registry and native-format adapters. Each adapter discovers roots and sessions, resolves changed paths, and selects appended-line or whole-store reading. Whole stores include JSON documents, SQLite databases, and Markdown logs. transcript-lake sources --json lists the registered runtimes and their roots on this machine. src/hook_segments/ separately ingests Tama closed segments or the legacy hook log. Adapters interpret vendor fields; shared readers own IO and canonicalization owns masking. Malformed or torn line records yield no events, without discarding complete preceding records.

Stream layer#

src/commands/stream/ installs native filesystem watches before catch-up, so changes during initialization are queued. Shared readers follow the adapter's reading mode: consume complete appended lines or re-read a changed whole store. Canonical events map to deterministic source/date partitions. See the ingestion reference for store modes and native-store qualification.

The stream status file moves through catching-up, running, degraded, and final stopped state. Human logs use <timestamp> stream <kind> key=value; --json emits one object per log with ts, kind, and detail fields.

State and leases#

LAKE_DATA/cursors.json maps source paths to mtimeMs, size, and offset. Publication writes .cursors.json.tmp-<pid>-<uuid>, fsyncs it, renames it, then fsyncs the data directory under cursors.lock.

stream.lock/owner.json identifies one writer by host, pid, start time, and random token. An atomic prepared-directory rename claims the lease. Live owners cause an immediate refusal. Dead owners may be removed after liveness checks. Release verifies the token before removal.

Authoritative partitions#

Canonical events are append-only NDJSON at:

text
LAKE_DATA/events/runtime=<runtime>/date=<YYYY-MM-DD>/part-<12-hex-path-digest>.ndjson

The event date comes from the timestamp prefix or unknown. NDJSON partitions are the canonical masked store. Parquet and Oko exports are derived and may be deleted/rebuilt.

Oko projection#

Conversation events are projected to:

text
LAKE_DATA/exports/oko/runtime=<runtime>/<sha256(runtime + newline + session-id)>.jsonl

Each oko-import-v1 line includes a deterministic 32-hex UUID and the canonical conversation fields. Session files are deduplicated by UUID and ordered by timestamp then UUID. export-cursors.json records each partition's size, mtimeMs, and physicalSize after successful publication. Truncation, same-size rewrite, first run, or projection rebuild --target oko selects a full rebuild through bounded staging; ordinary appends are incremental. Oko scans this tree read-only.

Read model#

The binary embeds sql/views.sql and sql/signals.sql. Core views are events, sessions, interrupted_sessions, tools_daily, tokens_daily, hook_decisions, blocks_by_hook, and labels. Signal views attach Oko's SQLite catalogue only when requested. The CLI invokes DuckDB; streaming itself has no DuckDB dependency.

Labels#

LAKE_DATA/labels/labels.ndjson is an append-only operator annotation store with ts, session_id, runtime, aspect, value, note, and source. It is neither authoritative transcript evidence nor masked by the event masker. It does not share the event writer lease.

Failure containment#

  • Vendor stores are always read-only.
  • Invalid cursor state is refused rather than silently reset.
  • Source shrink or rewrite is refused rather than merged.
  • Recovery writes only to a different empty root.
  • Malformed Oko rebuild rows refuse cursor advancement and do not modify authoritative partitions.
  • clean targets only derived Parquet/Oko state and previews unless --apply is present.

See the ingestion reference, configuration, and normative data contract.