This document defines the smallest provider-neutral system that fulfills Transcript Lake's README promise. Integrations may consume or extend these contracts but cannot redefine them.
Workflow: stream local evidence#
- Actor: one supervised local stream process.
- Initial state: a selected
LAKE_DATA; supported local source stores; no concurrent Lake writer. - Input: filesystem notifications naming transcript or closed-hook-segment files.
- Authoritative inputs: vendor transcript stores and closed hook segments.
- Authoritative Lake state: masked NDJSON partitions and durable source checkpoints.
- Success: every complete new source line reaches its masked partition and affected Oko session before its byte cursor advances.
- Source failure: the cursor does not advance past failed outputs; the failure is logged and other source files continue.
- Restart: a consumed prefix is verified before an append resumes.
- Recovery: rewritten or truncated sources and legacy cursors replay against retained canonical occurrences. Missing history is restored without deleting prior evidence or duplicating retained occurrences. Recovery keeps its old cursor until the whole source succeeds. Corrupt cursor or canonical state remains a refusal requiring preserved-state diagnosis or separate reconstruction.
- Security: raw text is masked before partition or projection writes. No credential is required. Source stores remain read-only.
The stream enumerates source candidates once during startup. Its live loop receives changed paths and verifies each consumed prefix before reading new bytes or recovering the affected source. It does not periodically scan all transcripts or launch a refresh command. rebuild remains the explicit separate-root reconstruction operation.
Workflow: discover and inspect state#
transcript-lake paths, sources, doctor, and status are read-only. Together they report resolved state and integration paths, discovered provider roots and file counts, cursor integrity, optional dependency presence, runtime partition counts and bytes, live stream state, and Oko freshness. --json exposes stable structured command results for automation.
These commands never create configuration, start streaming, repair state, contact providers, or claim that a missing optional integration invalidates core partitions. Corrupt authoritative metadata makes doctor and status exit non-zero. Missing optional DuckDB or Oko is a warning until the corresponding capability is invoked.
Workflow: query evidence#
transcript-lake query "<sql>" loads the frozen DuckDB views compiled into the binary over the selected Lake and executes the supplied SQL, so an installed CLI is never separable from its views. Named read commands (sessions, events, search, show, stats, hooks, signals, label list, label aspects) cover bounded common reads over the same views without operator SQL; search treats its term as literal text, escaping LIKE wildcards before matching, and show reconstructs one whole conversation in chronological order without per-event truncation. The query process does not mutate NDJSON, cursors, or source stores. DuckDB is an explicit optional prerequisite; absence is an actionable dependency failure, not a fallback to another engine.
The canonical events view pins column names and types and tolerates only a torn final partition line during a concurrent read. Aggregate views expose sessions, tools, tokens, and hook decisions. User SQL can itself create external files or perform DuckDB mutations; the operator owns the supplied SQL. Transcript Lake does not label arbitrary SQL as read-only.
Workflow: compact derived state#
transcript-lake compact creates per-runtime Parquet mirrors under LAKE_DATA/parquet. NDJSON remains authoritative and is never deleted. Existing output conflicts or DuckDB failure stop that runtime and produce a non-zero exit. Retrying is safe only after the operator inspects or removes the conflicting derived artifact.
Public command semantics#
| Command | Mutation | Success | Failure |
|---|---|---|---|
no command / help [command] / --help | None | General or command-specific guidance | Unknown topic is non-zero |
--version | None | Canonical product version, compiled in from Cargo.toml | The version travels inside the binary, so an artifact cannot report a version it was not built from |
paths / sources | None | Resolved paths and discovered supported stores | Permission or adapter error is explicit |
doctor | None | Health report; missing optional tools are warnings | Corrupt authoritative state or broken adapter is non-zero |
status | None | Human or JSON inventory and freshness | Corrupt cursor/summary state is non-zero without repair |
stream | LAKE_DATA only | Long-running direct source-to-Lake/Oko commits; clean stop on SIGINT/SIGTERM | Startup/configuration failure is non-zero; source failures are logged without advancing that source cursor |
rebuild | Separate empty target only | Historical replay and projection in the new root | Current root preserved; invalid/non-empty target is non-zero |
sessions / events | None | Filtered normalized evidence | Non-zero dependency or input error |
search | None | Newest-first literal substring matches over event text | Non-zero dependency or input error |
show | None | One conversation, oldest turn first, untruncated masked text, with a rendered/matched footer | Unknown session, unknown event type, or dependency error is non-zero |
label add | LAKE_DATA/labels only | Appended label record with namespaced provenance (manual/human/brama:<model>/model:<artifact>); unknown or ambiguous session is rejected | Non-zero dependency or input error |
label list / label aspects | None | Latest-assignment labels or aspect summaries | Non-zero dependency or input error |
stats / hooks | None | Bounded aggregates or hook decisions | Non-zero dependency or input error |
query | User SQL may have DuckDB-defined effects | DuckDB result | Non-zero dependency or SQL error |
compact | Derived Parquet only | Filterable per-runtime size/path report | Non-zero; NDJSON preserved |
projection rebuild --target oko | Derived Oko projection only | Full reconstruction summary | Non-zero; partitions and cursors preserved |
projection refresh --target oko | Oko-owned index through oko-cli | Child-process success | Non-zero or actionable missing-CLI guidance |
clean | Derived Parquet/Oko only with --apply | Dry-run by default; explicit removal report | Active writer or filesystem failure is non-zero |
Global --data-dir <path> selects the root for one invocation and may appear before or after the command. Unknown commands, flags, duplicate flags, contradictory input, and out-of-range limits are rejected before mutation. --source accepts only the declared runtime set.
Canonical event contract#
Every durable event has:
- ISO timestamp or explicit unknown partition placement;
- normalized runtime and machine identity;
- runtime-native session identity and best-effort project path;
- event type;
- bounded masked text;
- optional tool and model names;
- normalized input and output usage counters;
- bounded, depth-limited, recursively masked extra metadata.
The detailed field table is frozen in the Lake data contract. Provider adapters emit the same in-memory intent. They never write state or perform I/O while mapping one source line.
Mutable state and ownership#
| Resource | Authority | Writer | Concurrency | Recovery and retention |
|---|---|---|---|---|
| Vendor transcripts | Vendor runtime | Vendor runtime | External | Never changed by Lake |
| Closed hook segments | Tama hook runtime until handed off | Tama then Lake handoff | Segment protocol | Preserve segment evidence on failure |
events/ NDJSON | Transcript Lake | State writer lease | Append-only | Backup or rebuild into a separate root |
cursors.json | Transcript Lake | Locked read-modify-write transaction | Atomic merge under state lease | Hard-fail on corruption; never silently reset |
stream-status.json | Transcript Lake stream | Current stream process | Atomic replacement | Diagnostic; regenerated on start, stop, and commit |
parquet/ | DuckDB compaction | State writer lease | No concurrent Lake mutation | clean --target parquet --apply; rebuild from NDJSON |
exports/oko/ | Lake stream or recovery exporter | Same transaction as the corresponding canonical events | Atomic session/cursor writes | clean --target oko --apply; rebuild from NDJSON |
labels/labels.ndjson | Operator via Transcript Lake | Append-only single-line writes | No writer lease; independent of stream | Deleting loses only labels; events and exports unaffected |
| Oko SQLite index | Oko | Oko | External single-writer contract | Rebuild from Lake export |
A writer lease fails fast when another live owner holds the root. Stale same-host claims are reclaimed only after process identity no longer matches. Shared cross-host mutation is unsupported.
Failure classification#
- Invalid input: unknown command, flag, or runtime; reject before mutation.
- Missing optional dependency: DuckDB or Oko unavailable; the affected operation fails while unrelated core commands remain usable.
- Permission failure: name the failed source or state path; never silently skip it as success.
- Corrupt canonical history: refuse recovery with the partition path, line, and parse cause; preserve its bytes and the source cursor.
- State conflict: an active writer causes immediate refusal and retains ownership.
- Corrupt authoritative metadata: cursor parse or numeric validation failure; preserve state and require separate-root recovery.
- Source failure: preserve completed outputs, log the named file, and continue other paths.
- Interrupted recovery: keep the pre-replay cursor and compare retained occurrences again on the next attempt.
- Storage exhaustion: the cursor cannot advance past a failed write. The failure names the canonical partition and operating-system cause. A later source notification or supervised startup retries it after storage is available.
There is no external retry scheduler. The supervised process remains live after a source-local failure and retries that path on its next filesystem notification; startup and authoritative-state failures exit for the supervisor to restart.
Configuration#
--data-dir <path>: global per-invocation state-root selection, resolved to an absolute path.LAKE_DATA: automation default when--data-diris absent. Default~/.transcript-lake.OKO_CLI: optional integration executable override. Default path discovery searchesPATHonly when the integration is invoked.HOOKS_ADAPTIVE_SEGMENTS_READY: optional Tama handoff location used only by hook discovery and streaming.TRANSCRIPT_LAKE_SQL: optional directory of view definitions replacing the ones compiled into the binary. Used to iterate on views against an installed CLI. A directory that is set but does not contain the requested script is an error, never a silent fall back to the compiled copy.
Unknown CLI flags are rejected. Environment variables not documented here are not part of the public product contract. paths exposes resolved locations without transcript contents or credentials.
Security and privacy boundary#
Masking protects credential-shaped and high-entropy text before durable Lake writes, but it is not anonymization. Session identifiers, machine name, timestamps, project path, model, tool name, sizes, counts, and short non-matching text can remain sensitive. Protect LAKE_DATA with local filesystem permissions and do not publish it.
The masker is deterministic and idempotent. Nested metadata is depth bounded and text fields are size capped. Full tool-result files are referenced but never followed. No core command transmits transcript data over the network.
Resource behavior#
The stream reads changed files from newline-aligned byte cursors and bounds each in-memory commit. Individual text fields are capped and nested extra metadata is depth bounded. Session count and total Lake disk usage are not globally capped because they reflect the history appended by supported runtimes; operators control storage with filesystem quotas and use explicit separate-root rebuilds for historical scope.
The product supports one writer and concurrent read-only status or query operations per local state root. It defines no shared-filesystem, multi-host, or distributed-write topology.
Evolution#
Removal or incompatible change to commands, flags, runtime identifiers, configuration, event fields, cursor semantics, partition layout, masking guarantees, or integration export schemas follows release policy. Stored-state changes require explicit migration and rollback analysis. Obsolete paths are removed after callers migrate; permanent aliases and silent dual formats are not supported.