Transcript Lake

Documentation / Operate

Operator runbook

Preserve evidence before intervention. Never edit vendor transcripts, partitions, or cursors in place. Use transcript-lake paths, sources, doctor, and status --json to identify the selected root and failure domain.

Preserve evidence before intervention. Never edit vendor transcripts, partitions, or cursors in place. Use transcript-lake paths, sources, doctor, and status --json to identify the selected root and failure domain.

No supported roots#

Message: stream found no supported source roots on this machine

Run sources. Start at least one supported runtime as the same user or correct HOME. For Tama, verify HOOKS_ADAPTIVE_SEGMENTS_READY. Do not create fake directories in a production home merely to silence the refusal.

Writer lease is held#

Message: state writer lock is held by <host> pid <pid>

Run status; check the supervisor/service for the reported process. If it is alive, let that one stream own the root and stop the duplicate. Do not delete stream.lock for a live owner. The lease code reclaims a dead owner on its own: an owner on this host whose pid /bin/ps no longer reports, or reports with another start time than the one recorded. An owner on another host is always taken as alive. When the process identity cannot be established the lease is refused, not broken: cannot run /bin/ps to see whether process <pid> is alive: <cause> or /bin/ps answered <status> for process <pid> with <output>: <stderr>; preserve the root and repair ps rather than forcing mutation. A writer that cannot read its own start time takes no lease. An owner record that is absent, or is not a whole record of host, pid, start time and token, is abandoned, and stderr says which (a lease is published whole by one rename).

Cursor store is unreadable or corrupt#

Messages:

  • cursor store is unreadable; preserve it and recover into an empty LAKE_DATA: <error>
  • cursor store is corrupt; preserve it and recover into an empty LAKE_DATA: <error>
  • cursor store must be a JSON object; preserve it and recover into an empty LAKE_DATA
  • cursor record contains invalid numeric state

Stop the writer. Preserve the complete Lake and cursor file. Do not reset cursors.json beside existing partitions: replay could duplicate evidence. Choose a different empty destination and run:

sh
transcript-lake rebuild --to "$HOME/.transcript-lake-rebuild"
transcript-lake --data-dir "$HOME/.transcript-lake-rebuild" doctor
transcript-lake --data-dir "$HOME/.transcript-lake-rebuild" status

Switch consumers only after inspecting the new root. Retain the old root until the operator approves disposal.

Source shrank or changed without an append#

Recovery reasons:

  • legacy cursor has no verified source prefix
  • source is shorter than its consumed prefix
  • source bytes before the cursor changed

These messages announce automatic recovery, not a request to discard the Lake. The stream reads that source through masking and compares retained canonical occurrences. Its diagnostic includes the path, previous offset, and observed size. Missing instructions and Oko rows are restored; archived history stays intact. Per-runtime summaries include replayedFiles. Legacy cursors migrate on the next catch-up even without a new append.

If recovery fails later, its previous cursor remains. Correct the named storage or access failure; the next source notification or supervised startup repeats the retained-history comparison safely. A corrupt partition is refused as cannot replay source against corrupt canonical partition <path> at line <line>: <cause>. Preserve that evidence. Separate reconstruction remains available for damaged authoritative state, with these guardrails:

  • rebuild requires --to <empty-path>
  • rebuild target must differ from the current Lake
  • rebuild requires an empty LAKE_DATA root so replay cannot duplicate or erase existing evidence

Stream watcher failures#

Messages:

  • stream could not start: <error>
  • stream could not watch <path>: <error>
  • stat failed for <path>: <error>

Check path existence, current-user read permission, file-descriptor/watch limits, and whether the root disappeared during startup. status reports degraded with paths/error after catch-up or path failures. Fix the underlying filesystem issue and restart; cursor recovery catches up complete appends.

Hook segment conflict#

Message: hook segment output conflict: <path>

An immutable Tama segment would map to an existing Lake output with different content. Stop publication, preserve both files and Tama segment metadata, and investigate producer/release identity. Never overwrite the existing partition. A correct replay of identical content is idempotent.

DuckDB unavailable or fails#

Messages:

  • duckdb failed to start: <error>
  • duckdb terminated by signal
  • duckdb exited with status <code>: <stderr>
  • compact: duckdb failed for <runtime>

Streaming is unaffected. Install the supported DuckDB 1.5.x CLI on PATH, then retry only the read/compact command. When TRANSCRIPT_LAKE_SQL is set, missing <path> (TRANSCRIPT_LAKE_SQL is set but incomplete) means the override directory lacks the requested views.sql or signals.sql; complete it or unset the override.

Oko integration unavailable#

Message: oko-cli is not on PATH; install Oko or set OKO_CLI, then run: oko-cli transcripts reindex

Core Lake health is unaffected. Install a compatible CLI or set OKO_CLI to the executable. projection rebuild --target oko can materialize export files without Oko; --reindex and projection refresh --target oko require it.

Malformed export refusals:

  • full Oko export refused malformed Lake rows; authoritative partitions were not modified
  • incremental Oko export refused malformed Lake rows; export cursor was not advanced

Preserve the partition and export state. Identify the malformed canonical row without modifying the authoritative file. Repair at the source/code level and rebuild to qualified data; never advance export-cursors.json manually.

Invalid flags and bounded values#

Common messages:

  • unknown <command> flag: --<name>
  • <command> received duplicate --<name>
  • --<name> requires a value
  • --limit must be a whole number of at least one; omit it to list every …
  • --days must be a whole number of at least one; omit it to count the whole Lake
  • unknown source "<value>" (expected one of: <every runtime of the adapter registry>, hooks; `transcript-lake sources` shows each with its roots on this machine)

Use transcript-lake help <command>. Do not make automation parse human help; prefer supported --json output.

Label refusals#

  • --aspect must be a non-empty string
  • --value must be a non-empty string
  • --source must match ^(manual|human|model|brama)(:[A-Za-z0-9._/-]+)?$ (manual, human, model, or brama, with an optional :detail suffix)
  • unknown session "<id>": not present in the selected Lake (check the id or start the stream first)

Confirm the selected root and session id. If the same id exists in multiple runtimes, provide --runtime. Never place a secret in --value or --note; labels are not masked.

Local goal model failures#

  • local goal model runs on Ster: install ster on PATH or name it with TRANSCRIPT_LAKE_STER
  • <ENV_KEY> does not name a file: <path>
  • Ster could not run the goal model (<ster> generate exited <status>): <reason> — Ster's own refusal follows the colon: a tensor the GGUF does not hold, a shape the base config disagrees with, or a prompt longer than the context
  • <repository>@<revision> declares no base_model in its card
  • local goal model returned no goal tag
  • local goal model returned no opening goal tag
  • local goal model returned an invalid <n>-word title

Check configuration, pinned artifact integrity, and local disk/network availability for first acquisition. This feature is optional and does not affect streaming. Do not replace the pinned model/prompt with an unqualified artifact.

Derived cleanup#

Always preview:

sh
transcript-lake clean --target all

Expected footer: preview only; add --apply to remove derived data. Apply only after confirming paths. Invalid targets return --target must be parquet, oko, or all. clean never removes events, cursors, or labels.

Escalation bundle#

Provide public-safe, synthetic or redacted evidence only:

  • exact binary version and immutable release coordinate;
  • paths --json, doctor --json, status --json with local usernames/paths/session identifiers redacted;
  • exact error string and exit status;
  • supervisor logs around the failure;
  • whether the old root and vendor source were preserved.

Never attach real transcripts or a production Lake to an issue.