Transcript Lake

Documentation / Understand

Masking guarantees and limits

Masking is the boundary between adapter output and every durable Lake event. Adapters parse unmasked vendor records in memory; only the stream canonicalizer may write partitions or Oko projections.

Masking is the boundary between adapter output and every durable Lake event. Adapters parse unmasked vendor records in memory; only the stream canonicalizer may write partitions or Oko projections.

What never leaves the masking boundary#

For a detected hit, none of the matched plaintext bytes, including a prefix or suffix, are written to LAKE_DATA. The complete hit becomes:

text
[masked:<class>:<character-count>:<sha256-prefix>]

The fingerprint is the leading 32-bit word of SHA-256 over the complete hit, in lowercase hex without leading zeros. It is deterministic and non-reversible, but permits equality correlation: the same secret produces the same marker fingerprint.

What counts as a secret is declared. TRANSCRIPT_LAKE_SECRET_FORMATS names a JSON file with exactly three keys, each required; null declares that the class recognises nothing. Without the setting the masker reads the declaration that ships with Transcript Lake: the credential prefixes providers document (GitHub ghp_, gho_, ghu_, ghs_, ghr_, github_pat_; Anthropic sk-ant-; OpenAI sk-proj-, sk-svcacct-, sk-admin-; Slack xox?-; AWS AKIA; Google AIza; GitLab glpat-) as its token class, each hit masked from its prefix through its last token character, and no assignment or entropy class, whose lengths and floors nobody stated. A file the setting names has this shape:

json
{
  "assignment": "<regex for NAME=value hits>",
  "token": "<regex for provider-shaped tokens>",
  "entropy": { "pattern": "<regex for candidate runs>", "min_distinct": <n>, "min_groups": <n> }
}

With a named file that cannot be read, a missing or unknown key, or a pattern that does not compile, stream, adopt and recovery replay refuse to start and name the file; no text is written under an unstated rule. doctor --json reports the shipped declaration, when no file is named, as a valid masking-policy. Masking is applied in this order:

An omitted class differs from explicit null: {} refuses with the missing field name instead of disabling masking. The real CLI journey cargo test --test masking-declaration omits each required class separately, verifies no canonical events are published, then adopts an isolated native store with explicitly null classes and checks the persisted message. Its reports retain the tested source revision, binary digest, command, output and exit status; test source alone is not a passing qualification result.

  1. assignment — the declared pattern; the complete match is removed.
  2. token — the declared pattern; the complete match is removed.
  3. entropy — matches of the declared pattern, accepted only with at least min_distinct distinct characters drawn from at least min_groups of the lowercase, uppercase, digit and other groups.

Assignments run first so a whole NAME=value becomes one marker. Provider tokens run before the generic entropy rule. The transform stays idempotent as long as no declared pattern matches [, : or ], the marker's separators.

The stream applies the same transform to text and to every string in extra, at every depth, and keeps each value whole. Oversized Claude tool results are referenced, never followed. Kimi configuration and system-prompt records never contribute their text.

Demonstrated with synthetic evidence#

The executed fixture produced:

text
[masked:assignment:37:5be474f8]
[masked:token:36:e3bebba1]
[masked:entropy:44:d0fa7c09]

The same synthetic assignment appeared in both a user message and tool result with the same fingerprint. A literal scan for the three invented plaintext values returned zero matches throughout that isolated Lake, including the canonical partition and Oko export.

What is not guaranteed#

Masking is pattern-based, not semantic secret detection and not anonymization.

The following may remain:

  • ordinary prose and source code;
  • short passwords, tokens outside the supported alphabets, bearer strings without the documented shape, or secrets split by punctuation/whitespace;
  • timestamps, runtime, machine hostname, session id, absolute project path, event type, tool name, model, token counters, and selected source-specific metadata;
  • the marker class, original character count, and stable equality fingerprint;
  • operator labels and notes. Labels are a separate append-only operator store and are not passed through the event masker.

Therefore:

  • Treat all Lake roots as sensitive local data.
  • Never publish a Lake based only on the word “masked.”
  • Rotate any credential pasted into a transcript; masking the archive does not revoke it.
  • Use synthetic fixtures for documentation, demonstrations, bug reports, and qualification.

Durability relationship#

The writer lease covers partition append, affected Oko session projection, and cursor checkpoint. The cursor cannot advance ahead of durable masked output. A failure before checkpoint causes replay rather than skipped source bytes. Hook closed segments use temp-file write, file fsync, rename, and parent-directory fsync; normal transcript partitions append complete event batches, and cursor publication is atomic.

See masking concept, ingestion reference, and runbook.