PII and data boundaries

Why it matters

A model-backed system copies data into more places than a classic one: the prompt, the retrieval index, the memory store, the trace log, the eval set, and a provider's servers in a region you did not choose. Each is a place personal or confidential data can leak from, and a deletion request has to reach all of them. Sensitive information disclosure is second on the OWASP list for a reason. Interviewers ask because the answer shows whether you know where data goes, not only where it is stored. Encryption, secrets and access control are covered by the Security module in the System Design track; this lesson is about the flows a model adds.

The idea

Draw the flows first

Enumerate every place data lands: user input and uploads; the index and the chunks retrieval returns; memory persisted across sessions; the model call, where the provider sees the whole window; the output, shown to this user and perhaps cached for others; logs and traces with their payloads; eval sets built from traces; any fine-tuning data. Draw the boundary, the line past which the data is no longer under your control, and mark each crossing. Everything below is a control at a crossing.

Data minimisation

Send only what the task needs. An agent summarising a support ticket does not need the card number or the home address; strip those fields when the prompt is assembled. Retrieval returns chunks, not whole records, and only the columns the task uses. The Context engineering lesson decides what goes in the window; the rule here is that what never enters cannot leak.

Redact or pseudonymise before the model

Detect personal data (names, emails, phone numbers, account ids, addresses, health terms) with patterns and a small classifier, and replace it before the prompt is built. Redaction removes; pseudonymisation replaces with a stable token so the model can still reason about one customer across a conversation, and your system re-identifies on the way out. The mapping lives inside your boundary and never in the prompt. Pseudonyms keep most tasks working. Free-text clinical notes are harder, and the detector's miss rate is your leak rate, so measure it on a labelled sample.

raw     : "Priya Nair (acct 48812-7) called about a charge on 3 Nov; reach her at priya.n@example.com"
to model: "[PERSON_1] (acct [ACCT_1]) called about a charge on 3 Nov; reach her at [EMAIL_1]"
map     : PERSON_1=Priya Nair  ACCT_1=48812-7  EMAIL_1=priya.n@example.com   (inside the boundary only)

Tenant and user boundaries in retrieval and memory

The index filters by permission before ranking, so a user cannot retrieve what they could not open; the Permission-aware and multi-tenant retrieval lesson has the mechanics, and the test is an employee asking about a colleague's salary and getting nothing. Memory is scoped per user and per workspace, and recall filters by scope before similarity. A memory shared across users is a disclosure channel with a friendly name.

Logs, traces and eval sets

Traces carry full prompts and outputs by default. Redact payloads at the exporter, keep hashes and token counts, and put raw payloads in a locked store with short retention and an access log. Eval sets built from production traces, as in the Building an eval set from production traces lesson, inherit whatever the traces held; scrub them before they become a golden set someone copies into a spreadsheet.

Provider data terms

Read three things and write them on the diagram: whether inputs are used for training (paid API tiers mostly say no; consumer tiers may say yes), how long inputs are retained (zero-retention options exist on some tiers, and abuse monitoring may keep samples regardless), and which region processes the request. A provider is a boundary crossing with a contract attached, and the contract is a control only if someone has read it.

The right to be forgotten

A deletion request means deleting by subject from the index (chunks and their embeddings), the memory store including superseded versions, logs, traces, caches, eval sets and backups. Data that went into fine-tuning cannot be removed from the weights, which is the reason not to tune on personal data at all. Make it one job with one query per store, and log that it ran.

Regulated data and self-hosting

Health, payment and some financial data carry rules about where they may be processed and by whom. When no provider offers the region, the contract and the audit evidence you need, a self-hosted open-weight model inside your own boundary is the only option; the Self-host versus API lesson weighs the cost. It trades capability and operations for control. Many teams keep the API for unregulated flows and self-host only the regulated path, with redaction good enough that most traffic never needs the second one.