Replace identifying details before sending a request

Explanation

evaluator
operator

Purpose. Document the entity types LQ.AI’s Anonymization Layer (PRD §4.7) recognizes by default, the deliberately-disabled defaults, and how to customize the recognizer set for a specific deployment’s matter-numbering convention or domain-specific entities.

Status (2026-05-17). M2-B3 complete: the gateway middleware now pseudonymizes outbound chat/skill content and rehydrates the response (streaming and non-streaming). M2-D2 wires the retrieval-context skip flag so source chunks reach the provider un-pseudonymized (see Retrieval-context skip below). M2-B2’s custom legal recognizers + Presidio AnalyzerEngine configuration remain unchanged; M2-A3’s PseudonymMapper is the request-scoped substitution table.

In short

An optional gateway feature swaps names, organizations, emails, phone numbers, bank numbers, locations, and case/matter numbers for labels before a chat or skill request reaches an AI model, then restores them in the response. It ships off by default.

When it runs. The example config turns it on for tiers 3-5 and leaves Tier 1 (local) inference out; it also always skips privileged chats and retrieved source documents.
What it can miss. It only redacts entities it detects; recall and precision on real legal documents haven’t been measured, so treat this as a mitigation, not a guarantee. Several identifier types (Social Security, passport, driver’s license, IBAN, IP, cryptocurrency, medical license numbers) aren’t recognized by default.
What the status flag means. A logged “ran” status means the pass executed, not that anything was found and replaced.
If a miss is unacceptable, route that matter to Tier 1 instead.

What’s validated vs what’s unvalidated

This section exists because LQ.AI’s founding transparency principle requires us to be explicit about where this layer has been measured and where it has not — so practicing attorneys can make informed professional judgments about confidentiality posture per matter.

What is validated

Custom legal recognizers (CaseNumberRecognizer, MatterNumberRecognizer): ~24 unit tests in gateway/tests/anonymization/test_recognizers.py exercise the pattern matchers on hand-crafted positive and negative examples (federal reporter cites, state reporter cites, docket-number variants, alpha-year-sequence matter numbers, dotted matter numbers, and adversarial near-misses). The recognizers do what they claim against the patterns we anticipated.
Middleware pre/post integration: pre-anonymization fires before the request leaves the gateway, post-anonymization rehydrates the response (including SSE-streaming tail-buffer for pseudonyms that span chunk boundaries). Pinned by test_middleware.py + test_round_trip.py + the M2-C3 17-test round-trip correctness suite.
Edge cases pinned in M2-D4: long entity names, multi-line entities (address blocks spanning \n), the retrieval-context skip flag, the privileged-project tier-floor invariant. See test_edge_cases.py.
Pseudonym format stability: {ENTITY_TYPE}_{NNNN} is deterministic per-request and idempotent on re-substitution.

What is NOT validated

Presidio default-recognizer accuracy on legal-document corpus. The Anonymization Layer enables 6 of Presidio’s default recognizers (PERSON, ORGANIZATION, EMAIL_ADDRESS, PHONE_NUMBER, US_BANK_NUMBER, LOCATION) and disables 7 others (UsSsnRecognizer, UsPassportRecognizer, UsLicenseRecognizer, CryptoRecognizer, IbanRecognizer, IpRecognizer, MedicalLicenseRecognizer). The choices reflect engineering judgment about typical legal-document corpus, not empirical recall + precision measurements on a curated corpus of contracts, briefs, and correspondence.

Specifically, the following are unmeasured:

Recall per entity type — when a real contract contains 10 PERSON names, how many does Presidio actually catch? 95%? 80%? Unknown.
Precision per entity type — when Presidio flags a string as a PERSON, how often is that flag correct on legal prose vs the news/social-media corpus Presidio was developed against? Unknown.
Disabled-recognizer trade-offs — UsSsnRecognizer was disabled because the 123-45-6789 shape collides with case numbers and exhibit IDs in general legal corpus. But personal-injury, employment-discrimination, immigration, and benefits practices routinely handle real SSNs. The disable choice is correct for general civil litigation; it is potentially wrong for those specific practice areas, and the trade-off has not been measured.
Custom-recognizer performance on real legal text — the unit tests cover anticipated patterns. Recall against unanticipated drafting styles (international citation conventions, pre-2010 docket-number formats, firm-specific matter-number conventions) is unmeasured.

Why this matters — a miss is a silent confidentiality incident

For citation verification, a miss surfaces in the user interface as an “unverified” chip — the lawyer sees the system’s uncertainty and can react. For anonymization, a miss is silent: a PERSON name slips through, the unredacted text reaches the model provider, the response comes back rehydrated as if nothing happened, and the lawyer has no in-app signal that client confidentiality was breached. Operational telemetry cannot recover the leak post-hoc — by the time the miss is observable, the unredacted content has already been transmitted, logged, and possibly used in provider-side training (depending on the routed Inference Tier).

This is a meaningfully different load-bearing posture than the citation-verification layer’s “wrong answers visibly marked unverified” story. We document it explicitly so practicing attorneys can apply the professional-judgment standard that confidentiality work warrants — and so the project does not overclaim a privacy guarantee it has not yet empirically supported.

What to do if you can’t accept the unvalidated risk

Operators with confidentiality requirements that demand validated PII recognition (e.g., privileged matter work, regulated-data practices, or any matter where a single PII miss is unacceptable) have one principal mitigation today: route to a Tier 1 (fully local) inference path so the question is moot. Per PRD §1.5.2, Tier 1 keeps the chat content inside the operator’s environment — Ollama via docker compose --profile local up is the default — and no provider call leaves the deployment at all. With the shipped defaults the Anonymization Layer does not run on a Tier 1 route: anonymization.enabled defaults to false, and the tier gate (routed_tier in apply_at_tiers, gateway/app/anonymization/middleware.py) skips Tier 1 under the gateway.yaml.example value apply_at_tiers: [3, 4, 5] — the code default for apply_at_tiers (AnonymizationConfig, gateway/app/config.py) is an empty list, which fires at no tier until the operator sets one. An operator who enables the layer and adds 1 to apply_at_tiers gets the pseudonym/rehydrate round-trip on local inference too, but its failure mode is then “leak inside your own infrastructure” rather than “leak to a third party,” which is a categorically different risk surface.

Other paths to consider (without full Tier 1 routing):

Disable anonymization entirely (anonymization.enabled: false in gateway.yaml) and rely on the routed Inference Tier’s contractual protections (ZDR / no-training agreements at Tier 3 and 4). Honest posture: the operator has decided the provider’s contractual commitments are sufficient, and the gateway is no longer attempting recognition.
Pre-redact at upload time outside the LQ.AI pipeline so the chat-send path operates on already-redacted content. Suitable for narrow matter sets where the operator’s existing tooling handles PII removal.
Manual review on a per-message basis — operators who route highly sensitive chats can flip the per-chat anonymization toggle off (where exposed in the UI) and inspect each outbound chat themselves before sending. Slow; only viable for a small volume of work.

Path to closing the gap

This gap is tracked at PRD §9 / DE-282 — Anonymization Layer empirical validation on legal-document corpus. The DE is scoped as a community-contribution opportunity: it needs (a) practicing-attorney judgment for entity-type prioritization and ground-truth annotation, and (b) bounded technical work for the eval runner and metrics. The original M2-F2 plan (curate ~50 legal documents, annotate per entity type, build the eval runner, report recall/precision/F1, document the baseline in this file) is preserved as the contribution-ready scope. Contributors from specific practice areas — particularly those whose work routinely involves SSN / driver-license / passport recognition, or matter-number conventions outside the current defaults — are explicitly welcomed to extend the corpus and propose recognizer-set changes.

What gets pseudonymized

Per the M2-1 decision locked at M2 kickoff, in ordinary chat anonymization applies only to chat and skill content sent to the model. Retrieved source documents stay un-pseudonymized so the model sees intact source quotes for citation grounding and the user-facing retrieval surface continues to render real document text. The skip mechanism is the lq_ai_skip_anonymization field on ChatCompletionMessage: the api/ marks the retrieval-context system message with True; the gateway middleware honors the flag and leaves that message’s content alone. See Retrieval-context skip below for the data flow. The alternative (Option A — pseudonymize the document corpus too) is filed as DE-269 for future consideration.

A governed run holds its tool text as originals, and the gateway covers it. A governed run (ADR 0038) reads the matter’s documents through tools, so what its tasks read, the passages a search returns and the arguments of the calls its models make are the owner’s own text. The backend holds that text as stored while it passes between its own agents, from the orchestrator to a task and back, and nothing is rewritten there. It tells the gateway so with lq_ai_restore_tool_arguments, and the gateway is where the text is covered: the string values of each tool message’s JSON content and of the arguments of every tool call an assistant message carries are pseudonymized with the rest of the request, by the same mapper, so one name is the same pseudonym in the task, in a call and in a result. The detector reads each string alone and is weakest on a short isolated one, such as a search query, so a name it found anywhere in the request is also replaced wherever else it stands as a whole token in that tool text. A caller that does not set the field, which is ordinary chat, is not touched: its research tools return published sources, its follow-up calls repeat what they return, and nothing restores a pseudonym in them, so its tool messages and call arguments are sent as they are. Cost, measured once on the real detector (en_core_web_lg) on a developer laptop and not a benchmark: one JSON string value of 41,120 characters took about 1.35 s. A task’s loop sends every earlier tool result again on each turn, so a late turn reads again everything read so far, up to the run’s input limit. The pass runs on one worker thread, off the event loop, and the detector never runs two analyses at once (a process-wide lock in engine.py).

Recognizer set

The gateway’s AnalyzerEngine runs with this configuration (gateway/app/anonymization/engine.py):

Enabled by default

PERSON

Source

Presidio default (spaCy NER)

What it catches

Names of parties, judges, counsel, witnesses.

Notes

The single highest-value detector.

ORGANIZATION

Source

Presidio default (spaCy NER)

What it catches

Corporate entities, firms, agencies.

Notes

Surfaces under Presidio’s ORG label internally.

EMAIL_ADDRESS

Source

Presidio default

What it catches

Counsel email, party email in correspondence.

Notes

Requires a recognized TLD; .example test addresses won’t match.

PHONE_NUMBER

Source

Presidio default

What it catches

Contact numbers in correspondence.

Notes

US conventions catch best; international support varies.

US_BANK_NUMBER

Source

Presidio default

What it catches

Bank account numbers in settlement statements, escrow docs.

Notes

Pseudonyms carry the detector label verbatim (US_BANK_NUMBER_0001); there is no rename to an ACCOUNT_NUMBER domain.

LOCATION

Source

Presidio default (spaCy NER)

What it catches

Addresses, courthouses, jurisdictions.

Notes

Pseudonyms carry the detector label verbatim (LOCATION_0001); there is no rename to an ADDRESS domain.

CASE_NUMBER

Source

Custom — CaseNumberRecognizer

What it catches

Federal/state reporter cites (Smith v. Jones, 123 F.3d 456 (9th Cir. 2024)), In re X form, docket numbers (Case No. 1:24-cv-00123).

Notes

Requires structural anchoring; bare case captions intentionally not matched.

MATTER_NUMBER

Source

Custom — MatterNumberRecognizer

What it catches

Alpha-year-sequence (LQ-2026-0042), dotted (2026.0042).

Notes

Deployment-specific; defaults are conservative — extend per the “Customizing” section below.

Disabled by default

These recognizers ship in Presidio’s default set but produce a high false-positive rate on legal corpus or cover entity types that are irrelevant to in-house legal practice. The gateway removes them from the analyzer’s registry so they don’t fire even when an operator’s text accidentally pattern-matches.

UsPassportRecognizer

What Presidio would catch US passport numbers

Why disabled

High false-positive rate on contract numbers, exhibit indexes, dates. Real passports are rare in routine corpus; the downside of redacting “Exhibit A-1234567” as a passport outweighs the upside.

UsLicenseRecognizer

What Presidio would catch US driver’s license numbers

Why disabled

Same reasoning — wide pattern, low actual presence in legal corpus.

UsSsnRecognizer

What Presidio would catch US Social Security numbers

Why disabled

The 123-45-6789 shape collides with case numbers (Case 12-345-6789), exhibit IDs, and pinpoint cites. Real SSNs in briefs are rare and ought to be redacted by other means before reaching the model.

CryptoRecognizer

What Presidio would catch Bitcoin/Ethereum addresses

Why disabled

Irrelevant for legal corpus; the patterns collide with random hex strings.

IbanRecognizer

What Presidio would catch IBAN bank identifiers

Why disabled

US-centric deployments rarely see them; when they do, US_BANK_NUMBER covers the use case.

IpRecognizer

What Presidio would catch IPv4/IPv6 addresses

Why disabled

Incidental in evidence logs but extremely high false-positive rate against version numbers (192.168.1.1 as a section reference), page references, and dotted numeric identifiers.

MedicalLicenseRecognizer

What Presidio would catch Medical license numbers

Why disabled

Niche to healthcare practice areas; the shape collides with case numbers in unrelated corpora.

A healthcare-practice deployment can re-enable MedicalLicenseRecognizer via the operator-customization pattern below.

Customizing the recognizer set

Re-enabling a disabled default

Edit gateway/app/anonymization/engine.py and remove the entry from DISABLED_DEFAULT_RECOGNIZERS. Rebuild the gateway image.

# Before:
DISABLED_DEFAULT_RECOGNIZERS: tuple[str, ...] = (
    "UsPassportRecognizer",
    "UsLicenseRecognizer",
    "UsSsnRecognizer",
    "CryptoRecognizer",
    "IbanRecognizer",
    "IpRecognizer",
    "MedicalLicenseRecognizer",  # ← remove this line for healthcare deployments
)

The disabled list is currently a compile-time constant. A future task (M2-C3 or later) could surface it via gateway.yaml for runtime configuration.

Adding a deployment-specific recognizer

The MatterNumberRecognizer defaults catch the two most common shapes, but every firm’s numbering convention is different. Add a recognizer file at gateway/app/anonymization/recognizers/<name>.py mirroring the existing pattern:

from presidio_analyzer import Pattern, PatternRecognizer


class CustomMatterRecognizer(PatternRecognizer):
    """Recognize ``YYYY/NNNN``-style matter numbers (slash-separated)."""

    def __init__(self) -> None:
        patterns = [
            Pattern(
                name="slash_year_sequence",
                regex=r"(?<![\d/])(?:19|20)\d{2}/\d{3,6}(?![\d/])",
                score=0.8,
            ),
        ]
        super().__init__(
            supported_entity="MATTER_NUMBER",  # Reuse the existing entity type
            name="CustomMatterRecognizer",
            patterns=patterns,
        )

Then register it in engine.py:

from app.anonymization.recognizers.custom_matter import CustomMatterRecognizer

# Inside get_analyzer_engine():
registry.add_recognizer(CustomMatterRecognizer())

Add a unit test in gateway/tests/anonymization/test_recognizers.py covering both positive matches (the matter numbers you expect to see) and negatives (similarly-shaped non-matter strings the regex must reject).

Adding a new entity type

If the deployment needs an entity type that isn’t in Presidio’s vocabulary (e.g. CLIENT_CODE):

Create the recognizer with supported_entity="CLIENT_CODE".
Register it in engine.py.
The new entity type flows through to PseudonymMapper.assign("CLIENT_CODE", ...) and gets pseudonyms like CLIENT_CODE_0001.

No further configuration is needed — the pseudonym format is generic per-entity-type.

Calibrating against your corpus

The plan §M2-F2 explicitly calls for an acceptance corpus of legal documents to measure the false-positive / false-negative trade-off at the analyzer level. Until that ships, the conservative-by-default posture means:

The defaults under-match (some real entities slip through unredacted).
They almost never over-match (false positives are rare in normal prose).

If your deployment surfaces under-matching in practice (the M2 Anonymization round-trip tests in M2-C3 will help), the right response is to add deployment-specific recognizers rather than loosening the existing patterns globally.

Middleware behavior (M2-B3)

The gateway runs two passes around the provider call:

Auth → Router → Rate Limit → Tier Derivation
                 → Anonymization-Pre  (substitute)
                 → Provider Adapter
                 → Anonymization-Post (rehydrate)
              → Cost Tracker → Telemetry

When the middleware fires

All four conditions must hold; the first that fails short-circuits to a no-op for the entire pass (provider receives unmodified content; response is not touched; audit row records anonymization_applied = false).

gateway.yaml anonymization.enabled = true

Source

Operator config

Default

false — feature flag stays off until the deployment opts in.

Request’s routed tier is in anonymization.apply_at_tiers

Source

Operator config

Default

[] in code (AnonymizationConfig.apply_at_tiers, gateway/app/config.py) — an unset list fires at no tier. gateway.yaml.example sets [3, 4, 5] so local Tier 1 / Tier 2 inference skips because the data never leaves the operator’s environment.

Request’s lq_ai_privileged is false

Source

Backend forwards Project.privileged

Default

False for chats outside any project, or in non-privileged projects.

Request’s anonymize is true

Source

Per-call body field

Default

True. Callers send anonymize: false only when they need the raw text on the provider call (evaluation, raw-passthrough scenarios).

What the pre-pass touches

Every messages[*] whose role is user, assistant, or system and whose content is non-null, except a message carrying lq_ai_skip_anonymization.
When the request sets lq_ai_restore_tool_arguments: every messages[*] whose role is tool. Its content is a JSON document in a string (a document read, a search result), so the detector is given each string value and not the document: an escape is never put against a name, and a result with no entity in it is sent byte for byte as it was. A result that is not a JSON object or array, and one with more than 2,000 string values (analysed once, as one text), is treated as text. Member names, numbers, the tool schemas and a message’s other fields are not rewritten.
When the request sets the same field: the arguments of every tool call an assistant message carries (tool_calls[*].function.arguments, a JSON document in a string, or an object), the same way. Arguments that cannot be walked (a member named twice, NaN, nesting too deep) are treated as text, so the name still goes and nothing is repaired on the way. Then, in that tool text only, a name the request has already replaced is replaced wherever it stands as a whole token, for names of three characters or more. What the detector recognises is what is replaced: a miss is silent, as everywhere in this layer.
Every string leaf inside lq_ai_skill_inputs (recursive — dicts and lists are walked). Numbers, booleans, and null pass through untouched.

What the post-pass touches

Non-streaming. Each choices[*].message.content is rehydrated in place. Every pseudonym the mapper assigned is replaced by its original; a pseudonym-shaped string the model produces that the mapper never assigned (e.g. an invented PERSON_0009) is left as-is.

Streaming. Each SSE chunk’s choices[*].delta.content is fed through a per-stream StreamingRehydrator. The rehydrator holds the tail of the stream when it ends in a partial pseudonym (e.g. PERSON_ or PERSON_0001 with no trailing space — could grow to PERSON_00010). Held text emits as soon as the pattern crystallizes or fails to grow. At [DONE], any held tail flushes as a synthesized terminal chunk so the caller doesn’t lose the last fragment. The hold pattern (_PARTIAL_PSEUDONYM_AT_END, [A-Z][A-Z_]*(?:_\d*)?$ in gateway/app/anonymization/middleware.py) has no fixed length cap: any trailing run of uppercase letters and underscores, optionally followed by _ and digits, is held until a non-matching character arrives or the stream flushes at [DONE]. For ordinary prose the held tail is one pseudonym long, but a long all-caps run (a shouted heading, an ALL_CAPS_IDENTIFIER) is held for its full length, so emission of that text is delayed until the run ends.

Per Decision D: the middleware rehydrates response content. Citation rehydration is incidental — the api/’s downstream citation extraction operates on already-rehydrated content, so cite quotes naturally carry originals. The gateway never touches message_citations rows directly.

Tool-call arguments, on the caller’s opt-in. A request may set lq_ai_restore_tool_arguments. On a non-streaming response to a request that was pseudonymized, the pseudonyms in each tool call’s arguments are then replaced by the originals as well, inside the JSON string values, and only where a pseudonym this request assigned stands as a whole token. Without the opt-in, arguments are returned as the model wrote them, as Decision D had it. A governed run’s calls that are offered tools, or that carry earlier tool calls and results (a task’s last turn), set it (ADR 0038 D7), because it acts on the arguments as the owner’s own text: a search runs on the name that was asked about and a note is kept with the name in it. The same opt-in is what puts the run’s tool text under the pre-pass (above), so the originals the backend holds and sends back as part of the next request are pseudonymized again on the way out. The opt-in changes what the gateway returns and sends nothing more to a provider; it has no effect on a request that was not pseudonymized or on a streaming response.

Audit log

Every routed request writes one row to inference_routing_log. The flag is written as anon_mapper is not None (gateway/app/api/inference.py): it is true on every row whose request passed all four firing conditions and so had a mapper allocated by the pre-pass — including rows where the upstream later failed (the pre-pass ran; the provider then returned an error). It does not record that any entity was detected or any text substituted: an eligible request in which the analyzer found nothing still gets true. Tier-floor refusals (which short-circuit before the pre-pass) and skipped requests leave the flag false because no mapper was allocated.

Where mappings live

Per Decision A and Decision B (i) locked in M2-B3 kickoff: a fresh PseudonymMapper is constructed inside the request scope, populated by the pre-pass, read by the post-pass, and dropped on function exit. It is never persisted, never logged, and never serialized to any side channel. A new request gets a new mapper; counters reset every time.

Privileged chats — why we skip (M2-B3 + M2-D3)

The privileged-Project skip (Decision A) is a deliberate trade-off. Privileged chats are work product the attorney-client privilege protects; replacing names with pseudonyms before the model sees them — even with the rehydration on the way back — risks corrupting that work product if any step in the pipeline behaves unexpectedly. The conservative posture is to leave privileged content untouched. Operators who want pseudonymization in privileged chats can flip lq_ai_privileged off at the api/ layer per chat, but the default protects the legal-work-product invariant.

The intended posture for privileged content is Tier 1 (local). Privileged matters are best handled by local-only inference (per PRD §1.5.2: “no outbound network is required; customer data, prompts, and model outputs never leave the deployment”) — that’s the only path where the substitution layer’s behavior is structurally irrelevant because no upstream provider ever sees the content in the first place. Operators configuring a privileged project typically pair privileged=true with minimum_inference_tier=1 so the tier-floor enforcement refuses any routing weaker than local. Where local capacity is insufficient and the operator must use a Tier-2 (ZDR enterprise) upstream, the anonymization skip means privileged content reaches the provider verbatim — the operator’s procurement / DPA terms with that provider are the binding control at that point.

Why anonymization complicates privilege analysis. Pseudonymization rewrites identifying terms (names, organizations, matter numbers, case numbers) before the model sees them. For non-privileged content this is a privacy boost: the operator sends "PERSON_0001 v COMPANY_0001" to the provider instead of "Smith v Acme". For privileged content the same transform creates two correctness risks:

Substitution errors corrupt the privileged work product. If Presidio misidentifies an entity (false-positive on a non-name token; false-negative on a non-standard name format), the rehydrator either over-restores or leaves a fragment unrestored. For ordinary content this is a minor annoyance; for privileged work product the corruption may render the work less useful as a referenceable artifact and may complicate later assertion of privilege over the artifact’s contents.
The substitution itself may be discoverable. In a future-litigation context where the operator must assert privilege over the AI-assisted work product, the existence of a rewritten-then-restored intermediate version (even if never persisted) is one more piece of context opposing counsel can probe. Keeping the content unsubstituted means there is exactly one version of the work product — the original — at every stage.

Audit-log shape for privileged-project chats (per M2-D3 verification):

audit_log (api/)

Column privilege_marked