Scope: OpenTelemetry traces, Prometheus metrics, audit actions, and deployment recipes for the
apiandgatewayservices. Covers what is shipped as of the M4 close: the M3-F OTel baseline (M3-F1 trace propagation, M3-F2 domain spans, M3-F3 recipes and this doc) plus the M4 Autonomous Layer spans and audit actions. What is not yet shipped is listed in §6 — What’s not yet shipped with links to the tracking DE entries.The governed-run spans, the worker tracing and the routing-log purpose labels for runs (ADR 0038, proposed) are implemented and are marked as such where they appear. They are in no release.
Two kinds of visibility are always on: structured logs, and Prometheus’s /metrics
endpoints on the api and gateway. /metrics needs no configuration or authentication
and is reachable from the Docker host’s loopback interface and from inside the
Compose network. Traces are opt-in: nothing leaves the deployment until you set an
OTLP endpoint, at which point the api and gateway emit domain-specific spans —
citation checking, anonymization, skills, inference dispatch, playbooks, autonomous
tool calls — alongside standard HTTP instrumentation. The autonomous-session audit
log is separate from tracing and always on: M4 autonomous-session lifecycle events
write rows to the local audit_log table regardless of OTel configuration.
Anonymization spans carry only entity counts and type labels — never raw entity
values — even when traces go to a third-party backend; other domain spans record
their own structured attributes (resource IDs, enum/status labels, numeric metrics
such as cost and confidence) but never raw document text or request parameters.
Coverage isn’t complete: the streaming inference path, some playbook nodes, and
log-to-trace correlation aren’t instrumented yet, and two documented outcomes — a
refused metric label and the autonomous_session.started audit action — are
defined but never actually written. See §6 and the
“Known gap” notes below before building an alert on either.
For the architectural context see docs/architecture.md §OBS and the “What the diagram doesn’t show” section. For the deployment recipes see deploy/observability/README.md.
LQ.AI initializes the OTel SDK only when at least one of the endpoint variables
below is set. If none are set, the services start cleanly and spans are silently
dropped — no traces leave the deployment (per PRD §5.7).
Prometheus /metrics is always on regardless of OTel configuration.
Transport is OTLP/HTTP for all signals.
OTEL_EXPORTER_OTLP_ENDPOINT
http://otel-collector:4318.OTEL_EXPORTER_OTLP_TRACES_ENDPOINT
OTEL_EXPORTER_OTLP_METRICS_ENDPOINT
OTEL_SERVICE_NAME
service.name attached to every span.lq-ai-api (api service), lq-ai-gateway (gateway service). Under ADR 0038 the Compose files and both recipes also set lq-ai-arq-worker for the arq-worker service; see Governed runs.OTEL_RESOURCE_ATTRIBUTES
key=value,key=value pairs appended to the resource. Use for deployment environment, region, or cluster tags. Example: deployment.environment=production,k8s.cluster.name=prod-east.OTEL_TRACES_SAMPLER
parentbased_always_on (sample everything). For production, use parentbased_traceidratio and set OTEL_TRACES_SAMPLER_ARG=0.1 (10% head sampling).parentbased_always_onOTEL_TRACES_SAMPLER_ARG
parentbased_traceidratio, the sampling ratio (0.0–1.0).1.0Recommended production configuration:
OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318OTEL_SERVICE_NAME=lq-ai-api # or lq-ai-gateway per serviceOTEL_TRACES_SAMPLER=parentbased_traceidratioOTEL_TRACES_SAMPLER_ARG=0.1OTEL_RESOURCE_ATTRIBUTES=deployment.environment=productionW3C traceparent propagates across the full api → gateway → provider call chain
(M3-F1). A single chat-send surfaces as one end-to-end trace, not three separate
traces — verified by test_trace_propagation.py in both api/ and gateway/.
Beyond HTTP auto-instrumentation (FastAPI + httpx), the services emit domain spans for the high-value LQ.AI operations. These spans are no-ops when OTel is disabled.
citation.verifyWraps the full Citation Engine cascade for one citation candidate. Child spans cover each stage that runs.
Child span Emitted when
citation.stage.exact_match citation.stage.tolerant_match citation.stage.paraphrase_judge citation.stage.ensemble Attributes on each citation.stage.* child span:
Attribute Description
citation.stage.verified citation.stage.confidence Attributes on citation.verify:
Attribute Description
citation.method exact_match, tolerant_match, paraphrase_judge, ensemble_strict, or ensemble_majoritycitation.confidence citation.partial true when the source partially supports the claimcitation.tier_envelope document.id Additional attributes on citation.stage.ensemble:
Attribute Description
citation.ensemble.n_judges citation.ensemble.rule strict or majoritySpan events:
Event Meaning
exact_match.hit tolerant_match.hit ensemble.budget_fallback anonymization.pre / anonymization.postEmitted by the gateway anonymization middleware (gateway/app/anonymization/middleware.py).
anonymization.pre wraps the entity-detection + pseudonymization pass before the
model call; anonymization.post wraps the rehydration pass on the response.
The domain attributes below are recorded on anonymization.pre only.
anonymization.post is created by the @traced("anonymization.post") decorator and
carries no domain attributes — it exists to time the rehydration pass and capture an
exception if one is raised, nothing more. (The per-entity rehydration count is not
recorded; capturing it on the post span is a future enhancement, not a shipped claim.)
Attributes on anonymization.pre:
Attribute Description
anonymization.enabled enabled flaganonymization.tier pre span)anonymization.skip_reason disabled (config off), tier_floor (routed tier not in apply_at_tiers), privileged (privileged matter), request_opt_out (anonymize=false on the request)anonymization.entity_count 0 when skipped)anonymization.entity_types PERSON,MATTER_NUMBER). Never raw entity values — see §4. Recorded only when the pass actually runs (not on the skip path).Span event on anonymization.pre:
Event Meaning
anonymization.skip.{reason} {reason} is the skip_reason value above (e.g., anonymization.skip.disabled)Span events:
Event Meaning
anonymization.skip.{skip_reason} skip_reason value (disabled, tier_floor, privileged, or request_opt_out)skill.executeOne span per skill applied to the request.
Attribute Description
skill.slug nda-review)skill.version skill.author SKILL.md frontmatter (may be null for built-ins until DE-316 lands)project.id project.privileged true for privileged matterschat.id inference.dispatchEmitted by the gateway on the non-streaming chat_completions path. Carries the
post-dispatch cost and outcome data. Note: the streaming path does not yet emit
this span — tracked at DE-317.
Attribute Description
inference.provider name from gateway.yaml (for example anthropic-prod, ollama-local) — the operator-chosen entry name, not the adapter typeinference.model inference.tier inference.outcome success, provider_error, network_error, or unavailable (the last when no adapter could be instantiated). The provider_error / network_error labels come from outcome_label_from_error in gateway/app/router.py; unavailable is set on the NoAdapterAvailableError path in gateway/app/api/inference.py.inference.tokens_in inference.tokens_out inference.cost_usd On the unavailable path (no adapter could be instantiated for the resolved target), only inference.provider and inference.outcome are set — inference.model, inference.tier, inference.tokens_in/inference.tokens_out, and inference.cost_usd are absent, because no native model or tier was resolved.
playbook.execute / tabular.executeTop-level spans wrapping a full Playbook or Tabular Review execution.
playbook.execute attributes (api/app/playbooks/executor.py): playbook.id,
playbook.contract_type, position.count, document.id. Has per-position child
spans (playbook.position) carrying playbook.position.id and
playbook.position.order (api/app/playbooks/nodes.py).
tabular.execute attributes (api/app/tabular/executor.py):
tabular.document_count, tabular.column_count. Has per-cell child spans
(tabular.cell) carrying tabular.document.id and tabular.column.name
(api/app/tabular/nodes.py).
Note on playbook.position coverage: child spans are emitted in the classify
node but not yet in the redline node — tracked at
DE-318.
autonomous.execute / autonomous.tool_callEmitted by the M4 Autonomous Layer. autonomous.execute
(api/app/autonomous/executor.py) wraps a full autonomous-session graph
invocation; autonomous.tool_call (api/app/autonomous/guard.py) wraps the
per-tool chokepoint that enforces the autonomy brakes (halt, phase-grant, cost
cap) and dispatches the tool. Both record counts, type labels, IDs, and enum
values only — never raw parameters or document text.
Attributes on autonomous.execute:
Attribute Description
autonomous.session_id autonomous.trigger_kind autonomous.halt_state Attributes on autonomous.tool_call:
Attribute Description
autonomous.session_id autonomous.phase autonomous.tool autonomous.halt_state autonomous.outcome external_halt (R5 halt latch), tool_not_granted (R6 phase-grant miss), cost_cap_reached (R4 cap), or — on the dispatch path — the ToolResult.outcome from the executed toolautonomous.cost_usd The autonomous spans also write audit rows — see §2.4 — Audit actions.
orchestration.job / orchestration.run / orchestration.effectADR 0038 D2, proposed; implemented and in no release. A
governed run executes on the arq worker, not in the API request that accepted or
approved it. These spans put the worker’s part of a run in the same trace as that
request. Attributes are identifiers and labels only — never prompt, document or
answer text. autonomous.tool_call, above, still wraps the chokepoint beneath
each orchestration.effect.
orchestration.job
api/app/workers/orchestration_worker.pyorchestration.run
api/app/autonomous/orchestration/executor.pyorchestration.job.orchestration.effect
api/app/autonomous/orchestration/effects.pyorchestration.intent refused and orchestration.step_kind tool_refused, and no autonomous.tool_call beneath it, because nothing runs. It also leaves a zero-cost effect row, a step shown to the owner and an orchestration.tool_refused audit row.orchestration.root_id
orchestration.session_id
orchestration.outcome
orchestration.run: completed, waiting_children, busy or stopped; not set when the run span fails. On orchestration.job: one of those, or disabled (the run’s profile is switched off) or interrupted (an error ended the invocation; the recovery sweep resumes from stored state). Every job that opens a span records one. A job that arrives when no profile is enabled at all returns disabled before any span opens.error.type
autonomous.tool_callquiet_span in api/app/observability_helpers.py).orchestration.profile
model_demo_v1 (the 0.8.0 fictional profile) or new_harness_v1 (New harness)orchestration.role
root or childorchestration.stage
planning before a plan exists, execution afterorchestration.phase
orchestration.intent
orchestration.skill
new_harness/orchestrator.v2. A prompt file is not a skill, and the name is the file’s name.orchestration.step_kind
model_turn or document_read; empty when the effect has noneHow a run’s trace is connected.
POST /api/v1/chats/{chat_id}/runs) or
approves it is traced like any other request.enqueue_orchestration_job in api/app/workers/queue.py). orchestration.job
continues that context, so the worker’s spans share the request’s trace. A job
queued from inside a worker job, such as the wake-up of a run’s tasks, carries that
job’s context the same way. The carrier is empty when tracing is not configured. A
job woken by the recovery sweep, which runs outside any request, has nothing to
continue and starts its own trace. So does a job whose carrier is malformed: the
carrier is data from the queue, it grants nothing, and it cannot fail the job.orchestration.run wraps each claimed session and
orchestration.effect wraps each effect. A model call leaves the worker through the
same gateway client code the API uses, so its context continues to the gateway’s
spans.OTEL_EXPORTER_OTLP_ENDPOINT set. The worker
initializes tracing from the same variables as the API; without an endpoint its
spans are dropped and a run’s trace ends at the API request. docker-compose.yml
and docker-compose.release.yml pass OTEL_EXPORTER_OTLP_ENDPOINT (empty by default)
and OTEL_SERVICE_NAME (default lq-ai-arq-worker, overridden by
LQ_AI_ARQ_WORKER_OTEL_SERVICE_NAME) to arq-worker. Both recipes under
deploy/observability/ set arq-worker to http://otel-collector:4318 with service
name lq-ai-arq-worker and the same sampler default as the API.What a span does not carry. A span holds identifiers and labels. It holds no
prompt, no opening message, no model reply, no tool-call argument, no document text and
no search query or passage. How a run’s calls were set up and what its model wrote back
are read from the run’s own records through a route for the run’s owner (and, with an
audit row for each read, an admin or an auditor),
GET /api/v1/autonomous/orchestration/{root_id}/sessions/{session_id}/trace, described in the
backend OpenAPI sketch. That read is not a span, a log or an
export to a Collector. It rebuilds each call’s opening from the records and returns each
turn’s text and tool calls to that reader. It does not hold document text a read
returned or passages a search returned, nor a later turn’s full request.
What was checked. The plan’s
Implementation status
lists test_runs_in_chat.py and test_run_tracing.py as the evidence for these spans and
for the trace context carried through queued jobs. It also lists no full Docker stack
smoke, so a run’s trace through a Collector on a running Compose stack with these recipes
has not been observed.
Prometheus metrics are served by the api (:8000/metrics) and gateway
(:8001/metrics). These endpoints are always on and carry no authentication (the
gateway’s key check is attached to its /v1 routers, not to /metrics). They are
not container-internal: docker-compose.yml publishes both services on the host’s
loopback interface by default (127.0.0.1:8000, 127.0.0.1:8001), so any process on
the Docker host can scrape them. They stay off other interfaces unless the operator
changes API_BIND_ADDR / GATEWAY_BIND_ADDR or routes them through a reverse proxy;
the shipped proxy recipes forward only /lq-ai-api/v1/* to the api, so /metrics is
not on the proxied path unless added.
lq_ai_api_http_requests_total
lq_ai_api_http_request_duration_seconds
lq_ai_gateway_http_requests_total
lq_ai_gateway_http_request_duration_seconds
lq_ai_gateway_inference_requests_total
_record_inference_outcome in gateway/app/router.py, which emits success, provider_error, or network_error. See the note below on the refused label.Known gap — the refused outcome label is never emitted. The
INFERENCE_REQUESTS_TOTAL docstring in gateway/app/observability.py lists
refused as a possible outcome value, but no code path passes it:
_record_inference_outcome is called only with success or the two
outcome_label_from_error values (provider_error / network_error). The
tier-floor refusal path (_write_refusal in gateway/app/api/inference.py) returns
a tier_below_minimum gateway error without incrementing this counter. Treat
refused as documentation-only until a refusal counter increment is wired up.
The M4 Autonomous Layer writes structured audit rows to the audit_log table for
session lifecycle events, distinct from OTel spans. Coverage is not complete: the
started action is reserved but never written at session creation (see the known
gap below). All autonomous-session
writes flow through autonomous_audit (api/app/autonomous/audit.py), which
stamps a canonical autonomous_session.<event> action and gates the details
dict to counts, type labels, IDs, costs, and enums — never raw entity values.
The details dict carries only structured scalars (e.g., to_phase, reason,
tool, outcome, cost_total_usd, findings_count, artifacts_count,
projected_usd).
autonomous_session.phase_transition
api/app/autonomous/phases.pydetails.to_phase)autonomous_session.tool_call
api/app/autonomous/guard.pyoutcome ∈ started, the executed tool’s outcome, or tool_not_grantedautonomous_session.halted
api/app/autonomous/guard.py (external halt), api/app/workers/autonomous_worker.py (idle timeout)details.reason = external_halt or idle_timeout)autonomous_session.cost_cap_reached
api/app/autonomous/guard.pydetails.projected_usd)autonomous_session.completed
api/app/autonomous/nodes.pydetails.cost_total_usd, findings_count, artifacts_count)autonomous_session.halt_requested
api/app/api/autonomous.pyPOST /sessions/{id}/halt — the API-layer request that precedes the executor’s halted eventKnown gap — autonomous_session.started is defined but never emitted. The
closed-action set _ACTIONS in api/app/autonomous/audit.py includes started,
and the module docstring shows it as an example, but no production code path writes
it (the only "started" string in the executor is the outcome="started" detail
on a tool_call row, a different field). Session creation does not currently write
a started audit row. Treat started as a reserved-but-unused action until a
creation-time write is wired up.
Governed-run audit rows. Governed runs (ADR 0035) write orchestration.<event>
rows through the run store (api/app/autonomous/orchestration/store.py), with
resource_type = autonomous_session and the run’s root session as resource_id.
details hold identifiers, counts, labels and amounts, never prompt, document or answer
text. ADR 0038 (proposed) adds orchestration.tool_refused (a call
the application refused: the registry tool name or unregistered, and a code, never the
model’s own text) and orchestration.run_closed (an owner closed an uncertain run: the
number of effects closed and the amount charged). A run the orchestrator answered by
itself, with no plan, writes orchestration.answered_directly in place of
orchestration.root_delivered, with verification: unverified and the chat and message
identifiers; a plan that fails writes orchestration.planning_failed with the stop reason
as a label (invalid_proposal, unfunded_plan, plan_not_handed_over or
plan_has_placeholders). For a run hosted in a chat, orchestration.root_delivered
carries the chat and message identifiers. For a profile
whose answers are checked against the run’s sources (the documents selected for it, and files a knowledge-base search found) it also records verification
(citations_checked or unverified), the counts of quotations verified, unverified and
unresolved, the number of quoted passages that carry no source marker and so were not
checked (citations_unmarked), and the fiduciary-gate status. If the checks themselves fail, the answer is
still delivered and the row records verification: unverified and checks: failed. The
chat receipts feed reads the orchestration.* rows through the run message’s link to its
root session, so they appear in the chat’s receipts drawer. Attaching a chat attachment to
the matter for a run writes project.file_attached with source: run_selection; that row
is keyed to the matter, so it is in the admin audit log and not in the chat’s receipts.
The gateway writes one inference_routing_log row for every inference request: provider,
model, tier, tokens, latency, cost and whether anonymization ran. Two things tie a row to
what caused it. chat_id and message_id come from the lq_ai_chat_id and
lq_ai_message_id fields the backend sends. purpose is a label for the kind of
traffic, from the request’s lq_ai_purpose field (embedding on the embeddings path).
The gateway keeps a closed set of labels. A request with no label records chat, and so
does a label outside the set. The column is varchar(32), so a label must fit it.
purpose Recorded for
chat embedding judge_paraphrase judge_case_content judge_authority_content judge_treatment tabular_extraction tabular_bulk_op playbook_executor playbook_easy_extract autonomous_executor orchestration_run Proposed (ADR 0038 D2). Only chat, embedding, judge_paraphrase and
tabular_bulk_op were in the set before. The other labels were already sent by the backend and recorded as
chat; recording them as themselves, and the chat and message identifiers on a run’s
model calls, are implemented. Two Easy Playbook assembly labels,
playbook_easy_assemble_describe_position and playbook_easy_assemble_describe_tier,
are longer than the column and are not in the set, so those rows still record chat. The
set is _KNOWN_PURPOSES in gateway/app/api/inference.py. Chat and message identifiers
are also passed on tool_call_log rows, but that table records only calls to external
tool providers and neither shipped run profile grants one, so a run writes none.
Structured logs are emitted by both services. Log-trace correlation (injecting
trace_id / span_id into log lines) is not yet shipped — tracked at
DE-300.
Until DE-300 lands, pivoting from a span in Tempo or Honeycomb to the logs for that
request requires matching on timestamp + service.name manually.
Before attributing a timeout in the logs to the model, check which layer timed out: the gateway’s provider adapters (Anthropic, OpenAI and Ollama) default to 600 s, but the api’s own client to the gateway waits 60 s and its streaming call inherits that. See Something is wrong for both layers.
Two ready-made Docker Compose overlays are in deploy/observability/.
Read deploy/observability/README.md first — it explains
how to switch between them and what each adds to the base stack.
Recipe: deploy/observability/grafana-tempo-loki/
Use this when you are starting fresh and want a self-contained observability stack
with no third-party SaaS accounts. The overlay adds Grafana, Tempo (traces),
Loki (logs), Prometheus (metrics), and an OTel Collector — all running alongside
the main LQ.AI Compose stack. The OTel Collector is the collection point; api and
gateway send spans to it over OTLP/HTTP; it fans out to Tempo. Prometheus scrapes
/metrics directly.
Time-to-first-trace once the overlay is running: under 15 minutes with the bundled Grafana dashboard.
Recipe: deploy/observability/otel-collector-standalone/
Use this when you already run Honeycomb, Datadog, Lightstep, Splunk, or any other OTLP-compatible backend and you do not want Grafana or Tempo locally. The overlay adds only an OTel Collector configured to forward spans to your backend. You supply the backend endpoint and API key via environment variables in the Collector config; the api and gateway are wired to the local Collector, which handles batching and export.
The Anonymization Layer (Inference Gateway; docs/security/anonymization.md) pseudonymizes sensitive entities — names, matter numbers, organization names, contact details — before they reach a cloud inference provider. The same posture extends to telemetry: span attributes carry entity counts and type labels only, never raw entity values.
Concretely: if a request triggers anonymization of three person names and one matter
number, the anonymization.entity_count attribute is 4 and
anonymization.entity_types is the sorted array ["MATTER_NUMBER", "PERSON"]. The actual names and the
matter number do not appear in any span attribute, span event, or metric label.
This invariant is enforced by gateway/tests/test_anonymization_observability.py.
Operators forwarding trace data to a third-party backend (Honeycomb, Datadog, Lightstep) receive the same guarantee: the telemetry side-channel cannot leak what the Anonymization Layer is designed to protect.
Per PRD §5.7: until OTEL_EXPORTER_OTLP_ENDPOINT
or a per-signal endpoint is set, no traces leave the deployment. The OTel SDK is
present in the service images but the TracerProvider is never initialized, and spans
are silently dropped.
Prometheus /metrics is always on and unauthenticated. With the default Compose
bindings it is reachable from the Docker host’s loopback interface (127.0.0.1:8000
for api, 127.0.0.1:8001 for gateway) as well as from inside the Compose network;
reaching it from anywhere else requires the operator to rebind the port or route it
through a reverse proxy. The metrics
endpoint does not emit data to any external destination — it is a pull surface, not
a push surface.
This behavior is pinned by tests/test_observability.py (cross-cutting integration
test, added alongside this doc in M3-F3).
The items below are honestly deferred. Each links to its PRD §9 tracking entry with the acceptance criteria and contributor profile.
OTel Deepening DEs (PRD §9, added at M3-F close):
orchestration.job span, which is not general arq instrumentation.service.name; alignment or disable decision pendingStreaming coverage gaps:
StreamingRehydrator path not yet instrumentedinference.dispatch span on the streaming pathPlaybook coverage gap:
playbook.position child spans on the redline nodeLLM-specific tracing sink (Langfuse) — not implemented. PRD §5.4
describes an optional self-hosted Langfuse sink for LLM-specific tracing. Nothing in api/
or gateway/ implements it. The telemetry: block in gateway.yaml.example, with its
opentelemetry and langfuse entries, is accepted by the gateway’s configuration loader
and read by no code. The LANGFUSE_* variables in .env.example and the Compose files
are read by no code either, so setting them has no effect. Trace export is controlled only
by the OTEL_* variables in §1. Spans, audit rows and
routing-log rows carry identifiers, counts and labels, not prompt or document text, and the autonomous, tabular, orchestration and citation spans record a failure’s class and never its message (the other spans still record it). Under
ADR 0038 (proposed), a run’s owner can read how each call was set up and
what the model wrote back through the trace read described under
the governed-run spans; that read draws on the run’s own records and is
not this sink.
SLO + RUM catalog (from docs/proposals/opentelemetry-deepening.md):
DE-E (Browser RUM, same as DE-303 above), DE-F (Published SLO catalog), and DE-G
(Performance regression tracking) are downstream of the shipped M3-F signals and are
deferred to whichever milestone first makes the signal inventory stable enough to
define meaningful targets.
Maintained by the maintainer team. Updates land alongside observability changes. If a claim in this doc doesn’t match the code, the code is canonical — please open an issue.