Dashboards: observability/dashboards/generate.py (Grafana Foundation SDK)
writes k8s/grafana/dashboards/*.json -- a RED dashboard per HTTP service
and a system overview -- shipped through a file provider and
configMapGenerators, so `kubectl apply -k k8s/` rebuilds them. Verified by
deleting Grafana and its volume and re-applying. Adds tracesToMetrics to
the Tempo datasource.
Alerts: Prometheus rule files delivering to Alertmanager, under a
labels/annotations contract enforced by observability/alerts/check_contract.py,
with runbooks in docs/runbooks/.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
RED per service, the service map and exemplar click-through are verified
live. Build step 3's second metrics path and step 6's TraceQL test
migration are not done.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Tempo's metrics-generator defaults the exemplar label to traceID, which
the Prometheus datasource's exemplarTraceIdDestinations (trace_id) never
matched: exemplars rendered in Grafana but didn't link to Tempo. Set
trace_id_label_name in the overrides so trace_id is the one correlation
key everywhere.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This reverts commit a1f1d80. Phase 5 step 4 is done: the burst minted
68 series across 60 account_id values.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Phase 5 step 4: a deliberate cardinality explosion. Every account that
transacts mints its own series. To be reverted after the burst.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
czy publishes no service ports, so Traefik exposes each service's :9464
as https://ledgerline.vione.cloud/metrics/<service> behind basic auth,
and the platform's Prometheus gets a `ledgerline` job with one target
per service. The targets share an address; a `service` label tells them
apart, and relabels derive __metrics_path__ and instance from it.
create-secrets.sh now also creates the prometheus-scrape-auth secret
(mounted optional, so a missing secret fails only this job) and writes
the gitignored htpasswd file Traefik reads, which is copied to czy like
.env.observability.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- ledgerline.transactions.total (counter: kind, status, currency) and
ledgerline.transaction.duration (histogram, seconds: kind) in gateway,
the only place that sees the final outcome, failures included. currency
is client input, so values outside the seeded set are recorded as
"other".
- ledgerline.fraud.score (histogram, buckets in 0.1 steps to 1.0) in fraud.
- ledgerline.stream.lag (observable gauge: consumer_group) in audit and
notifier, read from XINFO GROUPS through a separate sync Redis client,
because callbacks run on the scrape thread; the call is untraced so
scrapes don't create traces.
- ledgerline.db.pool.in_use (observable up-down counter: pool) in every
service with a DB engine. The SQLAlchemy instrumentation's own pool
metric names a pool by DSN, which merges the normal and starved engines.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Alloy 404s OTLP metrics, so each service gets a MeterProvider with a
PrometheusMetricReader (same Resource as traces) served by
prometheus_client on its own port instead of a FastAPI route: gateway's
port 8000 is publicly routed by Traefik, and the audit worker has no HTTP
app. Host/port come from OTEL_EXPORTER_PROMETHEUS_HOST/PORT. Setting the
provider before the instrumentors also turns on their built-in HTTP and
DB pool metrics.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Phase 4 records what was built, the two skipped Build steps (OTLP-logs
experiment, LogQL test migration) and what carries forward. Phase 3
records its outcome, with the chaos-nightmare and blocking_event_loop
criteria attested by the user.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Without an explicit service_name, Loki derived it from the container
label (ledgerline-sanctions-1), while spans carry service.name
"sanctions". Grafana's Tempo->Loki link selects on
{service_name="<service.name>"}, so it matched no stream. The czy agent
now sets service_name from com.docker.compose.service; `container` is
unchanged. CLAUDE.md's description of the labels is updated to match.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Prose f-strings become past-tense events with consistent field names
across services (account_id, transaction_id, amount_minor, risk_score,
status_code, consumer_group, message_id, chaos_flag). The orchestrator
and both consumers move from logging.getLogger to structlog so they can
pass fields.
New events cover what was silent before: the gateway's UpstreamFailure
path (transaction_failed, with the underlying cause), fraud's score and
sanctions' screening outcomes, and each successfully processed stream
message. holder_name is no longer logged at all.
"lookup" and "healthz" still appear in exactly one event each, so the
pathology tests' log-line counts are unchanged until they move to LogQL.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A processor in the shared chain masks sensitive field names (credential
fragments like password/token/authorization, plus exact names such as
card_number and holder_name) at any depth, for structlog and stdlib
records alike. It sees field names only: values interpolated into the
event string still leak until call sites pass them as fields.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A processor in the shared chain reads the active span context and adds
its IDs as lowercase hex, so both structlog and stdlib records carry
them. The lowercase `trace_id` name is what the provisioned Loki derived
field's case-sensitive regex matches; outside a span nothing is added.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
configure_logging now builds one processor chain shared by structlog
calls and foreign stdlib records (uvicorn, the consumers' getLogger),
rendered as one JSON object per line on stdout. Tracebacks stay inside
their line instead of splitting into one Docker log entry per frame.
uvicorn.access is deliberately left untouched: --no-access-log works by
disabling its propagation, and re-enabling it would bring access logs
back (and inflate the healthz/lookup line counts the tests rely on).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Step 6: downstream_client() no longer sends X-Request-Id, and request_id
is gone from the internal request bodies, models, function signatures
and log lines; correlation now travels as traceparent. The gateway's
public request_id stays (the contract tests require it) but is now the
trace ID, 32 hex digits and still a valid UUID, so the value the
frontend shows can be pasted into Tempo.
Step 7: delete tests/test_no_observability.py. It guarded the Phase 0
"no observability" state, which Phase 3 ends by design.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Where an error is caught and logged under a span that would otherwise
stay clean, also record_exception and set ERROR on that span: the
CONSUMER span when a stream message fails, fraud.score_transaction when
the sanctions call times out or fails, and the SERVER span of every
/readyz check (a 503 already made it ERROR, but without the exception).
Left alone where instrumentation already records it: ledger's app-wide
exception handler (FastAPI SERVER span), caught httpx errors (CLIENT
span), and the stream publish failure (PRODUCER span). The consumers'
failed reads run outside any span, so they stay log-only.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Manual INTERNAL spans for the business steps the auto-instrumentation
can't name: fraud.score_transaction, sanctions.lookup_watchlist,
ledger.settle and insights.aggregate_120d, with ledgerline.* attributes
(account, amount, risk score, match outcome). holder_name stays off the
spans. Each span starts after its handler's chaos flag reads and
injection points, and no chaos-derived values become attributes.
The gateway's /api/health/all fan-out now runs under
suppress_instrumentation: with the route excluded, its five downstream
/healthz calls were each a parentless root trace.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Redis Streams have no headers, so the gateway injects traceparent into
the message fields inside a PRODUCER span. The notifier and audit
consumers extract it and open one CONSUMER span per message, so their
DB and XACK spans join the transaction's trace. The idle XREADGROUP poll
runs under suppress_instrumentation (it was a ~2s root trace every loop),
and audit strips the trace fields so its stored payload is unchanged.
The gateway also stops tracing its frontend's /api/health/ polling.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
setup_telemetry now instruments httpx and redis process-wide. Both patch
the client classes, so clients created at import time (gateway's
_proxy_client, common.redis_client) are covered too. httpx CLIENT spans
inject traceparent, so the downstream FastAPI SERVER spans join the
caller's trace. gateway and fraud get setup_telemetry, the FastAPI span,
and the OTLP auth header. All seven services now export traces.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
sanctions, notifier, ledger, insights and audit now call setup_telemetry
and instrument their engines with one SQLAlchemyInstrumentor call per
process (a second instrument() call is ignored, leaving that engine with
connect spans only). ledger and insights pass both their normal and
starved engines. The FastAPI services also get the server span, with the
same /healthz and receive/send exclusions as sanctions.
Compose: OTEL_SEMCONV_STABILITY_OPT_IN=http,database for the stable
attribute names, the OTLP endpoint moves to the shared env, and the four
newly instrumented services get the auth header from .env.observability.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
FastAPIInstrumentor.instrument_app gives sanctions its SERVER span back,
skipping /healthz (the Docker healthcheck flooded Tempo search) and the
ASGI receive/send INTERNAL spans. Drop the OTEL_* env vars only the
zero-code configurator read, and the unused SERVICE_NAME.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
common/telemetry.py builds the Resource, TracerProvider, BatchSpanProcessor
and OTLP/HTTP exporter by hand; sanctions calls setup_telemetry() at
import instead of running under opentelemetry-instrument. No instrumentor
is enabled yet, so sanctions emits no spans for now.
service.version comes from OTEL_RESOURCE_ATTRIBUTES: deploy.sh exports
DEPLOY_HASH (the server's HEAD after the pull) and compose falls back to
"dev". deployment.environment.name=prod is dropped.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Phase 2 traced sanctions through the zero-code wrapper, so there is no
common/telemetry.py yet; note that build step 1 writes it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
First auto-instrumented sanctions trace verified in Tempo. Records what
carries forward: /healthz trace noise, the prod environment label in the
base compose file, and the red G10 guard test.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Add opentelemetry-distro/exporter-otlp (pinned pair) and run
opentelemetry-bootstrap at build time; wrap the sanctions command with
opentelemetry-instrument and configure the SDK via OTEL_* env (traces
only, OTLP/HTTP to otlp.vione.cloud). create-secrets.sh now writes the
base64 OTLP auth header into .env.observability.
The G10 guard test (no observability libraries in requirements) is
expected to fail from here until the guard suite is retired in Phase 3.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Alloy only registers an OTLP/HTTP route for a signal with an output, so
POST /v1/metrics and /v1/logs return 404 rather than being accepted and
silently dropped (verified live against otlp.vione.cloud). Fix the claim
everywhere it appeared. Also mark Phase 2 in progress.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
k8s/ is now a faithful port of the mentor's Ansible Grafana stack
(Prometheus, Alertmanager, Loki, Tempo, Alloy, Grafana, node-exporter)
for the ledgerline-observability namespace on falken: same versions and
configs, one *.vione.cloud ingress per tool behind Traefik basic auth,
secrets generated by create-secrets.sh into gitignored files.
Two fixes over the Ansible role, both verified live: Tempo's
metrics-generator needs traces_storage or it runs no processors, and
Prometheus needs exemplar-storage to keep exemplars.
Ledgerline stays on czy; an Alloy agent in the Traefik override ships
its container logs to Loki. Roadmap and CLAUDE.md now point at k8s/.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
phase-1: deploy via deploy.sh, correct the container count, and note
that stripped host ports leave no path for the Phase 5 scrape.
phase-2: explain why Grafana is not on the Ledgerline server and why
Cloudflare's proxy never forwards :3000 or the other platform ports.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Wrap Submit and Load burst in one full-width .row and Result, Chaos and
Service health in a second. Each .row splits its width evenly between its
panels (gap subtracted first) and stacks below 900px.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Put Submit and Load burst side by side on the first row at half width,
and Result, Chaos and Service health on the second row at a third each.
Uses a 6-column grid so the gap is accounted for exactly; stacks to one
column below 900px.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Insert a new Phase 1 (deployment: Compose + Traefik on a server, marked
Done) between the base project and the OTel concepts phase, and shift
the old phases 1-10 to 2-11. Update summary, intro, README and CLAUDE.md
references (spoiler rules now target Phase 9, the bottleneck hunt).
Add docker-compose.traefik.yml: an override that fronts the gateway with
Traefik and Let's Encrypt and removes all other published host ports.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Replace the hardcoded "Phase 0a" line with an instruction to read
docs/roadmap/summary.md first, and add a verify-report-confirm rule
for marking a phase Done in the roadmap files.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
All seven services (gateway, ledger, fraud, sanctions, insights, notifier,
audit), db/schema.sql plus optional_indexes.sql, scripts/seed.py and
smoke.py, the gateway-served frontend, and docker-compose.yml. Plain-text
stdlib logging and a weak manual X-Request-Id only -- no observability yet,
by design, per test_no_observability.py. Implements all 13 named chaos-flag
pathologies from docs/PATHOLOGIES.md behind Redis-backed runtime flags.
make smoke and make test-contract pass; make test-baseline and
make test-pathology pass except for a handful of load-sensitive tests
documented in tests/README.md rather than papered over.
81 tests across 11 files plus a 5-module harness (load, pg, streams,
containers, payloads), covering all eleven Phase 0 goals and all 13 named
chaos-flag pathologies. Talks to the system only over HTTP, Postgres, Redis,
and the Docker CLI -- no imports of application code. RED-BASELINE.md records
the suite failing red against the empty repo before any implementation
existed; README.md documents the request-contract fields invented to fill
gaps in the spec, measurement methodology, and known-flaky tests.
This suite is a frozen contract from this point forward per CLAUDE.md.
Deploys the Grafana stack (Prometheus/Alertmanager/Loki/Tempo/Grafana with
Alloy as collector) and the ELK stack (Filebeat/Logstash/Elasticsearch/
Kibana) via Docker Compose on a target host. External to the application
repo, pre-built by the mentor.
Roadmap, phase specs, pathology catalog, and CLAUDE.md establishing
Ledgerline as an observability learning project built on a deliberately
imperfect seven-service payments backend.
