Add the five Ledgerline application metrics

- ledgerline.transactions.total (counter: kind, status, currency) and
  ledgerline.transaction.duration (histogram, seconds: kind) in gateway,
  the only place that sees the final outcome, failures included. currency
  is client input, so values outside the seeded set are recorded as
  "other".
- ledgerline.fraud.score (histogram, buckets in 0.1 steps to 1.0) in fraud.
- ledgerline.stream.lag (observable gauge: consumer_group) in audit and
  notifier, read from XINFO GROUPS through a separate sync Redis client,
  because callbacks run on the scrape thread; the call is untraced so
  scrapes don't create traces.
- ledgerline.db.pool.in_use (observable up-down counter: pool) in every
  service with a DB engine. The SQLAlchemy instrumentation's own pool
  metric names a pool by DSN, which merges the normal and starved engines.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Movses Aslanyan
2026-10-19 16:05:40 +03:30
co-authored by Claude Opus 5.5
parent 7cb3231226
commit 920712f32b
8 changed files with 107 additions and 2 deletions
+28 -1
View File
@@ -1,4 +1,5 @@
import asyncio
import time
import uuid
from pathlib import Path
@@ -7,7 +8,7 @@ from fastapi import FastAPI, Request
from fastapi.responses import FileResponse, JSONResponse
from fastapi.staticfiles import StaticFiles
from pydantic import BaseModel, field_validator
from opentelemetry import trace
from opentelemetry import metrics, trace
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor
from opentelemetry.instrumentation.utils import suppress_instrumentation
@@ -30,6 +31,25 @@ FastAPIInstrumentor.instrument_app(
exclude_spans=["receive", "send"],
)
meter = metrics.get_meter(__name__)
transactions_total = meter.create_counter(
"ledgerline.transactions.total",
unit="{transaction}",
description="Transactions submitted, by final outcome",
)
transaction_duration = meter.create_histogram(
"ledgerline.transaction.duration",
unit="s",
description="End-to-end time to process a submitted transaction",
# Seconds, not the SDK's millisecond-shaped defaults (0, 5, 10, ... 10000).
explicit_bucket_boundaries_advisory=[
0.005, 0.01, 0.025, 0.05, 0.075, 0.1, 0.25, 0.5, 0.75, 1, 2.5, 5, 7.5, 10
],
)
# currency is unvalidated client input; only the seeded
# set becomes a label value, so a bad client can't mint new series.
KNOWN_CURRENCIES = {"EUR", "USD", "GBP"}
STATIC_DIR = Path(__file__).parent / "static"
app.mount("/static", StaticFiles(directory=STATIC_DIR), name="static")
@@ -115,8 +135,11 @@ async def account_insights(account_id: str):
@app.post("/api/transactions", status_code=201)
async def submit_transaction(body: TransactionRequest):
request_id = _request_id()
start = time.perf_counter()
status = "failed"
try:
result = await orchestrator.create_transaction(body.model_dump())
status = result["status"]
except UpstreamFailure as exc:
log = logger.error if exc.status_code >= 500 else logger.warning
log(
@@ -129,6 +152,10 @@ async def submit_transaction(body: TransactionRequest):
return JSONResponse(
status_code=exc.status_code, content={"error": exc.message, "request_id": request_id}
)
finally:
transaction_duration.record(time.perf_counter() - start, {"kind": body.kind})
currency = body.currency if body.currency in KNOWN_CURRENCIES else "other"
transactions_total.add(1, {"kind": body.kind, "status": status, "currency": currency})
return JSONResponse(status_code=201, content={**result, "request_id": request_id})