Xray
daily rollups 1h 6h 24h 48h 3d 5d 7d 14d 30d ‹
Last 30 days▾
Ends now. Anything up to this cluster's 30-day retention.
Start and end times
Cancel
Interpreted in UTC (change it at the right of this bar). Earliest available: 2026-10-19 21:38:04.
› →|
updated 1d ago
Sample data. This database was generated by scripts/seed_dev_db.py — every service, deploy, person and figure on this page is synthetic. Names are sample.*@seed.invalid and image tags are prefixed SEED-. Nothing here happened.

Detectors 31

The programs that open findings, and whether each one has actually run. A detector that has silently died is invisible everywhere else — the Findings page just looks calm. ?Each detector runs on its own cadence inside the scheduler (every 5 min the due ones run). Stored detectors read SQLite only; live ones call the log backend, Managed Prometheus or BigQuery and are gated by OBS_DETECT_LIVE. A run that raised is recorded with its error and resolves nothing.

detection on live sources on
Registered detectors and their last runs
DetectorOpensScopeCadence Last runPasses (48 h)FoundErrors
outliers
Services unlike their peers on 5xx rate, p95, p99 or Neo4j queries per request — one row per service; a warning only on the measures you choose (by default the 5xx rate), informational otherwise.
outlier.service cluster 5 min never 0 0 none
budgets
Error budgets burning faster than the objective allows (1 h and 6 h burn rates).
budget.burning cluster 5 min never 0 0 none
shapes
Cypher shapes whose p95 in the second half of a 6 h window is ≥1.5× the first half — and is at least 1 s now. Spikes under a second are ignored as noise, whatever the ratio: 2 ms → 43 ms is 21× and means nothing to anyone.
shape.degrading cluster 15 min never 0 0 none
neo4j_errors live
A Neo4j database failing Cypher executions, from Neo4j's own metric (db_query_execution_failure_total, via Managed Prometheus) -- the failures the Neo4j tab charted but nothing raised (2026-10-23). Load and plan speed are anomaly.neo4j and shape.*; this is errors.
neo4j.errors fleet 5 min never 0 0 none
endpoint_4xx_spike
One finding per endpoint for its 4xx: answering far more 4xx than its own usual rate, or rejecting nearly every call. Permission denied (401/403) is critical; other 4xx a warning. A permissions change (403), broken or expired auth (401), a removed route or contract break (404). endpoint_failing only counts 5xx, and its 4xx signal needs ~every call rejected; a 403 storm on a busy endpoint was invisible (2026-10-23: docsvc folders 96 and onboardingsvc orgs 102 4xx in five minutes, no finding). The usual rate is the endpoint's own average over the last seven days (hourly rollup), so a chronically noisy endpoint is not flagged for being itself.
endpoint.4xx_spike cluster 5 min never 0 0 none
endpoint_failing
One endpoint of a service failing most of its calls (5xx) over the last 15 minutes — the unit the observation-window alerts count and resolve per endpoint (15-OBSERVATION-WINDOW-NOTIFICATIONS-RESEARCH.md §3.5). 4xx are shown beside it, not counted as failing: a 404 is the caller's problem, a 500 is ours — but an endpoint that rejects almost EVERY call with a 4xx is its own INFO signal (a broken client or contract), separate from the 5xx alert.
endpoint.failing, endpoint.rejecting cluster 5 min never 0 0 none
hand_changes
kubectl/Helm/console writes that bypassed ArgoCD in the last 24 h — production only, only in the tilda and neo4j-system namespaces, and never an image change.
change.hand cluster 5 min never 0 0 none
fanout
Services averaging ≥25 foreground Neo4j queries per API request over the last hour.
n1.fanout cluster 15 min never 0 0 none
ingest_stale
X-ray's own ingest has not succeeded for this cluster in 45 min (warning) or 3 h (critical).
ingest.stale cluster 5 min never 0 0 none
collector_stale
A push collector that has posted before has gone quiet for 3× its cadence.
collector.stale fleet 15 min never 0 0 none
baseline_refresh
Rebuilds the hour-of-week baselines for the cluster once a day (no findings of its own).
— cluster 1 d never 0 0 none
seasonal
Three of the last four complete 5-minute buckets far outside the hour-of-week baseline. A p95 that stays under 1 s is never reported — below a second nobody notices, whatever the deviation.
anomaly.traffic, anomaly.errors, anomaly.latency, anomaly.neo4j cluster 5 min never 0 0 none
budget_forecast
The date the 30-day budget runs out, if within 14 days — a constant recent-median scenario (Theil–Sen is only the gate that says the trend is upward at all, not the source of the date).
budget.forecast cluster 1 h never 0 0 none
shape_changepoint
For each deploy in the last 6 h, a chi-square test of every busy shape's latency histogram ±60 min around it.
shape.regressed cluster 15 min never 0 0 none
shape_churn
New Cypher shapes today vs the median of the prior 14 days — literals inlined into statements show up here.
shape.churn cluster 1 h never 0 0 none
disk_forecast live
Theil–Sen on 30 days of volume usage for every Neo4j and Weaviate node; a finding when a watermark is within 30 days. X-ray's own store is NOT covered here — that is obsdb_growth.
forecast.disk fleet 1 h never 0 0 none
obsdb_growth
A daily size sample of X-ray's own store — and, on the SQLite PVC, the date it fills 30 GiB (minus a 2 GiB reserve).
forecast.obsdb fleet 1 d never 0 0 none
llm_spend_forecast live
Month-end AI spend projection per cluster (BigQuery daily cost, Theil–Sen, weekend-weighted); a finding at >1.2× last month.
llm.spend cluster 1 h never 0 0 none
k8s_events
What the cluster itself reports going wrong, from its events (every namespace): pods that cannot start for 10+ minutes (image pull, config error, unschedulable), evictions, kernel OOM kills (including a process inside a running container), Jobs out of retries, ExternalSecrets not syncing, cert-manager or other platform pods down, autoscalers that cannot read their metric, and node problems. Judged on the current state: still being reported, and a normal wait (a pod pending two minutes, then scheduled) is not an incident.
pod.cannot_start, pod.evicted, pod.oom, job.backoff, secret.sync_failing, cert.unhealthy, platform.unhealthy, scaler.failing, node.unhealthy cluster 5 min never 0 0 none
pod_unstable
A workload whose containers restart in place: warning at three restarts in 30 minutes (or one OOM kill); critical once the run has lasted an hour and is still going. Stops being reported after 30 quiet minutes and closes about 15 minutes after that, once the resolve grace passes. Says why — crash loop, OOM, liveness, image — and since when.
pod.unstable cluster 5 min never 0 0 none
service_silent
A service that normally serves requests at this hour of the week (its baseline) and has served none for the sustain window. Absence as a signal: a crashed or cut-off service produces no anomaly, only silence.
service.silent cluster 5 min never 0 0 none
cost_anomaly
Daily net cost of each project × service × cluster cell against the median/MAD of its previous 28 days; a finding for the last two complete days.
cost.anomaly fleet 6 h never 0 0 none
cost_target
Every monthly target set on Costs → Forecast against the month-end projection; a finding when the projection is over it.
cost.target fleet 6 h never 0 0 none
cost_commitment
Commitment fees over the last 30 complete days that funded no usage; a finding per commitment family above the floor.
cost.commitment fleet 1 d never 0 0 none
custom_watches
Every enabled watch (a condition written in plain English, read into a closed spec) evaluated against the five-minute rollups.
custom.watch cluster 5 min never 0 0 none
query_alerts
Every enabled query alert — a request in plain English that the model turned into a query over X-ray's own tables or one of its read-only tools — run and compared with its threshold.
custom.query cluster 5 min never 0 0 none
errors_novel live
Error shapes seen for the first time in 90 days, ≥5 times in the last hour (learns silently on day one).
error.novel cluster 15 min never 0 0 none
job_failing
Failure lines of named background jobs, clustered into failed runs; a finding when a job keeps failing within the configured window (6 h by default). Severity alone is not the filter — a line carrying an error/exception field or a failure status counts too, because rag-agent.upload logs its failures at WARNING. The complement of errors_novel, which only sees NEW shapes.
job.failing cluster 1 h never 0 0 none
weaviate_stalled live
Weaviate tenant queues that are deep and have not drained in the last hour (GMP).
weaviate.stalled fleet 15 min never 0 0 none
llm_unattributed live
≥25% of the last 24 h of LLM spend carries no organisation (BigQuery).
llm.unattributed cluster 1 h never 0 0 none
openai_cost_drift live
What the LiteLLM log says a cluster spent on OpenAI against what OpenAI billed its project, over the last complete UTC days. Either direction. Each day's pair is kept in openai_reconciliation_daily as the audit trail.
llm.cost_drift cluster 6 h never 0 0 none

Baselines hour-of-week profiles per cluster, rebuilt daily

master
10591
28 services · 2–5 weeks · 2d ago
automation
1476
12 services · 2–6 weeks · 2d ago
nexus
681
13 services · 2–5 weeks · 2d ago
production
299
23 services · 2–5 weeks · 2d ago
delta
36
6 services · 2–5 weeks · 2d ago
test27
6438
29 services · 2–7 weeks · 2d ago
test26
479
25 services · 2–7 weeks · 2d ago

Recent passes

WhenDetectorScopemsFoundError
2026-11-16 07:08:43 UTCoutliersmaster 1154
2026-11-16 06:57:06 UTCbudgetsmaster 7050
2026-11-16 06:57:06 UTCbudgetsautomation 13550
2026-11-16 06:57:06 UTCbudgetsnexus 920
2026-11-16 06:57:06 UTCbudgetsproduction 4750
2026-11-16 06:57:06 UTCbudgetsdelta 710
2026-11-16 06:57:06 UTCbudgetstest27 3200
2026-11-16 06:57:06 UTCbudgetstest26 2090
2026-11-16 06:57:06 UTCcollector_stale* 690
2026-11-16 06:57:06 UTCcustom_watchesmaster 50
2026-11-16 06:57:06 UTCcustom_watchesautomation 60
2026-11-16 06:57:06 UTCendpoint_4xx_spikemaster 170
2026-11-16 06:57:06 UTCendpoint_4xx_spikeautomation 601
2026-11-16 06:57:06 UTCendpoint_4xx_spikenexus 150
2026-11-16 06:57:06 UTCendpoint_4xx_spikeproduction 180
2026-11-16 06:57:06 UTCendpoint_4xx_spikedelta 170
2026-11-16 06:57:06 UTCendpoint_4xx_spiketest27 531
2026-11-16 06:57:06 UTCendpoint_4xx_spiketest26 160
2026-11-16 06:57:06 UTCendpoint_failingmaster 250
2026-11-16 06:57:06 UTCendpoint_failingautomation 250
2026-11-16 06:57:06 UTCendpoint_failingnexus 230
2026-11-16 06:57:06 UTCendpoint_failingproduction 290
2026-11-16 06:57:06 UTCendpoint_failingdelta 290
2026-11-16 06:57:06 UTCendpoint_failingtest27 320
2026-11-16 06:57:06 UTCendpoint_failingtest26 260
2026-11-16 06:57:06 UTCfanoutmaster 491
2026-11-16 06:57:06 UTCfanoutautomation 230
2026-11-16 06:57:06 UTCfanoutnexus 190
2026-11-16 06:57:06 UTCfanoutproduction 210
2026-11-16 06:57:06 UTCfanoutdelta 220
2026-11-16 06:57:06 UTCfanouttest27 461
2026-11-16 06:57:06 UTCfanouttest26 391
2026-11-16 06:57:06 UTChand_changesmaster 120
2026-11-16 06:57:06 UTChand_changesautomation 90
2026-11-16 06:57:06 UTChand_changesnexus 80
2026-11-16 06:57:06 UTChand_changesproduction 90
2026-11-16 06:57:06 UTChand_changesdelta 90
2026-11-16 06:57:06 UTChand_changestest27 130
2026-11-16 06:57:06 UTChand_changestest26 90
2026-11-16 06:57:06 UTCingest_stalemaster 90
2026-11-16 06:57:06 UTCingest_staleautomation 90
2026-11-16 06:57:06 UTCingest_stalenexus 100
2026-11-16 06:57:06 UTCingest_staleproduction 100
2026-11-16 06:57:06 UTCingest_staledelta 120
2026-11-16 06:57:06 UTCingest_staletest27 110
2026-11-16 06:57:06 UTCingest_staletest26 100
2026-11-16 06:57:06 UTCk8s_eventsmaster 8802
2026-11-16 06:57:06 UTCk8s_eventsautomation 10272
2026-11-16 06:57:06 UTCk8s_eventsnexus 8283
2026-11-16 06:57:06 UTCk8s_eventsproduction 10302
2026-11-16 06:57:06 UTCk8s_eventsdelta 10054
2026-11-16 06:57:06 UTCk8s_eventstest27 8903
2026-11-16 06:57:06 UTCk8s_eventstest26 8934
2026-11-16 06:57:06 UTCneo4j_errors* 11022
2026-11-16 06:57:06 UTCoutliersmaster 1314
2026-11-16 06:57:06 UTCoutliersautomation 1222
2026-11-16 06:57:06 UTCoutliersnexus 150
2026-11-16 06:57:06 UTCoutliersproduction 441
2026-11-16 06:57:06 UTCoutliersdelta 170
2026-11-16 06:57:06 UTCoutlierstest27 872