Xray
hourly rollups 1h 6h 24h 48h 3d 5d 7d 14d 30d ‹
Last 48 hours▾
Ends now. Anything up to this cluster's 30-day retention.
Start and end times
Cancel
Interpreted in UTC (change it at the right of this bar). Earliest available: 2026-10-19 14:02:36.
› →|
updated 1d ago
Sample data. This database was generated by scripts/seed_dev_db.py — every service, deploy, person and figure on this page is synthetic. Names are sample.*@seed.invalid and image tags are prefixed SEED-. Nothing here happened.

Detectors 31 runs on production

The alerts X-ray ships with — the detectors that open findings for everyone. Their logic is code; their numbers (thresholds, windows, severity cut-offs), how often they run, and whether they run at all are yours to change here, like any alert. A change takes effect on the next run; reset returns to the shipped defaults.

Every setting has a system default: ↺ on a setting, a detector's reset button, or all of them here.
DetectorOpens findings of kindRunsLast runSettings
outliers
Services unlike their peers on 5xx rate, p95, p99 or Neo4j queries per request — one row per service; a warning only on the measures you choose (by default the 5xx rate), informational otherwise.
outlier.service every 5 min
per cluster
never 9 numbers · defaults
edit
budgets
Error budgets burning faster than the objective allows (1 h and 6 h burn rates).
budget.burning every 5 min
per cluster
never 1 number · defaults
edit
shapes
Cypher shapes whose p95 in the second half of a 6 h window is ≥1.5× the first half — and is at least 1 s now. Spikes under a second are ignored as noise, whatever the ratio: 2 ms → 43 ms is 21× and means nothing to anyone.
shape.degrading every 15 min
per cluster
never 7 numbers · defaults
edit
neo4j_errors live
A Neo4j database failing Cypher executions, from Neo4j's own metric (db_query_execution_failure_total, via Managed Prometheus) -- the failures the Neo4j tab charted but nothing raised (2026-10-23). Load and plan speed are anomaly.neo4j and shape.*; this is errors.
neo4j.errors every 5 min
fleet-wide
never 3 numbers · defaults
edit
endpoint_4xx_spike
One finding per endpoint for its 4xx: answering far more 4xx than its own usual rate, or rejecting nearly every call. Permission denied (401/403) is critical; other 4xx a warning. A permissions change (403), broken or expired auth (401), a removed route or contract break (404). endpoint_failing only counts 5xx, and its 4xx signal needs ~every call rejected; a 403 storm on a busy endpoint was invisible (2026-10-23: docsvc folders 96 and onboardingsvc orgs 102 4xx in five minutes, no finding). The usual rate is the endpoint's own average over the last seven days (hourly rollup), so a chronically noisy endpoint is not flagged for being itself.
endpoint.4xx_spike every 5 min
per cluster
never 6 numbers · defaults
edit
endpoint_failing
One endpoint of a service failing most of its calls (5xx) over the last 15 minutes — the unit the observation-window alerts count and resolve per endpoint (15-OBSERVATION-WINDOW-NOTIFICATIONS-RESEARCH.md §3.5). 4xx are shown beside it, not counted as failing: a 404 is the caller's problem, a 500 is ours — but an endpoint that rejects almost EVERY call with a 4xx is its own INFO signal (a broken client or contract), separate from the 5xx alert.
endpoint.failing endpoint.rejecting every 5 min
per cluster
never 6 numbers · defaults
edit
hand_changes
kubectl/Helm/console writes that bypassed ArgoCD in the last 24 h — production only, only in the tilda and neo4j-system namespaces, and never an image change.
change.hand every 5 min
per cluster
never 1 number · defaults
edit
fanout
Services averaging ≥25 foreground Neo4j queries per API request over the last hour.
n1.fanout every 15 min
per cluster
never 3 numbers · defaults
edit
ingest_stale
X-ray's own ingest has not succeeded for this cluster in 45 min (warning) or 3 h (critical).
ingest.stale every 5 min
per cluster
never 2 numbers · defaults
edit
collector_stale
A push collector that has posted before has gone quiet for 3× its cadence.
collector.stale every 15 min
fleet-wide
never 2 numbers · defaults
edit
baseline_refresh
Rebuilds the hour-of-week baselines for the cluster once a day (no findings of its own).
none (maintenance) every 24 h
per cluster
never no numbers to tune
edit
seasonal
Three of the last four complete 5-minute buckets far outside the hour-of-week baseline. A p95 that stays under 1 s is never reported — below a second nobody notices, whatever the deviation.
anomaly.traffic anomaly.errors anomaly.latency anomaly.neo4j every 5 min
per cluster
never 4 numbers · defaults
edit
budget_forecast
The date the 30-day budget runs out, if within 14 days — a constant recent-median scenario (Theil–Sen is only the gate that says the trend is upward at all, not the source of the date).
budget.forecast every 1 h
per cluster
never 2 numbers · defaults
edit
shape_changepoint
For each deploy in the last 6 h, a chi-square test of every busy shape's latency histogram ±60 min around it.
shape.regressed every 15 min
per cluster
never 4 numbers · defaults
edit
shape_churn
New Cypher shapes today vs the median of the prior 14 days — literals inlined into statements show up here.
shape.churn every 1 h
per cluster
never 2 numbers · defaults
edit
disk_forecast live
Theil–Sen on 30 days of volume usage for every Neo4j and Weaviate node; a finding when a watermark is within 30 days. X-ray's own store is NOT covered here — that is obsdb_growth.
forecast.disk every 1 h
fleet-wide
never 1 number · defaults
edit
obsdb_growth
A daily size sample of X-ray's own store — and, on the SQLite PVC, the date it fills 30 GiB (minus a 2 GiB reserve).
forecast.obsdb every 24 h
fleet-wide
never 1 number · defaults
edit
llm_spend_forecast live
Month-end AI spend projection per cluster (BigQuery daily cost, Theil–Sen, weekend-weighted); a finding at >1.2× last month.
llm.spend every 1 h
per cluster
never 2 numbers · defaults
edit
k8s_events
What the cluster itself reports going wrong, from its events (every namespace): pods that cannot start for 10+ minutes (image pull, config error, unschedulable), evictions, kernel OOM kills (including a process inside a running container), Jobs out of retries, ExternalSecrets not syncing, cert-manager or other platform pods down, autoscalers that cannot read their metric, and node problems. Judged on the current state: still being reported, and a normal wait (a pod pending two minutes, then scheduled) is not an incident.
pod.cannot_start pod.evicted pod.oom job.backoff secret.sync_failing cert.unhealthy platform.unhealthy scaler.failing node.unhealthy every 5 min
per cluster
never no numbers to tune
edit
pod_unstable
A workload whose containers restart in place: warning at three restarts in 30 minutes (or one OOM kill); critical once the run has lasted an hour and is still going. Stops being reported after 30 quiet minutes and closes about 15 minutes after that, once the resolve grace passes. Says why — crash loop, OOM, liveness, image — and since when.
pod.unstable every 5 min
per cluster
never 5 numbers · defaults
edit
service_silent
A service that normally serves requests at this hour of the week (its baseline) and has served none for the sustain window. Absence as a signal: a crashed or cut-off service produces no anomaly, only silence.
service.silent every 5 min
per cluster
never 4 numbers · defaults
edit
cost_anomaly
Daily net cost of each project × service × cluster cell against the median/MAD of its previous 28 days; a finding for the last two complete days.
cost.anomaly every 6 h
fleet-wide
never 3 numbers · defaults
edit
cost_target
Every monthly target set on Costs → Forecast against the month-end projection; a finding when the projection is over it.
cost.target every 6 h
fleet-wide
never 2 numbers · defaults
edit
cost_commitment
Commitment fees over the last 30 complete days that funded no usage; a finding per commitment family above the floor.
cost.commitment every 24 h
fleet-wide
never 2 numbers · defaults
edit
custom_watches
Every enabled watch (a condition written in plain English, read into a closed spec) evaluated against the five-minute rollups.
custom.watch every 5 min
per cluster
never no numbers to tune
edit
query_alerts
Every enabled query alert — a request in plain English that the model turned into a query over X-ray's own tables or one of its read-only tools — run and compared with its threshold.
custom.query every 5 min
per cluster
never no numbers to tune
edit
errors_novel live
Error shapes seen for the first time in 90 days, ≥5 times in the last hour (learns silently on day one).
error.novel every 15 min
per cluster
never 2 numbers · defaults
edit
job_failing
Failure lines of named background jobs, clustered into failed runs; a finding when a job keeps failing within the configured window (6 h by default). Severity alone is not the filter — a line carrying an error/exception field or a failure status counts too, because rag-agent.upload logs its failures at WARNING. The complement of errors_novel, which only sees NEW shapes.
job.failing every 1 h
per cluster
never 4 numbers · defaults
edit
weaviate_stalled live
Weaviate tenant queues that are deep and have not drained in the last hour (GMP).
weaviate.stalled every 15 min
fleet-wide
never 1 number · defaults
edit
llm_unattributed live
≥25% of the last 24 h of LLM spend carries no organisation (BigQuery).
llm.unattributed every 1 h
per cluster
never 2 numbers · defaults
edit
openai_cost_drift live
What the LiteLLM log says a cluster spent on OpenAI against what OpenAI billed its project, over the last complete UTC days. Either direction. Each day's pair is kept in openai_reconciliation_daily as the audit trail.
llm.cost_drift every 6 h
per cluster
never 5 numbers · defaults
edit

Two of these evaluate what people wrote themselves: custom_watches (condition alerts) and query_alerts (query alerts). Turning them off silences every alert of that kind.