Detectors 31
The programs that open findings, and whether each one has actually run. A detector that has silently died is invisible everywhere else — the Findings page just looks calm. ?Each detector runs on its own cadence inside the scheduler (every 5 min the due ones run). Stored detectors read SQLite only; live ones call the log backend, Managed Prometheus or BigQuery and are gated by OBS_DETECT_LIVE. A run that raised is recorded with its error and resolves nothing.
detection on
live sources on
| Detector | Opens | Scope | Cadence | Last run | Passes (48 h) | Found | Errors |
|---|---|---|---|---|---|---|---|
| outliers Services unlike their peers on 5xx rate, p95, p99 or Neo4j queries per request — one row per service; a warning only on the measures you choose (by default the 5xx rate), informational otherwise. |
outlier.service | cluster | 5 min | never | 0 | 0 | none |
| budgets Error budgets burning faster than the objective allows (1 h and 6 h burn rates). |
budget.burning | cluster | 5 min | never | 0 | 0 | none |
| shapes Cypher shapes whose p95 in the second half of a 6 h window is ≥1.5× the first half — and is at least 1 s now. Spikes under a second are ignored as noise, whatever the ratio: 2 ms → 43 ms is 21× and means nothing to anyone. |
shape.degrading | cluster | 15 min | never | 0 | 0 | none |
| neo4j_errors live A Neo4j database failing Cypher executions, from Neo4j's own metric (db_query_execution_failure_total, via Managed Prometheus) -- the failures the Neo4j tab charted but nothing raised (2026-10-23). Load and plan speed are anomaly.neo4j and shape.*; this is errors. |
neo4j.errors | fleet | 5 min | never | 0 | 0 | none |
| endpoint_4xx_spike One finding per endpoint for its 4xx: answering far more 4xx than its own usual rate, or rejecting nearly every call. Permission denied (401/403) is critical; other 4xx a warning. A permissions change (403), broken or expired auth (401), a removed route or contract break (404). endpoint_failing only counts 5xx, and its 4xx signal needs ~every call rejected; a 403 storm on a busy endpoint was invisible (2026-10-23: docsvc folders 96 and onboardingsvc orgs 102 4xx in five minutes, no finding). The usual rate is the endpoint's own average over the last seven days (hourly rollup), so a chronically noisy endpoint is not flagged for being itself. |
endpoint.4xx_spike | cluster | 5 min | never | 0 | 0 | none |
| endpoint_failing One endpoint of a service failing most of its calls (5xx) over the last 15 minutes — the unit the observation-window alerts count and resolve per endpoint (15-OBSERVATION-WINDOW-NOTIFICATIONS-RESEARCH.md §3.5). 4xx are shown beside it, not counted as failing: a 404 is the caller's problem, a 500 is ours — but an endpoint that rejects almost EVERY call with a 4xx is its own INFO signal (a broken client or contract), separate from the 5xx alert. |
endpoint.failing, endpoint.rejecting | cluster | 5 min | never | 0 | 0 | none |
| hand_changes kubectl/Helm/console writes that bypassed ArgoCD in the last 24 h — production only, only in the tilda and neo4j-system namespaces, and never an image change. |
change.hand | cluster | 5 min | never | 0 | 0 | none |
| fanout Services averaging ≥25 foreground Neo4j queries per API request over the last hour. |
n1.fanout | cluster | 15 min | never | 0 | 0 | none |
| ingest_stale X-ray's own ingest has not succeeded for this cluster in 45 min (warning) or 3 h (critical). |
ingest.stale | cluster | 5 min | never | 0 | 0 | none |
| collector_stale A push collector that has posted before has gone quiet for 3× its cadence. |
collector.stale | fleet | 15 min | never | 0 | 0 | none |
| baseline_refresh Rebuilds the hour-of-week baselines for the cluster once a day (no findings of its own). |
— | cluster | 1 d | never | 0 | 0 | none |
| seasonal Three of the last four complete 5-minute buckets far outside the hour-of-week baseline. A p95 that stays under 1 s is never reported — below a second nobody notices, whatever the deviation. |
anomaly.traffic, anomaly.errors, anomaly.latency, anomaly.neo4j | cluster | 5 min | never | 0 | 0 | none |
| budget_forecast The date the 30-day budget runs out, if within 14 days — a constant recent-median scenario (Theil–Sen is only the gate that says the trend is upward at all, not the source of the date). |
budget.forecast | cluster | 1 h | never | 0 | 0 | none |
| shape_changepoint For each deploy in the last 6 h, a chi-square test of every busy shape's latency histogram ±60 min around it. |
shape.regressed | cluster | 15 min | never | 0 | 0 | none |
| shape_churn New Cypher shapes today vs the median of the prior 14 days — literals inlined into statements show up here. |
shape.churn | cluster | 1 h | never | 0 | 0 | none |
| disk_forecast live Theil–Sen on 30 days of volume usage for every Neo4j and Weaviate node; a finding when a watermark is within 30 days. X-ray's own store is NOT covered here — that is obsdb_growth. |
forecast.disk | fleet | 1 h | never | 0 | 0 | none |
| obsdb_growth A daily size sample of X-ray's own store — and, on the SQLite PVC, the date it fills 30 GiB (minus a 2 GiB reserve). |
forecast.obsdb | fleet | 1 d | never | 0 | 0 | none |
| llm_spend_forecast live Month-end AI spend projection per cluster (BigQuery daily cost, Theil–Sen, weekend-weighted); a finding at >1.2× last month. |
llm.spend | cluster | 1 h | never | 0 | 0 | none |
| k8s_events What the cluster itself reports going wrong, from its events (every namespace): pods that cannot start for 10+ minutes (image pull, config error, unschedulable), evictions, kernel OOM kills (including a process inside a running container), Jobs out of retries, ExternalSecrets not syncing, cert-manager or other platform pods down, autoscalers that cannot read their metric, and node problems. Judged on the current state: still being reported, and a normal wait (a pod pending two minutes, then scheduled) is not an incident. |
pod.cannot_start, pod.evicted, pod.oom, job.backoff, secret.sync_failing, cert.unhealthy, platform.unhealthy, scaler.failing, node.unhealthy | cluster | 5 min | never | 0 | 0 | none |
| pod_unstable A workload whose containers restart in place: warning at three restarts in 30 minutes (or one OOM kill); critical once the run has lasted an hour and is still going. Stops being reported after 30 quiet minutes and closes about 15 minutes after that, once the resolve grace passes. Says why — crash loop, OOM, liveness, image — and since when. |
pod.unstable | cluster | 5 min | never | 0 | 0 | none |
| service_silent A service that normally serves requests at this hour of the week (its baseline) and has served none for the sustain window. Absence as a signal: a crashed or cut-off service produces no anomaly, only silence. |
service.silent | cluster | 5 min | never | 0 | 0 | none |
| cost_anomaly Daily net cost of each project × service × cluster cell against the median/MAD of its previous 28 days; a finding for the last two complete days. |
cost.anomaly | fleet | 6 h | never | 0 | 0 | none |
| cost_target Every monthly target set on Costs → Forecast against the month-end projection; a finding when the projection is over it. |
cost.target | fleet | 6 h | never | 0 | 0 | none |
| cost_commitment Commitment fees over the last 30 complete days that funded no usage; a finding per commitment family above the floor. |
cost.commitment | fleet | 1 d | never | 0 | 0 | none |
| custom_watches Every enabled watch (a condition written in plain English, read into a closed spec) evaluated against the five-minute rollups. |
custom.watch | cluster | 5 min | never | 0 | 0 | none |
| query_alerts Every enabled query alert — a request in plain English that the model turned into a query over X-ray's own tables or one of its read-only tools — run and compared with its threshold. |
custom.query | cluster | 5 min | never | 0 | 0 | none |
| errors_novel live Error shapes seen for the first time in 90 days, ≥5 times in the last hour (learns silently on day one). |
error.novel | cluster | 15 min | never | 0 | 0 | none |
| job_failing Failure lines of named background jobs, clustered into failed runs; a finding when a job keeps failing within the configured window (6 h by default). Severity alone is not the filter — a line carrying an error/exception field or a failure status counts too, because rag-agent.upload logs its failures at WARNING. The complement of errors_novel, which only sees NEW shapes. |
job.failing | cluster | 1 h | never | 0 | 0 | none |
| weaviate_stalled live Weaviate tenant queues that are deep and have not drained in the last hour (GMP). |
weaviate.stalled | fleet | 15 min | never | 0 | 0 | none |
| llm_unattributed live ≥25% of the last 24 h of LLM spend carries no organisation (BigQuery). |
llm.unattributed | cluster | 1 h | never | 0 | 0 | none |
| openai_cost_drift live What the LiteLLM log says a cluster spent on OpenAI against what OpenAI billed its project, over the last complete UTC days. Either direction. Each day's pair is kept in openai_reconciliation_daily as the audit trail. |
llm.cost_drift | cluster | 6 h | never | 0 | 0 | none |
Baselines hour-of-week profiles per cluster, rebuilt daily
master
10591
28 services · 2–5 weeks · 2d ago
automation
1476
12 services · 2–6 weeks · 2d ago
nexus
681
13 services · 2–5 weeks · 2d ago
production
299
23 services · 2–5 weeks · 2d ago
delta
36
6 services · 2–5 weeks · 2d ago
test27
6438
29 services · 2–7 weeks · 2d ago
test26
479
25 services · 2–7 weeks · 2d ago
Recent passes
| When | Detector | Scope | ms | Found | Error |
|---|---|---|---|---|---|
| 2026-11-16 07:08:43 UTC | outliers | master | 115 | 4 | |
| 2026-11-16 06:57:06 UTC | budgets | master | 705 | 0 | |
| 2026-11-16 06:57:06 UTC | budgets | automation | 1355 | 0 | |
| 2026-11-16 06:57:06 UTC | budgets | nexus | 92 | 0 | |
| 2026-11-16 06:57:06 UTC | budgets | production | 475 | 0 | |
| 2026-11-16 06:57:06 UTC | budgets | delta | 71 | 0 | |
| 2026-11-16 06:57:06 UTC | budgets | test27 | 320 | 0 | |
| 2026-11-16 06:57:06 UTC | budgets | test26 | 209 | 0 | |
| 2026-11-16 06:57:06 UTC | collector_stale | * | 69 | 0 | |
| 2026-11-16 06:57:06 UTC | custom_watches | master | 5 | 0 | |
| 2026-11-16 06:57:06 UTC | custom_watches | automation | 6 | 0 | |
| 2026-11-16 06:57:06 UTC | endpoint_4xx_spike | master | 17 | 0 | |
| 2026-11-16 06:57:06 UTC | endpoint_4xx_spike | automation | 60 | 1 | |
| 2026-11-16 06:57:06 UTC | endpoint_4xx_spike | nexus | 15 | 0 | |
| 2026-11-16 06:57:06 UTC | endpoint_4xx_spike | production | 18 | 0 | |
| 2026-11-16 06:57:06 UTC | endpoint_4xx_spike | delta | 17 | 0 | |
| 2026-11-16 06:57:06 UTC | endpoint_4xx_spike | test27 | 53 | 1 | |
| 2026-11-16 06:57:06 UTC | endpoint_4xx_spike | test26 | 16 | 0 | |
| 2026-11-16 06:57:06 UTC | endpoint_failing | master | 25 | 0 | |
| 2026-11-16 06:57:06 UTC | endpoint_failing | automation | 25 | 0 | |
| 2026-11-16 06:57:06 UTC | endpoint_failing | nexus | 23 | 0 | |
| 2026-11-16 06:57:06 UTC | endpoint_failing | production | 29 | 0 | |
| 2026-11-16 06:57:06 UTC | endpoint_failing | delta | 29 | 0 | |
| 2026-11-16 06:57:06 UTC | endpoint_failing | test27 | 32 | 0 | |
| 2026-11-16 06:57:06 UTC | endpoint_failing | test26 | 26 | 0 | |
| 2026-11-16 06:57:06 UTC | fanout | master | 49 | 1 | |
| 2026-11-16 06:57:06 UTC | fanout | automation | 23 | 0 | |
| 2026-11-16 06:57:06 UTC | fanout | nexus | 19 | 0 | |
| 2026-11-16 06:57:06 UTC | fanout | production | 21 | 0 | |
| 2026-11-16 06:57:06 UTC | fanout | delta | 22 | 0 | |
| 2026-11-16 06:57:06 UTC | fanout | test27 | 46 | 1 | |
| 2026-11-16 06:57:06 UTC | fanout | test26 | 39 | 1 | |
| 2026-11-16 06:57:06 UTC | hand_changes | master | 12 | 0 | |
| 2026-11-16 06:57:06 UTC | hand_changes | automation | 9 | 0 | |
| 2026-11-16 06:57:06 UTC | hand_changes | nexus | 8 | 0 | |
| 2026-11-16 06:57:06 UTC | hand_changes | production | 9 | 0 | |
| 2026-11-16 06:57:06 UTC | hand_changes | delta | 9 | 0 | |
| 2026-11-16 06:57:06 UTC | hand_changes | test27 | 13 | 0 | |
| 2026-11-16 06:57:06 UTC | hand_changes | test26 | 9 | 0 | |
| 2026-11-16 06:57:06 UTC | ingest_stale | master | 9 | 0 | |
| 2026-11-16 06:57:06 UTC | ingest_stale | automation | 9 | 0 | |
| 2026-11-16 06:57:06 UTC | ingest_stale | nexus | 10 | 0 | |
| 2026-11-16 06:57:06 UTC | ingest_stale | production | 10 | 0 | |
| 2026-11-16 06:57:06 UTC | ingest_stale | delta | 12 | 0 | |
| 2026-11-16 06:57:06 UTC | ingest_stale | test27 | 11 | 0 | |
| 2026-11-16 06:57:06 UTC | ingest_stale | test26 | 10 | 0 | |
| 2026-11-16 06:57:06 UTC | k8s_events | master | 880 | 2 | |
| 2026-11-16 06:57:06 UTC | k8s_events | automation | 1027 | 2 | |
| 2026-11-16 06:57:06 UTC | k8s_events | nexus | 828 | 3 | |
| 2026-11-16 06:57:06 UTC | k8s_events | production | 1030 | 2 | |
| 2026-11-16 06:57:06 UTC | k8s_events | delta | 1005 | 4 | |
| 2026-11-16 06:57:06 UTC | k8s_events | test27 | 890 | 3 | |
| 2026-11-16 06:57:06 UTC | k8s_events | test26 | 893 | 4 | |
| 2026-11-16 06:57:06 UTC | neo4j_errors | * | 1102 | 2 | |
| 2026-11-16 06:57:06 UTC | outliers | master | 131 | 4 | |
| 2026-11-16 06:57:06 UTC | outliers | automation | 122 | 2 | |
| 2026-11-16 06:57:06 UTC | outliers | nexus | 15 | 0 | |
| 2026-11-16 06:57:06 UTC | outliers | production | 44 | 1 | |
| 2026-11-16 06:57:06 UTC | outliers | delta | 17 | 0 | |
| 2026-11-16 06:57:06 UTC | outliers | test27 | 87 | 2 |