Xray
daily rollups 1h 6h 24h 48h 3d 5d 7d 14d 30d ‹
Max · 30 days▾
Ends now. Anything up to this cluster's 30-day retention.
Start and end times
Cancel
Interpreted in UTC (change it at the right of this bar). Earliest available: 2026-10-19 20:52:09.
› →|
updated 1d ago
Sample data. This database was generated by scripts/seed_dev_db.py — every service, deploy, person and figure on this page is synthetic. Names are sample.*@seed.invalid and image tags are prefixed SEED-. Nothing here happened.

Incidents 0 happening now on test27

What is wrong right now, by what it means for people. ?critical: users are affected now, or data is at risk — about zero a week. high: degraded or getting worse — fewer than five a week. medium: worth a look today, grouped. low: context, in the daily digest. An incident is judged on its current state and closes itself when it recovers; one that comes back within a day at no higher level is the same incident.

Dry-run. Messages are composed and logged under Deliveries; nothing external is sent unless a channel was armed on its own.
0 now 0 critical 0 high 0 medium 0 low
Nothing happening now on test27. 25 older findings still open below.
Open, but not observed for an hour 25 nothing has resolved them yet; they send nothing new unless they come back at a higher level
LevelIncidentWhatForSeenGoes to
medium regsvc (tilda) image pull failing for at least 117 min — 1 pod Pod cannot start 4d 1d ago in-app / digest only
medium pubsub.onboarding-test27-regsvc-sub: 19 failed runs in the last 6 h — failed to create triggers: failed to fetch existing triggers: failed to fetch triggers: Error from A Background job failing 16d 1d ago in-app / digest only
medium studioai.unknown: 5 failed runs in the last 6 h — Traceback (most recent call last): File ?, line ?, in urlopen response = self._make_request( ^^^^^^^ Background job failing 22d 1d ago in-app / digest only
medium rag-agent (tilda) image pull failing for at least 118 min — 1 pod Pod cannot start 24d 1d ago in-app / digest only
medium test27: LiteLLM's OpenAI cost is $2.03 (5.0%) over OpenAI's bill over 7 days (2026-11-10 to 2026-11-16) LLM cost off the provider's bill 37d 1d ago in-app / digest only
medium docproc: a new error shape, 10× in the last hour, never seen in 90 days (Pipeline failed - full context) Error never seen before 1d 1d ago in-app / digest only
medium virtual-assistant: a new error shape, 26× in the last hour, never seen in 90 days (Token refresh failed: Login failed: status=? error=too_many_attempts e) Error never seen before 2d 2d ago in-app / digest only
medium virtual-assistant: a new error shape, 27× in the last hour, never seen in 90 days (Error fetching document info: Login failed: status=? error=too_many_at) Error never seen before 2d 2d ago in-app / digest only
medium virtual-assistant: a new error shape, 53× in the last hour, never seen in 90 days (HTTP request failed: Login failed: status=? error=too_many_attempts er) Error never seen before 2d 2d ago in-app / digest only
medium studiosvc: a new error shape, ≥13006× in the last hour, never seen in 90 days (error on pubsub receive — rpc error: code = NotFound desc = Resource n) Error never seen before 4d 4d ago in-app / digest only
low docsvc: Neo4j queries per request 107.5 — 18.6× the fleet's usual 5.8 (also p95 latency (ms) 2.77 s) Service outlier 26d 1d ago in-app / digest only
low docproc: Neo4j queries per request 53.3 — 9.2× the fleet's usual 5.8 Service outlier 26d 1d ago in-app / digest only
low studiosvc Neo4j load 4,216 q/min vs a normal 0 q/min ± 20 q/min for Tue 10:00 UTC (4 of the last 4 checks) Neo4j load anomaly 70d 1d ago in-app / digest only
low studiosvc: Neo4j queries per request 161.0 — 27.8× the fleet's usual 5.8 (also p95 latency (ms) 9.24 s; p99 latency (ms) 18.30 s) Service outlier 83d 1d ago in-app / digest only
low studiosvc runs 270 foreground Neo4j queries per request (180,895 queries / 670 calls, last hour) N+1 fan-out 83d 1d ago in-app / digest only
low test27 (whole cluster) Neo4j load 4,441 q/min vs a normal 239 q/min ± 184 q/min for Tue 10:00 UTC (4 of the last 4 checks) Neo4j load anomaly 84d 1d ago in-app / digest only
low test27 (whole cluster) request rate 38.8 req/min vs a normal 20.5 req/min ± 0.7 req/min for Tue 10:00 UTC (4 of the last 4 checks) Traffic anomaly 84d 1d ago in-app / digest only
low enrollsvc: Neo4j queries per request 42.0 — 7.1× the fleet's usual 5.9 (also p95 latency (ms) 1.98 s) Service outlier 26d 1d ago in-app / digest only
low execsvc: p95 latency (ms) 3.05 s — 9.7× the fleet's usual 315 ms Service outlier 26d 1d ago in-app / digest only
low workplansvc: Neo4j queries per request 969.5 — 235.1× the fleet's usual 4.1 Service outlier 84d 1d ago in-app / digest only
low authzd shape 00000000: p95 1.25 s now vs 10 ms earlier (126.8×, 274 runs) Query shape degrading 1d 1d ago in-app / digest only
low chat-agent: p99 latency (ms) 42.90 s — 12.5× the fleet's usual 3.42 s (also p95 latency (ms) 9.61 s) Service outlier 76d 1d ago in-app / digest only
low virtual-assistant: Neo4j queries per request 47.5 — 22.2× the fleet's usual 2.1 (also p95 latency (ms) 5.58 s) Service outlier 84d 2d ago in-app / digest only
low ontologysvc: p95 latency (ms) 2.94 s — 8.1× the fleet's usual 364 ms (also p99 latency (ms) 4.59 s) Service outlier 3d 3d ago in-app / digest only
low patientsvc: p95 latency (ms) 9.09 s — 42.8× the fleet's usual 212 ms Service outlier 4d 4d ago in-app / digest only