Xray
daily rollups 1h 6h 24h 48h 3d 5d 7d 14d 30d ‹
Last 7 days▾
Ends now. Anything up to this cluster's 30-day retention.
Start and end times
Cancel
Interpreted in UTC (change it at the right of this bar). Earliest available: 2026-10-19 18:33:11.
› →|
updated 1d ago
Sample data. This database was generated by scripts/seed_dev_db.py — every service, deploy, person and figure on this page is synthetic. Names are sample.*@seed.invalid and image tags are prefixed SEED-. Nothing here happened.

Home master

What needs you first, on the cluster you selected — arranged the way you left it. ?Every card draws on data X-ray already has: findings, the headline numbers, time series with their normal band, forecasts, notifications, deploys, error budgets, tests, evals, your saved views. Cards follow the selected cluster and time range unless you pin a cluster in a card's settings. The layout is saved for you alone; nobody else's Home changes.

sign in to arrange this page

Broken right now · master

critical Error never seen before Owner: Madhava Mukkamala (backup: Shashank Agrawal) since Mon 16 Nov 16:55

execsvc · new error: failed to create document copy — could not add document context: could not writ… · 5× in the last hour, never seen in 90 days

started 75 min after deploy 29620eb70 by autosync

First look: Cause not clear from logs and recent changes; execsvc document copies failed on Neo4j DeadlockDetected in one burst at 16:49 UTC (5x here at doclib/copyDocument.go:74 via context.go:335 scope binders…

critical Error never seen before Owner: Madhava Mukkamala (backup: Shashank Agrawal) since Mon 16 Nov 16:55

execsvc · new error: failed to run cypher in transaction — Neo4jError: Neo.TransientError.Transactio… · 5× in the last hour, never seen in 90 days

started 75 min after deploy 29620eb70 by autosync

First look: Cause not clear from logs and recent changes; execsvc hit a burst of Neo4j DeadlockDetected errors, all at 16:49 UTC: 5× 'failed to run cypher in transaction' (tildadb/util.go:1004), plus 14× util.go…

Journeys not judged: request data for master stops at Tue 17 Nov 10:25.

Deployed in the last 2 hours · ArgoCD syncs

Nothing deployed to master in the last 2 hours.

Since you last looked

0 opened 0 escalated 0 resolved 0 reopened 164 open now · 21 critical · 67 info since 1d ago · across the fleet

Open findings, ranked

97 warning or worse across the fleet · 67 informational kept off the board · all findings

critical Neo4j queries failing master last seen 1d ago · since 25d ago · 366×

Neo4j shared Neo4j, database automation: 37 Cypher queries failed in the last 15 min

In the last 15 minutes, 37 Cypher queries failed in the shared Neo4j database (automation) for project tilda-trial-dev. Environment: shared. Database: automation. Failed queries: 37. Time window: 15 minutes.

First look low confidence cause not clear from logs and recent changes · read it →

critical Neo4j queries failing master last seen 1d ago · since 14d ago · 213×

Neo4j shared Neo4j, database matrix_b: 35 Cypher queries failed in the last 15 min

In the last 15 minutes, 35 Cypher queries failed on the shared Neo4j database matrix_b. Environment: shared. Database: matrix_b. Failed queries: 35. Time window: 15 minutes. Project: tilda-trial-dev.

First look low confidence cause not clear from logs and recent changes · read it →

critical Neo4j queries failing master last seen 1d ago · since 22d ago · 365×

Neo4j shared Neo4j, database test26: 36 Cypher queries failed in the last 15 min

Neo4j queries failing: Neo4j shared Neo4j, database test26: 36 Cypher queries failed in the last 15 min. Facts: env shared, database test26, failed 36, window_min 15, project tilda-trial-dev.

First look low confidence cause not clear from logs and recent changes · read it →

critical 4xx spike (403/401/404…) master last seen 1d ago · since 1d ago · 3×

feedsvc GET /studies/:id/feed/news: permission denied — 166 of 174 calls (95 %; 403×166) in the last 15 min — 1033× its usual 0.2

4xx spike (403/401/404…): feedsvc GET /studies/:id/feed/news: permission denied — 166 of 174 calls (95 %; 403×166) in the last 15 min — 1033× its usual 0.2. Facts: endpoint GET /studies/:id/feed/news, service feedsvc, ca…

First look low confidence cause not clear from logs and recent changes · read it →

critical Error never seen before delta Owner: Madhava Mukkamala (backup: Shashank Agrawal) last seen 1d ago · since 1d ago · 3×

execsvc: a new error shape, 9× in the last hour, never seen in 90 days (Request completed — ? · could not save Careplan activity V2)

execsvc logged “Request completed — ? · could not save Careplan activity V2” 9 times in the last hour and never before in 90 days on delta (first seen 2026-11-17 09:46 UTC, raised at observability/middleware.go:281). Exa…

First look low confidence cause not clear from logs and recent changes · read it →

critical Pod cannot start master last seen 1d ago · since 1d ago · 14×

x-ray-argocd-collector (x-ray) image pull failing for at least 93 min — 2 pods

Pod cannot start: x-ray-argocd-collector (x-ray) image pull failing for at least 93 min — 2 pods. Facts: namespace x-ray, workload x-ray-argocd-collector, why image pull failing, pending_minutes 93, since 1794906422.

First look medium confidence most likely x-ray-services-master sync 556a243c8ac9 (autosync) · read it →

critical Background job failing master last seen 1d ago · since 26d ago · 545×

workplan.summarization-sweep: 30 failed runs in the last 6 h — checkpoint scoring: unknown checkpoint source type ?

The background job workplan.summarization-sweep on master failed 30 runs recently (30 failure lines; lines within one run count once), detected via its structured error/exception field (below ERROR severity). A job that …

First look low confidence cause not clear from logs and recent changes · read it →

Fleet at a glance

What is running

detectorson · last pass never
notificationsdry-run (outbox only) · alerts
Ask X-rayready
master ingest1d ago
automation ingest1d ago
nexus ingest1d ago
production ingest1d ago
delta ingest1d ago
test27 ingest1d ago
test26 ingest1d ago