Xray
daily rollups 1h 6h 24h 48h 3d 5d 7d 14d 30d ‹
Last 7 days▾
Ends now. Anything up to this cluster's 30-day retention.
Start and end times
Cancel
Interpreted in UTC (change it at the right of this bar). Earliest available: 2026-10-19 19:08:27.
› →|
updated 1d ago
Sample data. This database was generated by scripts/seed_dev_db.py — every service, deploy, person and figure on this page is synthetic. Names are sample.*@seed.invalid and image tags are prefixed SEED-. Nothing here happened.

Noise on master, last 7 days

How loud each level is against what it is allowed to be. Nobody reads a channel that cries wolf. ?Replayed from this cluster's finding events exactly as the level rules apply them: each event's level at its own moment, a reopen within a day at no higher level is the same incident, an escalation inside a level is not news. critical and high that land within 30 minutes of each other are one incident — one message.

Dry-run. Messages are composed and logged under Deliveries; nothing external is sent unless a channel was armed on its own.
critical
0.0/week
0 in 7 d · budget ≤ 1/week
high
23.0/week
23 in 7 d · budget ≤ 5/week
endpoint.4xx_spike 6
neo4j.errors 5
job.failing 4
pod.cannot_start 2
endpoint.failing 2
anomaly.errors 2
pod.unstable 1
error.novel 1
medium
41.0/week
41 in 7 d · budget grouped
error.novel 23
job.failing 4
scaler.failing 3
pod.cannot_start 2
neo4j.errors 2
budget.burning 2
node.unhealthy 1
endpoint.4xx_spike 1
low
30.0/week
30 in 7 d · budget digest
anomaly.neo4j 10
anomaly.traffic 7
anomaly.latency 6
n1.fanout 2
pod.unstable 2
outlier.service 1
shape.regressed 1
node.unhealthy 1

critical + high as messages (30-minute correlation): 19.0/week · medium as hourly messages: 20 in 7 d · damped as the same incident: 118

critical + high incidents 19

high1d ago pubsub.training-assignment-master-trainingsub: 22 failed runs in the last 6 h — ReconcileTrainingAssignments failed for training ?: failed to fetch ActivityV2 ? for training ?: obj
high1d ago x-ray-argocd-collector (x-ray) image pull failing for at least 35 min — 1 pod
high1d ago feedsvc GET /studies/:id/feed/news: permission denied — 44 of 47 calls (94 %; 403×44) in the last 15 min — it usually has none
high1d ago execsvc: a new error shape, 700× in the last hour, never seen in 90 days (could not match participant questionnaire/question/table/doc-upload to) + 3 more in the same half hour
high2d ago workplan.summarization-sweep: 28 failed runs in the last 6 h — checkpoint scoring: unknown checkpoint source type ?
high2d ago Neo4j shared Neo4j, database master: 35 Cypher queries failed in the last 15 min
high2d ago studiosvc GET /api/studio/events: permission denied — 21 of 49 calls (43 %; 401×21) in the last 15 min — 28× its usual 0.8
high2d ago onboardingsvc GET /api/myprofile/visibility/v2: permission denied — 36 of 75 calls (48 %; 401×36) in the last 15 min — 17× its usual 2.1
high2d ago docsvc POST /api/documents/folders: 8 of 8 calls failed (100 %) in the last 15 min + 1 more in the same half hour
high2d ago Neo4j shared Neo4j, database test26: 662 Cypher queries failed in the last 15 min
high2d ago docproc.process: 23 failed runs in the last 6 h — Traceback (most recent call last): File ?, line ?, in _refresh self._do_password_grant() File ?, lin
high3d ago workplansvc on master has restarted 14 times in 15 min — crash loop, image suresh-eng8690-8981379926
high3d ago feedsvc GET /studies/:id/feed: permission denied — 22 of 25 calls (88 %; 403×21, other 4xx×1) in the last 15 min — 8× its usual 2.6
high3d ago x-ray-weaviate-stats (x-ray) image pull failing for at least 39 min — 1 pod
high3d ago Neo4j shared Neo4j, database automation: 30 Cypher queries failed in the last 15 min
high3d ago Neo4j shared Neo4j, database matrix_a: 31 Cypher queries failed in the last 15 min
high4d ago Neo4j shared Neo4j, database matrix_b: 45 Cypher queries failed in the last 15 min
high4d ago onboardingsvc GET /api/myprofile/visibility/v2/study/:id: permission denied — 23 of 47 calls (49 %; 401×23) in the last 15 min — 73× its usual 0.3
high4d ago pubsub.training-assignment-master-trainingsub: 27 failed runs in the last 6 h — ReconcileTrainingAssignments failed for training ?: failed to fetch ActivityV2 ? for training ?: obj

What each rule sent

Rulemessagesby status