Noise on master, last 7 days
How loud each level is against what it is allowed to be. Nobody reads a channel that cries wolf. ?Replayed from this cluster's finding events exactly as the level rules apply them: each event's level at its own moment, a reopen within a day at no higher level is the same incident, an escalation inside a level is not news. critical and high that land within 30 minutes of each other are one incident — one message.
critical
0.0/week
0 in 7 d · budget ≤ 1/week
high
23.0/week
23 in 7 d · budget ≤ 5/week
endpoint.4xx_spike 6
neo4j.errors 5
job.failing 4
pod.cannot_start 2
endpoint.failing 2
anomaly.errors 2
pod.unstable 1
error.novel 1
medium
41.0/week
41 in 7 d · budget grouped
error.novel 23
job.failing 4
scaler.failing 3
pod.cannot_start 2
neo4j.errors 2
budget.burning 2
node.unhealthy 1
endpoint.4xx_spike 1
low
30.0/week
30 in 7 d · budget digest
anomaly.neo4j 10
anomaly.traffic 7
anomaly.latency 6
n1.fanout 2
pod.unstable 2
outlier.service 1
shape.regressed 1
node.unhealthy 1
critical + high as messages (30-minute correlation): 19.0/week · medium as hourly messages: 20 in 7 d · damped as the same incident: 118
critical + high incidents 19
| high | 1d ago | pubsub.training-assignment-master-trainingsub: 22 failed runs in the last 6 h — ReconcileTrainingAssignments failed for training ?: failed to fetch ActivityV2 ? for training ?: obj |
| high | 1d ago | x-ray-argocd-collector (x-ray) image pull failing for at least 35 min — 1 pod |
| high | 1d ago | feedsvc GET /studies/:id/feed/news: permission denied — 44 of 47 calls (94 %; 403×44) in the last 15 min — it usually has none |
| high | 1d ago | execsvc: a new error shape, 700× in the last hour, never seen in 90 days (could not match participant questionnaire/question/table/doc-upload to) + 3 more in the same half hour |
| high | 2d ago | workplan.summarization-sweep: 28 failed runs in the last 6 h — checkpoint scoring: unknown checkpoint source type ? |
| high | 2d ago | Neo4j shared Neo4j, database master: 35 Cypher queries failed in the last 15 min |
| high | 2d ago | studiosvc GET /api/studio/events: permission denied — 21 of 49 calls (43 %; 401×21) in the last 15 min — 28× its usual 0.8 |
| high | 2d ago | onboardingsvc GET /api/myprofile/visibility/v2: permission denied — 36 of 75 calls (48 %; 401×36) in the last 15 min — 17× its usual 2.1 |
| high | 2d ago | docsvc POST /api/documents/folders: 8 of 8 calls failed (100 %) in the last 15 min + 1 more in the same half hour |
| high | 2d ago | Neo4j shared Neo4j, database test26: 662 Cypher queries failed in the last 15 min |
| high | 2d ago | docproc.process: 23 failed runs in the last 6 h — Traceback (most recent call last): File ?, line ?, in _refresh self._do_password_grant() File ?, lin |
| high | 3d ago | workplansvc on master has restarted 14 times in 15 min — crash loop, image suresh-eng8690-8981379926 |
| high | 3d ago | feedsvc GET /studies/:id/feed: permission denied — 22 of 25 calls (88 %; 403×21, other 4xx×1) in the last 15 min — 8× its usual 2.6 |
| high | 3d ago | x-ray-weaviate-stats (x-ray) image pull failing for at least 39 min — 1 pod |
| high | 3d ago | Neo4j shared Neo4j, database automation: 30 Cypher queries failed in the last 15 min |
| high | 4d ago | Neo4j shared Neo4j, database matrix_a: 31 Cypher queries failed in the last 15 min |
| high | 4d ago | Neo4j shared Neo4j, database matrix_b: 45 Cypher queries failed in the last 15 min |
| high | 4d ago | onboardingsvc GET /api/myprofile/visibility/v2/study/:id: permission denied — 23 of 47 calls (49 %; 401×23) in the last 15 min — 73× its usual 0.3 |
| high | 4d ago | pubsub.training-assignment-master-trainingsub: 27 failed runs in the last 6 h — ReconcileTrainingAssignments failed for training ?: failed to fetch ActivityV2 ? for training ?: obj |
What each rule sent
| Rule | messages | by status |
|---|