Home test27
What needs you first, on the cluster you selected — arranged the way you left it. ?Every card draws on data X-ray already has: findings, the headline numbers, time series with their normal band, forecasts, notifications, deploys, error budgets, tests, evals, your saved views. Cards follow the selected cluster and time range unless you pin a cluster in a card's settings. The layout is saved for you alone; nobody else's Home changes.
Broken right now · test27
virtual-assistant · new error: HTTP request failed: Login failed: status=429 error=too_many_attempts error_des… · 53× in the last hour, never seen in 90 days
virtual-assistant · new error: Error fetching document info: Login failed: status=429 error=too_many_attempts … · 27× in the last hour, never seen in 90 days
virtual-assistant · new error: Token refresh failed: Login failed: status=429 error=too_many_attempts error_de… · 26× in the last hour, never seen in 90 days
studiosvc · new error: error on pubsub receive — rpc error: code = NotFound desc = Resource not found … · 13,006× in the last hour, never seen in 90 days
Journeys not judged: request data for test27 stops at Tue 17 Nov 10:30.
Deployed in the last 2 hours · ArgoCD syncs
Nothing deployed to test27 in the last 2 hours.
Since you last looked
Open findings, ranked
11 warning or worse on test27 · 14 informational kept off the board · all findings
rag-agent (tilda) image pull failing for at least 118 min — 1 pod
Pod cannot start: rag-agent (tilda) image pull failing for at least 118 min — 1 pod. Facts: namespace tilda, workload rag-agent, why image pull failing, pending_minutes 118, since 1794904947.
regsvc (tilda) image pull failing for at least 117 min — 1 pod
Pod cannot start: regsvc (tilda) image pull failing for at least 117 min — 1 pod. Facts: namespace tilda, workload regsvc, why image pull failing, pending_minutes 117, since 1794904982.
studiosvc: a new error shape, ≥13006× in the last hour, never seen in 90 days (error on pubsub receive — rpc error: code = NotFound desc = Resource n)
studiosvc logged “error on pubsub receive — rpc error: code = NotFound desc = Resource not found (resource=study-site-binding-test27-studiosvcsub).” 13,006 times in the last hour and never before in 90 days on test27 (fi…
virtual-assistant: a new error shape, 53× in the last hour, never seen in 90 days (HTTP request failed: Login failed: status=? error=too_many_attempts er)
virtual-assistant logged “HTTP request failed: Login failed: status=? error=too_many_attempts error_description=Your account has been blocked after multiple consecutive login attempts. W” 53 times in the last hour and ne…
authzd shape 00000000: p95 1.25 s now vs 10 ms earlier (126.8×, 274 runs)
A authzd query shape on test27 has p95 1.2 s in the second half of the window against 10 ms in the first (126.8×, 274 executions). The shape reads: “MATCH(a:Organization{deltimestamp:$a_deltimestamp,alias:$a_alias}) RETU…
studioai.unknown: 5 failed runs in the last 6 h — Traceback (most recent call last): File ?, line ?, in urlopen response = self._make_request( ^^^^^^^
The background job studioai.unknown on test27 failed 5 runs recently (532 failure lines; lines within one run count once), detected via its ERROR-severity log lines. A job that logs its failures below ERROR severity — a …
pubsub.onboarding-test27-regsvc-sub: 19 failed runs in the last 6 h — failed to create triggers: failed to fetch existing triggers: failed to fetch triggers: Error from A
The background job pubsub.onboarding-test27-regsvc-sub on test27 failed 19 runs recently (528 failure lines; lines within one run count once), detected via its ERROR-severity log lines. A job that logs its failures below…
docproc: a new error shape, 10× in the last hour, never seen in 90 days (Pipeline failed - full context)
docproc logged “Pipeline failed - full context” 10 times in the last hour and never before in 90 days on test27 (first seen 2026-11-16 22:35 UTC, raised at orchestration/document_orchestrator.py:5044). Example trace: a41…
Fleet at a glance
Number = findings warning or worse. Colour = worst open finding. Grey = never ingested. Click a cluster to show it here.
What is running
| detectors | on · last pass never |
| notifications | dry-run (outbox only) · alerts |
| Ask X-ray | ready |
| master ingest | 21h ago |
| automation ingest | 21h ago |
| nexus ingest | 21h ago |
| production ingest | 21h ago |
| delta ingest | 21h ago |
| test27 ingest | 21h ago |
| test26 ingest | 21h ago |
Ask X-ray
- Why is tilda/rag-agent showing 'pod cannot start' on test27 right now?
- Why is tilda/regsvc showing 'pod cannot start' on test27 right now?
- Why is studiosvc showing 'error never seen before' on test27 right now?
- What changed on test27 in the last 6 hours?
- Which service on test27 is closest to exhausting its error budget?