x-ray-weaviate-stats (x-ray) image pull failing for at least 115 min — 1 pod
critical
Pod cannot start
production
resolved
impact 78
opened 11d ago · last observed 2d ago · observed 142× · #16883 · 9efadf270c28e8edc5d7
What happened template
Pod cannot start: x-ray-weaviate-stats (x-ray) image pull failing for at least 115 min — 1 pod. Facts: namespace x-ray, workload x-ray-weaviate-stats, why image pull failing, pending_minutes 115, since 1794884707.
A workload's pods have been unable to start for 10+ minutes — image pull failing, a container config error, or no node that can take them — and are still being reported.
First look — investigated before anyone looked
Cause not clear from logs and recent changes; checked error lines (none about this image), 12 deploys in the 24 h before (all config rev 6db9855a7d34 to other apps, none to x-ray), this finding's history and 2 findings at the same time.
Pod x-ray-weaviate-stats-29914720-w48vb (namespace x-ray) cannot pull us-central1-docker.pkg.dev/tilda-dev-325721/artifacts/services/x-ray:3.7.1-5aab6ea; pending 32 min with 53 events since 02:54 UTC.
This recurs daily: episodes open around 02:50 UTC and clear around 05:20 UTC (Nov 11, 12, 13, 14, 16); 134 occurrences since Nov 8. The schedule points to a CronJob run.
A node.unhealthy finding (NodeNotReady) opened at the same minute but is not linked to this pull.
Next step: read the Kubernetes events for the actual pull error (not found vs. permission denied), then confirm tag 3.7.1-5aab6ea exists in the tilda-dev-325721 registry and the production nodes can read it.
low confidence
claude-opus-5-5 · 2d ago · $0.05 — every link above is evidence it read; nothing else was claimed.
The facts
| namespace | x-ray |
| workload | x-ray-weaviate-stats |
| why | image pull failing |
| pods | x-ray-weaviate-stats-29914720-w48vb |
| pending_minutes | 115 |
| since | 1,794,884,707 |
| events | 2 |
| last_message | Error: ImagePullBackOff |
| discriminator | image pull failing |
The story — No deploy or hand change is recorded in the hour before this opened.
- 2026-11-08 02:56:19 UTCthis finding x-ray-weaviate-stats (x-ray) image pull failing for at least 115 min — 1 pod
- 2026-11-08 06:38:31 UTCfinding · critical virtual-assistant GET /api/v1/va/sse/stream: permission denied — 20 of 22 calls (91 %; 401×20) in the last 15 min — 232× its usual 0.1
- 2026-11-09 14:28:17 UTCfinding · warning studiosvc on production has served nothing for 1.1 h; at this hour it normally serves about 108 calls/hour
- 2026-11-15 05:43:29 UTCfinding · critical ontologysvc: 1× “Traceback (most recent call last): File ?, line ?, in lifespan await receive() File ?, line ?, in receive return await self.receive_queue.ge” in 10 min — Production logs — panic / fatal
- 2026-11-16 00:15:00 UTClatency cluster p95 peaked at 4.55 s
- 2026-11-16 05:10:11 UTCdeploy [email protected] deployed data-monitor
- 2026-11-16 05:10:12 UTCdeploy [email protected] deployed chat-agent
- 2026-11-16 05:10:12 UTCdeploy [email protected] deployed dashrsvc
- 2026-11-16 05:10:12 UTCdeploy [email protected] deployed workplansvc-infra-production
- 2026-11-16 05:10:13 UTCdeploy [email protected] deployed execsvc
- 2026-11-16 05:10:13 UTCdeploy [email protected] deployed dashql
- 2026-11-16 05:10:13 UTCdeploy [email protected] deployed enrollsvc
- 2026-11-16 05:10:14 UTCdeploy [email protected] deployed exportsvc
- 2026-11-16 05:10:14 UTCdeploy [email protected] deployed docsvc
- 2026-11-16 05:10:14 UTCdeploy [email protected] deployed finsvc
- 2026-11-16 05:10:15 UTCdeploy [email protected] deployed notifsvc
- 2026-11-16 05:10:16 UTCdeploy [email protected] deployed onboardingsvc
- 2026-11-16 05:10:17 UTCdeploy [email protected] deployed patientsvc
- 2026-11-16 05:10:19 UTCdeploy [email protected] deployed docproc
- 2026-11-16 05:10:23 UTCdeploy [email protected] deployed ontologysvc
- 2026-11-16 05:10:23 UTCdeploy [email protected] deployed schedsvc
- 2026-11-16 05:10:24 UTCdeploy [email protected] deployed studiosvc
- 2026-11-16 05:10:24 UTCdeploy [email protected] deployed regsvc
- 2026-11-16 05:10:24 UTCdeploy [email protected] deployed tusker
- 2026-11-16 05:10:24 UTCdeploy [email protected] deployed tmfsync
- 2026-11-16 05:10:25 UTCdeploy [email protected] deployed studioaisvc
- 2026-11-16 05:10:25 UTCdeploy [email protected] deployed rag-agent
- 2026-11-16 05:10:25 UTCdeploy [email protected] deployed workplan-agent
- 2026-11-16 05:10:25 UTCdeploy [email protected] deployed virtual-assistant
- 2026-11-16 05:10:25 UTCdeploy [email protected] deployed workplansvc
- 2026-11-16 05:10:51 UTCdeploy [email protected] deployed asm
- 2026-11-16 16:32:01 UTCfinding · critical execsvc: a new error shape, 473× in the last hour, never seen in 90 days (failed to evaluate and raise auto query for participant question — no )
- 2026-11-16 16:32:01 UTCfinding · critical execsvc: a new error shape, 75× in the last hour, never seen in 90 days (failed to evaluate and raise auto query for participant question — eva)
- 2026-11-16 16:32:01 UTCfinding · critical execsvc: a new error shape, 75× in the last hour, never seen in 90 days (evalResult is not a bool)
- 2026-11-16 21:38:50 UTCfinding · warning execsvc.auto-query-reeval: 6 failed runs in the last 6 h — participant activity ?: auto query ? on question ?: no result from branch evaluation for branch, pro
- 2026-11-17 00:20:00 UTCerrors cluster 5xx peaked at 15 in one five-minute bucket (median 0 in this window)
- 2026-11-17 02:54:44 UTCfinding · warning execsvc.auto-query-reeval-sweep: 11 failed runs in the last 6 h — no result from branch evaluation for branch, protocolId=? activityRefId=?
History
- 2026-11-16 02:51:52 UTCreopened x-ray · severity=warning was=critical title=x-ray-weaviate-stats (x-ray) image pull failing for at least 11 min — 1 pod level=medium was_level=high finding={'id': 16883, 'fingerprint': '9efadf270c28e8edc5d7', 'kind': 'pod.cannot_start', 'severity': 'warning', 'cluster': 'production', 'subject': 'x-ray/x-ray-weaviate-stats', 'title': 'x-ray-weaviate-stats (x-ray) image pull failing for at least 11 min — 1 pod', 'facts': {'namespace': 'x-ray', 'workload': 'x-ray-weaviate-stats', 'why': 'image pull failing', 'pods': ['x-ray-weaviate-stats-29913280-wnlv9'], 'pending_minutes': 11, 'since': 1794796801, 'events': 20, 'last_message': 'Error: ImagePullBackOff', 'discriminator': 'image pull failing'}, 'evidence': [{'label': 'Kubernetes events', 'href': '/logs?cluster=production&from=1794793912&to=1794797512&source=events&q=x-ray-weaviate-stats'}, {'label': 'Pods', 'href': '/pods?cluster=production'}], 'provenance': {'source': 'x-ray', 'note': 'Kubernetes events (logs/events in BigQuery), last 45 min'}, 'impact': 42.5, 'status': 'open', 'first_seen': 1794106579, 'last_seen': 1794797512, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 116, 'narrative': 'Pod cannot start: x-ray-weaviate-stats (x-ray) image pull failing for at least 117 min — 1 pod. Facts: namespace x-ray, workload x-ray-weaviate-stats, why image pull failing, pending_minutes 117, since 1794625510.', 'narrative_kind': 'template', 'discriminator': 'image pull failing', 'level': 'medium', 'told_level': 'high', 'episode_at': 1794797512, 'chat_thread': 'xray-alert-10-748fd5a928-9efadf270c28e8edc5d7-1794113635', 'kind_label': 'Pod cannot start', 'group': 'Pods', 'age_s': 690952, 'stale_s': 19}
- 2026-11-16 03:12:35 UTCescalated x-ray · severity=critical was=warning title=x-ray-weaviate-stats (x-ray) image pull failing for at least 32 min — 1 pod level=high was_level=medium finding={'id': 16883, 'fingerprint': '9efadf270c28e8edc5d7', 'kind': 'pod.cannot_start', 'severity': 'critical', 'cluster': 'production', 'subject': 'x-ray/x-ray-weaviate-stats', 'title': 'x-ray-weaviate-stats (x-ray) image pull failing for at least 32 min — 1 pod', 'facts': {'namespace': 'x-ray', 'workload': 'x-ray-weaviate-stats', 'why': 'image pull failing', 'pods': ['x-ray-weaviate-stats-29913280-wnlv9'], 'pending_minutes': 32, 'since': 1794796801, 'events': 53, 'last_message': 'Error: ImagePullBackOff', 'discriminator': 'image pull failing'}, 'evidence': [{'label': 'Kubernetes events', 'href': '/logs?cluster=production&from=1794795155&to=1794798755&source=events&q=x-ray-weaviate-stats'}, {'label': 'Pods', 'href': '/pods?cluster=production'}], 'provenance': {'source': 'x-ray', 'note': 'Kubernetes events (logs/events in BigQuery), last 45 min'}, 'impact': 72.5, 'status': 'open', 'first_seen': 1794106579, 'last_seen': 1794798755, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 118, 'narrative': 'Pod cannot start: x-ray-weaviate-stats (x-ray) image pull failing for at least 19 min — 1 pod. Facts: namespace x-ray, workload x-ray-weaviate-stats, why image pull failing, pending_minutes 19, since 1794796801.', 'narrative_kind': 'template', 'discriminator': 'image pull failing', 'level': 'high', 'told_level': 'medium', 'episode_at': 1794797512, 'chat_thread': 'xray-alert-10-748fd5a928-9efadf270c28e8edc5d7-1794113635', 'kind_label': 'Pod cannot start', 'group': 'Pods', 'age_s': 692214, 'stale_s': 38}
- 2026-11-16 05:18:52 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 16883, 'fingerprint': '9efadf270c28e8edc5d7', 'kind': 'pod.cannot_start', 'severity': 'critical', 'cluster': 'production', 'subject': 'x-ray/x-ray-weaviate-stats', 'title': 'x-ray-weaviate-stats (x-ray) image pull failing for at least 119 min — 1 pod', 'facts': {'namespace': 'x-ray', 'workload': 'x-ray-weaviate-stats', 'why': 'image pull failing', 'pods': ['x-ray-weaviate-stats-29913280-wnlv9'], 'pending_minutes': 119, 'since': 1794798006, 'events': 4, 'last_message': 'Back-off pulling image "us-central1-docker.pkg.dev/tilda-dev-325721/artifacts/services/x-ray:3.7.1-5aab6ea"', 'discriminator': 'image pull failing'}, 'evidence': [{'label': 'Kubernetes events', 'href': '/logs?cluster=production&from=1794801556&to=1794805156&source=events&q=x-ray-weaviate-stats'}, {'label': 'Pods', 'href': '/pods?cluster=production'}], 'provenance': {'source': 'x-ray', 'note': 'Kubernetes events (logs/events in BigQuery), last 45 min'}, 'impact': 77.5, 'status': 'resolved', 'first_seen': 1794106579, 'last_seen': 1794805156, 'resolved_at': 1794806332, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 131, 'narrative': 'Pod cannot start: x-ray-weaviate-stats (x-ray) image pull failing for at least 119 min — 1 pod. Facts: namespace x-ray, workload x-ray-weaviate-stats, why image pull failing, pending_minutes 119, since 1794798006.', 'narrative_kind': 'template', 'discriminator': 'image pull failing', 'level': 'high', 'told_level': 'high', 'episode_at': 1794797512, 'chat_thread': 'xray-alert-10-748fd5a928-9efadf270c28e8edc5d7-1794113635', 'kind_label': 'Pod cannot start', 'group': 'Pods', 'age_s': 699790, 'stale_s': 1213}
- 2026-11-17 02:54:44 UTCreopened x-ray · severity=warning was=critical title=x-ray-weaviate-stats (x-ray) image pull failing for at least 14 min — 1 pod level=medium was_level=high finding={'id': 16883, 'fingerprint': '9efadf270c28e8edc5d7', 'kind': 'pod.cannot_start', 'severity': 'warning', 'cluster': 'production', 'subject': 'x-ray/x-ray-weaviate-stats', 'title': 'x-ray-weaviate-stats (x-ray) image pull failing for at least 14 min — 1 pod', 'facts': {'namespace': 'x-ray', 'workload': 'x-ray-weaviate-stats', 'why': 'image pull failing', 'pods': ['x-ray-weaviate-stats-29914720-w48vb'], 'pending_minutes': 14, 'since': 1794883201, 'events': 43, 'last_message': 'Back-off pulling image "us-central1-docker.pkg.dev/tilda-dev-325721/artifacts/services/x-ray:3.7.1-5aab6ea"', 'discriminator': 'image pull failing'}, 'evidence': [{'label': 'Kubernetes events', 'href': '/logs?cluster=production&from=1794880484&to=1794884084&source=events&q=x-ray-weaviate-stats'}, {'label': 'Pods', 'href': '/pods?cluster=production'}], 'provenance': {'source': 'x-ray', 'note': 'Kubernetes events (logs/events in BigQuery), last 45 min'}, 'impact': 42.5, 'status': 'open', 'first_seen': 1794106579, 'last_seen': 1794884084, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 132, 'narrative': 'Pod cannot start: x-ray-weaviate-stats (x-ray) image pull failing for at least 119 min — 1 pod. Facts: namespace x-ray, workload x-ray-weaviate-stats, why image pull failing, pending_minutes 119, since 1794798006.', 'narrative_kind': 'template', 'discriminator': 'image pull failing', 'level': 'medium', 'told_level': 'high', 'episode_at': 1794884084, 'chat_thread': 'xray-alert-10-748fd5a928-9efadf270c28e8edc5d7-1794113635', 'kind_label': 'Pod cannot start', 'group': 'Pods', 'age_s': 777531, 'stale_s': 26}
- 2026-11-17 03:12:58 UTCescalated x-ray · severity=critical was=warning title=x-ray-weaviate-stats (x-ray) image pull failing for at least 32 min — 1 pod level=high was_level=medium finding={'id': 16883, 'fingerprint': '9efadf270c28e8edc5d7', 'kind': 'pod.cannot_start', 'severity': 'critical', 'cluster': 'production', 'subject': 'x-ray/x-ray-weaviate-stats', 'title': 'x-ray-weaviate-stats (x-ray) image pull failing for at least 32 min — 1 pod', 'facts': {'namespace': 'x-ray', 'workload': 'x-ray-weaviate-stats', 'why': 'image pull failing', 'pods': ['x-ray-weaviate-stats-29914720-w48vb'], 'pending_minutes': 32, 'since': 1794883201, 'events': 53, 'last_message': 'Back-off pulling image "us-central1-docker.pkg.dev/tilda-dev-325721/artifacts/services/x-ray:3.7.1-5aab6ea"', 'discriminator': 'image pull failing'}, 'evidence': [{'label': 'Kubernetes events', 'href': '/logs?cluster=production&from=1794881578&to=1794885178&source=events&q=x-ray-weaviate-stats'}, {'label': 'Pods', 'href': '/pods?cluster=production'}], 'provenance': {'source': 'x-ray', 'note': 'Kubernetes events (logs/events in BigQuery), last 45 min'}, 'impact': 72.5, 'status': 'open', 'first_seen': 1794106579, 'last_seen': 1794885178, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 134, 'narrative': 'Pod cannot start: x-ray-weaviate-stats (x-ray) image pull failing for at least 23 min — 1 pod. Facts: namespace x-ray, workload x-ray-weaviate-stats, why image pull failing, pending_minutes 23, since 1794883201.', 'narrative_kind': 'template', 'discriminator': 'image pull failing', 'level': 'high', 'told_level': 'high', 'episode_at': 1794884084, 'chat_thread': 'xray-alert-10-748fd5a928-9efadf270c28e8edc5d7-1794113635', 'kind_label': 'Pod cannot start', 'group': 'Pods', 'age_s': 778621, 'stale_s': 22}
- 2026-11-17 05:17:45 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 16883, 'fingerprint': '9efadf270c28e8edc5d7', 'kind': 'pod.cannot_start', 'severity': 'critical', 'cluster': 'production', 'subject': 'x-ray/x-ray-weaviate-stats', 'title': 'x-ray-weaviate-stats (x-ray) image pull failing for at least 115 min — 1 pod', 'facts': {'namespace': 'x-ray', 'workload': 'x-ray-weaviate-stats', 'why': 'image pull failing', 'pods': ['x-ray-weaviate-stats-29914720-w48vb'], 'pending_minutes': 115, 'since': 1794884707, 'events': 2, 'last_message': 'Error: ImagePullBackOff', 'discriminator': 'image pull failing'}, 'evidence': [{'label': 'Kubernetes events', 'href': '/logs?cluster=production&from=1794888047&to=1794891647&source=events&q=x-ray-weaviate-stats'}, {'label': 'Pods', 'href': '/pods?cluster=production'}], 'provenance': {'source': 'x-ray', 'note': 'Kubernetes events (logs/events in BigQuery), last 45 min'}, 'impact': 77.5, 'status': 'resolved', 'first_seen': 1794106579, 'last_seen': 1794891647, 'resolved_at': 1794892665, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 142, 'narrative': 'Pod cannot start: x-ray-weaviate-stats (x-ray) image pull failing for at least 115 min — 1 pod. Facts: namespace x-ray, workload x-ray-weaviate-stats, why image pull failing, pending_minutes 115, since 1794884707.', 'narrative_kind': 'template', 'discriminator': 'image pull failing', 'level': 'high', 'told_level': 'high', 'episode_at': 1794884084, 'chat_thread': 'xray-alert-10-748fd5a928-9efadf270c28e8edc5d7-1794113635', 'kind_label': 'Pod cannot start', 'group': 'Pods', 'age_s': 786105, 'stale_s': 1037}
Check it yourself
Every link carries the finding's own time window, so it shows tomorrow what it shows now.
Actions
Sign in (IAP) to acknowledge, snooze or resolve.
Where the numbers come from
| source | x-ray |
| note | Kubernetes events (logs/events in BigQuery), last 45 min |
| detector | k8s_events |