x-ray-argocd-collector (x-ray) image pull failing for at least 93 min — 2 pods
critical
Pod cannot start
master
open
impact 62
opened 1d ago · last observed 20h ago · observed 14× · #17510 · a089fae4aa8b198aa1c4
What happened template
Pod cannot start: x-ray-argocd-collector (x-ray) image pull failing for at least 93 min — 2 pods. Facts: namespace x-ray, workload x-ray-argocd-collector, why image pull failing, pending_minutes 93, since 1794906422.
A workload's pods have been unable to start for 10+ minutes — image pull failing, a container config error, or no node that can take them — and are still being reported.
First look — investigated before anyone looked
Pod x-ray-argocd-collector-29914927-d557w (namespace x-ray) cannot pull image us-central1-docker.pkg.dev/tilda-dev-325721/artifacts/services/x-ray:3.7.1-56fc8bc: 51 events, Back-off pulling image, pending 35 min.
Pending since about 06:07 UTC; the finding opened at 06:29 UTC.
Likely cause: ArgoCD autosync of x-ray-services-master at 06:02 UTC, config revision 556a243c8ac9, about 5 min before the pod got stuck. It is the only x-ray change in the 24 h before.
The bundle does not show the image tag that sync set, so the link rests on timing and the matching app.
Next step: check that commit 556a243c8ac9 sets tag 3.7.1-56fc8bc and that this tag exists in Artifact Registry. If the tag is missing, push the image or revert the commit.
medium confidence config x-ray-services-master sync 556a243c8ac9 (autosync)
claude-opus-5-5 · 1d ago · $0.05 — every link above is evidence it read; nothing else was claimed.
The facts
| namespace | x-ray |
| workload | x-ray-argocd-collector |
| why | image pull failing |
| pods | x-ray-argocd-collector-29915107-rqnsj, x-ray-argocd-collector-29915167-zbmpl |
| pending_minutes | 93 |
| since | 1,794,906,422 |
| events | 104 |
| last_message | Error: ImagePullBackOff |
| discriminator | image pull failing |
The story — 2 changes landed in the hour before this opened — adjacency, not proof.
- 2026-11-17 05:43:34 UTCfinding · critical feedsvc GET /studies/:id/feed/news: permission denied — 166 of 174 calls (95 %; 403×166) in the last 15 min — 1033× its usual 0.2
- 2026-11-17 06:02:36 UTCdeploy automated deployed x-ray-services-master
- 2026-11-17 06:17:25 UTCdeploy automated deployed swaggersvc
- 2026-11-17 06:20:00 UTClatency cluster p95 peaked at ≥ 2.0 min
- 2026-11-17 06:29:15 UTCthis finding x-ray-argocd-collector (x-ray) image pull failing for at least 93 min — 2 pods
- 2026-11-17 07:05:31 UTCdeploy automated deployed studiosvc
- 2026-11-17 08:30:00 UTCerrors cluster 5xx peaked at 6 in one five-minute bucket (median 0 in this window)
- 2026-11-17 09:07:28 UTCdeploy automated deployed patientsvc
- 2026-11-17 09:07:36 UTCdeploy automated deployed dashql
- 2026-11-17 09:07:38 UTCdeploy automated deployed enrollsvc
- 2026-11-17 09:07:39 UTCdeploy automated deployed schedsvc
- 2026-11-17 09:07:41 UTCdeploy automated deployed tusker
- 2026-11-17 09:07:50 UTCdeploy automated deployed dashrsvc
- 2026-11-17 09:07:54 UTCdeploy automated deployed execsvc
- 2026-11-17 09:07:57 UTCdeploy automated deployed workplansvc
- 2026-11-17 09:07:58 UTCdeploy automated deployed onboardingsvc
- 2026-11-17 09:09:40 UTCdeploy automated deployed notifsvc
- 2026-11-17 09:09:49 UTCdeploy automated deployed studiosvc
- 2026-11-17 09:09:57 UTCdeploy automated deployed regsvc
- 2026-11-17 09:09:58 UTCdeploy automated deployed docsvc
- 2026-11-17 09:10:10 UTCdeploy automated deployed phase-2-master
- 2026-11-17 09:11:18 UTCdeploy automated deployed finsvc
- 2026-11-17 09:23:45 UTCdeploy automated deployed dashrsvc
- 2026-11-17 09:23:49 UTCdeploy automated deployed dashql
- 2026-11-17 09:23:51 UTCdeploy automated deployed workplansvc
- 2026-11-17 09:23:53 UTCdeploy automated deployed schedsvc
- 2026-11-17 09:23:56 UTCdeploy automated deployed docsvc
- 2026-11-17 09:25:23 UTCdeploy automated deployed enrollsvc
- 2026-11-17 09:25:42 UTCdeploy automated deployed onboardingsvc
- 2026-11-17 09:25:43 UTCdeploy automated deployed studiosvc
- 2026-11-17 09:25:44 UTCdeploy automated deployed tusker
- 2026-11-17 09:25:48 UTCdeploy automated deployed regsvc
- 2026-11-17 09:25:49 UTCdeploy automated deployed patientsvc
- 2026-11-17 09:25:52 UTCdeploy automated deployed execsvc
- 2026-11-17 09:25:53 UTCdeploy automated deployed finsvc
- 2026-11-17 09:27:09 UTCdeploy automated deployed notifsvc
History
- 2026-11-17 06:29:15 UTCopened x-ray · severity=warning title=x-ray-argocd-collector (x-ray) image pull failing for at least 22 min — 1 pod finding={'id': 17510, 'fingerprint': 'a089fae4aa8b198aa1c4', 'kind': 'pod.cannot_start', 'severity': 'warning', 'cluster': 'master', 'subject': 'x-ray/x-ray-argocd-collector', 'title': 'x-ray-argocd-collector (x-ray) image pull failing for at least 22 min — 1 pod', 'facts': {'namespace': 'x-ray', 'workload': 'x-ray-argocd-collector', 'why': 'image pull failing', 'pods': ['x-ray-argocd-collector-29914927-d557w'], 'pending_minutes': 22, 'since': 1794895625, 'events': 47, 'last_message': 'Back-off pulling image "us-central1-docker.pkg.dev/tilda-dev-325721/artifacts/services/x-ray:3.7.1-56fc8bc"', 'discriminator': 'image pull failing'}, 'evidence': [{'label': 'Kubernetes events', 'href': '/logs?cluster=master&from=1794893355&to=1794896955&source=events&q=x-ray-argocd-collector'}, {'label': 'Pods', 'href': '/pods?cluster=master'}], 'provenance': {'source': 'x-ray', 'note': 'Kubernetes events (logs/events in BigQuery), last 45 min'}, 'impact': 34.0, 'status': 'open', 'first_seen': 1794896955, 'last_seen': 1794896955, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 1, 'narrative': None, 'narrative_kind': None, 'discriminator': 'image pull failing', 'level': 'medium', 'told_level': None, 'episode_at': 1794896955, 'chat_thread': None, 'kind_label': 'Pod cannot start', 'group': 'Pods', 'age_s': 19, 'stale_s': 19}
- 2026-11-17 06:42:35 UTCescalated x-ray · severity=critical was=warning title=x-ray-argocd-collector (x-ray) image pull failing for at least 35 min — 1 pod level=high was_level=medium finding={'id': 17510, 'fingerprint': 'a089fae4aa8b198aa1c4', 'kind': 'pod.cannot_start', 'severity': 'critical', 'cluster': 'master', 'subject': 'x-ray/x-ray-argocd-collector', 'title': 'x-ray-argocd-collector (x-ray) image pull failing for at least 35 min — 1 pod', 'facts': {'namespace': 'x-ray', 'workload': 'x-ray-argocd-collector', 'why': 'image pull failing', 'pods': ['x-ray-argocd-collector-29914927-d557w'], 'pending_minutes': 35, 'since': 1794895625, 'events': 51, 'last_message': 'Back-off pulling image "us-central1-docker.pkg.dev/tilda-dev-325721/artifacts/services/x-ray:3.7.1-56fc8bc"', 'discriminator': 'image pull failing'}, 'evidence': [{'label': 'Kubernetes events', 'href': '/logs?cluster=master&from=1794894155&to=1794897755&source=events&q=x-ray-argocd-collector'}, {'label': 'Pods', 'href': '/pods?cluster=master'}], 'provenance': {'source': 'x-ray', 'note': 'Kubernetes events (logs/events in BigQuery), last 45 min'}, 'impact': 58.0, 'status': 'open', 'first_seen': 1794896955, 'last_seen': 1794897755, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 2, 'narrative': 'The pod x-ray-argocd-collector in namespace x-ray cannot start due to image pull failures lasting at least 22 minutes. One pod (x-ray-argocd-collector-29914927-d557w) is affected. The system is attempting to pull image "us-central1-docker.pkg.dev/tilda-dev-325721/artifacts/services/x-ray:3.7.1-56fc8bc" and backing off. Since timestamp: 1794895625.', 'narrative_kind': 'ai', 'discriminator': 'image pull failing', 'level': 'high', 'told_level': None, 'episode_at': 1794896955, 'chat_thread': None, 'kind_label': 'Pod cannot start', 'group': 'Pods', 'age_s': 819, 'stale_s': 19}
Check it yourself
Every link carries the finding's own time window, so it shows tomorrow what it shows now.
Actions
Sign in (IAP) to acknowledge, snooze or resolve.
Where the numbers come from
| source | x-ray |
| note | Kubernetes events (logs/events in BigQuery), last 45 min |
| detector | k8s_events |