← findings
finsvc (tilda) image pull failing for at least 119 min — 1 pod
warning
Pod cannot start
test26
open
impact 38
opened 3d ago · last observed 1d ago · observed 322× · #17312 · 21cd8dd15617909e4e16
What happened template
Pod cannot start: finsvc (tilda) image pull failing for at least 119 min — 1 pod. Facts: namespace tilda, workload finsvc, why image pull failing, pending_minutes 119, since 1794904858.
A workload's pods have been unable to start for 10+ minutes — image pull failing, a container config error, or no node that can take them — and are still being reported.
The facts
| namespace | tilda |
| workload | finsvc |
| why | image pull failing |
| pods | finsvc-7dc78f6585-5btdz |
| pending_minutes | 119 |
| since | 1,794,904,858 |
| events | 43 |
| last_message | Back-off pulling image "us-central1-docker.pkg.dev/tilda-dev-325721/artifacts/services/finsvc:satish-ledger-1006" |
| severity_capped | critical → warning: test26 is a test-tier cluster |
| discriminator | image pull failing |
The story — 1 change landed in the hour before this opened — adjacency, not proof.
- 2026-11-15 06:28:58 UTCcypher execsvc shape 00000000 p95 went 33 ms → 124 ms across the window (14,505 runs)
- 2026-11-15 06:32:33 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/chat-agent by hand (kubectl/v1.36.1 (darwin/arm64) kubernetes/7569396,gzip(gfe))
- 2026-11-15 07:17:13 UTCfinding · info ReadOnlyFileSystemDetected: gk3-test26-nap-1tqh1p5r-518ae407-zdm7 (1× in 30 min)
- 2026-11-15 07:28:58 UTCthis finding finsvc (tilda) image pull failing for at least 119 min — 1 pod
- 2026-11-15 08:21:26 UTCfinding · info ReadOnlyFileSystemDetected: gk3-test26-nap-1tqh1p5r-ab0f2c65-rhuj (1× in 30 min)
- 2026-11-15 08:32:27 UTCfinding · warning rag-agent (tilda) image pull failing for at least 119 min — 1 pod
- 2026-11-15 09:24:04 UTCfinding · info ReadOnlyFileSystemDetected: gk3-test26-nap-1tqh1p5r-ab0f2c65-8i4p (1× in 30 min)
- 2026-11-15 15:00:33 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/chat-agent by hand (kubectl/v1.36.1 (darwin/arm64) kubernetes/7569396,gzip(gfe))
- 2026-11-15 15:17:23 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/chat-agent by hand (kubectl/v1.36.1 (darwin/arm64) kubernetes/7569396,gzip(gfe))
- 2026-11-15 15:21:07 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/chat-agent by hand (kubectl/v1.36.1 (darwin/arm64) kubernetes/7569396,gzip(gfe))
- 2026-11-15 15:29:18 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/chat-agent by hand (kubectl/v1.36.1 (darwin/arm64) kubernetes/7569396,gzip(gfe))
- 2026-11-15 15:35:57 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/chat-agent by hand (kubectl/v1.36.1 (darwin/arm64) kubernetes/7569396,gzip(gfe))
- 2026-11-16 00:11:50 UTCfinding · warning virtual-assistant: a new error shape, 25× in the last hour, never seen in 90 days (HTTP request failed: Login failed: status=? error=too_many_attempts er)
- 2026-11-16 00:11:50 UTCfinding · warning virtual-assistant: a new error shape, 12× in the last hour, never seen in 90 days (Token refresh failed: Login failed: status=? error=too_many_attempts e)
- 2026-11-16 00:50:21 UTCfinding · warning docproc: a new error shape, 8× in the last hour, never seen in 90 days (ServiceTokenManager refresh failed (? Client Error: Too Many Requests )
- 2026-11-16 01:09:28 UTCfinding · warning docproc: a new error shape, 5× in the last hour, never seen in 90 days (Failed to create document processing AIRun — aborting pipeline (refuse)
- 2026-11-16 02:09:29 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/ontologysvc by hand (kubectl/v1.36.1 (darwin/arm64) kubernetes/7569396,gzip(gfe))
- 2026-11-16 05:02:37 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/ontologysvc by hand (kubectl/v1.36.1 (darwin/arm64) kubernetes/7569396,gzip(gfe))
- 2026-11-16 05:05:00 UTCerrors cluster 5xx peaked at 98 in one five-minute bucket (median 0 in this window)
- 2026-11-16 13:26:38 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/docsvc by hand (kubectl/v1.35.1 (darwin/arm64) kubernetes/8fea90b,gzip(gfe))
- 2026-11-16 13:27:10 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/studiosvc by hand (kubectl/v1.35.9 (darwin/arm64) kubernetes/d834de3,gzip(gfe))
- 2026-11-16 14:00:19 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/onboardingsvc by hand (kubectl/v1.30.2 (darwin/arm64) kubernetes/3968350,gzip(gfe))
- 2026-11-16 14:01:04 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/enrollsvc by hand (kubectl/v1.30.2 (darwin/arm64) kubernetes/3968350,gzip(gfe))
- 2026-11-16 14:01:33 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/docsvc by hand (kubectl/v1.30.2 (darwin/arm64) kubernetes/3968350,gzip(gfe))
- 2026-11-16 16:05:00 UTClatency cluster p95 peaked at 8.51 s
- 2026-11-16 17:37:59 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/execsvc by hand (kubectl/v1.30.2 (darwin/arm64) kubernetes/3968350,gzip(gfe))
- 2026-11-16 17:45:31 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/feedsvc by hand (kubectl/v1.36.1 (darwin/arm64) kubernetes/7569396,gzip(gfe))
- 2026-11-16 19:49:16 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/feedsvc by hand (kubectl/v1.36.1 (darwin/arm64) kubernetes/7569396,gzip(gfe))
- 2026-11-17 00:15:40 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/feedsvc by hand (kubectl/v1.36.1 (darwin/arm64) kubernetes/7569396,gzip(gfe))
- 2026-11-17 00:30:51 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/feedsvc by hand (kubectl/v1.36.1 (darwin/arm64) kubernetes/7569396,gzip(gfe))
- 2026-11-17 00:34:22 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/feedsvc by hand (kubectl/v1.36.1 (darwin/arm64) kubernetes/7569396,gzip(gfe))
- 2026-11-17 00:53:22 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/feedsvc by hand (kubectl/v1.36.1 (darwin/arm64) kubernetes/7569396,gzip(gfe))
- 2026-11-17 01:11:14 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/feedsvc by hand (kubectl/v1.36.1 (darwin/arm64) kubernetes/7569396,gzip(gfe))
- 2026-11-17 02:15:09 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/feedsvc by hand (kubectl/v1.36.1 (darwin/arm64) kubernetes/7569396,gzip(gfe))
- 2026-11-17 07:50:22 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/execsvc by hand (kubectl/v1.30.2 (darwin/arm64) kubernetes/3968350,gzip(gfe))
- 2026-11-17 08:04:21 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/execsvc by hand (kubectl/v1.36.1 (darwin/arm64) kubernetes/7569396,gzip(gfe))
- 2026-11-17 09:12:33 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/onboardingsvc by hand (kubectl/v1.30.2 (darwin/arm64) kubernetes/3968350,gzip(gfe))
History
- 2026-11-15 07:28:58 UTCopened x-ray · severity=warning title=finsvc (tilda) image pull failing for at least 13 min — 1 pod finding={'id': 17312, 'fingerprint': '21cd8dd15617909e4e16', 'kind': 'pod.cannot_start', 'severity': 'warning', 'cluster': 'test26', 'subject': 'tilda/finsvc', 'title': 'finsvc (tilda) image pull failing for at least 13 min — 1 pod', 'facts': {'namespace': 'tilda', 'workload': 'finsvc', 'why': 'image pull failing', 'pods': ['finsvc-7dc78f6585-5btdz'], 'pending_minutes': 13, 'since': 1794726947, 'events': 32, 'last_message': 'Error: ImagePullBackOff', 'discriminator': 'image pull failing'}, 'evidence': [{'label': 'Kubernetes events', 'href': '/logs?cluster=test26&from=1794724138&to=1794727738&source=events&q=finsvc'}, {'label': 'Pods', 'href': '/pods?cluster=test26'}], 'provenance': {'source': 'x-ray', 'note': 'Kubernetes events (logs/events in BigQuery), last 45 min'}, 'impact': 21.2, 'status': 'open', 'first_seen': 1794727738, 'last_seen': 1794727738, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 1, 'narrative': None, 'narrative_kind': None, 'discriminator': 'image pull failing', 'level': 'medium', 'told_level': None, 'episode_at': 1794727738, 'chat_thread': None, 'kind_label': 'Pod cannot start', 'group': 'Pods', 'age_s': 21, 'stale_s': 21}
- 2026-11-15 09:32:05 UTCescalated x-ray · severity=warning was=warning title=finsvc (tilda) image pull failing for at least 116 min — 1 pod level=high was_level=medium finding={'id': 17312, 'fingerprint': '21cd8dd15617909e4e16', 'kind': 'pod.cannot_start', 'severity': 'warning', 'cluster': 'test26', 'subject': 'tilda/finsvc', 'title': 'finsvc (tilda) image pull failing for at least 116 min — 1 pod', 'facts': {'namespace': 'tilda', 'workload': 'finsvc', 'why': 'image pull failing', 'pods': ['finsvc-7dc78f6585-5btdz'], 'pending_minutes': 116, 'since': 1794728112, 'events': 44, 'last_message': 'Error: ImagePullBackOff', 'severity_capped': 'critical → warning: test26 is a test-tier cluster', 'discriminator': 'image pull failing'}, 'evidence': [{'label': 'Kubernetes events', 'href': '/logs?cluster=test26&from=1794731525&to=1794735125&source=events&q=finsvc'}, {'label': 'Pods', 'href': '/pods?cluster=test26'}], 'provenance': {'source': 'x-ray', 'note': 'Kubernetes events (logs/events in BigQuery), last 45 min'}, 'impact': 32.5, 'status': 'open', 'first_seen': 1794727738, 'last_seen': 1794735125, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 15, 'narrative': 'Pod cannot start: finsvc (tilda) image pull failing for at least 118 min — 1 pod. Facts: namespace tilda, workload finsvc, why image pull failing, pending_minutes 118, since 1794727511.', 'narrative_kind': 'template', 'discriminator': 'image pull failing', 'level': 'high', 'told_level': None, 'episode_at': 1794727738, 'chat_thread': None, 'kind_label': 'Pod cannot start', 'group': 'Pods', 'age_s': 7406, 'stale_s': 19}
Check it yourself
Every link carries the finding's own time window, so it shows tomorrow what it shows now.
Actions
Sign in (IAP) to acknowledge, snooze or resolve.
Where the numbers come from
| source | x-ray |
| note | Kubernetes events (logs/events in BigQuery), last 45 min |
| detector | k8s_events |