← findings
execsvc (tilda) image pull failing for at least 13 min — 1 pod
warning
Pod cannot start
test26
resolved
impact 26
opened 21d ago · last observed 1d ago · observed 9× · #8069 · cec3813a64bb1a4e873f
What happened template
Pod cannot start: execsvc (tilda) image pull failing for at least 13 min — 1 pod. Facts: namespace tilda, workload execsvc, why image pull failing, pending_minutes 13, since 1794901864.
A workload's pods have been unable to start for 10+ minutes — image pull failing, a container config error, or no node that can take them — and are still being reported.
The facts
| namespace | tilda |
| workload | execsvc |
| why | image pull failing |
| pods | execsvc-66c7974fcc-5svfx |
| pending_minutes | 13 |
| since | 1,794,901,864 |
| events | 32 |
| last_message | Failed to pull image "us-central1-docker.pkg.dev/tilda-dev-325721/artifacts/services/execsvc:cc27b3c-shashank": rpc error: code = NotFound desc = failed to pull and unpack image "us-central1-docker.pkg.dev/tilda-dev-325721/artifacts/services/execsvc:cc27b3c-shashank": failed to resolve reference "us-central1-docker.pkg.dev/tilda-dev-325721/artifacts/services/execsvc:cc27b3c-shashank": us-central1- |
| discriminator | image pull failing |
The story — No deploy or hand change is recorded in the hour before this opened.
- 2026-10-28 02:02:34 UTCthis finding execsvc (tilda) image pull failing for at least 13 min — 1 pod
- 2026-10-28 03:47:44 UTCfinding · info feedsvc: Neo4j queries per request 162.4 — 22.3× the fleet's usual 7.3
- 2026-10-28 04:49:11 UTCfinding · info feedsvc request rate 25.8 req/min vs a normal 0.0 req/min ± 5.0 req/min for Tue 01:00 UTC (4 of the last 4 checks)
- 2026-10-28 18:05:55 UTCfinding · warning feedsvc shape 00000000: p95 4.85 s now vs 28 ms earlier (172.7×, 3,394 runs)
- 2026-10-29 01:01:22 UTCfinding · info docsvc request rate 28.8 req/min vs a normal 0.0 req/min ± 5.0 req/min for Mon 06:00 UTC (3 of the last 4 checks)
- 2026-11-02 05:09:15 UTCfinding · warning docsvc GET /api/documents/:id: 4xx — 123 of 130 calls (95 %; other 4xx×123) in the last 15 min — it usually has none
- 2026-11-08 09:54:20 UTCfinding · warning regsvc (tilda) image pull failing for at least 116 min — 1 pod
- 2026-11-11 06:35:31 UTCfinding · warning studiosvc GET /api/studio/events: 4xx — 100 of 100 calls (100 %; other 4xx×100) in the last 15 min — it rejects nearly every call
- 2026-11-12 13:28:10 UTCfinding · info ReadOnlyFileSystemDetected: gk3-test26-nap-1tqh1p5r-ab0f2c65-mw2n (1× in 30 min)
- 2026-11-14 10:16:00 UTCdeploy automated deployed studiosvc-infra-dev
- 2026-11-14 10:51:22 UTCdeploy [email protected] deployed studiosvc-infra-dev
- 2026-11-14 11:00:00 UTClatency cluster p95 peaked at ≥ 2.0 min
- 2026-11-14 11:14:54 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/execsvc by hand (kubectl/v1.35.9 (darwin/arm64) kubernetes/d834de3,gzip(gfe))
- 2026-11-14 11:16:20 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/studiosvc by hand (kubectl/v1.35.9 (darwin/arm64) kubernetes/d834de3,gzip(gfe))
- 2026-11-14 11:52:45 UTCdeploy [email protected] deployed studiosvc-infra-dev
- 2026-11-14 14:16:14 UTCdeploy automated deployed studiosvc-infra-dev
- 2026-11-14 14:27:08 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/studiosvc by hand (kubectl/v1.35.9 (darwin/arm64) kubernetes/d834de3,gzip(gfe))
- 2026-11-14 14:27:26 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/execsvc by hand (kubectl/v1.35.9 (darwin/arm64) kubernetes/d834de3,gzip(gfe))
- 2026-11-14 14:30:14 UTCdeploy [email protected] deployed studiosvc-infra-dev
- 2026-11-14 14:45:39 UTCdeploy automated deployed studiosvc-infra-dev
- 2026-11-14 18:24:25 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/onboardingsvc by hand (kubectl/v1.30.2 (darwin/arm64) kubernetes/3968350,gzip(gfe))
- 2026-11-14 20:00:17 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/docsvc by hand (kubectl/v1.36.1 (darwin/arm64) kubernetes/7569396,gzip(gfe))
- 2026-11-14 20:00:18 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/feedsvc by hand (kubectl/v1.36.1 (darwin/arm64) kubernetes/7569396,gzip(gfe))
- 2026-11-14 20:17:22 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/studiosvc by hand (kubectl/v1.35.9 (darwin/arm64) kubernetes/d834de3,gzip(gfe))
- 2026-11-14 20:17:42 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/execsvc by hand (kubectl/v1.35.9 (darwin/arm64) kubernetes/d834de3,gzip(gfe))
- 2026-11-14 23:49:52 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/feedsvc by hand (kubectl/v1.36.1 (darwin/arm64) kubernetes/7569396,gzip(gfe))
- 2026-11-15 02:06:58 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/virtual-assistant by hand (kubectl/v1.36.1 (darwin/arm64) kubernetes/7569396,gzip(gfe))
- 2026-11-15 02:06:59 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/chat-agent by hand (kubectl/v1.36.1 (darwin/arm64) kubernetes/7569396,gzip(gfe))
- 2026-11-15 02:24:26 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/execsvc by hand (kubectl/v1.35.9 (darwin/arm64) kubernetes/d834de3,gzip(gfe))
- 2026-11-15 02:25:02 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/studiosvc by hand (kubectl/v1.35.9 (darwin/arm64) kubernetes/d834de3,gzip(gfe))
- 2026-11-15 05:02:55 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/chat-agent by hand (kubectl/v1.36.1 (darwin/arm64) kubernetes/7569396,gzip(gfe))
- 2026-11-15 05:16:48 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/chat-agent by hand (kubectl/v1.36.1 (darwin/arm64) kubernetes/7569396,gzip(gfe))
- 2026-11-15 06:11:31 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/chat-agent by hand (kubectl/v1.36.1 (darwin/arm64) kubernetes/7569396,gzip(gfe))
- 2026-11-15 06:32:33 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/chat-agent by hand (kubectl/v1.36.1 (darwin/arm64) kubernetes/7569396,gzip(gfe))
- 2026-11-15 15:00:33 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/chat-agent by hand (kubectl/v1.36.1 (darwin/arm64) kubernetes/7569396,gzip(gfe))
- 2026-11-15 15:17:23 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/chat-agent by hand (kubectl/v1.36.1 (darwin/arm64) kubernetes/7569396,gzip(gfe))
- 2026-11-15 15:21:07 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/chat-agent by hand (kubectl/v1.36.1 (darwin/arm64) kubernetes/7569396,gzip(gfe))
- 2026-11-15 15:29:18 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/chat-agent by hand (kubectl/v1.36.1 (darwin/arm64) kubernetes/7569396,gzip(gfe))
- 2026-11-15 15:35:57 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/chat-agent by hand (kubectl/v1.36.1 (darwin/arm64) kubernetes/7569396,gzip(gfe))
- 2026-11-16 02:09:29 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/ontologysvc by hand (kubectl/v1.36.1 (darwin/arm64) kubernetes/7569396,gzip(gfe))
- 2026-11-16 05:02:37 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/ontologysvc by hand (kubectl/v1.36.1 (darwin/arm64) kubernetes/7569396,gzip(gfe))
- 2026-11-16 05:05:00 UTCerrors cluster 5xx peaked at 98 in one five-minute bucket (median 0 in this window)
- 2026-11-16 13:26:38 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/docsvc by hand (kubectl/v1.35.1 (darwin/arm64) kubernetes/8fea90b,gzip(gfe))
- 2026-11-16 13:27:10 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/studiosvc by hand (kubectl/v1.35.9 (darwin/arm64) kubernetes/d834de3,gzip(gfe))
- 2026-11-16 14:00:19 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/onboardingsvc by hand (kubectl/v1.30.2 (darwin/arm64) kubernetes/3968350,gzip(gfe))
- 2026-11-16 14:01:04 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/enrollsvc by hand (kubectl/v1.30.2 (darwin/arm64) kubernetes/3968350,gzip(gfe))
- 2026-11-16 14:01:33 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/docsvc by hand (kubectl/v1.30.2 (darwin/arm64) kubernetes/3968350,gzip(gfe))
- 2026-11-16 17:37:59 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/execsvc by hand (kubectl/v1.30.2 (darwin/arm64) kubernetes/3968350,gzip(gfe))
- 2026-11-16 17:45:31 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/feedsvc by hand (kubectl/v1.36.1 (darwin/arm64) kubernetes/7569396,gzip(gfe))
- 2026-11-16 19:49:16 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/feedsvc by hand (kubectl/v1.36.1 (darwin/arm64) kubernetes/7569396,gzip(gfe))
- 2026-11-17 00:15:40 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/feedsvc by hand (kubectl/v1.36.1 (darwin/arm64) kubernetes/7569396,gzip(gfe))
- 2026-11-17 00:30:51 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/feedsvc by hand (kubectl/v1.36.1 (darwin/arm64) kubernetes/7569396,gzip(gfe))
- 2026-11-17 00:34:22 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/feedsvc by hand (kubectl/v1.36.1 (darwin/arm64) kubernetes/7569396,gzip(gfe))
- 2026-11-17 00:53:22 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/feedsvc by hand (kubectl/v1.36.1 (darwin/arm64) kubernetes/7569396,gzip(gfe))
- 2026-11-17 01:11:14 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/feedsvc by hand (kubectl/v1.36.1 (darwin/arm64) kubernetes/7569396,gzip(gfe))
- 2026-11-17 02:15:09 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/feedsvc by hand (kubectl/v1.36.1 (darwin/arm64) kubernetes/7569396,gzip(gfe))
- 2026-11-17 07:50:22 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/execsvc by hand (kubectl/v1.30.2 (darwin/arm64) kubernetes/3968350,gzip(gfe))
- 2026-11-17 08:04:21 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/execsvc by hand (kubectl/v1.36.1 (darwin/arm64) kubernetes/7569396,gzip(gfe))
History
- 2026-11-17 08:04:09 UTCreopened x-ray · severity=warning was=warning title=execsvc (tilda) image pull failing for at least 13 min — 1 pod level=medium was_level=medium finding={'id': 8069, 'fingerprint': 'cec3813a64bb1a4e873f', 'kind': 'pod.cannot_start', 'severity': 'warning', 'cluster': 'test26', 'subject': 'tilda/execsvc', 'title': 'execsvc (tilda) image pull failing for at least 13 min — 1 pod', 'facts': {'namespace': 'tilda', 'workload': 'execsvc', 'why': 'image pull failing', 'pods': ['execsvc-66c7974fcc-5svfx'], 'pending_minutes': 13, 'since': 1794901864, 'events': 32, 'last_message': 'Failed to pull image "us-central1-docker.pkg.dev/tilda-dev-325721/artifacts/services/execsvc:cc27b3c-shashank": rpc error: code = NotFound desc = failed to pull and unpack image "us-central1-docker.pkg.dev/tilda-dev-325721/artifacts/services/execsvc:cc27b3c-shashank": failed to resolve reference "us-central1-docker.pkg.dev/tilda-dev-325721/artifacts/services/execsvc:cc27b3c-shashank": us-central1-', 'discriminator': 'image pull failing'}, 'evidence': [{'label': 'Kubernetes events', 'href': '/logs?cluster=test26&from=1794899049&to=1794902649&source=events&q=execsvc'}, {'label': 'Pods', 'href': '/pods?cluster=test26'}], 'provenance': {'source': 'x-ray', 'note': 'Kubernetes events (logs/events in BigQuery), last 45 min'}, 'impact': 21.2, 'status': 'open', 'first_seen': 1793152954, 'last_seen': 1794902649, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 9, 'narrative': 'Pod cannot start: execsvc (tilda) image pull failing for at least 92 min — 1 pod. Facts: namespace tilda, workload execsvc, why image pull failing, pending_minutes 92, since 1793151919.', 'narrative_kind': 'template', 'discriminator': 'image pull failing', 'level': 'medium', 'told_level': None, 'episode_at': 1794902649, 'chat_thread': None, 'kind_label': 'Pod cannot start', 'group': 'Pods', 'age_s': 1749724, 'stale_s': 29}
- 2026-11-17 08:26:41 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 8069, 'fingerprint': 'cec3813a64bb1a4e873f', 'kind': 'pod.cannot_start', 'severity': 'warning', 'cluster': 'test26', 'subject': 'tilda/execsvc', 'title': 'execsvc (tilda) image pull failing for at least 13 min — 1 pod', 'facts': {'namespace': 'tilda', 'workload': 'execsvc', 'why': 'image pull failing', 'pods': ['execsvc-66c7974fcc-5svfx'], 'pending_minutes': 13, 'since': 1794901864, 'events': 32, 'last_message': 'Failed to pull image "us-central1-docker.pkg.dev/tilda-dev-325721/artifacts/services/execsvc:cc27b3c-shashank": rpc error: code = NotFound desc = failed to pull and unpack image "us-central1-docker.pkg.dev/tilda-dev-325721/artifacts/services/execsvc:cc27b3c-shashank": failed to resolve reference "us-central1-docker.pkg.dev/tilda-dev-325721/artifacts/services/execsvc:cc27b3c-shashank": us-central1-', 'discriminator': 'image pull failing'}, 'evidence': [{'label': 'Kubernetes events', 'href': '/logs?cluster=test26&from=1794899049&to=1794902649&source=events&q=execsvc'}, {'label': 'Pods', 'href': '/pods?cluster=test26'}], 'provenance': {'source': 'x-ray', 'note': 'Kubernetes events (logs/events in BigQuery), last 45 min'}, 'impact': 26.2, 'status': 'resolved', 'first_seen': 1793152954, 'last_seen': 1794902649, 'resolved_at': 1794904001, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 9, 'narrative': 'Pod cannot start: execsvc (tilda) image pull failing for at least 13 min — 1 pod. Facts: namespace tilda, workload execsvc, why image pull failing, pending_minutes 13, since 1794901864.', 'narrative_kind': 'template', 'discriminator': 'image pull failing', 'level': 'medium', 'told_level': None, 'episode_at': 1794902649, 'chat_thread': None, 'kind_label': 'Pod cannot start', 'group': 'Pods', 'age_s': 1751132, 'stale_s': 1437}
Check it yourself
Every link carries the finding's own time window, so it shows tomorrow what it shows now.
Actions
Sign in (IAP) to acknowledge, snooze or resolve.
Where the numbers come from
| source | x-ray |
| note | Kubernetes events (logs/events in BigQuery), last 45 min |
| detector | k8s_events |