workplansvc on master has restarted 15 times in 10 min — crash loop, image suresh-eng8690-8981379926
critical
Pod keeps restarting
master
resolved
impact 65
opened 3d ago · last observed 3d ago · observed 4× · #17317 · b807db4266bc4a7340a1
What happened template
workplansvc in tilda on master has restarted in place 15 times since 2026-11-15 13:40 UTC (8 in the last 30 minutes, 15 in 24 hours), most recently at 2026-11-15 13:50 UTC. Kubernetes' account of it: 27 crash events, 17 sched events. Last event: BackOff — “Back-off restarting failed container istio-proxy in pod workplansvc-6787748cc8-nvgzf_tilda(911901ce-cd4e-43ab-ab4f-d47978721f92)”. It is running image suresh-eng8690-8981379926. Containers restarting: istio-proxy ×8, workplansvc ×7. The workload carried 42.9% of the cluster's API calls in the last day. Restarts are counted from the container restart counter, so a rollout or a spot preemption does not count; a crash does. It resolves on its own after thirty quiet minutes.
A workload's containers restart in place over and over — crash loop, OOM kills, liveness kills or image pulls failing.
First look — investigated before anyone looked
workplansvc (ns tilda) on master is crash-looping: 14 restarts in 15 min on 1 pod (workplansvc 7, istio-proxy 7), image suresh-eng8690-8981379926.
Started 13:40 UTC. From 13:46 workplansvc/main.go:64 exits on 'observability init failed': otel resource detection cannot reach the GCE metadata server (dial tcp 169.254.169.254, from observability/observability.go:133), 6x each, never seen in 90 days.
The istio-proxy sidecar is also in BackOff. A node went NotReady and cilium logged foreign-IP errors at 13:17, so pod networking to the metadata server is the likely break.
No workplansvc deploy is in the 24 h list. The same image runs on workplan-agent and virtual-assistant, which are also restarting (unschedulable).
Next: check istio-proxy logs and egress to 169.254.169.254 from the pod. Make tracing init non-fatal at main.go:64 so the service starts without otel.
medium confidence dependency GCE metadata server 169.254.169.254 unreachable
claude-opus-5-5 · 3d ago · $0.05 — every link above is evidence it read; nothing else was claimed.
The facts
| workload | workplansvc |
| namespace | tilda |
| restarts_30m | 8 |
| restarts_1h | 15 |
| restarts_24h | 15 |
| restarts_this_run | 15 |
| run_minutes | 10 |
| since | 1,794,750,000 |
| last_restart_at | 1,794,750,600 |
| pods | 1 |
| containers | {"istio-proxy": 8, "workplansvc": 7} |
| reasons | {"sched": 17, "crash": 27} |
| dominant_reason | crash loop |
| image | us-central1-docker.pkg.dev/tilda-dev-325721/artifacts/services/workplansvc:suresh-eng8690-8981379926 |
| last_event | {"reason": "BackOff", "message": "Back-off restarting failed container istio-proxy in pod workplansvc-6787748cc8-nvgzf_tilda(911901ce-cd4e-43ab-ab4f-d47978721f92)", "at": 1794750986} |
| share_of_api_calls_24h | 0.43 |
| discriminator | |
| window | 2026-11-15 13:40:00 UTC → 2026-11-15 14:13:43 UTC |
The story — 12 changes landed in the hour before this opened — adjacency, not proof.
- 2026-11-15 13:09:11 UTChand change [email protected] changed apps/v1/namespaces/default/deployments/x-ray by hand (kubectl/v1.34.1 (darwin/arm64) kubernetes/93248f9,gzip(gfe))
- 2026-11-15 13:09:15 UTChand change [email protected] changed apps/v1/namespaces/default/deployments/x-ray by hand (kubectl/v1.34.1 (darwin/arm64) kubernetes/93248f9,gzip(gfe))
- 2026-11-15 13:09:20 UTChand change [email protected] changed apps/v1/namespaces/default/deployments/x-ray-ingester by hand (kubectl/v1.34.1 (darwin/arm64) kubernetes/93248f9,gzip(gfe))
- 2026-11-15 13:09:24 UTChand change [email protected] changed apps/v1/namespaces/default/deployments/x-ray-ingester by hand (kubectl/v1.34.1 (darwin/arm64) kubernetes/93248f9,gzip(gfe))
- 2026-11-15 13:09:29 UTChand change [email protected] changed apps/v1/namespaces/default/deployments/x-ray-log-alerter by hand (kubectl/v1.34.1 (darwin/arm64) kubernetes/93248f9,gzip(gfe))
- 2026-11-15 13:09:33 UTChand change [email protected] changed batch/v1/namespaces/default/cronjobs/x-ray-argocd-collector by hand (kubectl/v1.34.1 (darwin/arm64) kubernetes/93248f9,gzip(gfe))
- 2026-11-15 13:09:38 UTChand change [email protected] changed batch/v1/namespaces/default/cronjobs/x-ray-compass by hand (kubectl/v1.34.1 (darwin/arm64) kubernetes/93248f9,gzip(gfe))
- 2026-11-15 13:20:00 UTClatency cluster p95 peaked at ≥ 2.0 min
- 2026-11-15 13:28:12 UTChand change [email protected] changed batch/v1/namespaces/default/cronjobs/test-gaps-agent by hand (kubectl/v1.34.1 (darwin/arm64) kubernetes/93248f9,gzip(gfe))
- 2026-11-15 13:28:17 UTChand change [email protected] changed batch/v1/namespaces/default/jobs/test-gaps-agent-manual-1115-12 by hand (kubectl/v1.34.1 (darwin/arm64) kubernetes/93248f9,gzip(gfe))
- 2026-11-15 13:45:56 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/workplan-agent by hand (kubectl/v1.32.2 (darwin/amd64) kubernetes/67a30c0,gzip(gfe))
- 2026-11-15 13:45:57 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/workplansvc by hand (kubectl/v1.32.2 (darwin/amd64) kubernetes/67a30c0,gzip(gfe))
- 2026-11-15 13:45:58 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/virtual-assistant by hand (kubectl/v1.32.2 (darwin/amd64) kubernetes/67a30c0,gzip(gfe))
- 2026-11-15 13:51:32 UTCfinding · warning workplansvc: a new error shape, 7× in the last hour, never seen in 90 days (otel resource detection failed — error detecting resource: partial res)
- 2026-11-15 13:51:32 UTCfinding · warning workplansvc: a new error shape, 7× in the last hour, never seen in 90 days (observability: failed to initialize tracing — create otel resource: er)
- 2026-11-15 13:51:32 UTCfinding · warning workplansvc: a new error shape, 7× in the last hour, never seen in 90 days (observability init failed — observability: init tracing: create otel r)
- 2026-11-15 13:51:32 UTCfinding · critical virtual-assistant on master has restarted 17 times in 15 min — unschedulable, image suresh-eng8690-8981379926
- 2026-11-15 13:51:32 UTCfinding · critical workplan-agent on master has restarted 13 times in 15 min — unschedulable, image suresh-eng8690-8981379926
- 2026-11-15 13:51:32 UTCthis finding workplansvc on master has restarted 15 times in 10 min — crash loop, image suresh-eng8690-8981379926
- 2026-11-15 13:56:41 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/workplan-agent by hand (kubectl/v1.32.2 (darwin/amd64) kubernetes/67a30c0,gzip(gfe))
- 2026-11-15 13:56:42 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/workplansvc by hand (kubectl/v1.32.2 (darwin/amd64) kubernetes/67a30c0,gzip(gfe))
- 2026-11-15 13:56:43 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/virtual-assistant by hand (kubectl/v1.32.2 (darwin/amd64) kubernetes/67a30c0,gzip(gfe))
- 2026-11-15 14:21:03 UTChand change [email protected] changed batch/v1/namespaces/default/cronjobs/test-gaps-agent by hand (kubectl/v1.34.1 (darwin/arm64) kubernetes/93248f9,gzip(gfe))
- 2026-11-15 14:21:08 UTChand change [email protected] changed batch/v1/namespaces/default/jobs/test-gaps-agent-manual-1115-13 by hand (kubectl/v1.34.1 (darwin/arm64) kubernetes/93248f9,gzip(gfe))
- 2026-11-15 14:25:30 UTChand change [email protected] changed apps/v1/namespaces/default/deployments/x-ray by hand (kubectl/v1.34.1 (darwin/arm64) kubernetes/93248f9,gzip(gfe))
- 2026-11-15 14:25:35 UTChand change [email protected] changed apps/v1/namespaces/default/deployments/x-ray by hand (kubectl/v1.34.1 (darwin/arm64) kubernetes/93248f9,gzip(gfe))
- 2026-11-15 14:25:40 UTChand change [email protected] changed apps/v1/namespaces/default/deployments/x-ray-ingester by hand (kubectl/v1.34.1 (darwin/arm64) kubernetes/93248f9,gzip(gfe))
- 2026-11-15 14:25:44 UTChand change [email protected] changed apps/v1/namespaces/default/deployments/x-ray-ingester by hand (kubectl/v1.34.1 (darwin/arm64) kubernetes/93248f9,gzip(gfe))
- 2026-11-15 14:25:49 UTChand change [email protected] changed apps/v1/namespaces/default/deployments/x-ray by hand (kubectl/v1.34.1 (darwin/arm64) kubernetes/93248f9,gzip(gfe))
- 2026-11-15 14:25:53 UTChand change [email protected] changed apps/v1/namespaces/default/deployments/x-ray-log-alerter by hand (kubectl/v1.34.1 (darwin/arm64) kubernetes/93248f9,gzip(gfe))
- 2026-11-15 14:25:58 UTChand change [email protected] changed batch/v1/namespaces/default/cronjobs/x-ray-argocd-collector by hand (kubectl/v1.34.1 (darwin/arm64) kubernetes/93248f9,gzip(gfe))
- 2026-11-15 14:26:02 UTChand change [email protected] changed batch/v1/namespaces/default/cronjobs/x-ray-compass by hand (kubectl/v1.34.1 (darwin/arm64) kubernetes/93248f9,gzip(gfe))
History
- 2026-11-15 13:51:32 UTCopened x-ray · severity=warning title=workplansvc on master has restarted 8 times in 5 min — crash loop, image suresh-eng8690-8981379926 finding={'id': 17317, 'fingerprint': 'b807db4266bc4a7340a1', 'kind': 'pod.unstable', 'severity': 'warning', 'cluster': 'master', 'subject': 'tilda/workplansvc', 'title': 'workplansvc on master has restarted 8 times in 5 min — crash loop, image suresh-eng8690-8981379926', 'facts': {'workload': 'workplansvc', 'namespace': 'tilda', 'restarts_30m': 8, 'restarts_1h': 8, 'restarts_24h': 8, 'restarts_this_run': 8, 'run_minutes': 5, 'since': 1794750000, 'last_restart_at': 1794750300, 'pods': 1, 'containers': {'istio-proxy': 4, 'workplansvc': 4}, 'reasons': {'sched': 16, 'crash': 25}, 'dominant_reason': 'crash loop', 'image': 'us-central1-docker.pkg.dev/tilda-dev-325721/artifacts/services/workplansvc:suresh-eng8690-8981379926', 'last_event': {'reason': 'BackOff', 'message': 'Back-off restarting failed container istio-proxy in pod workplansvc-6787748cc8-nvgzf_tilda(911901ce-cd4e-43ab-ab4f-d47978721f92)', 'at': 1794750430}, 'share_of_api_calls_24h': 0.4223, 'recent_deploys': [], 'window': [1794750000, 1794750692], 'discriminator': ''}, 'evidence': [{'label': 'Reliability', 'href': '/reliability?cluster=master&range=24h'}, {'label': 'Pods', 'href': '/pods?cluster=master'}, {'label': 'workplansvc error logs', 'href': '/logs?cluster=master&service=workplansvc&severity=ERROR&range=24h'}], 'provenance': {'source': 'x-ray', 'note': 'pod_rollup: Monitoring restart_count deltas + Kubernetes events, per five minutes'}, 'impact': 40.4, 'status': 'open', 'first_seen': 1794750692, 'last_seen': 1794750692, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 1, 'narrative': None, 'narrative_kind': None, 'discriminator': '', 'level': 'medium', 'told_level': None, 'episode_at': 1794750692, 'chat_thread': None, 'kind_label': 'Pod keeps restarting', 'group': 'Reliability', 'age_s': 22, 'stale_s': 22}
- 2026-11-15 13:59:09 UTCescalated x-ray · severity=critical was=warning title=workplansvc on master has restarted 14 times in 15 min — crash loop, image suresh-eng8690-8981379926 level=high was_level=medium finding={'id': 17317, 'fingerprint': 'b807db4266bc4a7340a1', 'kind': 'pod.unstable', 'severity': 'critical', 'cluster': 'master', 'subject': 'tilda/workplansvc', 'title': 'workplansvc on master has restarted 14 times in 15 min — crash loop, image suresh-eng8690-8981379926', 'facts': {'workload': 'workplansvc', 'namespace': 'tilda', 'restarts_30m': 14, 'restarts_1h': 14, 'restarts_24h': 14, 'restarts_this_run': 14, 'run_minutes': 15, 'since': 1794750000, 'last_restart_at': 1794750900, 'pods': 1, 'containers': {'workplansvc': 7, 'istio-proxy': 7}, 'reasons': {'sched': 17, 'crash': 27}, 'dominant_reason': 'crash loop', 'image': 'us-central1-docker.pkg.dev/tilda-dev-325721/artifacts/services/workplansvc:suresh-eng8690-8981379926', 'last_event': {'reason': 'BackOff', 'message': 'Back-off restarting failed container istio-proxy in pod workplansvc-6787748cc8-nvgzf_tilda(911901ce-cd4e-43ab-ab4f-d47978721f92)', 'at': 1794750986}, 'share_of_api_calls_24h': 0.422, 'recent_deploys': [], 'window': [1794750000, 1794751149], 'discriminator': ''}, 'evidence': [{'label': 'Reliability', 'href': '/reliability?cluster=master&range=24h'}, {'label': 'Pods', 'href': '/pods?cluster=master'}, {'label': 'workplansvc error logs', 'href': '/logs?cluster=master&service=workplansvc&severity=ERROR&range=24h'}], 'provenance': {'source': 'x-ray', 'note': 'pod_rollup: Monitoring restart_count deltas + Kubernetes events, per five minutes'}, 'impact': 64.4, 'status': 'open', 'first_seen': 1794750692, 'last_seen': 1794751149, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 2, 'narrative': 'The workplansvc workload in the tilda namespace has restarted 8 times since 2026-11-15 13:40 UTC, most recently at 13:45 UTC. Kubernetes logs 25 crash events and 16 scheduling events. The last event shows BackOff restarting the failed istio-proxy container in pod workplansvc-6787748cc8-nvgzf_tilda (911901ce-cd4e-43ab-ab4f-d47978721f92). It runs image suresh-eng8690-8981379926 with istio-proxy ×4 and workplansvc ×4 containers restarting. This workload carried 42.2% of cluster API calls in 24 hours. Note: restarts are counted from container restart counters only, excluding rollouts or spot preemptions, and resolve automatically after thirty quiet minutes.', 'narrative_kind': 'ai', 'discriminator': '', 'level': 'high', 'told_level': None, 'episode_at': 1794750692, 'chat_thread': None, 'kind_label': 'Pod keeps restarting', 'group': 'Reliability', 'age_s': 527, 'stale_s': 70}
- 2026-11-15 14:33:42 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 17317, 'fingerprint': 'b807db4266bc4a7340a1', 'kind': 'pod.unstable', 'severity': 'critical', 'cluster': 'master', 'subject': 'tilda/workplansvc', 'title': 'workplansvc on master has restarted 15 times in 10 min — crash loop, image suresh-eng8690-8981379926', 'facts': {'workload': 'workplansvc', 'namespace': 'tilda', 'restarts_30m': 8, 'restarts_1h': 15, 'restarts_24h': 15, 'restarts_this_run': 15, 'run_minutes': 10, 'since': 1794750000, 'last_restart_at': 1794750600, 'pods': 1, 'containers': {'istio-proxy': 8, 'workplansvc': 7}, 'reasons': {'sched': 17, 'crash': 27}, 'dominant_reason': 'crash loop', 'image': 'us-central1-docker.pkg.dev/tilda-dev-325721/artifacts/services/workplansvc:suresh-eng8690-8981379926', 'last_event': {'reason': 'BackOff', 'message': 'Back-off restarting failed container istio-proxy in pod workplansvc-6787748cc8-nvgzf_tilda(911901ce-cd4e-43ab-ab4f-d47978721f92)', 'at': 1794750986}, 'share_of_api_calls_24h': 0.4286, 'recent_deploys': [], 'window': [1794750000, 1794752023], 'discriminator': ''}, 'evidence': [{'label': 'Reliability', 'href': '/reliability?cluster=master&range=24h'}, {'label': 'Pods', 'href': '/pods?cluster=master'}, {'label': 'workplansvc error logs', 'href': '/logs?cluster=master&service=workplansvc&severity=ERROR&range=24h'}], 'provenance': {'source': 'x-ray', 'note': 'pod_rollup: Monitoring restart_count deltas + Kubernetes events, per five minutes'}, 'impact': 65.0, 'status': 'resolved', 'first_seen': 1794750692, 'last_seen': 1794752023, 'resolved_at': 1794753222, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 4, 'narrative': "workplansvc in tilda on master has restarted in place 15 times since 2026-11-15 13:40 UTC (8 in the last 30 minutes, 15 in 24 hours), most recently at 2026-11-15 13:50 UTC. Kubernetes' account of it: 27 crash events, 17 sched events. Last event: BackOff — “Back-off restarting failed container istio-proxy in pod workplansvc-6787748cc8-nvgzf_tilda(911901ce-cd4e-43ab-ab4f-d47978721f92)”. It is running image suresh-eng8690-8981379926. Containers restarting: istio-proxy ×8, workplansvc ×7. The workload carried 42.9% of the cluster's API calls in the last day. Restarts are counted from the container restart counter, so a rollout or a spot preemption does not count; a crash does. It resolves on its own after thirty quiet minutes.", 'narrative_kind': 'template', 'discriminator': '', 'level': 'high', 'told_level': None, 'episode_at': 1794750692, 'chat_thread': None, 'kind_label': 'Pod keeps restarting', 'group': 'Reliability', 'age_s': 2549, 'stale_s': 1218}
Check it yourself
Every link carries the finding's own time window, so it shows tomorrow what it shows now.
Actions
Sign in (IAP) to acknowledge, snooze or resolve.
Where the numbers come from
| source | x-ray |
| note | pod_rollup: Monitoring restart_count deltas + Kubernetes events, per five minutes |
| detector | pod_unstable |