Xray
1h 6h 24h 48h 3d 5d 7d 14d 30d ‹
Last 1 hour▾
Ends now. Anything up to this cluster's 30-day retention.
Start and end times
Cancel
Interpreted in UTC (change it at the right of this bar). Earliest available: 2026-10-19 08:13:42.
› →|
updated 21h ago
Sample data. This database was generated by scripts/seed_dev_db.py — every service, deploy, person and figure on this page is synthetic. Names are sample.*@seed.invalid and image tags are prefixed SEED-. Nothing here happened.
← findings

workplan-agent on master has restarted 13 times in 15 min — unschedulable, image suresh-eng8690-8981379926

critical Pod keeps restarting master resolved impact 41 opened 2d ago · last observed 2d ago · observed 5× · #17319 · e50cd05f14c4d47d50f5

What happened template

workplan-agent in tilda on master has restarted in place 13 times since 2026-11-15 13:40 UTC (0 in the last 30 minutes, 13 in 24 hours), most recently at 2026-11-15 13:55 UTC. Kubernetes' account of it: 47 sched events, 12 crash events, 4 probe events. Last event: BackOff — “Back-off restarting failed container istio-proxy in pod workplan-agent-7f4d75b46c-8w7x9_tilda(5371afad-548c-4493-99b5-94989841e269)”. It is running image suresh-eng8690-8981379926. Containers restarting: workplan-agent ×7, istio-proxy ×6. The workload carried 0.0% of the cluster's API calls in the last day. Restarts are counted from the container restart counter, so a rollout or a spot preemption does not count; a crash does. It resolves on its own after thirty quiet minutes.

A workload's containers restart in place over and over — crash loop, OOM kills, liveness kills or image pulls failing.

First look — investigated before anyone looked

Cause not clear from logs and recent changes; workplan-agent (ns tilda) restarted 14 times in 15 min since 13:40 UTC: 47 unschedulable, 12 crash and 4 probe events, both istio-proxy and workplan-agent containers (last event: BackOff on istio-proxy).
The same image tag suresh-eng8690-8981379926 is failing at the same time on workplansvc (crash loop) and virtual-assistant (unschedulable).
workplansvc logs show a new error since 13:46: otel resource detection cannot reach the metadata server 169.254.169.254 (main.go:64, observability.go:133).
A node went NotReady and cilium-agent logged a foreign-IP error at 13:17, but nothing links either to this pod.
No deploy of these three services shows in the 24 h before. The argocd collector was stale for 224 min, so a rollout of this branch-named image may be missing.
Next step: run kubectl describe on pod workplan-agent-7f4d75b46c-8w7x9 to get the unschedulable reason, and check who rolled out tag suresh-eng8690-8981379926 to master.

low confidence

claude-opus-5-5 · 2d ago · $0.05 — every link above is evidence it read; nothing else was claimed.

The facts

workloadworkplan-agent
namespacetilda
restarts_30m0
restarts_1h13
restarts_24h13
restarts_this_run13
run_minutes15
since1,794,750,000
last_restart_at1,794,750,900
pods1
containers{"istio-proxy": 6, "workplan-agent": 7}
reasons{"sched": 47, "crash": 12, "probe": 4}
dominant_reasonunschedulable
imageus-central1-docker.pkg.dev/tilda-dev-325721/artifacts/services/workplan-agent:suresh-eng8690-8981379926
last_event{"reason": "BackOff", "message": "Back-off restarting failed container istio-proxy in pod workplan-agent-7f4d75b46c-8w7x9_tilda(5371afad-548c-4493-99b5-94989841e269)", "at": 1794751000}
share_of_api_calls_24h0.00
discriminator
window2026-11-15 13:40:00 UTC → 2026-11-15 14:20:01 UTC

The story — 12 changes landed in the hour before this opened — adjacency, not proof.

History

  • 2026-11-15 13:51:32 UTCopened x-ray · severity=warning title=workplan-agent on master has restarted 6 times in 5 min — unschedulable, image suresh-eng8690-8981379926 finding={'id': 17319, 'fingerprint': 'e50cd05f14c4d47d50f5', 'kind': 'pod.unstable', 'severity': 'warning', 'cluster': 'master', 'subject': 'tilda/workplan-agent', 'title': 'workplan-agent on master has restarted 6 times in 5 min — unschedulable, image suresh-eng8690-8981379926', 'facts': {'workload': 'workplan-agent', 'namespace': 'tilda', 'restarts_30m': 6, 'restarts_1h': 6, 'restarts_24h': 6, 'restarts_this_run': 6, 'run_minutes': 5, 'since': 1794750000, 'last_restart_at': 1794750300, 'pods': 1, 'containers': {'istio-proxy': 4, 'workplan-agent': 2}, 'reasons': {'sched': 47, 'crash': 11, 'probe': 3}, 'dominant_reason': 'unschedulable', 'image': 'us-central1-docker.pkg.dev/tilda-dev-325721/artifacts/services/workplan-agent:suresh-eng8690-8981379926', 'last_event': {'reason': 'Killing', 'message': 'Container workplan-agent failed liveness probe, will be restarted', 'at': 1794750434}, 'share_of_api_calls_24h': 0.0, 'recent_deploys': [], 'window': [1794750000, 1794750692], 'discriminator': ''}, 'evidence': [{'label': 'Reliability', 'href': '/reliability?cluster=master&range=24h'}, {'label': 'Pods', 'href': '/pods?cluster=master'}, {'label': 'workplan-agent error logs', 'href': '/logs?cluster=master&service=workplan-agent&severity=ERROR&range=24h'}], 'provenance': {'source': 'x-ray', 'note': 'pod_rollup: Monitoring restart_count deltas + Kubernetes events, per five minutes'}, 'impact': 23.1, 'status': 'open', 'first_seen': 1794750692, 'last_seen': 1794750692, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 1, 'narrative': None, 'narrative_kind': None, 'discriminator': '', 'level': 'low', 'told_level': None, 'episode_at': 1794750692, 'chat_thread': None, 'kind_label': 'Pod keeps restarting', 'group': 'Reliability', 'age_s': 22, 'stale_s': 22}
  • 2026-11-15 13:59:09 UTCescalated x-ray · severity=critical was=warning title=workplan-agent on master has restarted 14 times in 15 min — unschedulable, image suresh-eng8690-8981379926 level=low was_level=low finding={'id': 17319, 'fingerprint': 'e50cd05f14c4d47d50f5', 'kind': 'pod.unstable', 'severity': 'critical', 'cluster': 'master', 'subject': 'tilda/workplan-agent', 'title': 'workplan-agent on master has restarted 14 times in 15 min — unschedulable, image suresh-eng8690-8981379926', 'facts': {'workload': 'workplan-agent', 'namespace': 'tilda', 'restarts_30m': 14, 'restarts_1h': 14, 'restarts_24h': 14, 'restarts_this_run': 14, 'run_minutes': 15, 'since': 1794750000, 'last_restart_at': 1794750900, 'pods': 1, 'containers': {'istio-proxy': 7, 'workplan-agent': 7}, 'reasons': {'sched': 47, 'crash': 12, 'probe': 4}, 'dominant_reason': 'unschedulable', 'image': 'us-central1-docker.pkg.dev/tilda-dev-325721/artifacts/services/workplan-agent:suresh-eng8690-8981379926', 'last_event': {'reason': 'BackOff', 'message': 'Back-off restarting failed container istio-proxy in pod workplan-agent-7f4d75b46c-8w7x9_tilda(5371afad-548c-4493-99b5-94989841e269)', 'at': 1794751000}, 'share_of_api_calls_24h': 0.0, 'recent_deploys': [], 'window': [1794750000, 1794751149], 'discriminator': ''}, 'evidence': [{'label': 'Reliability', 'href': '/reliability?cluster=master&range=24h'}, {'label': 'Pods', 'href': '/pods?cluster=master'}, {'label': 'workplan-agent error logs', 'href': '/logs?cluster=master&service=workplan-agent&severity=ERROR&range=24h'}], 'provenance': {'source': 'x-ray', 'note': 'pod_rollup: Monitoring restart_count deltas + Kubernetes events, per five minutes'}, 'impact': 40.6, 'status': 'open', 'first_seen': 1794750692, 'last_seen': 1794751149, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 2, 'narrative': 'The workplan-agent workload in the tilda namespace has restarted 6 times since 2026-11-15 13:40 UTC, most recently at 13:45 UTC. Kubernetes recorded 47 scheduling events, 11 crashes, and 3 probe failures. The last event was a liveness probe failure triggering a container restart. The workload runs image suresh-eng8690-8981379926 with istio-proxy (×4) and workplan-agent (×2) containers restarting. It carried 0.0% of cluster API calls in 24 hours. Note: restart counts exclude rollouts and preemptions; crashes count. Resolution occurs after thirty quiet minutes.', 'narrative_kind': 'ai', 'discriminator': '', 'level': 'low', 'told_level': None, 'episode_at': 1794750692, 'chat_thread': None, 'kind_label': 'Pod keeps restarting', 'group': 'Reliability', 'age_s': 527, 'stale_s': 70}
  • 2026-11-15 14:40:40 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 17319, 'fingerprint': 'e50cd05f14c4d47d50f5', 'kind': 'pod.unstable', 'severity': 'critical', 'cluster': 'master', 'subject': 'tilda/workplan-agent', 'title': 'workplan-agent on master has restarted 13 times in 15 min — unschedulable, image suresh-eng8690-8981379926', 'facts': {'workload': 'workplan-agent', 'namespace': 'tilda', 'restarts_30m': 0, 'restarts_1h': 13, 'restarts_24h': 13, 'restarts_this_run': 13, 'run_minutes': 15, 'since': 1794750000, 'last_restart_at': 1794750900, 'pods': 1, 'containers': {'istio-proxy': 6, 'workplan-agent': 7}, 'reasons': {'sched': 47, 'crash': 12, 'probe': 4}, 'dominant_reason': 'unschedulable', 'image': 'us-central1-docker.pkg.dev/tilda-dev-325721/artifacts/services/workplan-agent:suresh-eng8690-8981379926', 'last_event': {'reason': 'BackOff', 'message': 'Back-off restarting failed container istio-proxy in pod workplan-agent-7f4d75b46c-8w7x9_tilda(5371afad-548c-4493-99b5-94989841e269)', 'at': 1794751000}, 'share_of_api_calls_24h': 0.0, 'recent_deploys': [], 'window': [1794750000, 1794752401], 'discriminator': ''}, 'evidence': [{'label': 'Reliability', 'href': '/reliability?cluster=master&range=24h'}, {'label': 'Pods', 'href': '/pods?cluster=master'}, {'label': 'workplan-agent error logs', 'href': '/logs?cluster=master&service=workplan-agent&severity=ERROR&range=24h'}], 'provenance': {'source': 'x-ray', 'note': 'pod_rollup: Monitoring restart_count deltas + Kubernetes events, per five minutes'}, 'impact': 41.1, 'status': 'resolved', 'first_seen': 1794750692, 'last_seen': 1794752401, 'resolved_at': 1794753640, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 5, 'narrative': "workplan-agent in tilda on master has restarted in place 13 times since 2026-11-15 13:40 UTC (0 in the last 30 minutes, 13 in 24 hours), most recently at 2026-11-15 13:55 UTC. Kubernetes' account of it: 47 sched events, 12 crash events, 4 probe events. Last event: BackOff — “Back-off restarting failed container istio-proxy in pod workplan-agent-7f4d75b46c-8w7x9_tilda(5371afad-548c-4493-99b5-94989841e269)”. It is running image suresh-eng8690-8981379926. Containers restarting: workplan-agent ×7, istio-proxy ×6. The workload carried 0.0% of the cluster's API calls in the last day. Restarts are counted from the container restart counter, so a rollout or a spot preemption does not count; a crash does. It resolves on its own after thirty quiet minutes.", 'narrative_kind': 'template', 'discriminator': '', 'level': 'low', 'told_level': None, 'episode_at': 1794750692, 'chat_thread': None, 'kind_label': 'Pod keeps restarting', 'group': 'Reliability', 'age_s': 2968, 'stale_s': 1259}

Check it yourself

Every link carries the finding's own time window, so it shows tomorrow what it shows now.

Actions

Sign in (IAP) to acknowledge, snooze or resolve.

Where the numbers come from

sourcex-ray
notepod_rollup: Monitoring restart_count deltas + Kubernetes events, per five minutes
detectorpod_unstable