ReadOnlyFileSystemDetected: gk3-delta-nap-d9uprtrn-f6208abb-8glb (1× in 30 min)
Node problem: ReadOnlyFileSystemDetected: gk3-delta-nap-d9uprtrn-f6208abb-8glb (1× in 30 min). Facts: reason ReadOnlyFileSystemDetected, count_30m 1, last_message .
What X-ray is prepared to defend about delta right now — each one a claim
with its numbers, the evidence to check it, and what normal looks like. Detectors last ran
never.
?A finding is opened by a detector — a statistical test over the stored rollups, or a
live check of a source — and stays ONE row while the condition persists: it updates,
escalates, and resolves itself fifteen minutes after the detector stops seeing it.
Acknowledge to say you have it; snooze to hide it for a while; nothing here is a
threshold somebody configured.
Every card is a claim X-ray can back with numbers you can click through to. Findings rank by severity first, then by an impact score (severity band + the subject's share of the cluster's traffic + how long it has been open), so the top of the page is the thing to look at first.
What this page will not tell you
Node problem: ReadOnlyFileSystemDetected: gk3-delta-nap-d9uprtrn-f6208abb-8glb (1× in 30 min). Facts: reason ReadOnlyFileSystemDetected, count_30m 1, last_message .
Node problem: ReadOnlyFileSystemDetected: gk3-delta-nap-d9uprtrn-f6208abb-bvdv (1× in 30 min). Facts: reason ReadOnlyFileSystemDetected, count_30m 1, last_message .
execsvc logged “Request completed — ? · could not save Careplan activity V2” 9 times in the last hour and never before in 90 days on delta (first seen 2026-11-17 09:46 UTC, raised at observability/middleware.go:281). Example trace: 9bff744b81af07042b1e3e7de4963df4.
First look low confidence
Cause not clear from logs and recent changes; execsvc returned 500 "could not save Careplan activity V2" 9× between 09:46 and 09:54 UTC on 2026-11-17 (new shape, logged at observability/middleware.go:281).
Checked: no execsvc deploys in the 24 h before onset (running build deployed 2026-11-16 05:11 UTC by [email protected]), no other findings at the same time.
Alongside it: "no result from branch evaluation" (branchV2.go:77, 2×), "Question object not found for question" (questionV2.go:1863, 3×, new at 09:53), and pubsub "transaction was never committed" (2×).
The code shown (careplan.go:1044) is the 403 branch; the 500 comes from a later error path after execution.SaveActivityV2, whose underlying error is not in the bundle.
Next step: open the example trace to read the wrapped error from SaveActivityV2. Then check whether it matches the branch-evaluation or missing-question errors for that protocol.
· its evidence →
execsvc logged "evalResult is not a bool" 55 times within a one-hour window ending at execlib/branchV2.go:96, with no prior occurrences in the preceding 90 days (first seen 2026-11-16 18:51 UTC). Example trace: 76e7e01f9377ba990c4164bcd594d9c8.
First look low confidence
Cause not clear from logs and recent changes; execsvc logged 'evalResult is not a bool' 55× (18:51–19:21 UTC) at execlib/branchV2.go:96, alongside 'no result from branch evaluation' 150× at branchV2.go:91 since 18:42 (finding 17479).
The same two errors already ran from 14:20 UTC at branchV2.go:77/82, last at 18:31, via the participant auto-query path (activityV2.go:882).
The line numbers moved by 14 between 18:31 and 18:42, which suggests a new execsvc build started then, but no image deploy is recorded.
The only recorded deploy is config-only (6db9855a7d34, [email protected], 05:11 UTC, 820 min before), and nothing links it to these errors.
The error strings were not found in the default-branch code, so the running code is unknown.
Next step: confirm which execsvc build has run since about 18:40, then read execlib/branchV2.go:91-96 in it to see why branch evaluation returns no or non-bool results.
· its evidence →
execsvc logged “no result from branch evaluation” 150 times in the last hour and never before in 90 days on delta (first seen 2026-11-16 18:42 UTC, raised at execlib/branchV2.go:91). Example trace: 76e7e01f9377ba990c4164bcd594d9c8.
First look low confidence
Cause not clear from logs and recent changes; this looks like an existing error that moved, not a new one.
execsvc logged 'no result from branch evaluation' 101× at execlib/branchV2.go:91 from 18:42 to 19:13 UTC, plus 'evalResult is not a bool' 38× at branchV2.go:96 from 18:51 UTC.
The same two messages came from branchV2.go:77 (106×, seen since Nov 7) and :82 (4×) until 18:31 UTC, then stopped. The activityV2.go:882 wrapper lines also stopped at 18:31.
That 14-line shift suggests a new execsvc build rolled out around 18:31–18:42, but the only recorded deploy is [email protected] at 05:11 UTC, 810 min earlier. Code and commits could not be read because the GitHub budget ran out.
Next step: find what execsvc version started running around 18:31–18:42 UTC, diff branchV2.go, and check why branch evaluation returns no result (failing since Nov 7).
· its evidence →
execsvc logged “message was never published because transaction was never committed” 6 times in the last hour and never before in 90 days on delta (first seen 2026-11-16 14:28 UTC, raised at pubsublib/pubsub.go:131). Example trace: 7bce6fd5fb57a5f7ec3d2edce7a8fdb0.
execsvc logged “evalResult is not a bool” 46 times in the last hour and never before in 90 days on delta (first seen 2026-11-16 14:20 UTC, raised at execlib/branchV2.go:82). Example trace: 7bce6fd5fb57a5f7ec3d2edce7a8fdb0.
execsvc logged “failed to evaluate and raise auto query for participant question — evalResult is not a bool” 46 times in the last hour and never before in 90 days on delta (first seen 2026-11-16 14:20 UTC, raised at execution/activityV2.go:882). Example trace: 7bce6fd5fb57a5f7ec3d2edce7a8fdb0.
execsvc logged “failed to evaluate and raise auto query for participant question — no result from branch evaluation for branch, protocolId=? activityRefId=?” 247 times in the last hour and never before in 90 days on delta (first seen 2026-11-16 14:20 UTC, raised at execution/activityV2.go:882). Example trace: 7bce6fd5fb57a5f7ec3d2edce7a8fdb0.
First look low confidence
Cause not clear from logs and recent changes; execsvc logged 122 'failed to evaluate and raise auto query for participant question — no result from branch evaluation' errors (execution/activityV2.go:882) from 14:20 to 14:26 UTC on 2026-11-16.
Alongside it, 19 'evalResult is not a bool' errors (execlib/branchV2.go:82) also began at 14:20, plus 1 auto query re-evaluation failure at 14:23.
The inner error 'no result from branch evaluation' (execlib/branchV2.go:77) has existed since 2026-11-07. Only the auto-query wrapper at activityV2.go:882 is new.
The only execsvc deploy was a config change (6db9855a7d34) by [email protected] at 05:11 UTC. That is 548 min before onset, and nothing links it to these errors. Code and commits could not be read because the GitHub budget ran out.
Next step: open the example trace and check the branch and auto-query definitions for protocolId 77bcfccb-cc59-4aea-bff8-0991eaa0c788. Look for a protocol edit published around 14:20 that makes branches return no result or a non-bool.
· its evidence →
Pod cannot start: schema-migrate (tilda) image pull failing for at least 15 min — 1 pod. Facts: namespace tilda, workload schema-migrate, why image pull failing, pending_minutes 15, since 1794911087.
neo4j-differential-backup-delta in neo4j-system on delta has restarted in place 4 times since 2026-11-17 10:15 UTC (4 in the last 30 minutes, 4 in 24 hours), most recently at 2026-11-17 10:25 UTC. Kubernetes' account of it: 104 sched events, 3 crash events. Last event: BackOff — “Back-off restarting failed container neo4j-differential-backup in pod neo4j-differential-backup-delta-29915180-wmvtw_neo4j-system(3b547b5e-8f5d-4404-91e7-1dbe31”. It is running image 5.26.28-diff-backup-2. Containers restarting: neo4j-differential-backup ×4. The workload carried 0.0% of the cluster's API calls in the last day. Restarts are counted from the container restart counter, so a rollout or a spot preemption does not count; a crash does. It resolves on its own after thirty quiet minutes.
tmfsync is spending its error budget 8.4× faster than the objective allows over the last hour (0.8× over six hours). The objective is 99.5%, derived — a baseline, not a promise. 26 bad events against 6,583 calls in the window.
delta's AI spend is $160.19 so far this month and projects to $295.16 by month end (10.91/day, weekends weighted 0.84×); last month closed at $1.24. A projection is a line through the past — the AI page shows which operation and org moved.
Autoscaler blind: HorizontalPodAutoscaler/tmfsync-hpa (tilda) cannot read its scaling metric — 28× in 2 h. Facts: namespace tilda, object HorizontalPodAutoscaler/tmfsync-hpa, failures_2h 28, last_message No recommendation.
Node problem: pods on an unready node: 12 affected in 30 min. Facts: reason NodeNotReady, count_30m 12, last_message Node is not ready.
Autoscaler blind: HorizontalPodAutoscaler/ontologysvc-hpa (tilda) cannot read its scaling metric — 28× in 2 h. Facts: namespace tilda, object HorizontalPodAutoscaler/ontologysvc-hpa, failures_2h 28, last_message No recommendation.