Xray
daily rollups 1h 6h 24h 48h 3d 5d 7d 14d 30d ‹
Last 7 days▾
Ends now. Anything up to this cluster's 30-day retention.
Start and end times
Cancel
Interpreted in UTC (change it at the right of this bar). Earliest available: 2026-10-19 16:48:58.
› →|
updated 1d ago
Sample data. This database was generated by scripts/seed_dev_db.py — every service, deploy, person and figure on this page is synthetic. Names are sample.*@seed.invalid and image tags are prefixed SEED-. Nothing here happened.

Findings 10

stored age unknown all 300detectors over stored rollups and live sources

What X-ray is prepared to defend about master right now — each one a claim with its numbers, the evidence to check it, and what normal looks like. Detectors last ran never. ?A finding is opened by a detector — a statistical test over the stored rollups, or a live check of a source — and stays ONE row while the condition persists: it updates, escalates, and resolves itself fifteen minutes after the detector stops seeing it. Acknowledge to say you have it; snooze to hide it for a while; nothing here is a threshold somebody configured.

How to read this page — how findings work

Every card is a claim X-ray can back with numbers you can click through to. Findings rank by severity first, then by an impact score (severity band + the subject's share of the cluster's traffic + how long it has been open), so the top of the page is the thing to look at first.

anomaly
Three of the last four five-minute buckets far outside what this hour of THIS weekday normally sees — median ± MAD over the last 2–8 weeks, not a threshold.
outlier
A service unlike the rest of the fleet on one measure, over six hours. Compares with peers, not with its own history — the two catch different things.
budget
Burn rate against an error objective derived from the service's own last week, and the date the 30-day budget runs out if the trend holds.
resolves itself
A finding a healthy detector run no longer observes resolves after fifteen minutes. A detector whose SOURCE fails resolves nothing — it says so on the Detectors page instead.

What this page will not tell you

  • That a finding is a diagnosis. It is a measured departure from normal with adjacent changes listed; the narrative says 'adjacency, not proof' and means it.
  • Anything about hours the ingester did not cover. A bucket with no rows is a gap, not a quiet period, and the baselines skip it.
  • Anything on the first day for the novel-error detector: its 90-day memory learns silently for 24 hours before it will call anything new.
show active resolved all severity everything warning + critical only clusters
all clusters
just master · nothing ticked means master, the cluster selected up top
or just close the list
group all AnomaliesReliabilityNeo4jChangesJourneyX-rayWeaviateAIPodsJobsPlatformErrorsCustomCostCapacity 21 critical · 76 warning · 67 info detectors
execsvc: a new error shape 12 targets worst: critical 9× in the last hour, never seen in 90 days (Request completed — ? · could not save Careplan activity V2, 787× in the last hour, never seen in 90 days (could not match participant questionnaire/question/table/doc-upload to, 55× in the last hour, never seen in 90 days (evalResult is not a bool, 150× in the last hour, never seen in 90 days (no result from branch evaluation, 5× in the last hour, never seen in 90 days (failed to run cypher in transaction — Neo4jError: Neo.TransientError.T, 5× in the last hour, never seen in 90 days (failed to add document to scope binders — could not write context scop and 6 more
critical Error never seen before delta last seen · since · 3×

execsvc: a new error shape, 9× in the last hour, never seen in 90 days (Request completed — ? · could not save Careplan activity V2)

execsvc logged “Request completed — ? · could not save Careplan activity V2” 9 times in the last hour and never before in 90 days on delta (first seen 2026-11-17 09:46 UTC, raised at observability/middleware.go:281). Example trace: 9bff744b81af07042b1e3e7de4963df4.

First look low confidence
Cause not clear from logs and recent changes; execsvc returned 500 "could not save Careplan activity V2" 9× between 09:46 and 09:54 UTC on 2026-11-17 (new shape, logged at observability/middleware.go:281).
Checked: no execsvc deploys in the 24 h before onset (running build deployed 2026-11-16 05:11 UTC by [email protected]), no other findings at the same time.
Alongside it: "no result from branch evaluation" (branchV2.go:77, 2×), "Question object not found for question" (questionV2.go:1863, 3×, new at 09:53), and pubsub "transaction was never committed" (2×).
The code shown (careplan.go:1044) is the 403 branch; the 500 comes from a later error path after execution.SaveActivityV2, whose underlying error is not in the bundle.
Next step: open the example trace to read the wrapped error from SaveActivityV2. Then check whether it matches the branch-evaluation or missing-question errors for that protocol. · its evidence →

critical Error never seen before master resolved last seen · since · 2×

execsvc: a new error shape, 787× in the last hour, never seen in 90 days (could not match participant questionnaire/question/table/doc-upload to)

execsvc logged “could not match participant questionnaire/question/table/doc-upload to activity graph” 787 times in the last hour and never before in 90 days on master (first seen 2026-11-17 04:05 UTC, raised at execution/activity_details_v2.go:2007). Example trace: b4f239ac36290842023cca51c51d0e33.

First look low confidence
Cause not clear from logs and recent changes; GET activity status in execsvc returned 500 about 700 times between 04:05 and 04:16 UTC on 2026-11-17, logged as 'could not match participant question to activity graph'.
One question appears in the samples, questionId="5571182f-6c21-491e-b8c6-51fce3166e1e", with empty refId, variableName and title, so it may be one participant record polled repeatedly.
The handler error (handlers/activity_details_v2.go:457) has existed since 2026-11-04. Only the log line at execution/activity_details_v2.go:2007 is new.
The 5 deploys in the prior 24 h were autosync config-only with no image commits. The latest, d8d21151cd28, landed 151 min before onset, and nothing links any of them to this error.
Next step: look up question 5571182f-6c21-491e-b8c6-51fce3166e1e and its activity graph, and check why it has no refId, variableName or title. Start from the example trace.
This is also driving the error budget burn on execsvc (18x over 1 h). · its evidence →

critical Error never seen before delta resolved last seen · since · 2×

execsvc: a new error shape, 55× in the last hour, never seen in 90 days (evalResult is not a bool)

execsvc logged "evalResult is not a bool" 55 times within a one-hour window ending at execlib/branchV2.go:96, with no prior occurrences in the preceding 90 days (first seen 2026-11-16 18:51 UTC). Example trace: 76e7e01f9377ba990c4164bcd594d9c8.

First look low confidence
Cause not clear from logs and recent changes; execsvc logged 'evalResult is not a bool' 55× (18:51–19:21 UTC) at execlib/branchV2.go:96, alongside 'no result from branch evaluation' 150× at branchV2.go:91 since 18:42 (finding 17479).
The same two errors already ran from 14:20 UTC at branchV2.go:77/82, last at 18:31, via the participant auto-query path (activityV2.go:882).
The line numbers moved by 14 between 18:31 and 18:42, which suggests a new execsvc build started then, but no image deploy is recorded.
The only recorded deploy is config-only (6db9855a7d34, [email protected], 05:11 UTC, 820 min before), and nothing links it to these errors.
The error strings were not found in the default-branch code, so the running code is unknown.
Next step: confirm which execsvc build has run since about 18:40, then read execlib/branchV2.go:91-96 in it to see why branch evaluation returns no or non-bool results. · its evidence →

critical Error never seen before delta resolved last seen · since · 3×

execsvc: a new error shape, 150× in the last hour, never seen in 90 days (no result from branch evaluation)

execsvc logged “no result from branch evaluation” 150 times in the last hour and never before in 90 days on delta (first seen 2026-11-16 18:42 UTC, raised at execlib/branchV2.go:91). Example trace: 76e7e01f9377ba990c4164bcd594d9c8.

First look low confidence
Cause not clear from logs and recent changes; this looks like an existing error that moved, not a new one.
execsvc logged 'no result from branch evaluation' 101× at execlib/branchV2.go:91 from 18:42 to 19:13 UTC, plus 'evalResult is not a bool' 38× at branchV2.go:96 from 18:51 UTC.
The same two messages came from branchV2.go:77 (106×, seen since Nov 7) and :82 (4×) until 18:31 UTC, then stopped. The activityV2.go:882 wrapper lines also stopped at 18:31.
That 14-line shift suggests a new execsvc build rolled out around 18:31–18:42, but the only recorded deploy is [email protected] at 05:11 UTC, 810 min earlier. Code and commits could not be read because the GitHub budget ran out.
Next step: find what execsvc version started running around 18:31–18:42 UTC, diff branchV2.go, and check why branch evaluation returns no result (failing since Nov 7). · its evidence →

critical Error never seen before master last seen · since · 3×

execsvc: a new error shape, 5× in the last hour, never seen in 90 days (failed to run cypher in transaction — Neo4jError: Neo.TransientError.T)

execsvc logged “failed to run cypher in transaction — Neo4jError: Neo.TransientError.Transaction.DeadlockDetected (ForsetiClient[transactionId=?, clientId=?] can't acquire Excl” 5 times in the last hour and never before in 90 days on master (first seen 2026-11-16 16:49 UTC, raised at tildadb/util.go:1004). Example trace: f91346437409440b0cc533d78b9297f2.

First look low confidence
Cause not clear from logs and recent changes; execsvc hit a burst of Neo4j DeadlockDetected errors, all at 16:49 UTC: 5× 'failed to run cypher in transaction' (tildadb/util.go:1004), plus 14× util.go:993, 12× bulk-write (doclib/createDocument.go:1015), 12+5× document copy (doclib/copyDocument.go:74) and 5× scope binders (doclib/context.go:335).
The errors have not recurred since then (last seen 16:49 per error lines).
The 5 deploys in the prior 24 h were config-only autosync; the closest was 15:39 (69 min before), with no image or commit change and nothing tying it to these files.
The util.go:993 deadlock shape has existed since 2026-11-15, so deadlocks are not new; what is new is the document-copy write path hitting them.
Next step: open the example trace and check which concurrent document copies (bulk doc write vs. context scope binders) took ExclusiveLocks on the same nodes, then confirm the copy path retries on TransientError. · its evidence →

critical Error never seen before master last seen · since · 3×

execsvc: a new error shape, 5× in the last hour, never seen in 90 days (failed to add document to scope binders — could not write context scop)

execsvc logged “failed to add document to scope binders — could not write context scope binders: Neo4jError: Neo.TransientError.Transaction.DeadlockDetected (ForsetiClient[tran” 5 times in the last hour and never before in 90 days on master (first seen 2026-11-16 16:49 UTC, raised at doclib/context.go:335). Example trace: f91346437409440b0cc533d78b9297f2.

First look low confidence
Cause not clear from logs and recent changes; execsvc hit a burst of Neo4j DeadlockDetected errors at 16:49 UTC while copying documents.
This finding is 5× 'failed to add document to scope binders' (doclib/context.go:335). Same burst: 14× tildadb/util.go:993, 12× bulk-write (doclib/createDocument.go:1015), 12× + 5× copy failures (doclib/copyDocument.go:74).
All lines are from 16:49 UTC only. The util.go:993 deadlock shape was already seen on 2026-11-15, so deadlocks are not new.
Checked: the 5 deploys in the prior 24 h were config-only autosyncs (latest 15:39, 69 min before) with no new commits. Nothing links them to this burst.
Next step: open the example trace to see which concurrent document copies collided on the same nodes. Check whether the copy path retries Neo4j TransientErrors. · its evidence →

critical Error never seen before master last seen · since · 3×

execsvc: a new error shape, 5× in the last hour, never seen in 90 days (failed to create document copy — could not add document context: could)

execsvc logged “failed to create document copy — could not add document context: could not write context scope binders: Neo4jError: Neo.TransientError.Transaction.DeadlockDetec” 5 times in the last hour and never before in 90 days on master (first seen 2026-11-16 16:49 UTC, raised at doclib/copyDocument.go:74). Example trace: f91346437409440b0cc533d78b9297f2.

First look low confidence
Cause not clear from logs and recent changes; execsvc document copies failed on Neo4j DeadlockDetected in one burst at 16:49 UTC (5x here at doclib/copyDocument.go:74 via context.go:335 scope binders).
The same burst produced 12x bulk-write deadlocks (createDocument.go:1015, copyDocument.go:74), 14x at tildadb/util.go:993 and 5x at util.go:1004, opening 4 sibling findings.
The deadlock shape at util.go:993 was already seen on 2026-11-15 07:38, so deadlocks predate this burst; the new part is which callers hit them.
Checked: the 5 deploys in the prior 24 h were autosync config-only changes (the latest at 15:39, 69 min before onset), with no image or commit change and nothing tying them to these writes.
Next step: open the example trace to see which concurrent copy transactions took the locks on the scope binders.
Also check whether copyDocument retries Neo.TransientError. · its evidence →

critical Error never seen before master last seen · since · 3×

execsvc: a new error shape, 12× in the last hour, never seen in 90 days (failed to bulk-write document nodes — Neo4jError: Neo.TransientError.T)

execsvc logged “failed to bulk-write document nodes — Neo4jError: Neo.TransientError.Transaction.DeadlockDetected (ForsetiClient[transactionId=?, clientId=?] can't acquire Excl” 12 times in the last hour and never before in 90 days on master (first seen 2026-11-16 16:49 UTC, raised at doclib/createDocument.go:1015). Example trace: f91346437409440b0cc533d78b9297f2.

First look low confidence
Cause not clear from logs and recent changes; execsvc hit a burst of Neo4j DeadlockDetected errors in one minute, 16:49 UTC on 2026-11-16.
Counts: 12x 'failed to bulk-write document nodes' (doclib/createDocument.go:1015), 12x copy failures (doclib/copyDocument.go:74), 14x tildadb/util.go:993, 5x scope binders (doclib/context.go:335).
All come from document-copy writes contending for the same lock; findings 17467/17469/17470/17471 are the same burst.
Checked: 5 autosync config-only deploys in the prior 24 h, the last at 15:39 UTC (70 min before), with no commits and nothing tying them to these writes. The util.go:993 deadlock shape was already seen on 2026-11-15 07:38.
Next step: open the example trace to see which concurrent copy requests collided, then check whether createDocument.go:1015 retries Neo.TransientError. · its evidence →

critical Error never seen before master last seen · since · 3×

execsvc: a new error shape, 12× in the last hour, never seen in 90 days (failed to create document copy — could not write document node and cor)

execsvc logged “failed to create document copy — could not write document node and core relationships: bulk doc write: Neo4jError: Neo.TransientError.Transaction.DeadlockDetect” 12 times in the last hour and never before in 90 days on master (first seen 2026-11-16 16:49 UTC, raised at doclib/copyDocument.go:74). Example trace: f91346437409440b0cc533d78b9297f2.

First look low confidence
Cause not clear from logs and recent changes; execsvc document copies failed on Neo4j DeadlockDetected in one burst at 16:49 UTC, last seen 16:55 UTC.
Counts: 12× copyDocument.go:74 (bulk doc write), 5× copyDocument.go:74 (context scope binders), 14× tildadb/util.go:993.
The util.go:993 deadlock shape has existed since 2026-11-15 07:38, so lock contention is not new; only the document-copy paths are.
Checked: 5 autosync deploys in the prior 24 h were config-only (no image or commit change, the last one 69 min before onset), with nothing linking them to the error. This finding has no earlier history.
Next step: open the example trace and find which concurrent transaction held the ExclusiveLock during the 16:49 copy burst.
Also check whether doclib retries Neo.TransientError, since this error class is retryable. · its evidence →

critical Error never seen before production resolved last seen · since · 3×

execsvc: a new error shape, 75× in the last hour, never seen in 90 days (evalResult is not a bool)

execsvc logged “evalResult is not a bool” 75 times in the last hour and never before in 90 days on production (first seen 2026-11-16 16:26 UTC, raised at execlib/branchV2.go:82). Example trace: b4a46af1511b00609ba4b32b4050ff0c.

First look low confidence
Cause not clear from logs and recent changes; execsvc logged 'evalResult is not a bool' at execlib/branchV2.go:82 68× (16:26–16:53 UTC), each one surfacing as a failed auto query at execution/activityV2.go:882.
It started at 16:26 UTC together with 350× 'no result from branch evaluation' (branchV2.go:77). That error has existed since 2026-11-11 but now also breaks auto queries.
The only execsvc deploy in 24 h was config revision 6db9855a7d34 by [email protected] at 05:10 UTC, 676 min before onset. Nothing links it to the error.
Code and commits were not read because the GitHub call budget ran out. A NodeNotReady and Neo4j transient errors at the same time are not linked either.
Next step: open the example trace to find which protocol's branch expression returns a non-bool, then read branchV2.go:77-82. · its evidence →

critical Error never seen before production resolved last seen · since · 3×

execsvc: a new error shape, 75× in the last hour, never seen in 90 days (failed to evaluate and raise auto query for participant question — eva)

execsvc logged “failed to evaluate and raise auto query for participant question — evalResult is not a bool” 75 times in the last hour and never before in 90 days on production (first seen 2026-11-16 16:26 UTC, raised at execution/activityV2.go:882). Example trace: b4a46af1511b00609ba4b32b4050ff0c.

First look low confidence
Cause not clear from logs and recent changes; execsvc raised 68 'evalResult is not a bool' errors (execlib/branchV2.go:82) between 16:26 and 16:53 UTC on 2026-11-16, each surfacing as 'failed to evaluate and raise auto query for participant question' at execution/activityV2.go:882.
At the same moment, 350 'no result from branch evaluation' errors (branchV2.go:77) began failing the same auto-query path. That shape existed since 2026-11-11, but the wrapped failure is new.
Checked: the only execsvc deploy was at 05:10 UTC by [email protected] (config 6db9855a7d34), 11 h before onset, with nothing linking it. Code and commits could not be read because the GitHub budget ran out.
The sudden start at 16:26 with no deploy points to data such as auto-query or branch expressions, for example protocolId=77bcfccb-cc59-4aea-bff8-0991eaa0c788.
Next step: open the example trace, see what type the branch expression returned, and check whether that protocol's auto-query rules changed near 16:26. Then read branchV2.go:77-82. · its evidence →

critical Error never seen before production resolved last seen · since · 3×

execsvc: a new error shape, 473× in the last hour, never seen in 90 days (failed to evaluate and raise auto query for participant question — no )

execsvc logged “failed to evaluate and raise auto query for participant question — no result from branch evaluation for branch, protocolId=? activityRefId=?” 473 times in the last hour and never before in 90 days on production (first seen 2026-11-16 16:26 UTC, raised at execution/activityV2.go:882). Example trace: b4a46af1511b00609ba4b32b4050ff0c.

First look low confidence
Cause not clear from logs and recent changes; execsvc logged 203 'failed to evaluate and raise auto query for participant question — no result from branch evaluation' errors at execution/activityV2.go:882 between 16:26 and 16:31 UTC on 2026-11-16, a shape never seen in 90 days.
The inner error 'no result from branch evaluation' (execlib/branchV2.go:77, 203×) has existed since 2026-11-11. The wrapping auto-query path is new from 16:26, along with 23× 'evalResult is not a bool' (branchV2.go:82).
The only execsvc deploy was a config revision 6db9855a7d34 by [email protected] at 05:10 UTC, 676 min before onset. Nothing links it to these errors.
Code and commits could not be read because the GitHub call budget ran out.
Next step: open the example trace and check whether all failures share one protocolId (e.g. 77bcfccb-…), which would point to a newly published protocol's branch/auto-query rule rather than a code change. · its evidence →

warning Out-of-memory kills automation last seen 1d ago · since 1d ago · 6×

the kernel OOM-killed “obs-serve” 2× in 30 min on gk3-automation-nap-1p2bubxp-41aa9f63-gpdz

Out-of-memory kills: the kernel OOM-killed “obs-serve” 2× in 30 min on gk3-automation-nap-1p2bubxp-41aa9f63-gpdz. Facts: node gk3-automation-nap-1p2bubxp-41aa9f63-gpdz, process obs-serve, kills_30m 2, last_message Memory cgroup out of memory: Killed process 2610801 (obs-serve) total-vm:2657908kB, anon-rss:1037496kB, file-rss:8984kB, shmem-rss:0kB, UID:1000 pgtables:2696kB oom_score_adj:985, note the kernel names the process and node, not the pod; likely_workloads are the ones recently placed on that node.

critical Pod cannot start master last seen 1d ago · since 1d ago · 14×

x-ray-argocd-collector (x-ray) image pull failing for at least 93 min — 2 pods

Pod cannot start: x-ray-argocd-collector (x-ray) image pull failing for at least 93 min — 2 pods. Facts: namespace x-ray, workload x-ray-argocd-collector, why image pull failing, pending_minutes 93, since 1794906422.

First look medium confidence
Pod x-ray-argocd-collector-29914927-d557w (namespace x-ray) cannot pull image us-central1-docker.pkg.dev/tilda-dev-325721/artifacts/services/x-ray:3.7.1-56fc8bc: 51 events, Back-off pulling image, pending 35 min.
Pending since about 06:07 UTC; the finding opened at 06:29 UTC.
Likely cause: ArgoCD autosync of x-ray-services-master at 06:02 UTC, config revision 556a243c8ac9, about 5 min before the pod got stuck. It is the only x-ray change in the 24 h before.
The bundle does not show the image tag that sync set, so the link rests on timing and the matching app.
Next step: check that commit 556a243c8ac9 sets tag 3.7.1-56fc8bc and that this tag exists in Artifact Registry. If the tag is missing, push the image or revert the commit.
Suspect: x-ray-services-master sync 556a243c8ac9 (autosync) · its evidence →

feedsvc GET /studies/:id/ 6 targets worst: critical feed/news: permission denied — 166 of 174 calls (95 %; 403×166) in the last 15 min — 1033× its usual 0.2, feed/news: 4xx — 30 of 30 calls (100 %; other 4xx×30) in the last 15 min — 130× its usual 0.2, feed/week: permission denied — 23 of 24 calls (96 %; 403×23) in the last 15 min — it rejects nearly every call, feed/week: permission denied — 33 of 52 calls (63 %; 403×33) in the last 15 min — 20× its usual 1.6, feed: permission denied — 52 of 64 calls (81 %; 403×52) in the last 15 min — 22× its usual 2.3, feed: permission denied — 108 of 108 calls (100 %; 403×108) in the last 15 min — it rejects nearly every call
critical 4xx spike (403/401/404…) master last seen · since · 3×

feedsvc GET /studies/:id/feed/news: permission denied — 166 of 174 calls (95 %; 403×166) in the last 15 min — 1033× its usual 0.2

4xx spike (403/401/404…): feedsvc GET /studies/:id/feed/news: permission denied — 166 of 174 calls (95 %; 403×166) in the last 15 min — 1033× its usual 0.2. Facts: endpoint GET /studies/:id/feed/news, service feedsvc, calls 174, c4xx 166, share_pct 95.40229885057471, c401 0.

First look low confidence
Cause not clear from logs and recent changes; feedsvc GET /studies/:id/feed/news returned 403 on 44 of 47 calls (94 %) in 15 min, against a usual 0.01 per window. Onset was 2026-11-17 05:20 UTC, and the first failing request in the log is at 05:27. Every sampled 403 comes from one org, "Teammates Demo Sandbox", and returns fast (13–21 ms). That points to a permission check on that org or its users · its evidence →

warning 4xx spike (403/401/404…) automation resolved last seen · since · 6×

feedsvc GET /studies/:id/feed/news: 4xx — 30 of 30 calls (100 %; other 4xx×30) in the last 15 min — 130× its usual 0.2

4xx spike (403/401/404…): feedsvc GET /studies/:id/feed/news: 4xx — 30 of 30 calls (100 %; other 4xx×30) in the last 15 min — 130× its usual 0.2. Facts: endpoint GET /studies/:id/feed/news, service feedsvc, calls 30, c4xx 30, share_pct 100.0, c401 0.

critical 4xx spike (403/401/404…) master resolved last seen · since · 57×

feedsvc GET /studies/:id/feed/week: permission denied — 23 of 24 calls (96 %; 403×23) in the last 15 min — it rejects nearly every call

4xx spike (403/401/404…): feedsvc GET /studies/:id/feed/week: permission denied — 23 of 24 calls (96 %; 403×23) in the last 15 min — it rejects nearly every call. Facts: endpoint GET /studies/:id/feed/week, service feedsvc, calls 24, c4xx 23, share_pct 95.83333333333333, c401 0.

First look low confidence
Cause not clear from logs and recent changes; feedsvc GET /studies/:id/feed/week returned 403 on 580 of 580 calls (100%) in 15 min, against a usual 2.3.
Onset 2026-11-16 07:10 UTC. This finding has recurred since 2026-11-12 18:03 (5 earlier episodes), and GET /studies/:id/feed is failing the same way (872 of 874 403s).
All 8 sampled 403s come from org "Tilda CRO" and return in 15-25 ms, so this is a fast permission rejection, not a crash. No feedsvc error lines are stored.
The only deploys were config-only autosyncs (81443681f910 at 01:17, c88737eb8000 at 22:28 the day before), 6+ hours before onset, with no commits. Nothing links them to the 403s.
Next step: open one failing trace to find which permission check rejects Tilda CRO. Then check that org's study access, or whether a client is hammering studies it cannot see. · its evidence →

critical 4xx spike (403/401/404…) automation resolved last seen · since · 13×

feedsvc GET /studies/:id/feed/week: permission denied — 33 of 52 calls (63 %; 403×33) in the last 15 min — 20× its usual 1.6

4xx spike (403/401/404…): feedsvc GET /studies/:id/feed/week: permission denied — 33 of 52 calls (63 %; 403×33) in the last 15 min — 20× its usual 1.6. Facts: endpoint GET /studies/:id/feed/week, service feedsvc, calls 52, c4xx 33, share_pct 63.46153846153846, c401 0.

critical 4xx spike (403/401/404…) automation resolved last seen · since · 22×

feedsvc GET /studies/:id/feed: permission denied — 52 of 64 calls (81 %; 403×52) in the last 15 min — 22× its usual 2.3

4xx spike (403/401/404…): feedsvc GET /studies/:id/feed: permission denied — 52 of 64 calls (81 %; 403×52) in the last 15 min — 22× its usual 2.3. Facts: endpoint GET /studies/:id/feed, service feedsvc, calls 64, c4xx 52, share_pct 81.25, c401 0.

critical 4xx spike (403/401/404…) master resolved last seen · since · 59×

feedsvc GET /studies/:id/feed: permission denied — 108 of 108 calls (100 %; 403×108) in the last 15 min — it rejects nearly every call

4xx spike (403/401/404…): feedsvc GET /studies/:id/feed: permission denied — 108 of 108 calls (100 %; 403×108) in the last 15 min — it rejects nearly every call. Facts: endpoint GET /studies/:id/feed, service feedsvc, calls 108, c4xx 108, share_pct 100.0, c401 0.

First look low confidence
Cause not clear from logs and recent changes; feedsvc GET /studies/:id/feed returned 403 on 872 of 874 calls (99.8 %) in 15 min, 329x its usual 2.7.
Onset 07:10 UTC on 2026-11-16; the first 403 in the request log is at 07:31 UTC. GET /studies/:id/feed/week shows the same spike (580 of 580 are 403).
All 8 sampled 403s are fast (17-22 ms) and come from org "Tilda CRO", so a caller is hitting the feed many times without permission.
This is a recurring finding: 5 earlier episodes since 2026-11-12, all cleared on their own.
Checked: no feedsvc error lines stored, and the only 2 deploys were config-only autosyncs, 352 and 521 min before onset, with no link to the 403s.
Next step: open a trace and see which user or client in "Tilda CRO" is calling, then check its access to those studies. · its evidence →

warning Background job failing production last seen 1d ago · since 1d ago · 8×

execsvc.auto-query-reeval-sweep: 11 failed runs in the last 6 h — no result from branch evaluation for branch, protocolId=? activityRefId=?

The background job execsvc.auto-query-reeval-sweep on production failed 11 runs recently (425 failure lines; lines within one run count once), detected via its ERROR-severity log lines. A job that logs its failures below ERROR severity — a WARNING carrying an error field — is invisible to the novel-shape detector and to the page's own failed count; this detector reads the error field itself, which is why the finding now appears. The sample failure is in the facts.

warning Background job failing delta resolved last seen 1d ago · since 2d ago · 17×

execsvc.auto-query-reeval-sweep: 7 failed runs in the last 6 h — no result from branch evaluation for branch, protocolId=? activityRefId=?

The background job execsvc.auto-query-reeval-sweep on delta failed 7 runs recently (99 failure lines; lines within one run count once), detected via its structured error/exception field (below ERROR severity). A job that logs its failures below ERROR severity — a WARNING carrying an error field — is invisible to the novel-shape detector and to the page's own failed count; this detector reads the error field itself, which is why the finding now appears. The sample failure is in the facts.