Xray
1h 6h 24h 48h 3d 5d 7d 14d 30d ‹
Last 1 hour▾
Ends now. Anything up to this cluster's 30-day retention.
Start and end times
Cancel
Interpreted in UTC (change it at the right of this bar). Earliest available: 2026-10-19 13:51:55.
› →|
updated 1d ago
Sample data. This database was generated by scripts/seed_dev_db.py — every service, deploy, person and figure on this page is synthetic. Names are sample.*@seed.invalid and image tags are prefixed SEED-. Nothing here happened.
← findings

execsvc server-error rate 89.19% vs a normal 0.00% ± 1.00% for Tue 04:00 UTC (3 of the last 4 checks) — mostly POST /api/trialexec/v2/study/:id/activity/status (787× 5xx)

critical Error-rate anomaly master resolved impact 62 Owner: Madhava Mukkamala (backup: Shashank Agrawal) opened 1d ago · last observed 1d ago · observed 1× · #17504 · 8593c9721f3d6c05f491

What happened ai

execsvc shows an 89.2% error rate on master, significantly above the normal 0.0% ± 1.0% baseline for this weekly time slot. The anomaly spans 3 of the last 4 five-minute buckets (z = 89.2), based on 4 weeks of historical data. The failing endpoint is POST /api/trialexec/v2/study/:id/activity/status, which experienced 787 errors out of 787 total calls. User impact depends on endpoint traffic distribution.

Error percentage far above its hour-of-week normal.

First look — investigated before anyone looked

Cause not clear from logs and recent changes; POST /api/trialexec/v2/study/:id/activity/status is failing 787 of 787 calls, putting execsvc 5xx at 89.19% against a normal 0.00%, since 2026-11-17 04:05 UTC.
All 700 error lines are one failure: "could not match participant question to activity graph" (execution/activity_details_v2.go:2007), sample questionId="5571182f-6c21-491e-b8c6-51fce3166e1e" with empty refId, variableName and title.
Checked: the 5 execsvc deploys in the prior 24 h were config-only autosyncs (last d8d21151cd28 at 01:33 UTC, 151 min before onset) and brought no commits, so nothing links them to the failure.
The empty refId/variableName/title points at study or activity-graph data rather than code; a smaller episode on 2026-11-09 (finding 16947) hit the same endpoint and was resolved.
Next step: open the trace, find the study whose question 5571182f-... is missing from its activity graph, and check what changed in that study's design around 04:05 UTC.

low confidence

claude-opus-5-5 · 1d ago · $0.05 — every link above is evidence it read; nothing else was claimed.

The facts

measureerr5_pct
unit%
value89.19
expected_median0.00
expected_mad1.00
z89.19
relinf
buckets_hit3
n24
weeks4
slotTue 04:00 UTC
bucket2026-11-17 04:15:00 UTC (1d ago)
failing_endpoints{'service': 'execsvc', 'endpoint': 'POST /api/trialexec/v2/study/:id/activity/status', 'c5xx': 787, 'calls': 787}
discriminatorerr5_pct
window2026-11-17 04:05:00 UTC → 2026-11-17 04:25:00 UTC

What is failing in execsvc — the bad events, decomposed

“Bad events” are HTTP 5xx answers from execsvc. In this finding's window (2026-11-17 04:05:00 UTC → 2026-11-17 04:25:00 UTC) there were 787; every line links to the grouped errors behind it.

POST /api/trialexec/v2/study/:id/activity/status 787 bad of 787 calls · 100% of the service's 5xx

The error shapes behind them

787× Request completed — 500 · failed to fetch activity status: could not match participant question to activity graph: quest… since 2026-10-16 12:11:42 UTC
787× Failed to fetch activity status — could not match participant question to activity graph: questionId="b3228925-d16e-41cf… since 2026-11-04 11:18:42 UTC
787× could not match participant questionnaire/question/table/doc-upload to activity graph since 2026-11-17 04:05:17 UTC
30× No cached node found for activation chain ref node since 2026-11-12 13:43:21 UTC
29× no result from branch evaluation since 2026-11-13 15:11:12 UTC

All execsvc errors, grouped → · service page · its logs

The story — No execsvc deploy or hand change is recorded in the hour before this opened. Everything below is execsvc only.

History

  • 2026-11-17 04:45:57 UTCopened x-ray · severity=critical title=execsvc server-error rate 89.19% vs a normal 0.00% ± 1.00% for Tue 04:00 UTC (3 of the last 4 checks) — mostly POST /api/trialexec/v2/study/:id/activity/status (787× 5xx) finding={'id': 17504, 'fingerprint': '8593c9721f3d6c05f491', 'kind': 'anomaly.errors', 'severity': 'critical', 'cluster': 'master', 'subject': 'execsvc', 'title': 'execsvc server-error rate 89.19% vs a normal 0.00% ± 1.00% for Tue 04:00 UTC (3 of the last 4 checks) — mostly POST /api/trialexec/v2/study/:id/activity/status (787× 5xx)', 'facts': {'measure': 'err5_pct', 'unit': '%', 'value': 89.1891891891892, 'expected_median': 0.0, 'expected_mad': 1.0, 'z': 89.1891891891892, 'rel': inf, 'buckets_hit': 3, 'n': 24, 'weeks': 4, 'slot': 'Tue 04:00 UTC', 'bucket': 1794888900, 'window': [1794888300, 1794889500], 'recent_deploys': [], 'failing_endpoints': [{'service': 'execsvc', 'endpoint': 'POST /api/trialexec/v2/study/:id/activity/status', 'c5xx': 787, 'calls': 787}], 'discriminator': 'err5_pct'}, 'evidence': [{'label': 'Service', 'href': '/service/execsvc?cluster=master&from=1794866700&to=1794889500'}, {'label': 'Fleet vs normal', 'href': '/fleet?cluster=master&from=1794866700&to=1794889500'}], 'provenance': {'source': 'rollup', 'collected_at': 1794890117, 'age_seconds': 650, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 61.9, 'status': 'open', 'first_seen': 1794890757, 'last_seen': 1794890757, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 1, 'narrative': None, 'narrative_kind': None, 'discriminator': 'err5_pct', 'level': 'high', 'told_level': None, 'episode_at': 1794890757, 'chat_thread': None, 'kind_label': 'Error-rate anomaly', 'group': 'Anomalies', 'age_s': 10, 'stale_s': 10}
  • 2026-11-17 04:46:53 UTCnotified x-ray · alert_id=3 channel=in-app status=delivered send_id=52005
  • 2026-11-17 04:46:53 UTCnotified x-ray · alert_id=3 channel=dev-notifications status=sent send_id=52005
  • 2026-11-17 05:34:42 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 17504, 'fingerprint': '8593c9721f3d6c05f491', 'kind': 'anomaly.errors', 'severity': 'critical', 'cluster': 'master', 'subject': 'execsvc', 'title': 'execsvc server-error rate 89.19% vs a normal 0.00% ± 1.00% for Tue 04:00 UTC (3 of the last 4 checks) — mostly POST /api/trialexec/v2/study/:id/activity/status (787× 5xx)', 'facts': {'measure': 'err5_pct', 'unit': '%', 'value': 89.1891891891892, 'expected_median': 0.0, 'expected_mad': 1.0, 'z': 89.1891891891892, 'rel': inf, 'buckets_hit': 3, 'n': 24, 'weeks': 4, 'slot': 'Tue 04:00 UTC', 'bucket': 1794888900, 'window': [1794888300, 1794889500], 'recent_deploys': [], 'failing_endpoints': [{'service': 'execsvc', 'endpoint': 'POST /api/trialexec/v2/study/:id/activity/status', 'c5xx': 787, 'calls': 787}], 'discriminator': 'err5_pct'}, 'evidence': [{'label': 'Service', 'href': '/service/execsvc?cluster=master&from=1794866700&to=1794889500'}, {'label': 'Fleet vs normal', 'href': '/fleet?cluster=master&from=1794866700&to=1794889500'}], 'provenance': {'source': 'rollup', 'collected_at': 1794890117, 'age_seconds': 650, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 61.9, 'status': 'resolved', 'first_seen': 1794890757, 'last_seen': 1794890757, 'resolved_at': 1794893682, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 1, 'narrative': 'execsvc shows an 89.2% error rate on master, significantly above the normal 0.0% ± 1.0% baseline for this weekly time slot. The anomaly spans 3 of the last 4 five-minute buckets (z = 89.2), based on 4 weeks of historical data. The failing endpoint is POST /api/trialexec/v2/study/:id/activity/status, which experienced 787 errors out of 787 total calls. User impact depends on endpoint traffic distribution.', 'narrative_kind': 'ai', 'discriminator': 'err5_pct', 'level': 'high', 'told_level': None, 'episode_at': 1794890757, 'chat_thread': None, 'kind_label': 'Error-rate anomaly', 'group': 'Anomalies', 'age_s': 2934, 'stale_s': 2934}

Notifications

  • 2026-11-17 04:46:53 UTCopenedin-app · delivered
  • 2026-11-17 04:46:53 UTCopeneddev-notifications · sent

Check it yourself

Every link carries the finding's own time window, so it shows tomorrow what it shows now.

Actions

Sign in (IAP) to acknowledge, snooze or resolve.

Where the numbers come from

sourcerollup
collected_at1794890117
age_seconds650
staleFalse
shownNone
totalNone
truncatedFalse
coverage_fromNone
retain_daysNone
noteNone
detectorseasonal