Xray
daily rollups 1h 6h 24h 48h 3d 5d 7d 14d 30d ‹
Last 7 days▾
Ends now. Anything up to this cluster's 30-day retention.
Start and end times
Cancel
Interpreted in UTC (change it at the right of this bar). Earliest available: 2026-10-19 14:02:47.
› →|
updated 1d ago
Sample data. This database was generated by scripts/seed_dev_db.py — every service, deploy, person and figure on this page is synthetic. Names are sample.*@seed.invalid and image tags are prefixed SEED-. Nothing here happened.

Findings 10

stored age unknown all 59detectors over stored rollups and live sources

What X-ray is prepared to defend about master right now — each one a claim with its numbers, the evidence to check it, and what normal looks like. Detectors last ran never. ?A finding is opened by a detector — a statistical test over the stored rollups, or a live check of a source — and stays ONE row while the condition persists: it updates, escalates, and resolves itself fifteen minutes after the detector stops seeing it. Acknowledge to say you have it; snooze to hide it for a while; nothing here is a threshold somebody configured.

How to read this page — how findings work

Every card is a claim X-ray can back with numbers you can click through to. Findings rank by severity first, then by an impact score (severity band + the subject's share of the cluster's traffic + how long it has been open), so the top of the page is the thing to look at first.

anomaly
Three of the last four five-minute buckets far outside what this hour of THIS weekday normally sees — median ± MAD over the last 2–8 weeks, not a threshold.
outlier
A service unlike the rest of the fleet on one measure, over six hours. Compares with peers, not with its own history — the two catch different things.
budget
Burn rate against an error objective derived from the service's own last week, and the date the 30-day budget runs out if the trend holds.
resolves itself
A finding a healthy detector run no longer observes resolves after fifteen minutes. A detector whose SOURCE fails resolves nothing — it says so on the Detectors page instead.

What this page will not tell you

  • That a finding is a diagnosis. It is a measured departure from normal with adjacent changes listed; the narrative says 'adjacency, not proof' and means it.
  • Anything about hours the ingester did not cover. A bucket with no rows is a gap, not a quiet period, and the baselines skip it.
  • Anything on the first day for the novel-error detector: its 90-day memory learns silently for 24 hours before it will call anything new.
show active resolved all severity everything warning + critical only clusters
master
just master · nothing ticked means master, the cluster selected up top
or just close the list
group all AnomaliesReliabilityNeo4jChangesJourneyX-rayWeaviateAIPodsJobsPlatformErrorsCustomCostCapacity + 4 fleet-wide 12 critical · 14 warning · 13 info detectors
warning 4xx spike (403/401/404…) master Owner: Suresh Boddapati (backup: Madhava Mukkamala) resolved last seen 1d ago · since 7d ago · 16×

studiosvc GET /api/studio/protocols/:id/visits/:id: 4xx — 37 of 210 calls (18 %; other 4xx×37) in the last 15 min — 34× its usual 1.1

4xx spike (403/401/404…): studiosvc GET /api/studio/protocols/:id/visits/:id: 4xx — 37 of 210 calls (18 %; other 4xx×37) in the last 15 min — 34× its usual 1.1. Facts: endpoint GET /api/studio/protocols/:id/visits/:id, service studiosvc, calls 210, c4xx 37, share_pct 17.61904761904762, c401 0.

critical 4xx spike (403/401/404…) master Owner: Suresh Boddapati (backup: Madhava Mukkamala) resolved last seen 1d ago · since 15d ago · 25×

studiosvc GET /api/studio/events: permission denied — 36 of 49 calls (73 %; 401×36) in the last 15 min — 44× its usual 0.8

4xx spike (403/401/404…): studiosvc GET /api/studio/events: permission denied — 36 of 49 calls (73 %; 401×36) in the last 15 min — 44× its usual 0.8. Facts: endpoint GET /api/studio/events, service studiosvc, calls 49, c4xx 36, share_pct 73.46938775510205, c401 36.

First look low confidence
Cause not clear from logs and recent changes; studiosvc GET /api/studio/events returned 401 on 21 of 49 calls (43 %) in 15 min, 28× its usual 0.8 per window.
Onset 15:25 UTC on 2026-11-16; the first 401 in the request log is at 15:28 UTC. This finding has recurred many times since 2026-11-03, and earlier episodes cleared on their own in about 20–30 min.
Checked: 0 stored error lines, 4 config-only autosync deploys (the last at 08:05 UTC, 7 h before) with no image or commit change, and the SSE handler. The handler returns 401 when authlib.GetRequestContext fails and logs a warning, but no such lines are stored.
Next step: open one of the 401 traces to see whether the auth context or Organization header is missing, and compare with the open sibling finding on /api/studio/triggers (100 % 401). · its evidence →

info Traffic anomaly master Owner: Suresh Boddapati (backup: Madhava Mukkamala) resolved last seen 1d ago · since 68d ago · 309×

studiosvc request rate 73.4 req/min vs a normal 2.6 req/min ± 3.9 req/min for Tue 10:00 UTC (3 of the last 4 checks)

studiosvc's request rate is 73.4 req/min on master, above a normal of 2.6 req/min ± 3.9 req/min for this hour of the week (3 of the last 4 five-minute buckets, z = 18.4, 4 weeks of history). automated changed studiosvc 56 minutes earlier — adjacency, not proof. Whether users noticed depends on which endpoints carried it — the service page shows the split.

warning Error budget burning master Owner: Madhava Mukkamala (backup: Shashank Agrawal) resolved last seen 1d ago · since 83d ago · 233×

execsvc is burning its error budget at 0.0× (1 h) / 8.3× (6 h) against a 95.00% objective

execsvc is spending its error budget 0.0× faster than the objective allows over the last hour (8.3× over six hours). The objective is 95.0%, derived — a baseline, not a promise. 787 bad events against 1,905 calls in the window.

info N+1 fan-out master Owner: Prasad Limaye (backup: Harish Kumar) resolved last seen 1d ago · since 83d ago · 177×

docsvc runs 45 foreground Neo4j queries per request (17,242 queries / 384 calls, last hour)

docsvc on master averaged 44.9 foreground Neo4j queries per API request over the last hour (17,242 queries for 384 calls). Dozens of queries per request is the N+1 signature: a round-trip per row instead of one query. The Journey page shows which screens pay for it, request by request.

critical 4xx spike (403/401/404…) master resolved last seen 1d ago · since 5d ago · 57×

feedsvc GET /studies/:id/feed/week: permission denied — 23 of 24 calls (96 %; 403×23) in the last 15 min — it rejects nearly every call

4xx spike (403/401/404…): feedsvc GET /studies/:id/feed/week: permission denied — 23 of 24 calls (96 %; 403×23) in the last 15 min — it rejects nearly every call. Facts: endpoint GET /studies/:id/feed/week, service feedsvc, calls 24, c4xx 23, share_pct 95.83333333333333, c401 0.

First look low confidence
Cause not clear from logs and recent changes; feedsvc GET /studies/:id/feed/week returned 403 on 580 of 580 calls (100%) in 15 min, against a usual 2.3.
Onset 2026-11-16 07:10 UTC. This finding has recurred since 2026-11-12 18:03 (5 earlier episodes), and GET /studies/:id/feed is failing the same way (872 of 874 403s).
All 8 sampled 403s come from org "Tilda CRO" and return in 15-25 ms, so this is a fast permission rejection, not a crash. No feedsvc error lines are stored.
The only deploys were config-only autosyncs (81443681f910 at 01:17, c88737eb8000 at 22:28 the day before), 6+ hours before onset, with no commits. Nothing links them to the 403s.
Next step: open one failing trace to find which permission check rejects Tilda CRO. Then check that org's study access, or whether a client is hammering studies it cannot see. · its evidence →

critical 4xx spike (403/401/404…) master resolved last seen 1d ago · since 5d ago · 59×

feedsvc GET /studies/:id/feed: permission denied — 108 of 108 calls (100 %; 403×108) in the last 15 min — it rejects nearly every call

4xx spike (403/401/404…): feedsvc GET /studies/:id/feed: permission denied — 108 of 108 calls (100 %; 403×108) in the last 15 min — it rejects nearly every call. Facts: endpoint GET /studies/:id/feed, service feedsvc, calls 108, c4xx 108, share_pct 100.0, c401 0.

First look low confidence
Cause not clear from logs and recent changes; feedsvc GET /studies/:id/feed returned 403 on 872 of 874 calls (99.8 %) in 15 min, 329x its usual 2.7.
Onset 07:10 UTC on 2026-11-16; the first 403 in the request log is at 07:31 UTC. GET /studies/:id/feed/week shows the same spike (580 of 580 are 403).
All 8 sampled 403s are fast (17-22 ms) and come from org "Tilda CRO", so a caller is hitting the feed many times without permission.
This is a recurring finding: 5 earlier episodes since 2026-11-12, all cleared on their own.
Checked: no feedsvc error lines stored, and the only 2 deploys were config-only autosyncs, 352 and 521 min before onset, with no link to the 403s.
Next step: open a trace and see which user or client in "Tilda CRO" is calling, then check its access to those studies. · its evidence →

info N+1 fan-out master Owner: Madhava Mukkamala (backup: Shashank Agrawal) resolved last seen 1d ago · since 21d ago · 23×

execsvc runs 26 foreground Neo4j queries per request (3,407 queries / 130 calls, last hour)

execsvc on master averaged 26.2 foreground Neo4j queries per API request over the last hour (3,407 queries for 130 calls). Dozens of queries per request is the N+1 signature: a round-trip per row instead of one query. The Journey page shows which screens pay for it, request by request.