Xray
1h 6h 24h 48h 3d 5d 7d 14d 30d ‹
Last 1 hour▾
Ends now. Anything up to this cluster's 30-day retention.
Start and end times
Cancel
Interpreted in UTC (change it at the right of this bar). Earliest available: 2026-10-19 08:59:47.
› →|
updated 22h ago
Sample data. This database was generated by scripts/seed_dev_db.py — every service, deploy, person and figure on this page is synthetic. Names are sample.*@seed.invalid and image tags are prefixed SEED-. Nothing here happened.

Ask views — 60 cases

Every Ask answer can now carry a view: charts, tables, timelines and buttons drawn from a small program the model writes (OpenUI Lang). Below is each test case — the question, the tools and components a good answer reaches for, and a sample answer drawn exactly as Ask draws one. Opening this page calls no model. Ask it live ↗ sends the question to Ask on the case's cluster.

Right now

A01 What is broken right now master real data Ask it live ↗
What is broken right now on master?
tools: whats_broken · components: Callout, List, Links, FollowUps

Two things are broken on master right now, and both are Neo4j: the shared database master is failing 218 Cypher queries in 15 min (finding #7603, open since Oct 27 15:34 UTC), and matrix_a is failing 9 in 15 min (#16175, since Nov 4 14:36 UTC) 1. Nothing was deployed in the last 2 h, and first look found no linked change for either 1.

✗ 2 things broken on master, both Neo4j query failures 1 — No deploy in the last 2 h. First look: cause not clear from logs or recent changes, for both 1.

Broken now on master (UTC) 1 - Neo4j · shared database master — 218 Cypher queries failed in 15 min · critical · first look: no linked change (since Oct 27 15:34) - Neo4j · shared database matrix_a — 9 Cypher queries failed in 15 min · warning · first look: cause not clear (since Nov 4 14:36)

User journeys on master, last 60 min 1 - Sign-in & profile: ok — working - Trial execution & work orders: ok — working - Documents: idle — no calls - AI document processing: idle — no calls - Notifications: idle — no calls

Finding #7603 · first look · Finding #16175 · first look · master Home

Ask next: Which services send the failing Cypher on the shared database master? · Show the Neo4j errors for database master in the last hour · Is matrix_a on master failing for the same reason as database master?

Neither row has a CODEOWNERS owner; the leads listed are ojas.kale and samba.bandarupalli 1. First look counted 53 failures in its own 15-min sample (onset Nov 6 08:47 UTC), so the per-15-min count moves between reads 1.

A02 Is production OK production real data Ask it live ↗
Is production OK right now? Give me the one-screen answer.
tools: whats_broken → overview · components: Callout, KpiRow, StatusGrid, FollowUps

Not quite. One journey is failing: Trial execution & work orders, 7 of 7 requests failed in the last hour, every one an execsvc 401 on POST /api/trialexec/v2/study/:id/activity/status 1. Nothing was deployed in the last 2 h 1, and over 24 h production served 84,085 requests with only 211 5xx 2.

✗ 1 thing broken: Trial execution & work orders, 7 of 7 failed in the last hour 1 — execsvc 401 on POST /api/trialexec/v2/study/:id/activity/status (7). Owner Madhava Mukkamala, backup Shashank Agrawal 1.

  • Requests: 84,085 (last 24 h 2)
  • Error rate: 14.3% (4xx + 5xx, mostly execsvc 2)
  • 5xx: 211 (201 from onboardingsvc 2)
  • Active users: 58 (27 orgs 2)

Production now: journeys, last 60 min 1 · services, last 24 h 2 - Trial execution & work orders: bad — 7 of 7 failed, last hour 1 - execsvc: bad — 89.1% of 13,010 calls failed 2 - onboardingsvc: warn — 201 5xx · 1.3% errors 2 - notifsvc: warn — 6.1% of 1,231 streams failed 2 - virtual-assistant: warn — p95 5.9 s 2 - docsvc: ok — 0.15% errors · p95 526 ms 2 - docproc: ok — 0.04% errors · p95 981 ms 2 - enrollsvc: ok — 1.1% errors · p95 422 ms 2 - Sign-in & profile: ok — working 1 - Documents: idle — no calls in 60 min 1 - AI document processing: idle — no calls in 60 min 1 - Notifications: idle — no calls in 60 min 1

Ask next: Show the failing execsvc 401 calls on production in the last hour · Which caller is getting execsvc's 401s on production, and why? · Why is onboardingsvc returning 5xx on production?

The 14.3% error rate is mostly execsvc 4xx rather than outages. The 24 h totals miss the last 5 min, which is a coverage gap 2.

A03 Cluster overview, 24 h production real data Ask it live ↗
How is production doing over the last 24 hours?
tools: overview → budgets · components: KpiRow, BarChart, Table

Production handled 84,085 requests in the last 24 h at a 14.3% error rate, but almost all of that is execsvc 4xx: there were only 211 5xx, and 201 of them came from onboardingsvc 1. onboardingsvc is the one service low on error budget (8.5% left), and the other twelve are healthy 2.

  • Requests: 84,085 (last 24 h 1)
  • Error rate: 14.3% (4xx + 5xx 1)
  • 5xx: 211 (201 from onboardingsvc 1)
  • Neo4j queries: 1,174,881 (p95 9.3 ms 1)
  • Active users: 58 (27 orgs 1)

Errors and latency Error rate by service, last 24 h 1 (%)

error %
execsvc89.1
notifsvc6.09
onboardingsvc1.31
enrollsvc1.09
virtual-assistant0.24
docsvc0.15
docproc0.04

p95 latency by service, last 24 h 1 (ms)

p95
virtual-assistant5,903
dashql1,955
patientsvc1,405
docproc981
studiosvc955
docsvc526
schedsvc478
regsvc474
tusker426
enrollsvc422
execsvc142
onboardingsvc86

#### Error budgets ⚠ onboardingsvc has 8.5% of its error budget left 2 — Every other service is healthy; execsvc's 4xx do not spend budget.

All 13 services, budget detail 2 Error budgets, last 24 h, all 13 services; objectives derived 2

ServiceCallsObjectiveAchievedBudget leftBurn 6 hVerdict
onboardingsvc21,97899%99.09%8.5%0.65×budget low
docsvc31,59699.9%99.98%84.2%0.0×healthy
execsvc13,01099.5%99.96%92.3%0.0×healthy
docproc12,52499.9%100.00%100%0.0×healthy
enrollsvc1,83699.9%100.00%100%0.0×healthy
notifsvc1,23199.9%100.00%100%0.0×healthy
tusker67199.9%100.00%100%—healthy
dashql44999.9%100.00%100%—healthy
virtual-assistant41399.9%100.00%100%—healthy
studiosvc24199.9%100.00%100%—healthy
schedsvc8499.5%100.00%100%—healthy
regsvc4499.9%100.00%100%—healthy
patientsvc899.9%100.00%100%—healthy

Ask next: Why is onboardingsvc burning its error budget on production? · Show example failing execsvc calls on production in the last 6 hours · Why is virtual-assistant's p95 5.9 s on production?

Every objective is derived from the service's own last week, not a declared SLO. Budgets count only 5xx, which is why execsvc's 4xx don't spend budget 2. The last 5 min of the window are a coverage gap 1.

A04 Open findings this week master real data Ask it live ↗
What is wrong on master this week? Summarise the open findings, most important first.
tools: list_findings · components: Donut, Table, FollowUps

410 findings were active on master in the last 7 days. 302 of them are error.novel (new error shapes), and 22 are high level. The two high-level ones on the latest page come first: Cypher failures on the shared database master (#7603, 218 in 15 min) and the workplan.summarization-sweep job failing (#5483, 28 failed runs in 6 h) 1.

Findings by level, last 7 days 1 - high: 22 - medium: 331 - low: 57

Findings by kind, top 8 of 23 kinds 1

findings
error.novel302
outlier.service12
neo4j.errors10
anomaly.neo4j10
endpoint.4xx_spike10
anomaly.latency9
node.unhealthy8
anomaly.traffic7

Open findings on master, most important first, from the 25 latest of 410 1

FindingLevelKindSubjectWhatSeen
#7603highneo4j.errorsneo4j shared/master218 Cypher queries failed in the last 15 min120×
#5483highjob.failingworkplan.summarization-sweep28 failed runs in 6 h · unknown checkpoint source type320×
#5693mediumneo4j.errorsneo4j shared/automation15 Cypher queries failed in 15 min234×
#16175mediumneo4j.errorsneo4j shared/matrix_a9 Cypher queries failed in 15 min31×
#16174mediumneo4j.errorsneo4j shared/matrix_b5 Cypher queries failed in 15 min50×
#1789mediumjob.failingpubsub.study-smo-binding-master-enrollsub5 failed runs in 6 h · ConstraintValidationFailed201×
#5452mediumscaler.failingScaledJob/rag-agent-workercannot read its scaling metric, 3× in 2 h1,713×
#5454mediumscaler.failingHPA/docproc-hpacannot read its scaling metric, 2× in 2 h410×
#5453mediumscaler.failingHPA/ontologysvc-hpacannot read its scaling metric, 2× in 2 h317×
#2843mediumnode.unhealthynode/NodeNotReadypods on an unready node: 1 affected in 30 min599×
#163lowbudget.burningexecsvcburning budget 7.7× (6 h) against 98.00%201×
#16245lowllm.spendai-spendAI spend projects to $1,538 this month, 1.4× last month's $1,06742×

Ask next: Why are Cypher queries failing on the shared database master? · Why is workplan.summarization-sweep failing on master? · Which error.novel findings on master appeared in the last 24 hours?

The table ranks only the latest page (25 of 410, page 1 of 17). 20 more high-level findings exist that this page doesn't show 1. I ranked by X-ray's level rather than raw severity, which marks 315 of the 410 as critical 1.

A05 What changed in 6 h production real data Ask it live ↗
What changed on production in the last 6 hours?
tools: change_feed → list_findings · components: Timeline, Compare

Nothing was deployed, hand-edited or rolled out on production in the last 6 hours: the change feed is empty 1. The change was in behaviour. 18 findings were active, two of them new tonight: the litellm HPA losing its scaling metric at 05:20 UTC (#16811) and a read-only filesystem on a node at 01:24 (#16786). Neo4j load ran at 99 q/min against a normal 32 for this hour 2.

  • No deploys, hand changes or rollouts on production in the last 6 h 1 — So none of the findings below can be tied to a change.

Production, last 6 h: findings by when X-ray saw them (UTC) 2 - 01:24 — Read-only filesystem on node gk3-production-nap-1sqli2in-94849021-hh7s: #16786 · first seen tonight, 2× - 01:39 — onboardingsvc blindedScopes: 12 of 12 calls failed: #4570, with budget burn 9.3× over 1 h (#1530) - 04:47 — Traffic at a normally idle hour: whole cluster 94.0 req/min vs 0.0 normal (#155) · docsvc Neo4j 326 q/min vs 0 (#649) - 05:02 — Whole-cluster Neo4j load 99 q/min vs 32 normal: #49 · 3 of the last 4 checks - 05:20 — litellm HPA cannot read its scaling metric: #16811 · first seen tonight, 4× in 2 h - 06:38 — exportsvc.activity_update still failing: #7927 · 18 failed runs in 6 h

Observed vs normal for Sat 04:00 UTC 2

normalobserved
Whole-cluster request rate0.0 req/min94.0 req/min
Whole-cluster Neo4j load32 q/min99 q/min
docsvc request rate0.0 req/min40.8 req/min
docsvc Neo4j load0 q/min326 q/min
docproc Neo4j load10 q/min43 q/min
onboardingsvc request rate0.0 req/min35.6 req/min

Ask next: What drove the extra Neo4j load on production after 04:00 UTC? · Why can't the litellm HPA read its scaling metric on production? · What changed on production in the last 24 hours?

Times are when X-ray last saw each finding, not exact onsets. The "normal 0.0 req/min" baselines are ± 5 req/min wide at Sat 04:00, so those comparisons are weak 2.

A06 Master vs production master real data Ask it live ↗
Compare master and production over the last 24 hours: traffic, errors, latency, Neo4j load. Where do they differ most?
tools: overview · components: Compare, BarChart

The biggest difference is Neo4j. Production ran 1,174,881 queries against master's 474,130, mostly from docsvc (816,705), yet master's Cypher is the slower one, at p95 51.1 ms against 9.3 ms 12. Production also carries far more traffic (84,085 vs 21,560 requests) and a 14.3% error rate, nearly all execsvc 4xx, against master's 2.2% 12.

master vs production, last 24 h 12

masterproduction
Requests21,56084,085
Error rate2.2%14.3%
5xx135211
Avg API latency32.5 s162 ms
Neo4j queries474,1301,174,881
Neo4j avg10.4 ms4.8 ms
Neo4j p9551.1 ms9.3 ms
Active users3558
Orgs4027

p95 latency by service, last 24 h 12 (ms)

masterproduction
execsvc3,881142
dashql3,6271,955
virtual-assistant3,0815,903
patientsvc2,8901,405
tusker1,823426
docsvc1,381526
docproc1,326981
schedsvc929478
enrollsvc464422
onboardingsvc15586

Neo4j queries by service, last 24 h 12

masterproduction
docsvc8,878816,705
workplansvc270,26817,430
docproc18,086200,342
execsvc65,39514,570
enrollsvc50,88110,407
virtual-assistant11,82942,726
onboardingsvc12,56634,474
tusker8,65015,409
studiosvc14,2541,053

Ask next: Why is studiosvc's p95 120 s on master? · Why does workplansvc run 270,268 Neo4j queries a day on master? · Why is master's Neo4j p95 51.1 ms when production's is 9.3 ms?

I left studiosvc (p95 120 s on master) and workplansvc (49.1 s) off the latency chart so it stays readable. studiosvc's 12,034,199 ms max is what drags master's average up to 32.5 s 1. Both windows end in a 5-min coverage gap 12.

A07 Failing journeys production real data Ask it live ↗
Which user journeys are failing or silent right now on production?
tools: whats_broken → journeys · components: StatusGrid, Callout

One journey is failing on production: Trial execution & work orders, 7 of 7 requests failed in the last hour, every one an execsvc 401 on POST /api/trialexec/v2/study/:id/activity/status 1. Three journeys are silent (Documents, AI document processing and Notifications had no calls in 60 min), and only Sign-in & profile is working 1. The last 2 h had just 5 user sessions, all signed out, with 0 errors 2.

✗ Trial execution & work orders: 7 of 7 failed in the last hour 1 — All execsvc 401 on POST /api/trialexec/v2/study/:id/activity/status. Owner Madhava Mukkamala, backup Shashank Agrawal 1.

Journeys on production, last 60 min 1 - Trial execution & work orders: bad — 7 of 7 failed · execsvc 401 - Documents: idle — no calls - AI document processing: idle — no calls - Notifications: idle — no calls - Sign-in & profile: ok — working

  • Sessions: 5 (last 2 h 2)
  • Requests: 8 (across those sessions 2)
  • Session errors: 0 (last 2 h 2)
  • Signed-in users: none (all 5 sessions signed out 2)
  • Silent is not the same as broken — With 5 signed-out sessions in 2 h 2, the three idle journeys may just have no users at this hour. No calls means nothing to judge 1.

The failing execsvc calls, 05:40–06:40 UTC

Ask next: Which caller sends execsvc POST /api/trialexec/v2/study/:id/activity/status on production? · When did the execsvc 401s on production start? · Is exportsvc.activity_update behind the execsvc 401s on production?

7 calls is a small sample, and the 2 h session read can't say which user or service sent them 2. Who calls the endpoint isn't in these results.

Performance

B01 Is docsvc slower than usual production real data Ask it live ↗
Is docsvc slower than normal for this time of week?
tools: timeseries → baseline · components: KpiRow, LineChart, Compare

There's no sign of it, though this exact slot can't be judged. docsvc has no baseline for Sat 06:00 UTC (the slot is normally idle) and has served no calls since 05:00 12. Its last busy hour, Sat 04:00 UTC, ran at p95 416 ms, at the low end of its weekly normal range of 340–3,125 ms 12.

  • p95, last busy hour: 416 ms (Sat 04:00 UTC, 1,056 calls 1)
  • Baseline for Sat 06:00: none (slot normally idle · 139 slots have history 2)
  • Weekly normal range: 340–3,125 ms (median p95 across the week's slots 2)
  • Calls since 05:00: 0 (nothing to judge right now 1)

docsvc p95 latency by hour, last 24 h (UTC) 1 (ms)

p95
Fri 06:001,163
07:00479
08:00465
09:00561
10:00479
11:00462
12:00949
13:00500
14:00886
15:00603
16:00516
17:00805
18:00795
19:00651
20:00462
21:00625
22:00883
23:00635
Sat 00:00—
01:00767
02:00—
03:00464
04:00416
05:00—
06:00—

Threshold: 340 low end of weekly normal 2

docsvc: Friday's busiest hour vs Saturday's last busy hour 1

Fri 08:00Sat 04:00
p95 latency465 ms416 ms
Calls3,9231,056
Error rate0.05%0.00%
Neo4j load1,936 q/min163 q/min

Ask next: Is docsvc slower than normal on production on weekday mornings this week? · Which docsvc endpoints drove the 949 ms p95 at Fri 12:00 UTC? · Show docsvc p99 on production for the last 24 hours

Empty hours are gaps, not zeros, and the 06:00 hour is still filling (measured=false) 1. Without a baseline for this slot, "normal for this time of week" can't be stated. The next busy weekday hour can be 2.

B02 p95 by service production real data Ask it live ↗
Rank every production service by p95 latency over the last 6 hours.
tools: overview · components: BarChart, Table

Over the last 6 h docsvc is the slowest production service at p95 479 ms (1,912 calls), followed by schedsvc at 375 ms and enrollsvc at 360 ms 1. Only 7 services served API calls in the window, and it was a quiet one: 4,481 requests from 6 users 1.

  • Requests: 4,481 (last 6 h 1)
  • Error rate: 1.27% (4xx + 5xx 1)
  • 5xx: 12 (all onboardingsvc 1)
  • Active users: 6 (2 orgs 1)

p95 latency by service, production, last 6 h 1 (ms)

p95
docsvc479
schedsvc375
enrollsvc360
docproc169
tusker121
onboardingsvc49
execsvc1.9

⚠ schedsvc and tusker rank on 5 and 6 calls 1 — A p95 over a handful of calls is one slow request, not a trend. execsvc looks fast only because 97.7% of its 43 calls were rejected 1.

Every production service with API calls, last 6 h 1

ServiceCallsp95AvgMaxErrors5xxNeo4j q/req
docsvc1,912479 ms165 ms4,627 ms0.00%010.5
schedsvc5375 ms237 ms375 ms0.00%06.6
enrollsvc169360 ms99 ms951 ms0.00%06.9
docproc298169 ms70 ms251 ms0.00%04.3
tusker6121 ms62 ms121 ms0.00%07.7
onboardingsvc1,84549 ms16 ms381 ms0.81%121.5
execsvc431.9 ms2.0 ms82 ms97.67%00.05

Ask next: Which docsvc endpoints have the highest p95 on production in the last 6 hours? · Rank production services by p95 over the last 24 hours · Why are 97.7% of execsvc calls failing on production?

virtual-assistant, regsvc and the background workers had no API calls in the window, so they aren't ranked 1. The last 5 min are a coverage gap.

B03 Why was this request slow production real data Ask it live ↗
Why was this docsvc request slow? Break down where the time went.
tools: api_calls → trace_detail · components: Waterfall, Code, Callout

Neo4j wasn't the cause. The request took 686 ms, but its 3 Cypher queries add up to only 12 ms, and the first one didn't start until 570 ms in 2. Most of the time passed inside docsvc before it touched the database, and none of the trace's log lines explain that gap 2.

⚠ 570 ms before the first query, and it isn't Neo4j 2 — GET /api/documents/collections/:id · 200 · Sat 04:36:09 UTC · [email protected] (tildainc). The 3 queries took 12 ms in total. The trace's 4 log lines are all INFO, and none of them falls in the gap 2.

Trace 8d7312f7…, GET /api/documents/collections/:id, 686 ms 2 - whole request: 686 ms (starts at 0 ms) - inside docsvc, before any Cypher: 570 ms (starts at 0 ms) - user and org access check: 3 ms (starts at 570 ms) - app time between queries: 84 ms (starts at 573 ms) - collection folder and its documents: 8 ms (starts at 657 ms) - actor lookup: 1 ms (starts at 680 ms)

The 3 Cypher queries this request waited for 2

OffsetTookQuery
570 ms3 msUser / org membership check
657 ms8 msCollection folder, study and documents
680 ms1 msActor by user, org and scope
MATCH (c:Folder{tildaid:$collectionFolderId, deltimestamp:0})
OPTIONAL MATCH (root:Folder{level:0, name:$rootName, deltimestamp:0})-[:INCLUDES]->(c)
WITH c, root
WHERE c.folderType = $ft AND c.level = 1 AND c.scope = $scope AND root IS NOT NULL
MATCH (o:Organization{tildaid:$orgId})<-[:STUDY_TO_ORG]-(st:Study{tildaid:c.scopeId})
OPTIONAL MATCH (c)-[:CREATED_BY]->(u:User)

Open trace 8d7312f7… · docsvc on production

Ask next: Is there a 500 ms gap before the first query in other docsvc collections requests today? · Show the slowest docsvc requests on production in the last 6 hours · What does docsvc do before its first query in GET /api/documents/collections/:id?

I picked the slowest of the 25 newest docsvc calls (686 ms). The window held 1,912, so this isn't the slowest request of the 6 h 1. What docsvc did during the 570 ms (auth, a downstream call, GC) isn't in the logs, so the code would have to say.

B04 N+1 / chatty endpoints production real data Ask it live ↗
Which endpoints are chatty with Neo4j (N+1 patterns) on production?
tools: outliers · components: BarChart, Table, Steps

Only one service stands out: virtual-assistant runs 103.5 Neo4j queries per request, against a peer median of 5.7 (robust z 16.1) 1. It is also the p95 (5.9 s) and p99 (12.6 s) outlier, which is what a per-request fan-out looks like. outliers compares services, not endpoints, though, so it doesn't name the endpoint 1.

✗ virtual-assistant is the only N+1 outlier on production, last 24 h 1 — 103.5 Neo4j queries per request vs a peer median of 5.7 across 9 peers. No other service crossed the bar.

Neo4j queries per request, last 24 h 1

queries / request
virtual-assistant103.5
peer median (9 services)5.7

Every outlier on production, 3 of 3 1

ServiceMeasureValuePeer medianScoreNotable
virtual-assistantNeo4j queries per request103.55.716.1yes
virtual-assistantp95 latency5,903 ms478 ms10.9yes
virtual-assistantp99 latency12,625 ms1,732 ms6.3yes

Find the chatty endpoint - [ ] Pick a slow virtual-assistant session — journeys on production, then journey_detail to see Cypher counts per screen - [ ] Find the repeated shape — one plan hash run once per document or folder inside a single request is the N+1 - [ ] Batch it — replace the per-item query with one UNWIND $ids query - [ ] Re-check outliers after the fix ships — queries per request should fall towards the peer median of 5.7

Ask next: Which Cypher shapes does virtual-assistant run most per request on production? · Show one slow virtual-assistant session on production with its Cypher · Is virtual-assistant also an N+1 outlier on master?

outliers judges a service against its peers, not its own history, and skips services with under 20 calls 1. The fan-out is the likely reading, but no trace has confirmed it yet.

B05 Slowest hours by service production real data Ask it live ↗
When during the day is production slowest? Show it by hour and service for the last 24 hours.
tools: timeseries · components: Heatmap, Callout

Production is slowest in the early afternoon UTC. On Friday the cluster p95 peaked at 858 ms at 14:00 and 832 ms at 12:00 1, the same hours docproc reached 6.2 s (12:00) and virtual-assistant 8.2–9.2 s (13:00–14:00) 35. docsvc and onboardingsvc stay fairly flat all day 24.

  • Cluster p95 peak: 858 ms (Fri 14:00 UTC 1)
  • docproc peak: 6.2 s (Fri 12:00 UTC 3)
  • virtual-assistant peak: 9.2 s (Fri 14:00 UTC, 95 calls 5)
  • docsvc range: 416–1,163 ms (hours with calls 2)

⚠ The slow stretch is Fri 12:00–14:00 UTC 1 — docproc hit p95 6.2 s at 12:00 3, and virtual-assistant 8.2 s and 9.2 s at 13:00 and 14:00 on 95 calls each 5. Overnight the cluster is close to idle 1.

p95 latency by hour and service, last 24 h (UTC) 1–5

Fri 060708091011121314151617181920212223Sat 00010203040506
whole cluster1487462497486647832473858454472457485479430613643471155466241727132
virtual-assistant1959107751,5508004334558,2149,1506501,0501,6501,650875185—9002,650———————
docproc—8437543,3001,3241,9426,2415352,6504516272006187865361,7001901,200—99—188171——
docsvc1,163479465561479462949500886603516805795651462625883635—767—464416——
onboardingsvc989492969288736893808587889365848682135492884822

Ask next: What made docproc's p95 6.2 s at 12:00 UTC Friday on production? · Why did virtual-assistant reach 9.2 s p95 at 14:00 UTC Friday? · Is the 12:00–14:00 UTC slowdown on production there every weekday?

This is one day, so it shows when Friday was slow, not a proven daily pattern. Blank cells are hours with no calls, and the near-zero cluster values overnight come from a few rejected calls 1. I read four services, not every one.

B06 Did the deploy slow it down production real data Ask it live ↗
Did the last docsvc deploy make it slower?
tools: change_feed → timeseries · components: Compare, Timeline, LineChart

No. The last docsvc deploy, v2.10.43-86c75b5 at Oct 30 14:21 UTC, was followed by a faster hour: p95 554 → 454 ms and p99 3,607 → 499 ms 1. Daily p95 has stayed between 474 and 512 ms on every weekday since 2.

docsvc, 1 h before vs 1 h after the Oct 30 14:21 UTC deploy 1

beforeafter
p50163 ms141 ms
p95554 ms454 ms
p993,607 ms499 ms
Error rate0.31%0.12%
Calls325808

docsvc on production (UTC) 12 - Oct 30 06:15 — v2.10.42-0f0f52e synced: revision 9dc97ef16 · manual sync by samba.bandarupalli · p95 4,421 → 629 ms, but only 27 calls before 1 - Oct 30 14:21 — v2.10.43-86c75b5 synced, the last deploy: revision 4273d9ebb · manual sync by samba.bandarupalli 1 - Oct 30 15:21 — One hour after: p95 454 ms, p99 499 ms: 808 calls, 0.12% errors 1 - Nov 7 — No docsvc deploy since · p95 479 ms so far today: 12

docsvc p95 by day since the deploy (UTC) 2 (ms)

p95
Oct 30470
Oct 31657
Nov 1838
Nov 2478
Nov 3474
Nov 4512
Nov 5512
Nov 6499
Nov 7479

Threshold: 554 p95 in the hour before the deploy 1

Ask next: What changed in docsvc between v2.10.42 and v2.10.43? · Why was docsvc's p95 838 ms on Sunday Nov 1 on production? · Did the Oct 30 06:15 docsvc deploy change its error rate on production?

The comparison covers one hour on each side, and calls rose from 325 to 808, so time of day is mixed in 1. Oct 30 and Nov 7 are partial days, and the weekend's 657–838 ms ran on only 633 and 405 calls 2.

B07 virtual-assistant p95 production real data Ask it live ↗
Why is virtual-assistant's p95 over 5 seconds?
tools: timeseries → traces → outliers · components: KpiRow, LineChart, Steps

Two busy hours explain it. On Friday at 13:00 and 14:00 UTC virtual-assistant served 95 calls each at p95 8.2 s and 9.2 s, while most other hours stayed under 1.7 s 1. It also runs 103.5 Neo4j queries per request against a peer median of 5.7, so a per-request fan-out is the first suspect 3. Nothing reached it in the last 6 h, so there was no request to trace 2.

  • p95, last 24 h: 5.9 s (peer median 478 ms 3)
  • p99, last 24 h: 12.6 s (peer median 1.7 s 3)
  • Neo4j queries / request: 103.5 (peer median 5.7 3)
  • Requests, last 6 h: 0 (nothing to trace 2)

virtual-assistant p95 and p99 by hour, Fri 06:00–23:00 UTC, no calls since 1 (ms)

p95p99
06:00195199
07:00910982
08:00775955
09:001,5501,910
10:00800960
11:00433487
12:00455491
13:008,21415,500
14:009,15020,600
15:00650930
16:001,0501,810
17:001,6501,930
18:001,6501,930
19:00875975
20:00185197
21:00——
22:00900980
23:002,6506,650

Threshold: 478 peer median p95 3

Confirm the cause - [ ] Pull the slow calls from Fri 13:00–15:00 UTC — api_calls for virtual-assistant in that window, slowest first - [ ] Trace two of them with Cypher — trace_detail with cypher=true: count the queries per request and see which shape repeats - [ ] Batch the repeated query — if one shape runs per document or folder, fold it into one UNWIND query - [ ] Re-check against the 478 ms peer median — outliers on production after the fix ships

Ask next: Show the slowest virtual-assistant calls on production Fri 13:00–15:00 UTC · Which Cypher shapes does virtual-assistant run per request on production? · Is virtual-assistant's p95 also over 5 seconds on master?

The fan-out is a lead, not a proven cause. outliers compares with peers, not with history 3, and the latency spikes still need a trace from Friday afternoon to tie them to the queries.

Neo4j & Cypher

C01 Cypher that got slower production real data Ask it live ↗
Which Cypher queries got slower on production in the last 24 hours?
tools: query_trends · components: Table, BarChart, Code

Only one shape really got slower: virtual-assistant's folder-tree query 32395444 went from p95 4.6 ms to 977 ms (210× drift) across the day, on 77 executions 1. The rest of the list roughly doubled from a low base. enrollsvc's 9D3836B5 (17.8 → 51.2 ms over 388 runs) is the only other one worth watching 1.

✗ 32395444 is the one real regression, p95 4.6 ms to 977 ms 1 — virtual-assistant · 77 runs · max 1,095 ms. The next biggest, enrollsvc 9D3836B5, went from 17.8 ms to 51.2 ms over 388 runs 1.

p95 early vs late in the window, top 10 degrading shapes, last 24 h 1 (ms)

p95 earlyp95 late
32395444 · virtual-assistant4.6977
81C0B9EE · tusker185365
856B8DA1 · regsvc48.597.5
9D3836B5 · enrollsvc17.851.2
017A8E7A · dashql9.718.8
30AA9AC1 · virtual-assistant23.9
CBF2C7D7 · docsvc23.9
A3BB82EA · docproc1.953.7
6E20C5D2 · dashrsvc23.5
C5AEF889 · (internal)0.951.95

Cypher shapes that got slower on production, last 24 h, top 10 of 808 1

ShapeServiceRunsp95 earlyp95 lateDriftMax
32395444(/neo/shape/32395444?cluster=production)virtual-assistant774.6 ms977 ms210.5×1,095 ms
9D3836B5enrollsvc38817.8 ms51.2 ms2.9×278 ms
C5AEF889(internal)20.95 ms1.95 ms2.1×1 ms
856B8DA1regsvc348.5 ms97.5 ms2.0×55 ms
81C0B9EEtusker15185 ms365 ms2.0×230 ms
30AA9AC1virtual-assistant1,4532.0 ms3.9 ms2.0×3 ms
017A8E7Adashql409.7 ms18.8 ms1.9×619 ms
CBF2C7D7docsvc3,5152.0 ms3.9 ms1.9×3 ms
A3BB82EAdocproc1461.95 ms3.7 ms1.9×3 ms
6E20C5D2dashrsvc472.0 ms3.5 ms1.8×46 ms
MATCH (st:Study {tildaid: $study_id})-[:USES]->(:FolderManager)-[:INCLUDES]->(tmf:Folder {name:?}) WHERE tmf.level = ? MATCH (tmf)-[:INCLUDES*?..?]->(d:Document) RETURN DISTINCT d.tildaid AS doc_id

Ask next: Why did Cypher shape 32395444 get 210× slower on production? · Which virtual-assistant requests run shape 32395444 on production? · Is enrollsvc shape 9D3836B5 also slower on master?

Six of the ten went from about 2 ms to about 4 ms, which is noise at that scale. Drift compares the early and late parts of the window, so it doesn't say when the change happened 1.

C02 Busiest Cypher shapes master real data Ask it live ↗
What are the busiest Cypher shapes on master, and who runs them?
tools: query_trends · components: BarChart, Donut, Table

The busiest Cypher shapes on master are cheap user lookups by execsvc: 308B13B1 and 046B6411 ran 13,759 and 13,074 times in 24 h at p95 under 2 ms 1. The shapes that cost database time are workplansvc's background polls. Five of the top ten belong to it, led by DCDC23FF at 77.7 ms average over 6,830 runs 1.

Executions by shape, master, last 24 h, top 10 of 1,473 1

executions
308B13B1 · execsvc13,759
046B6411 · execsvc13,074
E083AA2B · enrollsvc10,254
9BF74CB7 · workplansvc6,842
DCDC23FF · workplansvc6,830
38102FE8 · workplansvc6,446
6F0FF668 · workplansvc6,032
ECCD8C56 · enrollsvc5,840
1A26C00F · enrollsvc5,631
0A4540E4 · workplansvc5,549

Who runs the top 10: executions by service 1 - workplansvc: 31,699 - execsvc: 26,833 - enrollsvc: 21,725

Busiest Cypher shapes on master, last 24 h 1

ShapeServiceRunsAvgp95R/WWhat it does
308B13B1execsvc13,7590.9 ms1.9 msreadUser by email
046B6411execsvc13,0740.1 ms1.6 msreadUser by email (second plan)
E083AA2Benrollsvc10,2540.3 ms1.8 msreadStudy by tildaid
9BF74CB7workplansvc6,84222.8 ms48.5 msreadPending-work EXISTS checks
DCDC23FFworkplansvc6,83077.7 ms99.2 msreadMilestone items awaiting work orders
38102FE8workplansvc6,44625.6 ms48.6 msreadItems awaiting activation
6F0FF668workplansvc6,0320.5 ms1.9 msreadCheckpoint reject cascades
ECCD8C56enrollsvc5,8400.2 ms1.8 msreadOrganization by tildaid
1A26C00Fenrollsvc5,6311.0 ms2.0 msreadSites for a study and org
0A4540E4workplansvc5,5490.8 ms1.9 mswriteCreate WorkPlanContext
MATCH (wp:WorkPlanInstance)-[:WORKPLAN_INSTANCE_TO_MILESTONE]->(:WorkPlanMilestone)-[:REL_WORKPLAN_INSTANCE_MILESTONE_TO_ITEM]->(item:WorkplanInstanceMilestoneItem) WHERE wp.state <> ? AND coalesce(item.skippedAt, ?) = ? AND coalesce(item.notApplicableAt, ?) = ? AND (coalesce(item.workOrderCreateUnresolvableAt, ?) = ? OR $nowMs - item.workOrderCreateUnresolvableAt >= $windowMs) AND ( (item.itemType = ? AND (coalesce(item.readyAt, ?) > ? OR coalesce(item.vaWoInstanceFetchRequestSentAt, ?) > ?) AND coalesce(item.workOrderId, ?) = ? AND coalesce(item.expectedAttachmentUnresolvedAt, ?) = ? AND (coalesce(item.vaWoInstanceFetchRequestSentAt, ?) = ? OR $nowMs - item.vaWoInstanceFetchRequestSentAt >= $windowMs)) OR (item.itemType = ? AND coalesce(item.readyAt, ?) > ? AND coalesce(item.edlEventScopeId, ?) = ? AND coalesce(item.edlScopeCreateFailedAt, ?) = ? AND (coalesce(item.edlScopeCreateRequestSentAt, ?) = ? OR $nowMs - item.edlScopeCreateRequestSentAt >= $windowMs)) ) RETURN item.tildaid AS id LIMIT $limit

Ask next: Why does workplansvc poll shape DCDC23FF 6,830 times a day on master? · Why does execsvc's user-by-email lookup run under two plans on master? · Which Cypher shapes on master use the most total database time?

The donut sums only these ten shapes out of 1,473 1. 308B13B1 and 046B6411 are the same user-by-email statement under two plans. 308B13B1 has no late-window p95, so it stopped running part-way through the day, possibly when 046B6411 took over 1.

C03 Explain one shape production real data Ask it live ↗
Explain the slowest Cypher shape on production: why is it slow and what would fix it?
tools: query_trends → shape_detail · components: KpiRow, LineChart, Code, Steps

Strictly, the top of the p95 list is tusker's 522F6906 at 4.85 s, but it ran once: a one-off MERGE of Inbox nodes across every Organization 1. The slowest shape that runs repeatedly is virtual-assistant's 32395444, a folder-tree walk to every Document under a study's TMF folder, at p95 966 ms over 77 runs 1. It isn't a plan change. The same plan ran at p50 1.9 ms at 15:00 and 471 ms at 23:00 on Friday, so its cost follows what it walks 2.

  • p95, last 24 h: 966 ms (77 runs 1)
  • p95 at 15:00: 4.6 ms (12 runs, Fri UTC 2)
  • p95 at 23:00: 977 ms (65 runs, Fri UTC 2)
  • Max: 1,095 ms (23:00 bucket 2)

Shape 32395444 by bucket, Fri (UTC), its only two buckets 2 (ms)

p50p95
15:001.94.6
23:00471977
  • Why it's slow — The [:INCLUDES*?..?] walk visits every folder below the TMF folder before DISTINCT collapses the documents. The plan is the same at both times, so the jump in p50 points at a bigger tree being walked, not a worse plan 2.
MATCH (st:Study {tildaid: $study_id})-[:USES]->(:FolderManager)-[:INCLUDES]->(tmf:Folder {name:?}) WHERE tmf.level = ? MATCH (tmf)-[:INCLUDES*?..?]->(d:Document) RETURN DISTINCT d.tildaid AS doc_id

What would fix it - [ ] PROFILE it for a slow study — take a study_id from a 23:00 run and PROFILE the statement: count db hits on the INCLUDES expansion - [ ] Stop walking the tree — if Documents carry their study id, match them by an indexed property and keep only the folder filter - [ ] Cache per study in virtual-assistant — 65 runs in one hour suggests the same document list is fetched repeatedly - [ ] Confirm with shape_detail — p95 should fall back toward the 4.6 ms seen at 15:00

Ask next: Which virtual-assistant requests ran shape 32395444 at 23:00 UTC Friday? · Show Cypher shape 32395444 on production over the last 7 days · Is tusker's Inbox MERGE 522F6906 a startup job on production?

The shape has only two buckets of data, since it first ran Fri 14:50 UTC 2. "A bigger tree" is the likely reading, not something the tools measured, and PROFILE will confirm or rule it out.

C04 Neo4j health production real data Ask it live ↗
How healthy is Neo4j on production: memory, page cache, transactions?
tools: neo4j_health · components: Gauge, Meters, KpiRow

Healthy. All 3 of 3 production Neo4j servers are up with 0 restarts, the page cache serves 100% of reads, and CPU is near idle at about 0.04 cores each over the last 6 h 1. I couldn't read transactions, because that chart was trimmed from the result 1.

✓ Neo4j on production is healthy, last 6 h (00:50–06:50 UTC) 1 — 3 of 3 servers up, 0 restarts, page cache hit ratio 100%, page faults at most 0.012/s, around 01:42 UTC.

  • Servers up: 3 of 3 (0 restarts 1)
  • CPU: 0.04 cores (per server, latest · max 0.24 on server-1 1)
  • Memory: 37.2–37.7 GB (per server, flat over 6 h 1)
  • Transactions: not read (chart trimmed from the result 1)
  • Page cache hit ratio: 100% of 100 (100% on all 3 servers, every sample shown 1)

Page cache usage, 19G cache 1 - server-1-production-0: 93.1% of 100

  • server-2-production-0: 93.1% of 100
  • server-3-production-0: 93.2% of 100

Per server, last 6 h 1

ServerCPU latestCPU maxMemory latestRestartsPage cache used
server-1-production-00.0400.24037.5 GB093.1%
server-2-production-00.0410.07637.2 GB093.1%
server-3-production-00.0440.09237.7 GB093.2%

⚠ The disk reading looks wrong 1 — The volume metric reports 4,096 bytes used of 101 GB on every server. Treat disk as unknown, not as empty.

Ask next: Show Neo4j transactions and lock waits on production for the last 6 hours · Was the page-fault blip on production Neo4j at 01:42 UTC tied to a slow query? · How healthy is the shared Neo4j that master uses?

Memory is the container's non-evictable memory, not a heap vs page-cache split 1. 93% page-cache usage is fine. Being pinned at 100% would mean the hot set had outgrown the cache.

C05 Cypher failures on master master real data Ask it live ↗
Why are Cypher queries failing on the shared database master, and should we do something?
tools: whats_broken → errors → logs · components: Callout, Timeline, Code, Choice

It's mostly one bug. enrollsvc's study–SMO binding handler tries to create a ChartReviewStudy node that already exists, Neo4j rejects it with ConstraintValidationFailed, and the Pub/Sub message is retried: 874 such failures in the last hour 2, with one trace failing 4 times between 06:46 and 06:50 UTC 3. Yes, it's worth fixing. The database itself is fine, but those bindings never complete, and the retries keep finding #7603 critical at 218 failures in 15 min 1.

✗ enrollsvc keeps re-creating ChartReviewStudy nodes that already exist 23 — The handler logs that the user is already a member, then fails to CREATE the chart-review study node, returns an error, and the same message comes back. First look on #7603 found no cause, and this is the app-side evidence it lacked 1.

  • Failed Cypher, shared database master: 218 (last 15 min · finding #7603 1)
  • ChartReviewStudy constraint failures: 874 (enrollsvc, last hour 2)
  • Error groups on master: 295 (last hour, page 1 of 12 read 2)

Cypher failures on the shared database master (UTC) - Oct 27 15:34 — Finding #7603 opens: Cypher failures on database master: critical · leads ojas.kale, samba.bandarupalli 1 - Nov 6 08:47 — Onset of the current run, per first look: no linked change found 1 - Nov 7 05:51 — Already failing when the 1-hour read starts: ConstraintValidationFailed from internal/study.go:214 and internal/pubsub.go:115 2 - 06:46 — Trace 72d887ff…: binding message received, CREATE fails, handler returns error: 4 attempts between 06:46 and 06:50 3 - 06:50 — Still failing: last error 06:50:51 23

study smo binding msg received
User is already a member. Updating the same actor
could not add chart review study node while binding study to org — Neo4jError: Neo.ClientError.Schema.ConstraintValidationFailed …
pubsub message handler returned error

Shall I file a Linear ticket to make enrollsvc's ChartReviewStudy create idempotent (MERGE, or treat already-exists as done)? — Yes, file it for the enrollsvc owners / Raise a PR that MERGEs instead of CREATE / No, keep watching

The full error goes on: (Node(10378619) already exists with label ChartReviewStudy and properties orgid = '7374e427-32… 3. I read only page 1 of the 12 error groups 2, and matrix_a's 9 failures (#16175) weren't examined 1. The 874 in an hour lines up with #7603's 218 per 15 min, but the tools don't link the two directly.

Errors & logs

D01 What keeps failing production real data Ask it live ↗
What keeps failing on production, with an example?
tools: errors → logs · components: Table, BarChart, Code

The one app failure that keeps going is exportsvc's activity updates: 43 errors in 6 h, steady from 00:52 to 06:46 UTC. Each one is a 401 from execsvc, because the token exportsvc sends has expired 12. The bigger counts are platform noise (cilium-agent 1,563, fluentbit) or one-off bursts: docproc's pipeline failures all landed at 01:15, and onboardingsvc's twelve 500s at 01:11–01:13 1.

Error lines by group, production, last 6 h, top 10 of 173 1

platformapp
cilium-agent · foreign IP1,563—
cilium metrics collector196—
fluentbit-gke · parse154—
docproc · processing log—116
fluentbit-gke · parse (2)105—
docproc · slots in use—58
exportsvc · 401 activity update—43
docproc · pipeline failed—29
docproc · no agent token—29
metrics-server · scrape timeout26—

What keeps failing on production, last 6 h (UTC) 1

ServiceErrorCountFirstLastPattern
exportsvcError processing activity update: 401 Unauthorized from /api/trialexec/v2/study/:id/activity/status4300:5206:46steady
docprocPipeline failed - full context2901:1501:15one burst
docprocdocsvc checksum update skipped: no agent token (configure TildaSense bot token)2901:1501:15one burst
onboardingsvc500 · failed to get study: get with empty tildaId (blinded-scope lookup)1201:1101:13one burst
metrics-serverFailed to scrape node, timeout to access kubelet2601:1206:13platform
cilium-agentFound a foreign IP address with the ID of the current node1,56301:1306:13platform

⚠ exportsvc is calling execsvc with an expired token 2 — On the same trace, execsvc logs token validation failed and Unauthorized: token expired, and exportsvc logs the 401. It has repeated all night 1.

06:46:12 WARNING execsvc   token validation failed, sending 401
06:46:12 INFO    execsvc   Request completed — 401 · Unauthorized: token expired
06:46:12 WARNING exportsvc 401 Unauthorized - token may be expired or invalid
06:46:12 ERROR   exportsvc Error processing activity update: API request failed: Client error '401 Unauthorized' for url 'api.tilda.bio/api/trialexec/v2/study/… ↗

Trace f98f04f8… · Errors page

Ask next: Where does exportsvc get the token it sends to execsvc on production? · What made docproc's pipelines fail at 01:15 UTC on production? · Show onboardingsvc's blindedScopes 500s on production at 01:11 UTC

This is page 1 of 7 (25 of 173 groups) 1. Counts are log lines, not failed requests, and docproc writes several lines per failed pipeline.

D02 execsvc error rate production real data Ask it live ↗
Why does execsvc have an 89% error rate on production?
tools: overview → errors · components: Donut, Table, Callout

execsvc's 89.1% error rate on production comes almost entirely from 4xx rejections. The service isn't failing: of 13,010 calls in the last 24 h only 5 were 5xx 1, and its log has no ERROR-level groups in the last 6 h 2. Something is sending execsvc requests it refuses.

⚠ execsvc is refusing requests, not crashing 1 — About 11,593 of 13,010 calls in 24 h were 4xx; only 5 were 5xx, and execsvc wrote no ERROR log groups in the last 6 h 2.

  • Calls: 13,010 (last 24 h 1)
  • Error rate: 89.1% (4xx + 5xx 1)
  • 5xx: 5 (last 24 h 1)
  • p95 latency: 142 ms (avg 24 ms: fast refusals 1)

execsvc calls by outcome, last 24 h 1 - 4xx (refused): 11,593 - succeeded: 1,412 - 5xx: 5

  • No ERROR log groups for execsvc, last 6 h 2 — 4xx responses are logged below ERROR, so the errors tool has nothing to group. To see the status codes and endpoints, read api_calls.

Production services by error rate, last 24 h 1

ServiceCallsError rate5xxp95
execsvc13,01089.1%5142 ms
notifsvc1,231 streams6.1%0—
onboardingsvc21,9781.3%20186 ms
enrollsvc1,8361.1%0422 ms
docsvc31,5960.15%5526 ms
docproc12,5240.04%0981 ms

Ask next: Which execsvc endpoints return 4xx on production in the last hour? · Which orgs or callers hit execsvc's 4xx on production today? · Did execsvc's error rate change after its last production deploy?

The 4xx and success counts are derived (calls × error rate, minus the 5 5xx). I couldn't tell which status codes (401/403/404) or which caller from these tools. The overview also has a 5-minute coverage gap at 06:45 UTC 1.

D03 onboardingsvc 5xx production real data Ask it live ↗
Show the onboardingsvc 5xx errors over the last 24 hours and what they are.
tools: timeseries → errors · components: LineChart, Table, Code

onboardingsvc returned 201 5xx in the last 24 h. 145 of them came in one burst on Nov 6 between 12:15 and 12:59 UTC 1. In that burst, every failure is the same bug: a study lookup with an empty tildaId, which the data layer refuses 2.

  • 5xx, last 24 h: 201 (Nov 6 06:00 – Nov 7 06:00 UTC 1)
  • Worst hour: 145 (12:00 UTC Nov 6, 9.2% of calls 1)
  • p95 in that hour: 73 ms (the requests failed fast 1)

onboardingsvc 5xx per hour on production, Nov 6 06:00 – Nov 7 06:00 UTC 1

5xx
06:000
07:000
08:000
09:000
10:000
11:000
12:00145
13:000
14:000
15:000
16:006
17:0038
18:000
19:000
20:000
21:000
22:000
23:000
00:000
01:0012
02:000
03:000
04:000
05:000

Error groups, Nov 6 12:00–13:00 UTC 2

WhereMessageCountFirstLast
observability/middleware.go:281Request completed — 500 · failed to get study: … get with empty tildaId14512:1512:59
handlers/errors.go:68request failed — 50014512:1512:59
health/actor.go:49could not get study for blinded-scope lookup — … empty tildaId14512:1512:59
Request completed — 500 · failed to get study: objLabel Study: get with empty tildaId requires a struct with at least one non-inherited dbkey field; a {deltimestamp} filter would match a

Example failing trace

Ask next: What were the other 56 onboardingsvc 5xx on production (16:00, 17:00 and 01:00 UTC)? · Did a deploy to onboardingsvc on production precede 12:15 UTC Nov 6? · Which orgs hit the empty-tildaId study error on production?

The three groups share one trace, so they are most likely one failure logged three times: 145 failing requests, not 435. I read error logs only for 12:00–13:00 UTC, so the other 56 5xx (16:00, 17:00 and 01:00 UTC hours) are not explained here.

D04 Log patterns for a service production real data Ask it live ↗
Summarise docproc's error log patterns in the last hour.
tools: logs · components: Table, BarChart, Code

docproc wrote no WARNING or ERROR lines on production in the last hour 1. Over the last 6 h there was one burst, at 01:15 UTC on Nov 7: 380 lines in a single minute, which included 29 failed document pipelines and repeated bot-token refresh failures 2.

✓ Quiet for the last hour 1 — 0 WARNING+ lines from 05:52 to 06:52 UTC. The figures below cover the last 6 h 2.

  • Lines, last 1 h: 0 (WARNING and above 1)
  • Lines, last 6 h: 381 (349 ERROR · 32 WARNING 2)
  • Failed pipelines: 29 (01:15:05–01:15:08 UTC 2)
  • Burst: 380 lines (in the minute of 01:15 UTC 2)

docproc log patterns on production, last 6 h (top patterns) 2

lines
[…] ═══ banner116
N/20 slots in use58
Pipeline failed - full context29
docsvc checksum update skipped29
ServiceTokenManager: failed to get bot token28
Context … -> Download FAILED18
Connection pool is full2
Pub/Sub num_unacked: 5041

docproc patterns on production, last 6 h 2

PatternSeverityCountWhen (UTC)Note
Pipeline failed - full contextERROR2901:15:05–01:15:08one per failed document
ServiceTokenManager: failed to get bot token … in backoffWARNING2801:15:05–01:15:08same traces as the failures
docsvc checksum update skipped: no agent tokenERROR2901:15:05–01:15:08asks for a TildaSense bot token
[…] Context …s -> Download FAILEDERROR1801:15:05–01:15:087 variants of one line
[…] ═══ and N/20 slots in useERROR17401:15:05–01:15:08context dump printed at ERROR
Connection pool is full, discarding connection: storage.googleapis.comWARNING201:15:07pool size 10
Failed to fetch Pub/Sub num_unacked: 504 Deadline expiredWARNING102:30:13separate, one-off
ServiceTokenManager: failed to get bot token: ServiceTokenManager: token refresh in backoff after a recent failure and no usable cached token yet

Ask next: Why did docproc's bot-token refresh fail on production at 01:15 UTC Nov 7? · Show trace dcfb72e61b11e600d67cc8d448983a20 on production · Has docproc's token-backoff failure happened on production before this week?

The ERROR count overstates the failures: 174 of the 349 ERROR lines are the pipeline's context dump (banners and slot counts) logged at ERROR. The pattern list is the top patterns only, covering 283 of the 381 lines.

D05 When did the spike start master real data Ask it live ↗
When did the error spike on master start, and what was deployed around then?
tools: timeseries → change_feed · components: LineChart, Timeline, Callout

The 5xx spike on master began in the 11:00 UTC hour on Nov 6 with 29 5xx. It peaked at 12:00 with 95 5xx (6.2% of calls) and was gone by 13:00 1. The nearest deploy was docproc master-713dc89, which ArgoCD synced at 11:08 UTC. Four more services followed at 11:18–11:19 2. That timing makes both adjacent to the spike, not proven causes.

master error rate per hour, Nov 6 06:00 – Nov 7 05:00 UTC 1 (%)

all errors5xx
06:00——
07:000.980.05
08:002.180.25
09:000.20
10:000.490.03
11:002.931.03
12:006.336.2
13:001.750
14:0016.170
15:000.290
16:001.760
17:000.580
18:003.540
19:000.840
20:0000
21:0000
22:0000
23:0000
00:0000
01:0000
02:001.190
03:002.540
04:0000
05:0000

What changed on master around the spike, Nov 6 (UTC) 1 2 - 06:55 — Fleet sync to master-11306eb: ArgoCD rev 882d8d8dc: docsvc, execsvc, onboardingsvc, enrollsvc, notifsvc and more 2 - 09:07 — docproc → master-3d14e1b: ArgoCD rev 256ee762e 2 - 09:36 — x-ray 3.9.16 rolled out by hand: [email protected] via kubectl 2 - 09:53 — x-ray 3.9.17 rolled out by hand: [email protected] via kubectl 2 - 11:00 — 5xx begin: 29 in the hour (1.0%): 1 - 11:08 — docproc → master-713dc89: ArgoCD sync, rev 0476ae4b4 2 - 11:18 — rag-agent, exportsvc, feedsvc, studioaisvc → master-792bf41: ArgoCD rev 85850840a 2 - 12:00 — Peak: 95 5xx, 6.2% of calls: p95 latency 28.4 s in that hour 1 - 13:00 — 5xx back to 0: 1 - 14:00 — Separate 4xx rise: 16.2% errors, 0 5xx: 1 - 14:10 — Fleet sync to master-fe8a323: ArgoCD rev 580db29e1 2

⚠ Adjacent, not proven — docproc's deploy landed inside the first spike hour, but hourly buckets can't show whether the 5xx began before or after 11:08, and the cluster series doesn't name the failing service.

Ask next: Which service returned the 5xx on master between 11:00 and 13:00 UTC Nov 6? · What changed in docproc between master-3d14e1b and master-713dc89? · What caused the 4xx rise on master at 14:00 UTC Nov 6?

Traffic on master is low (about 25–60 requests/min in those hours), so a few dozen failures move the percentage a lot. The 06:00 bucket was not measured.

Pods & Kubernetes

E01 Restarts and OOMs production real data Ask it live ↗
Which pods restarted or were OOMKilled on production this week?
tools: reliability · components: Table, BarChart, Callout

Only 2 containers restarted on production in the last 7 days: one chat-agent pod and one docproc pod, once each 1. I can't confirm OOMKills either way. The Kubernetes event scan returned 0 events for the week, so an empty OOM list here means missing data, not proof that none happened 1.

  • Container restarts: 2 (2 pods, last 7 d 1)
  • OOMKilled: unknown (0 k8s events scanned 1)
  • Spot preemptions: 22 (each followed by an auto-repair 1)
  • Pods seen: 4,968 (last 7 d 1)

Restarts by workload, last 7 d 1

restarts
chat-agent1
docproc1

Restarted pods on production, last 7 d 1

PodNamespaceRestarts
chat-agent-856f5b5446-fzvvftilda1
docproc-9d58f6f65-dt8xhtilda1

Node events on production, last 7 d 1

events
Spot preemption22
Auto-repair (recreated)22

⚠ OOM status unknown for this week 1 — The Kubernetes events read covered 0 events, so OOM, eviction and scheduling counts are missing. Each workload's pod page reads about 2 days of events and can show the reason.

Ask next: Why did docproc-9d58f6f65-dt8xh restart on production? · Did any production pods get OOMKilled in the last 48 h? · Which production services threw 5xx after the Nov 7 01:12 UTC spot preemption?

The node-event list is paged (25 of 44 rows shown), but the counts above (22 + 22) are the tool's totals.

E02 Why does a pod keep restarting production real data Ask it live ↗
Why does docproc keep restarting?
tools: pod_detail → reliability · components: Timeline, Meters, Steps

docproc isn't restarting repeatedly: it restarted once in the last 7 days, on pod docproc-9d58f6f65-dt8xh 2, and there are no restart events since Nov 5 06:50 UTC 1. Over the last 48 h, memory peaked at 2.09 GiB against a 5 GiB limit and CPU peaked at 0.78 of 4 cores. Neither suggests an OOM kill or throttling 1.

✓ One restart in 7 days, none in the last 48 h 1 2 — That restart is older than X-ray's ~2-day event history (from Nov 5 06:50 UTC), so its reason — OOM, crash or probe — is not on record.

docproc on production (UTC) 1 2 - Oct 31 – Nov 5 — 1 container restart: pod docproc-9d58f6f65-dt8xh, inside the 7-day window 2 - Nov 5 06:50 — Restart-reason history begins: no restart events since 1 - Nov 6 17:56 — Spot node preempted on production: gk3-production-nap-1sqli2in-94849021-lblh; docproc did not restart 2 - Nov 7 01:12 — Spot node preempted on production: gk3-production-nap-1sqli2in-94849021-sxc4; docproc did not restart 2 - Nov 7 06:02 — Now: 1 pod, 0.87 GiB, 0.03 cores: 1

docproc resources against limits, last 48 h 1 - Memory now: 0.87 GiB of 5 (request 4 GiB)

  • Memory peak: 2.09 GiB of 5 (mean 1.26 GiB)
  • CPU now: 0.03 cores of 4 (request 1 core)
  • CPU peak: 0.78 cores of 4 (mean 0.16 cores)

To find that one restart's reason - [ ] Open docproc's pod page — docproc pods: restarts and last state per container - [ ] Read the restarted container's last state — kubectl -n tilda describe pod docproc-9d58f6f65-dt8xh: Last State shows OOMKilled, Error or a probe failure - [ ] Watch for a repeat — If it restarts again, the event falls inside the 2-day window and the reason will show here

Ask next: Show docproc's memory on production over the last 7 days · Did docproc log errors on production before its restart? · Which production workloads restarted in the last 48 h?

The spot preemptions are cluster-wide node events. Nothing ties them to docproc's restart.

E03 Node incidents master real data Ask it live ↗
Any node incidents or preemptions on master this week?
tools: node_incidents · components: Timeline, KpiRow

Yes: master had 8 node incidents this week, 3 spot preemptions (each followed by an auto-repair) and 5 live migrations for host maintenance 1. None was followed by 5xx on services sharing the node. All three preemptions hit the same spot pool, z5asznva-3730cfb4 1.

  • Incidents: 8 (11 node events merged, last 7 d 1)
  • Spot preemptions: 3 (all auto-repaired 1)
  • Live migrations: 5 (host maintenance 1)
  • 5xx after: 0 (5 min after each, same node 1)

Node incidents on master, last 7 d (UTC) 1 - Nov 3 09:22 — Live-migrated for host maintenance: gk3-master-nap-wcc7s7td-1469370b-7pvb - Nov 4 21:43 — Live-migrated for host maintenance: gk3-master-nap-wcc7s7td-1469370b-77g4 - Nov 4 22:03 — Spot node preempted, then auto-repaired: gk3-master-nap-z5asznva-3730cfb4-ndcm - Nov 4 23:59 — Spot node preempted, then auto-repaired: gk3-master-nap-z5asznva-3730cfb4-bd4q - Nov 5 15:04 — Live-migrated for host maintenance: gk3-master-nap-wcc7s7td-ed06d1b1-qgjq - Nov 5 16:48 — Spot node preempted, then auto-repaired: gk3-master-nap-z5asznva-3730cfb4-4nmf - Nov 5 22:19 — Live-migrated for host maintenance: gk3-master-nap-wcc7s7td-56b443e2-krmp - Nov 6 19:47 — Live-migrated for host maintenance: gk3-master-nap-wcc7s7td-228a94b5-v5cl

  • Pods on these nodes were not mapped 1 — Each incident lists 0 pods because X-ray could not map the node's pods (mapped = false), not because the node was empty. Watched namespaces: tilda, neo4j-system, weaviate.

Ask next: Which master pods ran on the gk3-master-nap-z5asznva-3730cfb4 nodes this week? · Any node incidents or preemptions on production this week? · Did master's error rate rise after the Nov 5 16:48 UTC preemption?

"0 5xx after" holds only for services X-ray could place on each node. With no pod mapping, a disruption elsewhere wouldn't show up here.

E04 ArgoCD in sync production real data Ask it live ↗
Is every app in sync in ArgoCD on production?
tools: argocd_in_sync · components: StatusGrid, Table

Yes: production matches git. ArgoCD reports it clean, with 0 objects changed by hand in the last 24 hours 1. This check runs per environment, not per app. It tells you nothing was changed outside git, but not whether every app's last sync succeeded.

ArgoCD drift, last 24 h 1 - production: ok — clean: matches git, 0 hand changes

In-sync check, last 24 h 1

EnvironmentStateChanged by handPeopleCapacity changes
productionclean0none0
  • What "in sync" covers here 1 — A scale change counts as capacity, not drift, because git doesn't set replica counts. Restarts and service-account changes are excluded, so every hand change listed would be a person.

Ask next: Was anything on production changed by hand in the last 7 days? · Is master in sync with git right now? · Which ArgoCD syncs ran on production in the last 24 h?

The window is 24 h by default. Ask for up to 168 h to cover the week.

E05 Where is docsvc deployed production real data Ask it live ↗
Which clusters deploy docsvc, and which image tags do they pin?
tools: k8s_map · components: Table, Tags

6 overlays in ash-nazg deploy docsvc: production, delta, nexus, master, automation and dev 1. They pin 4 different tags. Production runs v2.10.46-c571bfa, delta v2.10.38-beta-dafae8a and nexus production-d716e5c; master, automation and dev all pin master-c3412ef 2.

Tags pinned for docsvc 2: v2.10.46-c571bfa, v2.10.38-beta-dafae8a, production-d716e5c, master-c3412ef

docsvc overlays in ash-nazg 1 2

OverlayImage tagPatched byTag set in
productionv2.10.46-c571bfalive nodepool + environment, resources-patchservices/docsvc/overlays/production/kustomization.yaml
deltav2.10.38-beta-dafae8alive nodepool + environment, delta nodepool-patchservices/docsvc/overlays/delta/kustomization.yaml
nexusproduction-d716e5cnamed nodepool, nexus nodepool-patchservices/docsvc/overlays/nexus/kustomization.yaml
mastermaster-c3412efnamed nodepoolservices/docsvc/overlays/master/kustomization.yaml
automationmaster-c3412efnamed nodepool, env-patchservices/docsvc/overlays/automation/kustomization.yaml
devmaster-c3412efenv-patchservices/docsvc/overlays/dev/kustomization.yaml
  • Rendered from a Helm chart 1 — The Deployment comes from the tilda-service chart (release docsvc). The overlays only patch it and pin the tag.

Ask next: What changed in docsvc between v2.10.38-beta-dafae8a (delta) and v2.10.46-c571bfa (production)? · Is docsvc on production actually running v2.10.46-c571bfa right now? · Which overlays deploy docproc, and which tags do they pin?

This is what git pins on the default branch. What each cluster is running right now could differ if a sync is pending. The map doesn't say which cluster the dev overlay targets.

Budgets & SLOs

F01 Closest to its budget production real data Ask it live ↗
Which service is closest to exhausting its error budget, and when?
tools: budgets → forecasts · components: Meters, Table, Callout

onboardingsvc is closest: it has spent 92% of its 24-hour error budget and has 8.1% left, after 201 5xx against an allowance of 219 1. I can't say when it would run out. Every production budget forecast reports incomplete rolling-window history, so none has an exhaustion date 2.

Error budget left on production, last 24 h 1 - onboardingsvc: 8.1% of 100 (201 of 218.8 bad events spent)

  • docsvc: 84.1% of 100 (5 of 31.4 spent)
  • execsvc: 92.3% of 100 (5 of 64.6 spent)
  • docproc: 100% of 100 (0 of 12.5 spent)
  • enrollsvc: 100% of 100 (0 of 1.8 spent)
  • notifsvc: 100% of 100 (0 of 1.2 spent)

⚠ No exhaustion date available 2 — All 22 budget:production forecasts have eta = null (reason: incomplete rolling-window history). Over the last 6 h onboardingsvc burned at 0.65×, below the 1× pace that empties a budget on schedule 1.

Production budgets, last 24 h 1, forecasts 2

ServiceObjectiveAchievedBad / allowedBudget leftBurn 6 hForecast ETAVerdict
onboardingsvc99% (derived)99.08%201 / 218.88.1%0.65×nonebudget low
docsvc99.9% (derived)99.98%5 / 31.484.1%0×nonehealthy
execsvc99.5% (derived)99.96%5 / 64.692.3%0×nonehealthy
docproc99.9% (derived)100%0 / 12.5100%0×nonehealthy

Ask next: What were onboardingsvc's 201 5xx on production in the last 24 h? · Show onboardingsvc's error-budget burn on production over the last 7 days · Which production services have a declared rather than derived objective?

Every objective here is derived from the service's own last week, not a declared SLO. The budget is a rolling 24 h window, so onboardingsvc recovers as its Nov 6 12:00 UTC burst ages out.

F02 Burn rate for one service production real data Ask it live ↗
What is onboardingsvc's error-budget burn rate and how long until it runs out?
tools: budgets → forecasts · components: Gauge, KpiRow, LineChart

onboardingsvc is burning at 0.65× over the last 6 h, slower than the pace that would empty its budget. The last 24 h have still used 92% of it, leaving 8.1% 1. I can't give a run-out date: its forecast has no ETA because the rolling-window history is incomplete 2.

  • onboardingsvc error budget left, last 24 h: 8.1% of 100 (201 of 218.8 allowed bad events spent 1)
  • Burn rate, 6 h: 0.65× (1× empties the budget on schedule 1)
  • Burn rate, 1 h: no data (1)
  • Achieved: 99.08% (objective 99%, derived 1)
  • Forecast ETA: none (incomplete history 2)

onboardingsvc 5xx per hour on production, Nov 6 06:00 – Nov 7 06:00 UTC 3

5xx
06:000
07:000
08:000
09:000
10:000
11:000
12:00145
13:000
14:000
15:000
16:006
17:0038
18:000
19:000
20:000
21:000
22:000
23:000
00:000
01:0012
02:000
03:000
04:000
05:000

Threshold: 9.1 even burn ≈ 9.1/h (derived)

  • One burst spent most of it 3 — 145 of the 201 bad events came in the 12:00 UTC hour on Nov 6. Only two other hours went over the even-burn line: 17:00 (38) and 01:00 (12).

Ask next: What were onboardingsvc's 145 5xx at 12:00 UTC Nov 6 on production? · What is onboardingsvc's error budget on production over the last 7 days? · Did a deploy precede onboardingsvc's 5xx burst on production?

The even-burn line is derived (the 24 h allowance ÷ 24). The objective is derived from last week. Because the window rolls, the 12:00 burst stops counting about 24 h later, and the budget recovers then unless new errors come in.

F03 Budget forecast production illustrative data Ask it live ↗
Forecast error-budget exhaustion for every production service.
tools: forecasts → budgets · components: LineChart, Table

Only onboardingsvc is on course to exhaust its 30-day error budget on production, around Nov 22 (band Nov 18–27) 1. notifsvc follows around Dec 17. Every other production service is flat or not trending toward its threshold 1.

⚠ onboardingsvc runs out around Nov 22 1 — It has spent 52% of a 3,088-bad-event budget and is adding about 21 points a week. Its objective (99%) is derived from its own last week 2.

Budget spent, 30-day window: actual to Nov 7, Theil–Sen projection after 1 (%)

onboardingsvcnotifsvcdocsvc
Oct 241253
Oct 3131146
Nov 752279
Nov 14744011
Nov 21965413
Nov 281176715
Dec 5—8117

Threshold: 100 budget exhausted

Error-budget forecasts, production — 10 of 22 services with traffic 1 2

ServiceBudget (30 d)SpentTrend / weekETABand
onboardingsvc3,088 bad events52%+21 ptsNov 22Nov 18 – Nov 27
notifsvc68 bad events27%+13 ptsDec 17Dec 6 – Jan 2
docsvc309 bad events9%+2.5 ptsnot trending—
execsvc617 bad events9%flatnot trending—
docproc118 bad events0%flatnot trending—
enrollsvc25 bad events4%flatnot trending—
schedsvc11 bad events0%flatnot trending—
virtual-assistant10 bad events0%flatnot trending—
dashql9 bad events0%flatnot trending—
tusker7 bad events0%flatnot trending—

Ask next: What is driving onboardingsvc's error-budget burn on production this week? · Why is notifsvc's budget burn on production rising since Oct 31? · Show onboardingsvc's 5xx per day on production for the last 14 days

Forecasts are a straight-line fit with a band, not a promise. A single burst or a fix moves the date. "Not trending" means the fit doesn't reach the threshold within 90 days.

F04 Reliability scorecard production real data Ask it live ↗
Give me a reliability scorecard for production services.
tools: budgets · components: StatusGrid, Table, FollowUps

Production is healthy apart from onboardingsvc. Its error budget is down to 8.1% after 201 5xx in the last 24 h (verdict: "budget low"). The other 12 services with traffic are healthy, and 10 of them had no bad events at all 1.

Production reliability, last 24 h 1 - onboardingsvc: warn — 8.1% budget left · 201 bad - docsvc: ok — 84.1% left · 5 bad - execsvc: ok — 92.3% left · 5 bad - docproc: ok — 100% · 12,468 calls - enrollsvc: ok — 100% · 1,824 calls - notifsvc: ok — 100% · 1,228 calls - tusker: ok — 100% · 671 calls - dashql: ok — 100% · 449 calls - virtual-assistant: ok — 100% · 413 calls - studiosvc: ok — 100% · 240 calls - schedsvc: ok — 100% · 84 calls - regsvc: ok — 100% · 43 calls - patientsvc: ok — 100% · 8 calls

Error budgets on production, last 24 h, availability SLI 1

ServiceObjectiveAchievedCallsBadBudget leftp95p99Verdict
onboardingsvc99%99.08%21,8832018.1%86 ms163 msbudget low
docsvc99.9%99.98%31,409584.1%524 ms1,876 mshealthy
execsvc99.5%99.96%12,923592.3%141 ms552 mshealthy
docproc99.9%100%12,4680100%980 ms5,292 mshealthy
enrollsvc99.9%100%1,8240100%419 ms490 mshealthy
notifsvc99.9%100%1,2280100%——healthy
tusker99.9%100%6710100%426 ms1,911 mshealthy
dashql99.9%100%4490100%1,955 ms4,253 mshealthy
virtual-assistant99.9%100%4130100%5,903 ms12,625 mshealthy
studiosvc99.9%100%2400100%956 ms1,733 mshealthy
schedsvc99.5%100%840100%478 ms580 mshealthy
regsvc99.9%100%430100%454 ms491 mshealthy
patientsvc99.9%100%80100%1,600 ms1,920 mshealthy
  • Availability only: no latency targets set 1 — Every objective is derived from the service's own last week, and no service has a latency target. virtual-assistant's 5.9 s p95 and dashql's 2.0 s p95 therefore don't count against any budget.

Ask next: What were onboardingsvc's 201 5xx on production in the last 24 h? · Is virtual-assistant's 5.9 s p95 on production normal for it? · How does master's reliability scorecard compare with production's?

patientsvc (8 calls) and regsvc (43) have too little traffic for 100% to mean much.

AI & LLM spend

G01 AI spend, 24 h production illustrative data Ask it live ↗
How much did we spend on AI in the last 24 hours, and which org drove it?
tools: ai_usage · components: KpiRow, Donut, Table

Production spent $118.40 on AI in the last 24 h across 4,812 calls 1. sandboxThermofisher drove the most: $46.10, 39% of the total, mostly Tmf Document Language Detection 1.

  • AI spend: $118.40 (last 24 h, a floor 1)
  • Calls: 4,812 (14 orgs 1)
  • Tokens: 61.2 M (38% cached 1)
  • Failures: 9 (0.19% of calls 1)
  • Top org: sandboxThermofisher ($46.10 · 38.9% 1)

AI spend by org, last 24 h 1 - sandboxThermofisher: 46.1 - tildainc: 21.85 - acmeTrials: 17.3 - novaCro: 12.65 - brightPathCro: 8.4 - 9 other orgs: 12.1

AI spend by model, last 24 h 1 - gpt-5.6-terra: 71.2 - gpt-5.2: 19.85 - gpt-5.4: 13.1 - gemini-3.8-flash: 9.75 - text-embedding-3-small: 0.85 - other: 3.65

AI spend by org on production, last 24 h — top 5 of 14 1

OrgCallsCostShareTop operation
sandboxThermofisher1,640$46.1038.9%Tmf Document Language Detection
tildainc1,205$21.8518.5%Data Monitor Review
acmeTrials802$17.3014.6%virtual-assistant chat
novaCro611$12.6510.7%Work order reminder — draft prose
brightPathCro284$8.407.1%Wo Reply Agent

Ask next: What did sandboxThermofisher's AI calls cost on production over the last 7 days? · Which operations drove gpt-5.6-terra's spend on production today? · Which AI calls failed on production in the last 24 h?

Cost is a floor: 37 calls had no price in the proxy log. These are LiteLLM's own figures, not the OpenAI or GCP bill.

G02 Costliest AI operations production illustrative data Ask it live ↗
Which AI operations cost the most on production this week?
tools: ai_usage · components: BarChart, Table

This week's costliest AI operation on production is Data Monitor Review, at $212.40 of $642.80 (33%). virtual-assistant chat ($148.75) and Tmf Document Language Detection ($96.20) come next, and the top three are 71% of the week's spend 1.

AI cost by operation on production, last 7 days — top 8 of 23 1 (usd)

cost
Data Monitor Review212.4
virtual-assistant chat148.75
Tmf Document Language Detection96.2
Work order reminder — draft prose61.3
rag-agent document Q&A44.85
Wo Reply Agent18.6
tilda-weaviate:RagDocs:write-embedding4.4
Work order — status summary2.15
  • Data Monitor Review is the expensive one per call 1 — $0.18 a call, about 5× virtual-assistant chat. Each review sends ~150k tokens to gpt-5.6-terra.

AI operations on production, last 7 days 1

OperationCallsCostShareCost / callMain modelAvg latency
Data Monitor Review1,184$212.4033.0%$0.179gpt-5.6-terra11.2 s
virtual-assistant chat3,960$148.7523.1%$0.038gpt-5.6-terra4.8 s
Tmf Document Language Detection9,412$96.2015.0%$0.010gemini-3.8-flash2.6 s
Work order reminder — draft prose2,870$61.309.5%$0.021gpt-5.23.4 s
rag-agent document Q&A1,515$44.857.0%$0.030gpt-5.46.1 s
Wo Reply Agent1,102$18.602.9%$0.017gpt-5.21.9 s
tilda-weaviate:RagDocs:write-embedding11,656$4.400.7%$0.0004text-embedding-3-small0.3 s
Work order — status summary846$2.150.3%$0.003gpt-6-luna2.2 s

Ask next: Which orgs triggered Data Monitor Review on production this week? · How much of Data Monitor Review's input on production is cached? · What did virtual-assistant chat cost per day on production this week?

Cost is a floor where calls lack a price (52 this week). The remaining 15 operations add up to $54.15.

G03 OpenAI vs LiteLLM production real data Ask it live ↗
Why don't OpenAI's bill and LiteLLM's log match this month?
tools: openai_costs → openai_vs_litellm · components: LineChart, Table, Callout

For production the two match. OpenAI billed $630.75 for Nov 1–6 and LiteLLM logged $630.78, a $0.03 difference, and no day is off by more than $0.15 2. The gap you're seeing is most likely scope. OpenAI's org-wide bill for the same days is $1,245.14, and the other $614.39 belongs to other OpenAI projects (eval, automation, master, nexus, test27, delta, test26), which production's proxy log doesn't cover 1.

  • OpenAI bill, whole org: $1,245.14 (Nov 1–6, 8 projects 1)
  • OpenAI bill, production: $630.75 (Nov 1–6 2)
  • LiteLLM log, production: $630.78 (Nov 1–6 2)
  • Difference: −$0.03 (billed − logged 2)

OpenAI bill vs LiteLLM log per UTC day, Nov 1–6 1 2 (usd)

OpenAI, whole orgOpenAI, productionLiteLLM, production
Nov 121.55.665.66
Nov 2169.61110.16110.16
Nov 3229.43164.16164.04
Nov 4242.93106.6106.75
Nov 5389.64149.91149.91
Nov 6192.0394.2694.26

✓ Production reconciles to within cents 2 — Requests match model by model: gpt-5.6-terra 10,350 vs 10,350. 8 LiteLLM calls carried no cost (7 on gpt-5.4, 1 on gpt-5.2).

By OpenAI project OpenAI's bill by project, Nov 1–6 1

OpenAI projectBilled Nov 1–6In production's LiteLLM log?
production$630.75yes: logged $630.78
eval$197.49no
automation$157.20no
master$90.58no
nexus$69.39no
test27$56.44no
delta$25.76no
test26$17.52no

Production by model production, OpenAI vs LiteLLM by model, Nov 1–6 2

ModelOpenAI billedLiteLLM loggedBilled − loggedOpenAI requestsLiteLLM calls
gpt-5.6-terra$448.29$448.29$0.0010,35010,350
gpt-5.2$109.67$109.70−$0.039,5789,581
gpt-5.4$68.34$68.34$0.003,3733,380
text-embedding-3-small$4.44$4.44$0.00011,656

Ask next: Reconcile OpenAI's automation project with the automation LiteLLM log for Nov 1–6 · Reconcile OpenAI's master project with master's LiteLLM log for Nov 1–6 · Which production operations drove gpt-5.6-terra's $448 in Nov 1–6?

Nov 6 may still be filling in, since OpenAI's copy refreshes about every 12 h 1. I didn't reconcile the other projects, so whether eval, automation or master match their own logs is still open.

G04 Failing LLM calls master real data Ask it live ↗
Are LLM calls failing on master? Which models and operations?
tools: llm_calls → ai_usage · components: Donut, Table

Barely. 1 of 502 LLM calls on master failed in the last 6 h: an anthropic/claude-opus-5-5 call from X-ray's xray_optimizer, which ended in *Server disconnected* 1 2. No other model or operation had a failure 2.

LLM calls on master by status, last 6 h 2 - success: 501 - failure: 1

  • Calls: 502 (8 models, 14 operations 2)
  • Failures: 1 (0.2% 2)
  • Cost: $21.23 (a floor: 1 call unpriced 2)

Failed LLM calls on master, last 6 h 1

Time (UTC)ModelOperationOrgError
Nov 7 05:12anthropic/claude-opus-5-5xray_optimizersystem/xrayInternalServerError: Server disconnected

LLM calls on master by model, last 6 h 2

ModelCallsFailuresCost
anthropic/claude-sonnet-5-54160$15.73
gpt-6.1-terra380$0.47
gpt-5.6-terra170$4.42
anthropic/claude-opus-5-5111$0.60
gpt-6-luna100$0.004
gemini-3.8-flash40$0.005
openai/text-embedding-3-small40$0.00
anthropic/claude-haiku-4-520$0.001
litellm.InternalServerError: AnthropicException - Server disconnected. Handle with `litellm.InternalServerError`.

Ask next: Are LLM calls failing on production in the last 6 h? · Show xray_optimizer's calls on master in the last 24 h · What did the Test agent run operation cost on master today?

One disconnect from the provider looks transient, not a pattern. A wider window would show whether it repeats.

GCP cost

H01 Where the money goes production real data Ask it live ↗
Where is the money going this month?
tools: cost_overview → cost_breakdown · components: KpiRow, Donut, BarChart, LineChart

November's net GCP spend is $2,086.71 for Nov 1–5, on course for about $13,527 (range $12,571–$14,484) against October's $12,337.77. That's 12% ahead of the same point last month 1. Kubernetes Engine is 40% of it ($828.41), followed by Compute Engine ($318.97) and Pub/Sub ($138.85, up $39.75 on the previous five days) 2.

  • Month to date: $2,086.71 (Nov 1–5, net 1)
  • Projected November: $13,527 ($12,571–$14,484, derived 1)
  • October: $12,337.77 (full month 1)
  • vs same point last month: +12% ($1,867.70 by Oct 5 1)
  • Credits: $993.37 (CUD $711.97 · discounts $252.40 1)

Net by project, Nov 1–5 1 - tilda-trial-production: 678.76 - tilda-trial-dev: 602.32 - tilda-trial-automation: 458.51 - tilda-control-plane: 144.8 - tilda-dev-325721: 120.14 - other: 82.18

Net by GKE cluster, Nov 1–5 1 - not cluster-attributed: 972.49 - production: 279.54 - shared-neo4j-5-26: 237.8 - master: 108.26 - automation: 102.28 - test27: 101.37 - other clusters: 284.97

Net by service, Nov 1–5 vs Oct 27–31 — top 8 of 32 2 (usd)

Nov 1–5Oct 27–31
Kubernetes Engine828.41935.31
Compute Engine318.97288.22
Cloud Pub/Sub138.8599.1
Vertex AI121.92113.22
Artifact Registry117.3590.32
Cloud Monitoring83.19125.21
Anthos Service Mesh67.7686.72
Security Command Center66.6868.34

Average net per day by window, UTC (derived from window totals) 1 (usd)

net per day
Oct 9–22383
Oct 23–29451
Oct 30–Nov 5418

Threshold: 414.61 run rate $415/day

Ask next: What is Kubernetes Engine costing per cluster this month? · Why did Cloud Pub/Sub rise $39.75 in Nov 1–5 vs Oct 27–31? · What does the production cluster cost per namespace this month?

The figures are complete through Nov 5, and the export lags about 4 h. The projection is a Theil–Sen estimate, so it is derived, not billed. 47% of spend is project-level, so it can't be attributed to a cluster.

H02 Biggest cost spike production real data Ask it live ↗
Explain the biggest cost spike in the last 14 days.
tools: cost_anomalies → cost_explain · components: Timeline, BarChart, Callout

The biggest spike was on Oct 27: Cloud Translation on tilda-trial-production billed $188.23 against a normal $0.14 a day 1. That was 93.7% of the project's $200.66 jump that day. Almost all of it was the Translation V3 Document Translation SKU, and almost all of it landed in the 13:00 UTC hour 2. It wasn't a one-off: Translate also spiked on Oct 26 ($31.36) and Nov 2 ($46.72) 1.

Cost anomalies, last 14 days (UTC days) 1 - Oct 26 — Translate $31.36 (normal $0.14): tilda-trial-production, project-level · z 151 - Oct 27 — Translate $188.23 (normal $0.14): tilda-trial-production · z 911, the biggest - Oct 27 — Vertex AI $28.40 (normal $1.65): tilda-trial-dev · z 12 - Nov 2 — Translate $46.72 (normal $0.08): tilda-trial-production · z 393

What moved on Oct 27, tilda-trial-production vs the previous 7-day median 2 (usd)

change vs normal
Translate188.04
Vertex AI6.68
Cloud Vision API4.7
Cloud Pub/Sub1.64
Kubernetes Engine1.12
Compute Engine-1.57

tilda-trial-production net per hour, Oct 27 (UTC) 2 (usd)

net
00:004.4
01:004.48
02:004.3
03:004.24
04:004.63
05:004.33
06:006.34
07:004.69
08:004.54
09:004.64
10:004.65
11:007.1
12:007.06
13:00192.78
14:006.08
15:006.57
16:008.83
17:007.01
18:004.81
19:005.35
20:004.7
21:004.7
22:004.72
23:004.39
Neural Translation Model Online Prediction In Translation V3 Document Translation.

⚠ The bill can't name the caller 2 — The Translate cost is project-level, with no cluster, namespace or workload. On the same day, production's API calls ran 1.35× normal and Neo4j queries 2.4×, and rag-agent.upload and exportsvc.activity_update were failing. Those signals are adjacent, not shown to be the cause.

Ask next: Which production service calls Cloud Translation document translation? · Explain the Nov 2 Translate spike on tilda-trial-production · What has Translate cost per day on tilda-trial-production over the last 30 days?

The day's total for the project was $315.32 against a $114.66 median 2. This kind of usage-billed API spike comes from whoever submitted the documents, and nothing in the bill identifies them.

H03 Where could we save production real data Ask it live ↗
Where could we save money? Rank the best moves.
tools: cost_recommendations → cost_commitments → cost_ai · components: Table, Meters, Steps

The 10 cost moves listed add up to about $257 a month, and the best are AI caching and Kubernetes requests, not commitments 1. Turn on Gemini context caching for gemini-3.8-flash ($70.12/mo, an X-ray estimate) and gemini-3.0 ($38.54/mo), check x-ray's growth on master ($51.74/mo), and right-size two over-requested deployments ($39.00 and $31.83/mo, from Google) 1. Commitments leave nothing to reclaim: $6.07 of $4,272.77 went unused in 30 days 2.

Cost moves ranked by monthly saving — 10 cost items of 23 open recommendations 1

#MoveSourceSaving / monthEvidence
1Context caching for gemini-3.8-flashX-ray (estimate)$70.12$277 of input in 30 d, $0 cached 1; $323.49 billed 3
2Check x-ray workload growth on masterX-ray$51.74$40 in 14 d, +151% vs $16 1
3Right-size argocd-repo-server CPU requests (control-center-dev)Google$39.00over-provisioned 1
4Context caching for gemini-3.0X-ray (estimate)$38.54$152 of input, $0 cached 1; $234.52 billed 3
5Right-size x-ray CPU requests (master)Google$31.83over-provisioned 1
6Snapshot and delete idle disk pvc-f7471750 (production, us-west1-b)Google$11.61idle disk 1
7Snapshot and delete 4 more idle disks (production, dev)Google$13.94$5.81 + $5.81 + $1.16 + $1.16 1

Commitment utilisation, last 30 d 2 - Kubernetes Engine CUD (3 yr): 100% of 100 ($2,376 fee, $0 unused)

  • Compute Flexible CUD (3 yr): 99.7% of 100 ($1,800 fee, $5.17/mo unused)
  • Cloud SQL CUD (3 yr): 99% of 100 ($96.77 fee, $0.98/mo unused)

Gemini input served from cache, last 30 d 3 - gemini-3.8-flash: 0% of 100 ($323.49 billed)

  • gemini-3.0: 0% of 100 ($234.52 billed)

Order to do them in - [ ] Enable context caching on gemini-3.8-flash and gemini-3.0 — Find the operations that resend the same system prompt or documents. About $109/mo combined (estimates) 1 - [ ] Look at x-ray on master — X-ray ($51.74) and Google ($31.83) both flag it, so the two savings overlap; don't add them. Compare requests with use on x-ray pods - [ ] Right-size argocd-repo-server on control-center-dev — Google's CPU-request recommendation, $39.00/mo 1 - [ ] Snapshot and delete the 5 idle disks — $25.55/mo together. Confirm nothing mounts them first 1

⚠ Google's list is partial 1 — Cloud Run cost recommendations were denied (403) on tilda-trial-production, tilda-trial-dev and tilda-control-plane, one recommender failed on tilda-trial-automation (invalid location), and commitment recommendations were denied at the billing account.

Ask next: Which operations resend the same context to gemini-3.8-flash on production? · Why did x-ray's cost on master grow 151% in 14 days? · What does Translate cost per day on tilda-trial-production?

X-ray's AI-caching savings are estimates, while Google's figures are Google's own. The $257 counts x-ray on master twice. The list shows 12 of 23 open recommendations, so more may exist, and two of the 12 (PodDisruptionBudgets on master and automation) are reliability items with no dollar figure.

H04 Cost per cluster per day production real data Ask it live ↗
What does each GKE cluster cost per day over the last 14 days?
tools: cost_breakdown · components: BarChart, LineChart

Over the 14 days Oct 23 – Nov 5, production is the most expensive GKE cluster at $795.28 net, about $56.81 a day 1, followed by shared-neo4j-5-26 at $684.68 (~$48.91 a day) 1. Production's own daily bill ran between $52.24 and $64.09 2; the biggest riser is test27, up $122.31 on the previous 14 days 1.

  • production: $56.81/day ($795.28 over 14 days 1)
  • shared-neo4j-5-26: $48.91/day ($684.68 over 14 days 1)
  • Biggest rise: test27 +$122.31 ($247.03 → $369.33 vs Oct 9–22 1)
  • No cluster: $203.13/day ($2,843.78 project-level, 46.7% of the bill 1)

Average net cost per day by GKE cluster, Oct 23 – Nov 5 vs Oct 9 – 22 (UTC) 1 (usd)

Oct 23 – Nov 5Oct 9 – 22
production56.8152.72
shared-neo4j-5-2648.9145.04
test2726.3817.64
master22.0420.06
automation16.1914.2
control-center-dev14.4814.51
delta13.6213.61
argocd-control-center213.2513.4
test2610.089.72
nexus9.9410.5

production cluster, net cost per day (UTC) 2 (usd)

production
Oct 2358.04
Oct 2457.93
Oct 2557.88
Oct 2664.09
Oct 2756.78
Oct 2856.77
Oct 2955.31
Oct 3055.61
Oct 3153.33
Nov 152.24
Nov 254.23
Nov 354.95
Nov 456.87
Nov 561.25

Threshold: 56.81 14-day average

Net cost by cluster, Oct 23 – Nov 5 1

ClusterNet, 14 daysPer dayGrossCreditsPrevious 14 daysChange
production$795.28$56.81$1,374.24-$578.96$738.02+$57.25
shared-neo4j-5-26$684.68$48.91$1,180.95-$496.27$630.51+$54.17
test27$369.33$26.38$517.00-$147.67$247.03+$122.31
master$308.52$22.04$558.27-$249.75$280.85+$27.67
automation$226.63$16.19$404.32-$177.69$198.82+$27.81
control-center-dev$202.77$14.48$367.16-$164.39$203.14-$0.37
delta$190.70$13.62$240.41-$49.70$190.59+$0.11
argocd-control-center2$185.53$13.25$290.59-$105.06$187.67-$2.14
test26$141.17$10.08$167.50-$26.34$136.12+$5.05
nexus$139.14$9.94$165.74-$26.60$146.93-$7.79
  • Per-day figures for clusters other than production are 14-day averages — Only production's day-by-day series was read 2; the others need one cost_search per cluster. 46.7% of the bill has no cluster at all (project-level logging, networking, storage) 1.

Ask next: Why did test27's cost rise $122 over the previous 14 days? · Break production's cost down by namespace for the last 14 days · What is in the $2,843.78 that is not attributed to any cluster?

Cost is complete through Nov 5 (UTC usage days, net of credits); the export lags ~4 h 1. The per-day bars are derived (net ÷ 14), not read day by day.

H05 What does Neo4j cost production real data Ask it live ↗
What is Neo4j costing us?
tools: cost_search · components: KpiRow, BarChart, Table

Everything named Neo4j cost $2,084.76 over the last 30 days (Oct 7 – Nov 5), 17.1% of the $12,182.11 bill 1. Most of it is the shared-neo4j-5-26 GKE cluster ($1,402.03); the daily run-rate stepped up from about $65–68 to $72–74 from Oct 23 1.

  • Neo4j, 30 days: $2,084.76 (Oct 7 – Nov 5 1)
  • Share of the bill: 17.1% (of $12,182.11 1)
  • shared-neo4j-5-26: $1,402.03 (the cluster, all 30 days 1)
  • Per day: $72–74 (since Oct 23; was $65–68 1)

Neo4j net cost per day, Oct 7 – Oct 31 (UTC) 1 (usd)

Neo4j
Oct 765.78
Oct 864.79
Oct 964.8
Oct 1064.25
Oct 1166.49
Oct 1267.99
Oct 1368.08
Oct 1468.36
Oct 1567.86
Oct 1667.47
Oct 1766.99
Oct 1867.01
Oct 1967.8
Oct 2067.01
Oct 2167.89
Oct 2268.07
Oct 2372.88
Oct 2472.3
Oct 2572.05
Oct 2674.03
Oct 2773.77
Oct 2874.37
Oct 2973.4
Oct 3073.79
Oct 3170.91

Neo4j cost by named resource, last 30 days 1 (usd)

net
shared-neo4j-5-26 (GKE cluster)811.44
logs-raw-neo4j-bq (Pub/Sub subscription, 2 rows)110.14
production-tilda-neo4j-backups (bucket)54.49
logs-raw-neo4j (Pub/Sub topic, 2 rows)41.34
dev-tilda-neo4j-backups (bucket)18.01
delta-tilda-neo4j-backups (bucket)16.64
x-ray-logalert-master-neo4j (Pub/Sub subscription)13.64
neo4j-automation-log-forwarder (Pub/Sub subscription)13.51

Where "neo4j" matched on the bill 1

MatchedDimensionNet, 30 daysDays
shared-neo4j-5-26cluster$1,402.0330
neo4j-systemnamespace$1,282.4130
neo4j-differential-backup-deltaworkload$7.7730
neo4j-differential-backup-masterworkload$7.2930
neo4j-differential-backup-productionworkload$5.5530
neo4j-log-forwarderworkload$2.3730
neo4j-differential-backup-test26workload$0.1330
neo4j-differential-backup-test27workload$0.1330

⚠ Don't add these rows up — The dimensions overlap (the neo4j-system namespace is inside the cluster), and the per-resource ledger ($1,103.86) is already inside the $2,084.76 — each row of the bill is counted once 1.

Ask next: Break shared-neo4j-5-26's cost down by node pool for the last 30 days · What changed on shared-neo4j-5-26 around Oct 23, when its daily cost rose? · What do the logs-raw-neo4j Pub/Sub topic and subscriptions cost per day?

Complete through Nov 5 1; the daily line stops at Oct 31 because Nov 1–5 are on the result's second page, which I did not read. Named resources are kept 35 days, and this is a word match — Neo4j cost billed under rows that don't say "neo4j" is not in it.

H06 Commitments used production real data Ask it live ↗
Are our committed-use discounts actually used?
tools: cost_commitments · components: Gauge, Meters, Callout

Yes. Over the last 30 days (Oct 7 – Nov 5) the three commitments were 99.9% used: $4,266.70 of $4,272.77 in fees funded real usage and only $6.07 went unused 1. They also put $3,576.94 of usage below list price, most of it on Kubernetes Engine 1.

✓ Commitments are fully used: $6.07 unused in 30 days 1 — The GKE CUD funded every dollar of its fee; Compute Flexible left $5.10 and Cloud SQL $0.97 unused.

  • Utilisation, all commitments: 99.86% of 100 ($4,266.70 of $4,272.77 funded usage 1)

Usage billed below list price, 30 days 1 - Kubernetes Engine · 26.8% off: 2,750.84 - Compute Engine · 17.1% off: 722.32 - Cloud SQL · 33.0% off: 103.78

Utilisation by commitment, Oct 7 – Nov 5 1 - Kubernetes Engine CUD · us-central1 · 3 yr: 100% of 100 (fee $2,376.00 · unused $0.00 · 3 subscriptions)

  • Compute Flexible CUD · 3 yr: 99.7% of 100 (fee $1,800.00 · unused $5.10 ($5.17 a month))
  • Cloud SQL CUD · us-central1 · 3 yr: 99% of 100 (fee $96.77 · unused $0.97 · tilda-trial-automation)

What the commitments fund, top 8 clusters each 1

ClusterFunded by GKE CUDFunded by Compute Flexible CUD
production$698.61$535.27
shared-neo4j-5-26$507.30$503.15
master$365.50$131.25
test27$256.77$91.26
control-center-dev$248.07$88.87
automation$238.05$90.89
nexus$31.38not in top 8
test26$30.33not in top 8
argocd-control-center2not in top 8$216.12
deltanot in top 8$91.93

Ask next: Is there steady GKE usage on production above what the CUD covers? · What are Google's cost recommendations for tilda-trial-production? · What did Kubernetes Engine cost at list price vs what we paid over the last 30 days?

Complete through Nov 5 1. Full use says the commitments are not too big; whether a larger one would pay off is not in this result — cost_recommendations would show it.

H07 Cost per customer org production real data Ask it live ↗
What does each customer org cost us?
tools: cost_by_org · components: BarChart, Table, Callout

Over the last 30 days production's directly attributed $1,637.00 splits most heavily to thermofisher ($273.79), then calls with no org ($257.29) and perceiveBio ($172.99) 1. These are estimates: the cost is apportioned by each org's per-day share of 936,913 API calls, not billed per customer 1.

⚠ Apportioned, not billed 1 — The bill knows workloads, not customers. Production's $1,637.00 direct net cost is split by each org's share of API calls per day; project-level shared costs (logging, networking, storage) are not included.

  • Production direct cost: $1,637.00 (Oct 7 – Nov 5 1)
  • API calls: 936,913 (the basis of the split 1)
  • Top org: thermofisher ($273.79 · 18.4% of calls 1)
  • No org: $257.29 (19.7% of calls carry no org 1)

Apportioned cost by org, production, last 30 days (top 12 of 30) 1 (usd)

apportioned cost
thermofisher273.79
(no org)257.29
perceiveBio172.99
sandboxIconPlc130.61
ollin124.44
mintraHealth104.63
sandboxThermofisher72.25
tildainc66.59
surrozen56.3
aiTeammatesDemo46.18
uPennSANDBOXByDT40.68
sandboxOllin30.88

Same cost by product area 1 - TMF: 704.26 - Account & Org Settings: 424.29 - EDC: 160.6 - Studies: 142.02 - Home: 79.82 - Studio: 45.5 - 15 other areas: 80.5

Cost per org, production, Oct 7 – Nov 5 (25 of 30 orgs read) 1

OrgAPI callsShare of callsApportioned cost
thermofisher172,72218.4%$273.79
(no org)184,20119.7%$257.29
perceiveBio81,6888.7%$172.99
sandboxIconPlc67,0437.2%$130.61
ollin95,19810.2%$124.44
mintraHealth56,6886.1%$104.63
sandboxThermofisher42,1164.5%$72.25
tildainc32,3923.5%$66.59
surrozen38,8254.1%$56.30
aiTeammatesDemo8,3720.9%$46.18
uPennSANDBOXByDT33,8283.6%$40.68
sandboxOllin14,4471.5%$30.88
inlandPsychiatricMedicalGroupInc14,8631.6%$27.33
retinaMaculaInstituteOfArizona5,1070.6%$24.63
uPenn4,4680.5%$23.91
ophthalmicConsultantsOfBoston5,1710.6%$22.15
jade3,9910.4%$22.14
sandboxDemoOrg16,3021.7%$20.22
vedaAawc3,8700.4%$13.35
tildaScale5,9360.6%$12.17
1SouthernCollegeOfOptometry5,8470.6%$11.13
sandboxIuvo5,4460.6%$8.97
vedaCrcsi5,0980.5%$8.94
vedaAaecq3,2420.3%$5.88
sandboxEmerald3,3240.4%$5.68

Ask next: What does thermofisher cost on production per day over the last 30 days? · Which services do the (no org) calls on production go to? · What does each org cost on delta over the last 30 days?

Because the split is per day, an org's cost isn't simply proportional to its 30-day call count (ollin has more calls than perceiveBio but a smaller share) 1. Five small orgs on the result's second page were not read.

Users & journeys

I01 Logins and lockouts production real data Ask it live ↗
Any login failures or lockouts in Auth0 today?
tools: auth0 · components: KpiRow, LineChart, Table

No lockouts, and nothing that looks like an attack. In the last 6 h (about 01:00–06:55 UTC) Auth0 recorded 2 failed logins out of 11, and no event had a block or lockout outcome 1. One failure was the [email protected] service login, which succeeded in the same second; the other was a "couldn't find your session" error a minute after akhil.mekala logged in successfully 2.

✓ No lockouts or blocks in the last 6 h 1 — 6 of 44 tenant events failed: 2 logins and 4 refresh-token exchanges. Every user who failed also logged in successfully.

  • Successful logins: 9 (last 6 h 1)
  • Failed logins: 2 (18.2% of 11 attempts 1)
  • Lockouts / blocks: 0 (no block outcome recorded 1)
  • Users: 6 (1 org (tilda.bio) · 9 IPs 1)

Auth0 events per hour, production tenant, Nov 7 (UTC) 2

all eventssuccessful loginsfailures
01:001022
02:00512
03:00820
04:001742
05:00200
06:00200

Every failure, last 6 h 2

Time (UTC)EventWho / fromReason
01:11Token · refresh failed35.226.74.193 (Council Bluffs, US)Token could not be decoded or is missing in DB
01:15Token · refresh failed35.226.74.193Token could not be decoded or is missing in DB
02:25Failed login · password grant[email protected] · 35.226.74.193Unauthorized — a successful login followed in the same second
02:55Token · refresh failed35.226.74.193Token could not be decoded or is missing in DB
04:19Token · refresh failed103.66.213.3 (Mumbai)same second as [email protected]'s logout
04:31Failed loginHyderabad, Indiacouldn't find your session (back button / cookies) — 1 min after [email protected] logged in

Who signed in, last 6 h 1

UserLoginsFailsLast seen (UTC)From
[email protected]2003:43Nagpur, India
[email protected]2004:30India
[email protected]2102:2535.226.74.193
[email protected]1004:20Mumbai, India
[email protected]1001:10Yamuna Nagar, India
[email protected]1004:39India

Ask next: Show every Auth0 event from 35.226.74.193 in the last 6 hours · Any login failures or lockouts on the master tenant in the last 6 hours? · What did [email protected] do on production after logging in at 04:30?

"Today" here is the last 6 h; I kept the read small, so 00:00–00:55 UTC was not read. 17 of the 44 events are machine-to-machine tokens (Tilda Sense Bot, Vanta), all successful 1.

I02 One user's session production real data Ask it live ↗
What did the most active user on production do in their last session?
tools: journeys → journey_detail · components: Timeline, KpiRow

The most active user on production in the last 6 h was [email protected] (tildainc): 33 sessions and 1,001 requests 1. His last full working session (01:10–01:16 UTC, 120 requests) went sign-in → Home → Studies → Study › Files, where he spent 262 s and hit 12 failed requests, 6 of them 500s on GET /api/myprofile/visibility/study/:id/blindedScopes 2.

  • Session length: 6m 3s (01:10–01:16 UTC 2)
  • Requests: 120 (7 screens 1)
  • Failed requests: 12 (all on Study › Files 2)
  • App load: 4.9 s (main content at 9.0 s 2)

[email protected] on production, Nov 7 (UTC) 2 - 01:10:17 — Signed in with SSO: POST /register/discover-method · onboardingsvc - 01:10:30 — Home: app ready in 4.9 s, main content at 9.0 s · 18 requests - 01:10:42 — Pressed Studies: 7 requests, slowest 3,256 ms - 01:10:46 — Opened Study › Files: study 4a5c6740-6896-4c3d-ab01-7d52a2f4485f · 262 s, 76 requests - 01:10:49 — GET /api/documents took 4.0 s: 5 times on this screen - 01:11:12 — PUT storage.upload took 6.1 s: 4 uploads - 01:11:19 — GET …/blindedScopes failed with 500: 6 times, after pressing a grid cell - 01:16:20 — Session ends: Files › Collections was the last area visited

Where the session went (first 3 of 5 visits) 2

ScreenTime on screenRequestsFailedSlowestHealth
Home12 s180308 mshealthy
Studies3 s703,256 mswarning
Study › Files262 s76124,627 msfailed

One of the blindedScopes 500s · [email protected]

Ask next: Why did trace ef8d7018f29718b0e3063cdd3fcbfd8a return 500 on production? · Is blindedScopes failing for other users on production in the last 6 hours? · Why did GET /api/documents take 4 s for [email protected] on production?

"Most active" is by requests; by session count [email protected] leads with 38 1. Harish's very last session (01:49 UTC) was a 2-second view of one TMF collection, so I showed the last one with real work in it 1. The Neo4j log was not joined, so Cypher per call is unknown 2.

I03 Most active orgs production real data Ask it live ↗
Which orgs were most active on production today?
tools: overview → journeys · components: BarChart, Table

Only one org had sessions on production this morning: tildainc, with 124 of 129 sessions in the last 6 h, and every one of its users is @tilda.bio 2. Production is quiet overall, with 4,510 requests from 8 users in 4 orgs over 7 h 1.

  • No customer org opened a session in the last 6 h 2 — 124 of 129 sessions were tildainc, Tilda's own staff; the other 5 were signed out. Overview counts 4 orgs over 7 h 1; the other 3 had no sessions in the 6 h journeys read.
  • Requests: 4,510 (last 7 h 1)
  • Active users: 8 (4 orgs, last 7 h 1)
  • Sessions: 129 (last 6 h 2)
  • Error rate: 1.4% (12 5xx, last 7 h 1)

Sessions by org, production, last 6 h 2

sessions
tildainc124
(none)5

Who was active, last 6 h 2

UserOrgSessionsRequestsErrors
[email protected]tildainc331,00112
[email protected]tildainc387070
[email protected]tildainc206400
[email protected]tildainc316190
[email protected]tildainc2420
(signed out)(none)5110

API calls by service, production, last 7 h 1

API calls
docsvc1,912
onboardingsvc1,858
docproc298
enrollsvc169
execsvc50
tusker11
schedsvc5

Ask next: Which orgs made the most API calls on production over the last 30 days? · Why is execsvc failing 98% of its 50 calls on production? · What did [email protected] do on Study › Files on production?

"Today" is 00:00–06:55 UTC on a Saturday; overview covers 7 h and journeys 6 h, and overview has a 10-minute coverage gap at 06:45–06:55 1. Per-org request counts are not in overview, so the 3 orgs without sessions are not named.

Security & perimeter

J01 Security findings production real data Ask it live ↗
What are the open Security Command Center findings in production? What should we fix first?
tools: scc_findings · components: Donut, BarChart, Table, Steps

tilda-trial-production has 6,907 active, unmuted Security Command Center findings: 460 critical and 4,077 high 1. Fix first the critical CVEs in images running on the production cluster that already have a fixed version: regsvc's openssl, rag-agent-worker's pillow, and X-ray's own x-ray-weaviate-stats CronJob (Go 1.22.6). All three score CVSS 9.8 1.

  • Active findings: 6,907 (unmuted, exact count 1)
  • Critical: 460 (1)
  • High: 4,077 (1)
  • Threats: 2,266 (2,116 are RBAC privilege-escalation 1)

Findings by severity 1 - Critical: 460 - High: 4,077 - Medium: 2,148 - Low: 222

Findings by class 1 - Vulnerability: 4,536 - Threat: 2,266 - Misconfiguration: 105

Top categories, tilda-trial-production 1

findings
SOFTWARE_VULNERABILITY2,441
Privilege Escalation: sensitive Kubernetes RBAC changes2,116
OS_VULNERABILITY2,068
Persistence: New API Method147
PRIVATE_GOOGLE_ACCESS_DISABLED44
GKE_SECURITY_BULLETIN27
BUCKET_LOGGING_DISABLED17

Critical findings read (5 of 460) 1

CVECVSSWorkloadClusterPackageInstalled → fixed
CVE-2026-630739.8tilda/regsvc · regsvc:v2.10.43-86c75b5productionopenssl3.5.5-r0 → 3.5.8-r0
CVE-2026-1129009.8tilda/rag-agent-worker · rag-agent:v2.10.43-86c75b5productionpillow12.3.0 → 13
CVE-2026-271439.8x-ray/x-ray-weaviate-stats CronJob · x-ray:3.7.1-5aab6eaproductiongo1.22.6 → 1.25.9
CVE-2026-398219.6tilda/docsvc · docsvc:v2.10.38-beta-dafae8adeltago1.25.9 → 1.25.13
CVE-2026-89279.1tilda/rag-agent-worker · rag-agent:v2.10.38-beta-dafae8adeltacurl8.14.1-2+deb13u5 → 8.14.1-2+deb13u7

What to fix first - [ ] Rebuild regsvc and rag-agent on patched packages — openssl ≥ 3.5.8-r0 and pillow 13; both images are v2.10.43, running on production 1 - [ ] Rebuild X-ray's weaviate-stats image on Go ≥ 1.25.9 — x-ray:3.7.1-5aab6ea ships Go 1.22.6 1 - [ ] Roll delta forward from v2.10.38-beta — its docsvc (Go) and rag-agent-worker (curl) carry critical CVEs with fixes available 1 - [ ] Triage the 2,116 RBAC privilege-escalation threats — find the principal making the RBAC changes; if it is the deploy pipeline, mute by rule rather than one by one - [ ] Close the one-offs — PUBLIC_BUCKET_ACL (1), USER_MANAGED_SERVICE_ACCOUNT_KEY (3), SERVICE_ACCOUNT_KEY_NOT_ROTATED (3) 1

Ask next: List every CRITICAL OS_VULNERABILITY finding on the production cluster · Which principal made the RBAC changes behind the 2,116 privilege-escalation findings? · Which bucket has the PUBLIC_BUCKET_ACL finding in tilda-trial-production?

Counts are exact; only 50 findings were listed, so the table shows 5 of the 460 critical ones 1. I did not count findings per image, so how many a single rebuild clears is not known.

J02 What is exposed production illustrative data Ask it live ↗
What is exposed to the internet in production?
tools: gcp_network · components: KpiRow, Table, Callout

In tilda-trial-production, 3 firewall rules accept traffic from 0.0.0.0/0 and 2 external load balancers face the internet 1. The one to fix is default-allow-ssh: tcp:22 open to the whole internet on the default network 1. Both GKE control planes (production, delta) have public endpoints, limited by authorized networks 1.

✗ SSH is open to the internet on the default network 1 — default-allow-ssh allows tcp:22 from 0.0.0.0/0 to every instance on default; no load balancer or GKE node pool in this read uses that network.

  • Rules open to 0.0.0.0/0: 3 (of 41 ingress rules 1)
  • External load balancers: 2 (ingress-nginx on production and delta 1)
  • Public GKE endpoints: 2 (authorized networks on 1)
  • Reserved, unused IPs: 1 (held but not attached 1)

Firewall rules open to the internet, tilda-trial-production 1

RuleNetworkPortsSourceTargetsLogging
default-allow-sshdefaulttcp:220.0.0.0/0all instancesoff
k8s-fw-a8f3c1e2d4b5 (ingress-nginx)production-vpctcp:80, 4430.0.0.0/0gke-production nodesoff
k8s-fw-b71e09c4f2a6 (ingress-nginx)delta-vpctcp:80, 4430.0.0.0/0gke-delta nodeson

External load balancers 1

Forwarding ruleIPPortsBackendRegion
a8f3c1e2d4b5-production34.120.18.7780, 443ingress-nginx (production)us-central1
b71e09c4f2a6-delta34.83.142.2080, 443ingress-nginx (delta)us-west1

GKE control planes 1 - production: warn — public endpoint · 3 authorized CIDRs - delta: warn — public endpoint · 3 authorized CIDRs

Ask next: Who created default-allow-ssh in tilda-trial-production, and does anything run on the default network? · What did Cloudflare block in front of production.tilda.bio today? · What is open to the internet in tilda-trial-dev?

Logging is off on two of the three open rules 1, so hits on them can't be counted from GCP. The HTTP side is behind Cloudflare, whose firewall log shows what it blocked.

J03 Is anyone attacking us production real data Ask it live ↗
Is anyone attacking us? Show today's Cloudflare firewall events.
tools: cf_firewall_top → cf_firewall_events · components: BarChart, Heatmap, Table

Yes, but it is routine automated probing, and Cloudflare blocked all of it: 7,383 firewall events in the last 24 h (Nov 6 07:00 – Nov 7 07:00 UTC), every one a block 1234. Most were Bot Fight Mode (4,608) and the managed rule for the React RCE CVE-2025-55182 (2,187). That rule's events peaked at 2,056 between 07:00 and 13:00 on Nov 6 1.

  • Firewall events: 7,383 (last 24 h, by 6-hour block 1234)
  • Bot Fight Mode: 4,608 (62% of events 1234)
  • React RCE CVE-2025-55182: 2,187 (2,056 of them Nov 6 07–13 UTC 1)
  • Not blocked: 0 (every event's action was block 1234)

Blocked events by rule, 6-hour blocks (UTC) 1234

Nov 6 07–13Nov 6 13–19Nov 6 19–01Nov 7 01–07
Bot Fight Mode1,2032,6147901
React RCE CVE-2025-551822,05693389
WordPress broken access / file inclusion1653058610
bic13500
WordPress SQLi CVE-2026-601373000
WordPress RCE CVE-2026-630300010

Blocked events by rule, last 24 h 1234

blocks
Bot Fight Mode4,608
React RCE CVE-2025-551822,187
WordPress broken access / file inclusion566
bic18
WordPress SQLi CVE-2026-601373
WordPress RCE CVE-2026-630301

Newest blocked events (newest 30 read, capped) 5

Time (UTC)HostSource IPCountry · ASNRuleRequest
Nov 7 06:49automation.nexus.tilda.pizza34.116.77.178AU · 396982React RCE CVE-2025-55182POST / ×3
Nov 7 06:23tildasense.ai34.79.63.189BE · 396982React RCE CVE-2025-55182POST / ×3
Nov 7 06:18tilda.pizza34.79.63.189BE · 396982React RCE CVE-2025-55182POST / ×3
Nov 7 05:39automation.nexus.tilda.pizza34.116.77.178AU · 396982React RCE CVE-2025-55182POST / ×3
Nov 7 05:29automation.nexus.tilda.pizza34.116.77.178AU · 396982React RCE CVE-2025-55182POST / ×3
Nov 7 05:19api.master.tilda.pizza43.130.109.72US · 132203React RCE CVE-2025-55182POST / ×4
Nov 7 05:00automation.nexus.tilda.pizza34.116.77.178AU · 396982React RCE CVE-2025-55182POST / ×3
Nov 7 04:40production.tilda.bio54.234.25.230US · 14618React RCE CVE-2025-55182POST / ×3
  • Scanning, not a targeted attack 5 — The newest events are bursts of 3–4 POST / requests from a handful of cloud IPs, aimed at every hostname alike (tilda.pizza, tildasense.ai, production.tilda.bio). That spread looks like an automated CVE scanner.

Ask next: Which IPs and countries triggered Bot Fight Mode in the last 24 hours? · Did any request from 34.116.77.178 reach automation.nexus.tilda.pizza today? · How many threats did Cloudflare count per day this week?

This log holds only what Cloudflare blocked or challenged; requests it let through are in the traffic totals, not here. The event sample is capped at the newest 30 5.

J04 GCP inventory production illustrative data Ask it live ↗
How many buckets, VMs and service accounts do we have in production?
tools: gcp_assets · components: BarChart, Table

tilda-trial-production holds 31 buckets, 27 VM instances and 44 service accounts 1. All 27 VMs are GKE nodes of the production and delta clusters; there are no standalone VMs 2.

  • Buckets: 31 (1)
  • VM instances: 27 (all GKE nodes 1)
  • Service accounts: 44 (12 Google-managed 1)
  • Assets counted: 412 (not capped 1)

Resources by type, tilda-trial-production (top 10) 1

count
Disk58
ServiceAccount44
Firewall41
Bucket31
Instance27
Secret24
Subscription18
Topic12
Subnetwork9
Cluster2

Cloud Asset Inventory, counts by type 1

Asset typeCount
compute.googleapis.com/Disk58
iam.googleapis.com/ServiceAccount44
compute.googleapis.com/Firewall41
storage.googleapis.com/Bucket31
compute.googleapis.com/Instance27
secretmanager.googleapis.com/Secret24
pubsub.googleapis.com/Subscription18
pubsub.googleapis.com/Topic12
compute.googleapis.com/Subnetwork9
container.googleapis.com/Cluster2

Where the 27 VMs are 2

ClusterNode poolInstancesMachine type
productiondefault-pool12e2-standard-8
productionneo4j-pool3n2-highmem-8
deltadefault-pool9e2-standard-8
deltaneo4j-pool3n2-highmem-4

Ask next: List the 31 buckets in tilda-trial-production with their location and public access · Which of the 44 service accounts in tilda-trial-production have user-managed keys? · How many buckets, VMs and service accounts are in tilda-trial-dev?

Counts are complete (412 assets, under the 5,000 cap) 1. A count is not a cost: what the buckets cost comes from cost_resources.

J05 Edge traffic and blocks production real data Ask it live ↗
How much traffic did Cloudflare serve this week, and what share was blocked?
tools: cf_traffic_daily · components: BarChart, KpiRow

Cloudflare served 12.8 million requests (83.3 GB) in the 7 days Oct 31 – Nov 6, and counted 123,542 of them as threats, 0.97% 1. Nov 6 stands out: 66,935 threats, 3.9% of that day's requests and about 7× the average of the other six days 1.

  • Requests: 12.8 M (Oct 31 – Nov 6 1)
  • Threats: 123,542 (0.97% of requests 1)
  • Bandwidth: 83.3 GB (69% served from cache 1)
  • Cache hit ratio: 46% (of requests, 7 days 1)

Requests per day, cached vs from origin (UTC) 1

cachedfrom origin
Oct 31640,333939,548
Nov 1215,990564,422
Nov 21,201,4401,400,780
Nov 31,036,1861,226,399
Nov 4520,157865,712
Nov 51,342,2321,139,672
Nov 6950,619757,515

Threats per day (UTC) 1

threats
Oct 311,418
Nov 14,664
Nov 29,381
Nov 321,198
Nov 410,765
Nov 59,181
Nov 666,935

Daily Cloudflare totals, all zones 1

DayRequestsThreatsThreat shareCache hitUnique visitors
Oct 311,579,8811,4180.09%40.5%1,577
Nov 1780,4124,6640.60%27.7%1,914
Nov 22,602,2209,3810.36%46.2%1,632
Nov 32,262,58521,1980.94%45.8%2,463
Nov 41,385,86910,7650.78%37.5%1,787
Nov 52,481,9049,1810.37%54.1%2,446
Nov 61,708,13466,9353.92%55.7%1,799

⚠ Nov 6: 66,935 threats, about 7× the rest of the week 1 — The other six days averaged about 9,400. Requests that day were not unusual (1.7 M), so it was a burst of hostile traffic, not a traffic surge.

Ask next: What were the 66,935 threats on Nov 6: which rules, IPs and hosts? · Which zone served the most requests this week: tilda.pizza, tilda.bio or tildasense.ai? · Show Cloudflare requests and threats per day for the last 30 days

Nov 7 is partial (454,929 requests so far) and left out 1. "Threats" is Cloudflare's own count of blocked or challenged requests; the per-rule breakdown is in the firewall event log, which I didn't read here.

Work, code & approvals

K01 Open Linear issues master real data Ask it live ↗
Which Linear issues are open for the platform team, by assignee?
tools: linear_issues · components: BarChart, Table

Of the 25 most recently updated open ENG issues, 11 have no assignee, all of them [Coverage] tickets from Test Agent Automation 1. Among assigned issues, Monika Madan has the most: 3, all Urgent ICON QC bugs in Testing 1. Linear returned more than 100 open issues, so treat these counts as a floor 1.

  • Platform team read as Linear team ENG; 25 of 100+ open issues counted 1 — Open = started + unstarted. Linear returned 100 and said there are more; this view counts the 25 most recently updated. Name a project or label to narrow it.
  • Open issues counted: 25 of 100+ (most recently updated 1)
  • Unassigned: 11 (all [Coverage] API tickets 1)
  • Urgent: 6 (5 in Zero Tolerance Document QC - Q4 1)
  • Release / prod blockers: 3 (ENG-8488, ENG-8453, ENG-8525 1)

Open ENG issues by assignee (25 most recently updated) 1

in progressnot started
Unassigned92
Monika Madan30
[email protected]20
Satish Ashok10
[email protected]10
Bharghavi Podili10
[email protected]10
[email protected]10
[email protected]10
Ram Yalamanchili10
[email protected]01
Gedi01

Assigned open ENG issues (14 of the 25) 1

IssueTitleAssigneeStatePriorityProject
ENG-8489ICON- Inconsistent Missing Information Tagging When Scope Is Not DetectedMonika MadanTestingUrgentZero Tolerance Document QC - Q4
ENG-8488ICON- Documents Being Pushed to Ready for TMF Despite Missing Scope/TMF LevelMonika MadanTestingUrgentZero Tolerance Document QC - Q4
ENG-8530ICON- Inconsistency in QC Agent for Triggering Truncated content IssuesMonika MadanTestingUrgentZero Tolerance Document QC - Q4
ENG-8487ICON- Documents Taking Too Long to Process and Getting Stuck in AI ReviewSatish AshokIn DevelopmentUrgentZero Tolerance Document QC - Q4
ENG-8529QC fails to detect document truncation and redaction/formatting defects in ICON PLC staging[email protected]In ReviewUrgentZero Tolerance Document QC - Q4
ENG-8562Not Done Reason Selection Modal Opens Again for Regular Activities Even After Selecting a Not Done Reason for a Question[email protected]In DevelopmentUrgentCRO Bugs Backlog
ENG-8538When we clear the answer data from a question and mark it as Not Done by providing a reason, the "Marked as Not Done" text is not displayed on the question.[email protected]In DevelopmentHighSense User Guides and Trainings
ENG-8525[Intermittent]: Unable to generate plan for a Work Plan[email protected]TestingHighWork Plan Improvements Q4
ENG-6002Audit-trail gap: email uploads from non-member senders are recorded as "AI Teammate", losing the real sender[email protected]Ready for DevelopmentHighnone
ENG-8453Study Admin Unable to View Training Logs for Accessible Studies When Navigating Through the Trainings page[email protected]TestingMediumQMS Module
ENG-8322docproc API stalls under load: PDF rendering in worker threads holds the GIL and starves the event loop[email protected]TestingMediumSense Platform Improvements
ENG-7637Improve the warning for mismatched files on trainingsGediReady for DevelopmentMediumQMS Module
ENG-8053Give downloadable document instances user-friendly namesBharghavi PodiliTestingNo priorityQMS Module
ENG-7534LLM cost optimization for TMF agentsRam YalamanchiliProduct AcceptanceNo priorityTMF Improvements - Q4

Ask next: Show every open ENG issue assigned to Monika Madan · List every open ENG issue labelled Release Blocker or Prod Blocker · Which open ENG issues in Zero Tolerance Document QC - Q4 are still In Development?

The other 75+ open ENG issues were not counted, so the assignee ranking can change once all of them are read 1.

K02 File a ticket (approval) master real data Ask it live ↗
File a Linear ticket for the Cypher failures on the shared database master.
tools: whats_broken → linear_create · components: Approval, Callout

I've drafted the ticket, and it is waiting for your approval. Nothing has been filed yet 2. The evidence: the shared Neo4j database master has been failing Cypher queries since Oct 27 15:34 UTC, with 224 failures in 15 min at the last check (finding #7603) and no cause clear from logs or recent changes 1.

Action #318: File a Linear ticket: Cypher failures on shared Neo4j database master — approve or reject it on its card.

⚠ Not filed yet: waiting for your approval 2 — Open the gate card to approve or reject this exact draft. It is filed only if you approve.

What the draft ticket says 2

FieldDraft
TitleNeo4j shared database master: Cypher queries failing since Oct 27
TeamENG
PriorityHigh
Linksfinding #7603 · xray.tilda.pizza/findings/7603?cluste… ↗
Body224 Cypher queries failed in 15 min (whats_broken, Nov 7 06:38 UTC). First seen Oct 27 15:34 UTC. No deploy linked. First look: cause not clear from logs and recent changes. Leads: [email protected], [email protected].

Shared Neo4j failures seen from master (UTC) 1 - Oct 27 15:34 — Cypher failures start on database master: finding #7603 · no deploy linked - Nov 4 14:36 — Failures start on database matrix_a too: finding #16175 · not in this ticket - Nov 6 08:47 — Current burst on master begins: first look: 53 failed in 15 min, no linked change - Nov 7 06:38 — Still failing: 224 in 15 min: latest whats_broken check

Finding #7603: shared database master · Finding #16175: shared database matrix_a

The draft covers database master only. matrix_a (#16175, failing since Nov 4) is a separate finding I left out; tell me if it belongs in the same ticket. The root cause is not determined 1.

K03 Raise a PR (approval) master illustrative data Ask it live ↗
Raise a PR that makes the docs sweep log which repos it skipped.
tools: explore → pr_edit → github_pr · components: Approval, Code, Steps

I've drafted the PR in tildabio/tools, and it is waiting for your approval; no PR exists until you approve it 3. Today sweep() in obs_dashboard/docsweep.py adds every skipped pull request to one total. The draft also counts them per repo and reason, logs one line per repo at the end of a pass, and keeps the per-repo counts in the run's stats 12.

Action #319: Open PR in tildabio/tools: docs sweep logs skipped pull requests per repo — approve or reject it on its card.

⚠ Not opened yet: waiting for your approval 3 — The gate card shows the full diff; the PR is opened only if you approve it.

+from collections import Counter
...
+    skipped_by: dict[str, Counter] = {repo: Counter() for repo, _ in repos()}
         for repo, base in repos():
             for pr in merged(repo, base, since, until):
                 ...
                 if why:
                     stats["skipped"] += 1
+                    skipped_by[repo][why] += 1
                     _record(conn, pr, "skipped", why, 0, slot, dry_run)
                     continue
                 ...
                     if not shown.strip():
                         stats["skipped"] += 1
+                        skipped_by[repo]["no product code"] += 1
...
+        for repo, why in skipped_by.items():
+            log(f"[docs-sweep] {repo}: skipped {sum(why.values())} ({', '.join(f'{k} {n}' for k, n in why.most_common()) or 'none'})")
+        stats["skipped_by_repo"] = {r: dict(w) for r, w in skipped_by.items()}

What the PR does, and how to check it - [x] Found where sweep() skips a pull request — two places: skip_reason() (bot, no product code) and an empty product diff 1 - [x] Edited obs_dashboard/docsweep.py — Counter import, a per-repo tally in both skip branches, one log line per repo, stats["skipped_by_repo"] 2 - [x] Added a test in tests/test_docsweep.py — a bot PR and a docs-only PR in two repos give two log lines with the right counts 2 - [ ] Approve or reject the draft on the gate card — action 319 3 - [ ] After merge, read the next pass's log — expect one [docs-sweep] <repo>: skipped N (…) line per repo in docs_sweep_repos

A repo with nothing skipped still gets a line ("skipped 0 (none)"), so a quiet repo is easy to tell apart from one that was never read. Say if you'd rather log only repos with skips.

K04 Is it in production yet production illustrative data Ask it live ↗
Is ENG-4826 in production yet?
tools: where_is · components: Timeline, StatusGrid

Yes, ENG-4826 is in production. Its fix (PR #16655, commit 4be91c2) shipped in v2.10.43, which production has run since Oct 30 17:42 UTC 1. delta is still on v2.10.38-beta and does not have it 1.

✓ ENG-4826 is in production 1 — PR #16655 was merged, is in release v2.10.43, and has been running on production since Oct 30 17:42 UTC.

Does each cluster run it? 1 - delta: bad — v2.10.38-beta-dafae8a: does not include 4be91c2 - nexus: unknown — no running image recorded - production: ok — v2.10.43-86c75b5: includes it - test27: ok — v2.10.44-6023bae: includes it - automation: ok — v2.10.44-6023bae: includes it - master: ok — master builds since Oct 22

ENG-4826 from merge to production (UTC) 1 - Oct 21 14:05 — PR #16655 merged into main: commit 4be91c2 · regsvc - Oct 22 09:30 — On master: first master build after the merge - Oct 29 11:12 — Tagged v2.10.43: the release includes 4be91c2 - Oct 30 17:42 — v2.10.43 rolled out to production: regsvc:v2.10.43-86c75b5

regsvc on production

Ask next: When will delta get v2.10.43 or later? · What else shipped to production in v2.10.43? · Did regsvc's error rate on production change after the Oct 30 rollout?

"In production" means the commit is in the image production runs; whether the fix behaves as the ticket expects needs a look at regsvc's traffic since Oct 30.

K05 What shipped this week production real data Ask it live ↗
What shipped to production this week?
tools: change_feed → release_notes · components: Timeline, List

One product version went to production this week: studiosvc and studioaisvc v2.10.44-6023bae (Nov 2–3), plus a docproc deploy on Nov 6 1. The rest of the 20 changes were infrastructure: platform add-ons and per-service infra overlays on Nov 4 1. No release notes were written this week; the latest approved is Release 2.11, from Oct 23 2.

  • Changes on production: 20 (last 7 days 1)
  • App version: v2.10.44-6023bae (studiosvc, studioaisvc 1)
  • By hand: 1 (metrics-server, GKE-managed 1)
  • Release notes this week: 0 (latest approved Oct 23 2)

What changed on production, Nov 2 – Nov 6 (UTC) 1 - Nov 2 15:53 — studioaisvc → v2.10.44-6023bae: [email protected] · ash-nazg f162f90b6 - Nov 3 08:37 — studiosvc → v2.10.44-6023bae: ArgoCD sync - Nov 3 11:05 — metrics-server v0.8.0-gke.23: by hand · system:metrics-server-nanny (GKE) - Nov 3 17:11 — studioaisvc and studiosvc re-synced: [email protected] · ash-nazg 3c385e39f - Nov 3 18:08 — academy infra: [email protected] · ash-nazg 46cc997d5 - Nov 4 06:33 — cert-manager, external-secrets, litellm, chat-agent-postgres: [email protected] · ash-nazg 7982661b5 · 4 apps in 34 min - Nov 4 08:35 — Infra overlays for 7 services, plus asm: dashrsvc, enrollsvc, docsvc, regsvc, studioaisvc, execsvc, virtual-assistant · ash-nazg 6fa9ddbb2 - Nov 6 18:12 — docproc: [email protected] · ash-nazg 66d47f48b · image not recorded

Release notes in X-ray 2 - Release 2.11 — approved v26 by [email protected] · v2.10.0 → main 8a8e91e4c (Oct 23) - Release v2.10.39 — approved v7 by [email protected] · 5 open threads (Oct 23) - Release v2.10 U2 — approved v9 · v2.10.16 → v2.10.31 (Oct 12)

Ask next: What changed in studiosvc between v2.10.43 and v2.10.44? · Did studioaisvc's latency or errors on production change after the Nov 3 deploy? · What did the Nov 6 docproc deploy change in ash-nazg?

Deploys with no image recorded are config syncs from ash-nazg; what changed inside them needs the commit diff 1. I can't tell from the release list whether Release 2.11 (ending at main 8a8e91e4c) already covers v2.10.44 2.

K06 Eval run vs previous master real data Ask it live ↗
How did the latest eval run score compared with the previous one?
tools: evals · components: Compare, BarChart

The latest eval run (Quality Control, gpt-5.6-terra on qc_classification_v1_test, Nov 5 19:11 UTC) scored 92.38% (303/330), +0.96 pp over the previous run's 91.42% (299/330) at 18:17 1. That is inside the 91.42–93.08% band of the last seven QC runs, so it is not a real change 1.

  • Latest accuracy: 92.38% (303 / 330 · Nov 5 19:11 UTC 1)
  • Change: +0.96 pp (vs 91.42% at 18:17 1)
  • Cost of the run: $25.34 (669 calls · flex tier 1)
  • Eval spend, 30 days: $247.38 (48 runs, 1 broken 1)

qc_classification_v1_test · gpt-5.6-terra 1

v88 · Nov 5 18:17v89 · Nov 5 19:11
Accuracy91.42%92.38%
Passed299 / 330303 / 330
Errored documents00
Cost$25.70$25.34
Wall time857 s879 s
LLM calls665669

Quality Control accuracy, last 7 runs, Nov 4 17:48 – Nov 5 19:11 UTC 1 (%)

accuracy
144dpi93.08
72dpi91.84
v8691.63
v86 run292.23
8491 rules92.38
v8891.42
v8992.38

All seven runs are [email protected]'s, from a laptop. The best was the Nov 4 144 dpi run at 93.08% (305/330), and the 72 dpi run on the same day scored 91.84% 1.

Ask next: Which 27 documents did the v89 run get wrong on qc_classification_v1_test? · How does claude-haiku-5-5 compare with gemini-3.8-flash on TMF Artifact across datasets? · Why did TMF Metadata on claude-haiku-5-5 drop 2.46 pp on dataset17?

"Previous" means the previous full run of the same category, dataset and model 1. With 330 documents, one document is 0.3 pp, so the +0.96 pp is 4 documents.

K07 Failing API tests master real data Ask it live ↗
Which API tests are failing on master, and since when?
tools: tests · components: Table, Donut

Only 2 of 808 API tests failed in the latest master run, api-tests-master #588 (Nov 7 00:22 UTC) 1. The one chronic failure is GatedFacilityChangeFlow, which failed in all 12 runs of the last 14 days, so it has been failing since at least #573 on Oct 25 1. EDLCrud also failed in #588, but it fails only about one run in three 1.

  • Latest run: #588 (Nov 7 00:22 UTC · UNSTABLE 1)
  • Pass rate: 98.6% (797 of 808 1)
  • Failed: 2 (GatedFacilityChangeFlow, EDLCrud 1)
  • Chronic suites: 1 (failed 12 of 12 runs 1)

api-tests-master #588 1 - passed: 797 - failed: 2 - neither passed nor failed: 9

api-tests-master pass rate by build, last 14 days 1 (%)

pass rate
#57398.36
#57498.36
#575—
#57696.37
#5770.37
#57855.54
#57998.63
#58094.4
#58194.4
#582—
#583—
#584—
#585—
#586—
#587—
#58898.64

Threshold: 98.36 14-day median

Failing suites, api-tests-master, last 14 days (9 of 14 shown) 1

SuiteFailed runsFail rateClassLast failedFailing since
GatedFacilityChangeFlow12 of 12100%chronic#588every run in 14 days (since at least Oct 25)
WorkOrderTemplateCopy8 of 1266.7%flaky#586intermittent
GatedTemplatePreview8 of 1266.7%flaky#585intermittent
WorkPlanInstance5 of 1241.7%flaky#581intermittent
EDLTracking4 of 1233.3%flaky#585intermittent
EDLCrud4 of 1233.3%flaky#588intermittent
TwoStepUpload3 of 1225%flaky#581intermittent
BulkDocumentUpload3 of 1225%flaky#576intermittent
WorkOrderLanguage3 of 1225%flaky#581intermittent

⚠ Three runs were collapses, not test failures 1 — #575 (Oct 27) ran no tests; #577 (Oct 29) failed 794 of 803 in 2.8 min; #578 (Oct 30) failed 345. They are counted apart from suite failures.

Ask next: Show the failing GatedFacilityChangeFlow test in api-tests-master #588 and its error · Why did api-tests-master #577 collapse on Oct 29? · Which API tests are failing on production (api-tests-production)?

Builds #582–#587 were on a page of the result I did not read, so their points are blank 1. 14 days is the tool's window, so I can't see when GatedFacilityChangeFlow started failing, only that it was before Oct 25.

K08 Weaviate health master real data Ask it live ↗
Is Weaviate healthy on master?
tools: weaviate_health · components: KpiRow, Meters

Yes. Master's Weaviate (the shared cluster on shared-neo4j-5-26) is healthy: 3 of 3 nodes up, 0 restarts, no stalled indexing queues and no app errors in the last 6 h 1. The node to watch is weaviate-1, which holds most of the data: 55% disk free and 9.0 GB of memory, against 4.1 GB and 0.36 GB on the other two 1.

✓ Weaviate for master is healthy 1 — 3 of 3 servers up, no restarts, async indexing queues at 0, no app errors in the last 6 h.

  • Nodes up: 3 / 3 (last 6 h 1)
  • Restarts: 0 (last 6 h 1)
  • Stalled queues: 0 (async indexing queue at 0 1)
  • RagDocs_master: 2,770,918 (objects (chunks), flat over 6 h 1)

Disk used per node, of 105.1 GB each 1 - weaviate-1: 46.9 GB of 105.1 (55.4% free)

  • weaviate-0: 19.5 GB of 105.1 (81.4% free)
  • weaviate-2: 15.9 GB of 105.1 (84.9% free)

Memory per node, last 6 h 1 (GB)

now6 h peak
weaviate-19.019.38
weaviate-04.097.17
weaviate-20.360.38

Per node, last 6 h 1

NodeCPU nowCPU meanCPU peakMemory nowDisk free
weaviate-00.046 cores0.060 cores0.98 cores4.09 GB81.4%
weaviate-10.032 cores0.047 cores0.65 cores9.01 GB55.4%
weaviate-20.024 cores0.035 cores0.57 cores0.36 GB84.9%

Ask next: Which Weaviate classes and shards sit on weaviate-1 vs weaviate-2? · Is Weaviate healthy on production? · How fast has RagDocs_master grown over the last 7 days?

This Weaviate is shared by master, automation, nexus and three more app clusters, so its load is not master's alone 1. The result has no memory limit, so I can't say how close weaviate-1 is to one.