← findings
production AI spend projects to $2,313 this month, 2.0× last month's $1,131
warning
AI spend running high
production
open
impact 42
opened 13d ago · last observed 1d ago · observed 253× · #16246 · d3648191351bd4a98be4
What happened template
production's AI spend is $1,497.42 so far this month and projects to $2,313.44 by month end (89.32/day, weekends weighted 0.03×); last month closed at $1,131.10. A projection is a line through the past — the AI page shows which operation and org moved.
The month-end projection is more than 1.2× last month's total.
The facts
| spent | 1,497 |
| projected | 2,313 |
| trend_per_day | 89.32 |
| weekend_ratio | 0.03 |
| last_month | 1,131 |
| days_so_far | 17 |
| month_start | 1,793,491,200 |
| month_end | 1,796,083,200 |
| discriminator | |
| window | 2026-11-01 00:00:00 UTC → 2026-11-17 10:40:41 UTC |
The story — No deploy or hand change is recorded in the hour before this opened.
- 2026-11-05 00:54:34 UTCthis finding production AI spend projects to $2,313 this month, 2.0× last month's $1,131
- 2026-11-08 02:56:19 UTCfinding · critical x-ray-weaviate-stats (x-ray) image pull failing for at least 115 min — 1 pod
- 2026-11-08 06:38:31 UTCfinding · critical virtual-assistant GET /api/v1/va/sse/stream: permission denied — 20 of 22 calls (91 %; 401×20) in the last 15 min — 232× its usual 0.1
- 2026-11-09 14:28:17 UTCfinding · warning studiosvc on production has served nothing for 1.1 h; at this hour it normally serves about 108 calls/hour
- 2026-11-15 05:43:29 UTCfinding · critical ontologysvc: 1× “Traceback (most recent call last): File ?, line ?, in lifespan await receive() File ?, line ?, in receive return await self.receive_queue.ge” in 10 min — Production logs — panic / fatal
- 2026-11-16 00:15:00 UTClatency cluster p95 peaked at 4.55 s
- 2026-11-16 05:10:11 UTCdeploy [email protected] deployed data-monitor
- 2026-11-16 05:10:12 UTCdeploy [email protected] deployed chat-agent
- 2026-11-16 05:10:12 UTCdeploy [email protected] deployed dashrsvc
- 2026-11-16 05:10:12 UTCdeploy [email protected] deployed workplansvc-infra-production
- 2026-11-16 05:10:13 UTCdeploy [email protected] deployed execsvc
- 2026-11-16 05:10:13 UTCdeploy [email protected] deployed dashql
- 2026-11-16 05:10:13 UTCdeploy [email protected] deployed enrollsvc
- 2026-11-16 05:10:14 UTCdeploy [email protected] deployed exportsvc
- 2026-11-16 05:10:14 UTCdeploy [email protected] deployed docsvc
- 2026-11-16 05:10:14 UTCdeploy [email protected] deployed finsvc
- 2026-11-16 05:10:15 UTCdeploy [email protected] deployed notifsvc
- 2026-11-16 05:10:16 UTCdeploy [email protected] deployed onboardingsvc
- 2026-11-16 05:10:17 UTCdeploy [email protected] deployed patientsvc
- 2026-11-16 05:10:19 UTCdeploy [email protected] deployed docproc
- 2026-11-16 05:10:23 UTCdeploy [email protected] deployed ontologysvc
- 2026-11-16 05:10:23 UTCdeploy [email protected] deployed schedsvc
- 2026-11-16 05:10:24 UTCdeploy [email protected] deployed studiosvc
- 2026-11-16 05:10:24 UTCdeploy [email protected] deployed regsvc
- 2026-11-16 05:10:24 UTCdeploy [email protected] deployed tusker
- 2026-11-16 05:10:24 UTCdeploy [email protected] deployed tmfsync
- 2026-11-16 05:10:25 UTCdeploy [email protected] deployed studioaisvc
- 2026-11-16 05:10:25 UTCdeploy [email protected] deployed rag-agent
- 2026-11-16 05:10:25 UTCdeploy [email protected] deployed workplan-agent
- 2026-11-16 05:10:25 UTCdeploy [email protected] deployed virtual-assistant
- 2026-11-16 05:10:25 UTCdeploy [email protected] deployed workplansvc
- 2026-11-16 05:10:51 UTCdeploy [email protected] deployed asm
- 2026-11-16 16:32:01 UTCfinding · critical execsvc: a new error shape, 473× in the last hour, never seen in 90 days (failed to evaluate and raise auto query for participant question — no )
- 2026-11-16 16:32:01 UTCfinding · critical execsvc: a new error shape, 75× in the last hour, never seen in 90 days (failed to evaluate and raise auto query for participant question — eva)
- 2026-11-16 16:32:01 UTCfinding · critical execsvc: a new error shape, 75× in the last hour, never seen in 90 days (evalResult is not a bool)
- 2026-11-16 21:38:50 UTCfinding · warning execsvc.auto-query-reeval: 6 failed runs in the last 6 h — participant activity ?: auto query ? on question ?: no result from branch evaluation for branch, pro
- 2026-11-17 00:20:00 UTCerrors cluster 5xx peaked at 15 in one five-minute bucket (median 0 in this window)
History
Check it yourself
Every link carries the finding's own time window, so it shows tomorrow what it shows now.
Actions
Sign in (IAP) to acknowledge, snooze or resolve.
Where the numbers come from
| source | live |
| collected_at | None |
| age_seconds | None |
| stale | False |
| shown | None |
| total | None |
| truncated | False |
| coverage_from | None |
| retain_days | None |
| note | BigQuery daily cost, month to date |
| detector | llm_spend_forecast |