Xray
1h 6h 24h 48h 3d 5d 7d 14d 30d ‹
Last 1 hour▾
Ends now. Anything up to this cluster's 30-day retention.
Start and end times
Cancel
Interpreted in UTC (change it at the right of this bar). Earliest available: 2026-10-19 13:03:25.
› →|
updated 1d ago
Sample data. This database was generated by scripts/seed_dev_db.py — every service, deploy, person and figure on this page is synthetic. Names are sample.*@seed.invalid and image tags are prefixed SEED-. Nothing here happened.
← findings

docproc is burning its error budget at 0.0× (1 h) / 6.0× (6 h) against a 99.50% objective

warning Error budget burning master resolved impact 38 Owner: Sambasiva Bandarupalli (backup: Ahmed Aburaed) opened 11d ago · last observed 2d ago · observed 63× · #16828 · fa41081bd3d92369dc52

What happened template

docproc is spending its error budget 0.0× faster than the objective allows over the last hour (6.0× over six hours). The objective is 99.5%, derived — a baseline, not a promise. 6 bad events against 199 calls in the window.

Bad events are consuming the error budget faster than the objective allows.

First look — investigated before anyone looked

docproc returned 5xx on 3 of 35 calls (91.4% vs 99.5% objective), burning budget at 17.1× over 1 h and 6 h; p95 708 ms.
Started 21:16 UTC on 2026-11-15, when several new error shapes appeared together. The last error was the 500 at 21:18.
The chain in the logs: ServiceTokenManager refresh got 429 Too Many Requests from https://login.tilda.pizza/oauth/token with no usable cached token (tilda_auth/service_token_manager.py:185). This left 'No docsvc authorization', so docsvc PATCH aiEnabled failed (queries/ai_run_queries.py:294) and AIRun creation aborted the pipeline (orchestration/document_orchestrator.py:4514, handlers/pubsub_handl
The two autosync deploys before onset were config-only, 15–20 h earlier, and nothing links them to this.
Next: find out why the OAuth token endpoint rate-limited docproc (quota or refresh storm), and check that the token manager keeps and reuses a cached token through 429 backoff.

medium confidence dependency login.tilda.pizza OAuth token endpoint (429)

claude-opus-5-5 · 2d ago · $0.05 — every link above is evidence it read; nothing else was claimed.

The facts

objective99.50
objective_sourcederived
achieved96.98
calls199.00
bad6.00
c5xx6.00
budget0.99
spent6.03
remaining_pct0.00
burn_1h0.00
burn_6h6.03
verdictburning
p95_ms485.73
discriminator
window2026-11-15 17:26:00 UTC → 2026-11-15 23:26:00 UTC

What is failing in docproc — the bad events, decomposed

“Bad events” are HTTP 5xx answers from docproc. In this finding's window (2026-11-15 17:26:00 UTC → 2026-11-15 23:26:00 UTC) there were 6; every line links to the grouped errors behind it.

POST /api/v1/docproc/:id/split 6 bad of 6 calls · 100% of the service's 5xx

The error shapes behind them

38× Failed to create document processing AIRun — aborting pipeline (refuse to run untracked) — No docsvc authorization (call… since 2026-11-15 21:16:35 UTC
38× docsvc PATCH aiEnabled failed before creating AI run since 2026-11-15 21:16:35 UTC
29× ServiceTokenManager refresh failed (429 Client Error: Too Many Requests for url: https://login.tilda.pizza/oauth/token) … since 2026-11-15 21:16:35 UTC
23× [90814349-b97c-4cf0-b423-9decc2a42448] PubSub certify failed: AIRun creation failed for document 90814349-b97c-4cf0-b423… since 2026-11-01 21:47:55 UTC
10× Recovery failed — AIRun creation failed for document 708e1425-a271-4065-9830-6f906d9e1004 (document processing): No docs… since 2026-11-15 21:17:08 UTC

All docproc errors, grouped → · service page · its logs

The story — No docproc deploy or hand change is recorded in the hour before this opened. Everything below is docproc only.

History

  • 2026-11-15 21:33:35 UTCreopened x-ray · severity=critical was=warning title=docproc is burning its error budget at 17.1× (1 h) / 17.1× (6 h) against a 99.50% objective level=medium was_level=low finding={'id': 16828, 'fingerprint': 'fa41081bd3d92369dc52', 'kind': 'budget.burning', 'severity': 'critical', 'cluster': 'master', 'subject': 'docproc', 'title': 'docproc is burning its error budget at 17.1× (1 h) / 17.1× (6 h) against a 99.50% objective', 'facts': {'objective': 99.5, 'objective_source': 'derived', 'achieved': 91.42857142857143, 'calls': 35.0, 'bad': 3.0, 'c5xx': 3.0, 'budget': 0.175, 'spent': 17.142857142857142, 'remaining_pct': 0.0, 'burn_1h': 17.142857142857142, 'burn_6h': 17.142857142857142, 'verdict': 'burning fast', 'p95_ms': 708.3333333333334, 'window': [1794756815, 1794778415], 'recent_deploys': [], 'discriminator': ''}, 'evidence': [{'label': 'Error budgets', 'href': '/budgets?cluster=master&from=1794756815&to=1794778415'}, {'label': 'docproc service page', 'href': '/service/docproc?cluster=master&from=1794756815&to=1794778415'}], 'provenance': {'source': 'rollup', 'collected_at': 1794778340, 'age_seconds': 77, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 56.2, 'status': 'open', 'first_seen': 1794051896, 'last_seen': 1794778415, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 49, 'narrative': 'docproc is spending its error budget — faster than the objective allows over the last hour (12.8× over six hours). The objective is 99.5%, derived — a baseline, not a promise. 3 bad events against 47 calls in the window.', 'narrative_kind': 'template', 'discriminator': '', 'level': 'medium', 'told_level': None, 'episode_at': 1794778415, 'chat_thread': None, 'kind_label': 'Error budget burning', 'group': 'Reliability', 'age_s': 726521, 'stale_s': 2}
  • 2026-11-15 23:46:05 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 16828, 'fingerprint': 'fa41081bd3d92369dc52', 'kind': 'budget.burning', 'severity': 'warning', 'cluster': 'master', 'subject': 'docproc', 'title': 'docproc is burning its error budget at 0.0× (1 h) / 6.0× (6 h) against a 99.50% objective', 'facts': {'objective': 99.5, 'objective_source': 'derived', 'achieved': 96.98492462311557, 'calls': 199.0, 'bad': 6.0, 'c5xx': 6.0, 'budget': 0.995, 'spent': 6.030150753768845, 'remaining_pct': 0.0, 'burn_1h': 0.0, 'burn_6h': 6.030150753768844, 'verdict': 'burning', 'p95_ms': 485.7317073170731, 'window': [1794763560, 1794785160], 'recent_deploys': [], 'discriminator': ''}, 'evidence': [{'label': 'Error budgets', 'href': '/budgets?cluster=master&from=1794763560&to=1794785160'}, {'label': 'docproc service page', 'href': '/service/docproc?cluster=master&from=1794763560&to=1794785160'}], 'provenance': {'source': 'rollup', 'collected_at': 1794785070, 'age_seconds': 92, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 38.1, 'status': 'resolved', 'first_seen': 1794051896, 'last_seen': 1794785160, 'resolved_at': 1794786365, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 63, 'narrative': 'docproc is spending its error budget 0.0× faster than the objective allows over the last hour (6.0× over six hours). The objective is 99.5%, derived — a baseline, not a promise. 6 bad events against 199 calls in the window.', 'narrative_kind': 'template', 'discriminator': '', 'level': 'low', 'told_level': None, 'episode_at': 1794778415, 'chat_thread': None, 'kind_label': 'Error budget burning', 'group': 'Reliability', 'age_s': 734470, 'stale_s': 1206}

Check it yourself

Every link carries the finding's own time window, so it shows tomorrow what it shows now.

Actions

Sign in (IAP) to acknowledge, snooze or resolve.

Where the numbers come from

sourcerollup
collected_at1794785070
age_seconds92
staleFalse
shownNone
totalNone
truncatedFalse
coverage_fromNone
retain_daysNone
noteNone
detectorbudgets