docsvc GET /api/documents/:id: 4xx — 123 of 130 calls (95 %; other 4xx×123) in the last 15 min — it usually has none
warning
4xx spike (403/401/404…)
test26
resolved
impact 31
Owner: Prasad Limaye (backup: Harish Kumar)
opened 16d ago · last observed 2d ago · observed 8× · #14829 · 6d581a83b573f08c5cf3
What happened template
4xx spike (403/401/404…): docsvc GET /api/documents/:id: 4xx — 123 of 130 calls (95 %; other 4xx×123) in the last 15 min — it usually has none. Facts: endpoint GET /api/documents/:id, service docsvc, calls 130, c4xx 123, share_pct 94.61538461538461, c401 0.
An endpoint answering far more 4xx than its own usual rate — a permissions change (403), expired or broken auth (401), a missing route or contract break (404). Its usual rate is its last seven days.
The facts
| endpoint | GET /api/documents/:id |
| service | docsvc |
| calls | 130 |
| c4xx | 123 |
| share_pct | 94.62 |
| c401 | 0 |
| c403 | 0 |
| other_4xx | 123 |
| rejecting | no |
| usual_per_window | 0.04 |
| baseline_days | 7 |
| window_min | 15 |
| discriminator | GET /api/documents/:id |
| window | 2026-11-16 06:25:00 UTC → 2026-11-16 06:40:00 UTC |
What is failing in docsvc — the bad events, decomposed
“Bad events” are HTTP 5xx answers from docsvc. In this finding's window (2026-11-16 06:25:00 UTC → 2026-11-16 06:40:00 UTC) there were 0; every line links to the grouped errors behind it.
The error shapes behind them
| 156× | template backfill: could not move document — document has no primary version, cannot move to Templates since 2026-11-16 13:27:23 UTC |
The story — No docsvc deploy or hand change is recorded in the hour before this opened. Everything below is docsvc only.
- 2026-11-02 05:09:15 UTCthis finding docsvc GET /api/documents/:id: 4xx — 123 of 130 calls (95 %; other 4xx×123) in the last 15 min — it usually has none
- 2026-11-14 20:00:17 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/docsvc by hand (kubectl/v1.36.1 (darwin/arm64) kubernetes/7569396,gzip(gfe))
- 2026-11-15 00:00:00 UTClatency docsvc p95 peaked at 2.55 s
- 2026-11-16 00:00:00 UTCerrors docsvc 5xx peaked at 240 in one five-minute bucket (median 0 in this window)
- 2026-11-16 02:26:12 UTCfinding · warning docsvc: a new error shape, 11× in the last hour, never seen in 90 days (Request completed — ? · document cannot be promoted to TMF: document m)
- 2026-11-16 02:26:12 UTCfinding · warning docsvc PUT /api/documents/promote: 16 of 16 calls failed (100 %) in the last 15 min
- 2026-11-16 06:01:45 UTCfinding · warning docsvc: a new error shape, 41× in the last hour, never seen in 90 days (Request completed — ? · ?/? documents promoted to TMF and ? failed)
History
- 2026-11-16 06:42:08 UTCreopened x-ray · severity=warning was=warning title=docsvc GET /api/documents/:id: 4xx — 123 of 131 calls (94 %; other 4xx×123) in the last 15 min — it usually has none level=medium was_level=medium finding={'id': 14829, 'fingerprint': '6d581a83b573f08c5cf3', 'kind': 'endpoint.4xx_spike', 'severity': 'warning', 'cluster': 'test26', 'subject': 'docsvc', 'title': 'docsvc GET /api/documents/:id: 4xx — 123 of 131 calls (94 %; other 4xx×123) in the last 15 min — it usually has none', 'facts': {'endpoint': 'GET /api/documents/:id', 'service': 'docsvc', 'calls': 131, 'c4xx': 123, 'share_pct': 93.89312977099236, 'c401': 0, 'c403': 0, 'other_4xx': 123, 'rejecting': False, 'usual_per_window': 0.04, 'baseline_days': 7, 'window': [1794810000, 1794810900], 'window_min': 15, 'recent_deploys': [], 'discriminator': 'GET /api/documents/:id'}, 'evidence': [{'label': 'The 4xx calls (status, caller, path)', 'href': '/api-calls?cluster=test26&from=1794810000&to=1794810900&service=docsvc&status=4xx'}, {'label': 'docsvc service page', 'href': '/service/docsvc?cluster=test26&from=1794810000&to=1794810900'}], 'provenance': {'source': 'rollup', 'collected_at': 1794811328, 'age_seconds': 7, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 25.0, 'status': 'open', 'first_seen': 1793596155, 'last_seen': 1794811328, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 7, 'narrative': '4xx spike (403/401/404…): docsvc GET /api/documents/:id: permission denied — 361 of 362 calls (100 %; 403×361) in the last 15 min — 1988× its usual 0.2. Facts: endpoint GET /api/documents/:id, service docsvc, calls 362, c4xx 361, share_pct 99.72375690607734, c401 0.', 'narrative_kind': 'template', 'discriminator': 'GET /api/documents/:id', 'level': 'medium', 'told_level': None, 'episode_at': 1794811328, 'chat_thread': None, 'kind_label': '4xx spike (403/401/404…)', 'group': 'Errors', 'age_s': 1215180, 'stale_s': 7}
- 2026-11-16 07:17:55 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 14829, 'fingerprint': '6d581a83b573f08c5cf3', 'kind': 'endpoint.4xx_spike', 'severity': 'warning', 'cluster': 'test26', 'subject': 'docsvc', 'title': 'docsvc GET /api/documents/:id: 4xx — 123 of 130 calls (95 %; other 4xx×123) in the last 15 min — it usually has none', 'facts': {'endpoint': 'GET /api/documents/:id', 'service': 'docsvc', 'calls': 130, 'c4xx': 123, 'share_pct': 94.61538461538461, 'c401': 0, 'c403': 0, 'other_4xx': 123, 'rejecting': False, 'usual_per_window': 0.04, 'baseline_days': 7, 'window': [1794810300, 1794811200], 'window_min': 15, 'recent_deploys': [], 'discriminator': 'GET /api/documents/:id'}, 'evidence': [{'label': 'The 4xx calls (status, caller, path)', 'href': '/api-calls?cluster=test26&from=1794810300&to=1794811200&service=docsvc&status=4xx'}, {'label': 'docsvc service page', 'href': '/service/docsvc?cluster=test26&from=1794810300&to=1794811200'}], 'provenance': {'source': 'rollup', 'collected_at': 1794811767, 'age_seconds': 6, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 31.4, 'status': 'resolved', 'first_seen': 1793596155, 'last_seen': 1794811767, 'resolved_at': 1794813475, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 8, 'narrative': '4xx spike (403/401/404…): docsvc GET /api/documents/:id: 4xx — 123 of 130 calls (95 %; other 4xx×123) in the last 15 min — it usually has none. Facts: endpoint GET /api/documents/:id, service docsvc, calls 130, c4xx 123, share_pct 94.61538461538461, c401 0.', 'narrative_kind': 'template', 'discriminator': 'GET /api/documents/:id', 'level': 'medium', 'told_level': None, 'episode_at': 1794811328, 'chat_thread': None, 'kind_label': '4xx spike (403/401/404…)', 'group': 'Errors', 'age_s': 1217326, 'stale_s': 1714}
Check it yourself
Every link carries the finding's own time window, so it shows tomorrow what it shows now.
Actions
Sign in (IAP) to acknowledge, snooze or resolve.
Where the numbers come from
| source | rollup |
| collected_at | 1794811767 |
| age_seconds | 6 |
| stale | False |
| shown | None |
| total | None |
| truncated | False |
| coverage_from | None |
| retain_days | None |
| note | None |
| detector | endpoint_4xx_spike |