studiosvc GET /api/studio/events: 4xx — 100 of 100 calls (100 %; other 4xx×100) in the last 15 min — it rejects nearly every call
warning
4xx spike (403/401/404…)
test26
open
impact 37
Owner: Suresh Boddapati (backup: Madhava Mukkamala)
opened 7d ago · last observed 1d ago · observed 164× · #17063 · 001ce0a5dd0a7ac21dd0
What happened template
4xx spike (403/401/404…): studiosvc GET /api/studio/events: 4xx — 100 of 100 calls (100 %; other 4xx×100) in the last 15 min — it rejects nearly every call. Facts: endpoint GET /api/studio/events, service studiosvc, calls 100, c4xx 100, share_pct 100.0, c401 0.
An endpoint answering far more 4xx than its own usual rate — a permissions change (403), expired or broken auth (401), a missing route or contract break (404). Its usual rate is its last seven days.
The facts
| endpoint | GET /api/studio/events |
| service | studiosvc |
| calls | 100 |
| c4xx | 100 |
| share_pct | 100.00 |
| c401 | 0 |
| c403 | 0 |
| other_4xx | 100 |
| rejecting | yes |
| usual_per_window | 23.35 |
| baseline_days | 7 |
| window_min | 15 |
| discriminator | GET /api/studio/events |
| window | 2026-11-17 10:15:00 UTC → 2026-11-17 10:30:00 UTC |
The story — No studiosvc deploy or hand change is recorded in the hour before this opened. Everything below is studiosvc only.
- 2026-11-11 06:35:31 UTCthis finding studiosvc GET /api/studio/events: 4xx — 100 of 100 calls (100 %; other 4xx×100) in the last 15 min — it rejects nearly every call
- 2026-11-14 00:00:00 UTClatency studiosvc p95 peaked at 20.57 s
- 2026-11-14 11:16:20 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/studiosvc by hand (kubectl/v1.35.9 (darwin/arm64) kubernetes/d834de3,gzip(gfe))
- 2026-11-14 14:25:45 UTCfinding · warning studiosvc: a new error shape, ≥4357× in the last hour, never seen in 90 days (error on pubsub receive — rpc error: code = NotFound desc = Resource n)
- 2026-11-14 14:27:08 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/studiosvc by hand (kubectl/v1.35.9 (darwin/arm64) kubernetes/d834de3,gzip(gfe))
- 2026-11-14 20:17:22 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/studiosvc by hand (kubectl/v1.35.9 (darwin/arm64) kubernetes/d834de3,gzip(gfe))
- 2026-11-15 02:25:02 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/studiosvc by hand (kubectl/v1.35.9 (darwin/arm64) kubernetes/d834de3,gzip(gfe))
- 2026-11-16 13:27:10 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/studiosvc by hand (kubectl/v1.35.9 (darwin/arm64) kubernetes/d834de3,gzip(gfe))
- 2026-11-16 13:48:59 UTCfinding · info studiosvc request rate 23.6 req/min vs a normal 0.0 req/min ± 5.0 req/min for Mon 15:00 UTC (3 of the last 4 checks)
History
- 2026-11-15 03:00:37 UTCreopened x-ray · severity=warning was=warning title=studiosvc GET /api/studio/events: 4xx — 43 of 43 calls (100 %; other 4xx×43) in the last 15 min — 688× its usual 0.1 level=medium was_level=medium finding={'id': 17063, 'fingerprint': '001ce0a5dd0a7ac21dd0', 'kind': 'endpoint.4xx_spike', 'severity': 'warning', 'cluster': 'test26', 'subject': 'studiosvc', 'title': 'studiosvc GET /api/studio/events: 4xx — 43 of 43 calls (100 %; other 4xx×43) in the last 15 min — 688× its usual 0.1', 'facts': {'endpoint': 'GET /api/studio/events', 'service': 'studiosvc', 'calls': 43, 'c4xx': 43, 'share_pct': 100.0, 'c401': 0, 'c403': 0, 'other_4xx': 43, 'rejecting': True, 'usual_per_window': 0.06, 'baseline_days': 7, 'window': [1794710100, 1794711000], 'window_min': 15, 'recent_deploys': [], 'discriminator': 'GET /api/studio/events'}, 'evidence': [{'label': 'The 4xx calls (status, caller, path)', 'href': '/api-calls?cluster=test26&from=1794710100&to=1794711000&service=studiosvc&status=4xx'}, {'label': 'studiosvc service page', 'href': '/service/studiosvc?cluster=test26&from=1794710100&to=1794711000'}], 'provenance': {'source': 'rollup', 'collected_at': 1794711636, 'age_seconds': 7, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 22.5, 'status': 'open', 'first_seen': 1794378931, 'last_seen': 1794711637, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 2, 'narrative': '4xx spike (403/401/404…): studiosvc GET /api/studio/events: permission denied — 42 of 43 calls (98 %; 403×42) in the last 15 min — it usually has none. Facts: endpoint GET /api/studio/events, service studiosvc, calls 43, c4xx 42, share_pct 97.67441860465117, c401 0.', 'narrative_kind': 'template', 'discriminator': 'GET /api/studio/events', 'level': 'medium', 'told_level': None, 'episode_at': 1794711637, 'chat_thread': None, 'kind_label': '4xx spike (403/401/404…)', 'group': 'Errors', 'age_s': 332711, 'stale_s': 5}
- 2026-11-15 08:32:27 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 17063, 'fingerprint': '001ce0a5dd0a7ac21dd0', 'kind': 'endpoint.4xx_spike', 'severity': 'warning', 'cluster': 'test26', 'subject': 'studiosvc', 'title': 'studiosvc GET /api/studio/events: 4xx — 49 of 49 calls (100 %; other 4xx×49) in the last 15 min — 64× its usual 0.8', 'facts': {'endpoint': 'GET /api/studio/events', 'service': 'studiosvc', 'calls': 49, 'c4xx': 49, 'share_pct': 100.0, 'c401': 0, 'c403': 0, 'other_4xx': 49, 'rejecting': True, 'usual_per_window': 0.76, 'baseline_days': 7, 'window': [1794722700, 1794723600], 'window_min': 15, 'recent_deploys': [], 'discriminator': 'GET /api/studio/events'}, 'evidence': [{'label': 'The 4xx calls (status, caller, path)', 'href': '/api-calls?cluster=test26&from=1794722700&to=1794723600&service=studiosvc&status=4xx'}, {'label': 'studiosvc service page', 'href': '/service/studiosvc?cluster=test26&from=1794722700&to=1794723600'}], 'provenance': {'source': 'rollup', 'collected_at': 1794724138, 'age_seconds': 7, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 26.7, 'status': 'resolved', 'first_seen': 1794378931, 'last_seen': 1794724138, 'resolved_at': 1794731547, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 18, 'narrative': '4xx spike (403/401/404…): studiosvc GET /api/studio/events: 4xx — 49 of 49 calls (100 %; other 4xx×49) in the last 15 min — 64× its usual 0.8. Facts: endpoint GET /api/studio/events, service studiosvc, calls 49, c4xx 49, share_pct 100.0, c401 0.', 'narrative_kind': 'template', 'discriminator': 'GET /api/studio/events', 'level': 'medium', 'told_level': None, 'episode_at': 1794711637, 'chat_thread': None, 'kind_label': '4xx spike (403/401/404…)', 'group': 'Errors', 'age_s': 352622, 'stale_s': 7415}
- 2026-11-15 15:19:13 UTCreopened x-ray · severity=warning was=warning title=studiosvc GET /api/studio/events: 4xx — 21 of 21 calls (100 %; other 4xx×21) in the last 15 min — 24× its usual 0.9 level=medium was_level=medium finding={'id': 17063, 'fingerprint': '001ce0a5dd0a7ac21dd0', 'kind': 'endpoint.4xx_spike', 'severity': 'warning', 'cluster': 'test26', 'subject': 'studiosvc', 'title': 'studiosvc GET /api/studio/events: 4xx — 21 of 21 calls (100 %; other 4xx×21) in the last 15 min — 24× its usual 0.9', 'facts': {'endpoint': 'GET /api/studio/events', 'service': 'studiosvc', 'calls': 21, 'c4xx': 21, 'share_pct': 100.0, 'c401': 0, 'c403': 0, 'other_4xx': 21, 'rejecting': True, 'usual_per_window': 0.88, 'baseline_days': 7, 'window': [1794754500, 1794755400], 'window_min': 15, 'recent_deploys': [], 'discriminator': 'GET /api/studio/events'}, 'evidence': [{'label': 'The 4xx calls (status, caller, path)', 'href': '/api-calls?cluster=test26&from=1794754500&to=1794755400&service=studiosvc&status=4xx'}, {'label': 'studiosvc service page', 'href': '/service/studiosvc?cluster=test26&from=1794754500&to=1794755400'}], 'provenance': {'source': 'rollup', 'collected_at': 1794755953, 'age_seconds': 6, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 21.5, 'status': 'open', 'first_seen': 1794378931, 'last_seen': 1794755953, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 19, 'narrative': '4xx spike (403/401/404…): studiosvc GET /api/studio/events: 4xx — 49 of 49 calls (100 %; other 4xx×49) in the last 15 min — 64× its usual 0.8. Facts: endpoint GET /api/studio/events, service studiosvc, calls 49, c4xx 49, share_pct 100.0, c401 0.', 'narrative_kind': 'template', 'discriminator': 'GET /api/studio/events', 'level': 'medium', 'told_level': None, 'episode_at': 1794755953, 'chat_thread': None, 'kind_label': '4xx spike (403/401/404…)', 'group': 'Errors', 'age_s': 377028, 'stale_s': 6}
- 2026-11-15 18:14:30 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 17063, 'fingerprint': '001ce0a5dd0a7ac21dd0', 'kind': 'endpoint.4xx_spike', 'severity': 'warning', 'cluster': 'test26', 'subject': 'studiosvc', 'title': 'studiosvc GET /api/studio/events: 4xx — 40 of 40 calls (100 %; other 4xx×40) in the last 15 min — 46× its usual 0.9', 'facts': {'endpoint': 'GET /api/studio/events', 'service': 'studiosvc', 'calls': 40, 'c4xx': 40, 'share_pct': 100.0, 'c401': 0, 'c403': 0, 'other_4xx': 40, 'rejecting': True, 'usual_per_window': 0.88, 'baseline_days': 7, 'window': [1794757500, 1794758400], 'window_min': 15, 'recent_deploys': [], 'discriminator': 'GET /api/studio/events'}, 'evidence': [{'label': 'The 4xx calls (status, caller, path)', 'href': '/api-calls?cluster=test26&from=1794757500&to=1794758400&service=studiosvc&status=4xx'}, {'label': 'studiosvc service page', 'href': '/service/studiosvc?cluster=test26&from=1794757500&to=1794758400'}], 'provenance': {'source': 'rollup', 'collected_at': 1794758929, 'age_seconds': 6, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 36.9, 'status': 'resolved', 'first_seen': 1794378931, 'last_seen': 1794758929, 'resolved_at': 1794766470, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 26, 'narrative': '4xx spike (403/401/404…): studiosvc GET /api/studio/events: 4xx — 40 of 40 calls (100 %; other 4xx×40) in the last 15 min — 46× its usual 0.9. Facts: endpoint GET /api/studio/events, service studiosvc, calls 40, c4xx 40, share_pct 100.0, c401 0.', 'narrative_kind': 'template', 'discriminator': 'GET /api/studio/events', 'level': 'medium', 'told_level': None, 'episode_at': 1794755953, 'chat_thread': None, 'kind_label': '4xx spike (403/401/404…)', 'group': 'Errors', 'age_s': 387544, 'stale_s': 7546}
- 2026-11-16 00:11:50 UTCreopened x-ray · severity=warning was=warning title=studiosvc GET /api/studio/events: 4xx — 29 of 29 calls (100 %; other 4xx×29) in the last 15 min — 25× its usual 1.2 level=medium was_level=medium finding={'id': 17063, 'fingerprint': '001ce0a5dd0a7ac21dd0', 'kind': 'endpoint.4xx_spike', 'severity': 'warning', 'cluster': 'test26', 'subject': 'studiosvc', 'title': 'studiosvc GET /api/studio/events: 4xx — 29 of 29 calls (100 %; other 4xx×29) in the last 15 min — 25× its usual 1.2', 'facts': {'endpoint': 'GET /api/studio/events', 'service': 'studiosvc', 'calls': 29, 'c4xx': 29, 'share_pct': 100.0, 'c401': 0, 'c403': 0, 'other_4xx': 29, 'rejecting': True, 'usual_per_window': 1.15, 'baseline_days': 7, 'window': [1794786600, 1794787500], 'window_min': 15, 'recent_deploys': [], 'discriminator': 'GET /api/studio/events'}, 'evidence': [{'label': 'The 4xx calls (status, caller, path)', 'href': '/api-calls?cluster=test26&from=1794786600&to=1794787500&service=studiosvc&status=4xx'}, {'label': 'studiosvc service page', 'href': '/service/studiosvc?cluster=test26&from=1794786600&to=1794787500'}], 'provenance': {'source': 'rollup', 'collected_at': 1794787909, 'age_seconds': 7, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 21.2, 'status': 'open', 'first_seen': 1794378931, 'last_seen': 1794787910, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 27, 'narrative': '4xx spike (403/401/404…): studiosvc GET /api/studio/events: 4xx — 40 of 40 calls (100 %; other 4xx×40) in the last 15 min — 46× its usual 0.9. Facts: endpoint GET /api/studio/events, service studiosvc, calls 40, c4xx 40, share_pct 100.0, c401 0.', 'narrative_kind': 'template', 'discriminator': 'GET /api/studio/events', 'level': 'medium', 'told_level': None, 'episode_at': 1794787910, 'chat_thread': None, 'kind_label': '4xx spike (403/401/404…)', 'group': 'Errors', 'age_s': 408984, 'stale_s': 5}
- 2026-11-16 01:29:10 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 17063, 'fingerprint': '001ce0a5dd0a7ac21dd0', 'kind': 'endpoint.4xx_spike', 'severity': 'warning', 'cluster': 'test26', 'subject': 'studiosvc', 'title': 'studiosvc GET /api/studio/events: 4xx — 42 of 42 calls (100 %; other 4xx×42) in the last 15 min — 36× its usual 1.2', 'facts': {'endpoint': 'GET /api/studio/events', 'service': 'studiosvc', 'calls': 42, 'c4xx': 42, 'share_pct': 100.0, 'c401': 0, 'c403': 0, 'other_4xx': 42, 'rejecting': True, 'usual_per_window': 1.17, 'baseline_days': 7, 'window': [1794788400, 1794789300], 'window_min': 15, 'recent_deploys': [], 'discriminator': 'GET /api/studio/events'}, 'evidence': [{'label': 'The 4xx calls (status, caller, path)', 'href': '/api-calls?cluster=test26&from=1794788400&to=1794789300&service=studiosvc&status=4xx'}, {'label': 'studiosvc service page', 'href': '/service/studiosvc?cluster=test26&from=1794788400&to=1794789300'}], 'provenance': {'source': 'rollup', 'collected_at': 1794789844, 'age_seconds': 5, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 26.6, 'status': 'resolved', 'first_seen': 1794378931, 'last_seen': 1794789844, 'resolved_at': 1794792550, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 30, 'narrative': '4xx spike (403/401/404…): studiosvc GET /api/studio/events: 4xx — 42 of 42 calls (100 %; other 4xx×42) in the last 15 min — 36× its usual 1.2. Facts: endpoint GET /api/studio/events, service studiosvc, calls 42, c4xx 42, share_pct 100.0, c401 0.', 'narrative_kind': 'template', 'discriminator': 'GET /api/studio/events', 'level': 'medium', 'told_level': None, 'episode_at': 1794787910, 'chat_thread': None, 'kind_label': '4xx spike (403/401/404…)', 'group': 'Errors', 'age_s': 413624, 'stale_s': 2711}
- 2026-11-16 01:36:14 UTCreopened x-ray · severity=warning was=warning title=studiosvc GET /api/studio/events: 4xx — 38 of 38 calls (100 %; other 4xx×38) in the last 15 min — 30× its usual 1.3 level=medium was_level=medium finding={'id': 17063, 'fingerprint': '001ce0a5dd0a7ac21dd0', 'kind': 'endpoint.4xx_spike', 'severity': 'warning', 'cluster': 'test26', 'subject': 'studiosvc', 'title': 'studiosvc GET /api/studio/events: 4xx — 38 of 38 calls (100 %; other 4xx×38) in the last 15 min — 30× its usual 1.3', 'facts': {'endpoint': 'GET /api/studio/events', 'service': 'studiosvc', 'calls': 38, 'c4xx': 38, 'share_pct': 100.0, 'c401': 0, 'c403': 0, 'other_4xx': 38, 'rejecting': True, 'usual_per_window': 1.26, 'baseline_days': 7, 'window': [1794791700, 1794792600], 'window_min': 15, 'recent_deploys': [], 'discriminator': 'GET /api/studio/events'}, 'evidence': [{'label': 'The 4xx calls (status, caller, path)', 'href': '/api-calls?cluster=test26&from=1794791700&to=1794792600&service=studiosvc&status=4xx'}, {'label': 'studiosvc service page', 'href': '/service/studiosvc?cluster=test26&from=1794791700&to=1794792600'}], 'provenance': {'source': 'rollup', 'collected_at': 1794792974, 'age_seconds': 6, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 21.3, 'status': 'open', 'first_seen': 1794378931, 'last_seen': 1794792974, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 31, 'narrative': '4xx spike (403/401/404…): studiosvc GET /api/studio/events: 4xx — 42 of 42 calls (100 %; other 4xx×42) in the last 15 min — 36× its usual 1.2. Facts: endpoint GET /api/studio/events, service studiosvc, calls 42, c4xx 42, share_pct 100.0, c401 0.', 'narrative_kind': 'template', 'discriminator': 'GET /api/studio/events', 'level': 'medium', 'told_level': None, 'episode_at': 1794792974, 'chat_thread': None, 'kind_label': '4xx spike (403/401/404…)', 'group': 'Errors', 'age_s': 414048, 'stale_s': 5}
- 2026-11-16 02:26:12 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 17063, 'fingerprint': '001ce0a5dd0a7ac21dd0', 'kind': 'endpoint.4xx_spike', 'severity': 'warning', 'cluster': 'test26', 'subject': 'studiosvc', 'title': 'studiosvc GET /api/studio/events: 4xx — 56 of 56 calls (100 %; other 4xx×56) in the last 15 min — 45× its usual 1.3', 'facts': {'endpoint': 'GET /api/studio/events', 'service': 'studiosvc', 'calls': 56, 'c4xx': 56, 'share_pct': 100.0, 'c401': 0, 'c403': 0, 'other_4xx': 56, 'rejecting': True, 'usual_per_window': 1.26, 'baseline_days': 7, 'window': [1794793500, 1794794400], 'window_min': 15, 'recent_deploys': [], 'discriminator': 'GET /api/studio/events'}, 'evidence': [{'label': 'The 4xx calls (status, caller, path)', 'href': '/api-calls?cluster=test26&from=1794793500&to=1794794400&service=studiosvc&status=4xx'}, {'label': 'studiosvc service page', 'href': '/service/studiosvc?cluster=test26&from=1794793500&to=1794794400'}], 'provenance': {'source': 'rollup', 'collected_at': 1794794999, 'age_seconds': 6, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 25.9, 'status': 'resolved', 'first_seen': 1794378931, 'last_seen': 1794794999, 'resolved_at': 1794795972, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 34, 'narrative': '4xx spike (403/401/404…): studiosvc GET /api/studio/events: 4xx — 56 of 56 calls (100 %; other 4xx×56) in the last 15 min — 45× its usual 1.3. Facts: endpoint GET /api/studio/events, service studiosvc, calls 56, c4xx 56, share_pct 100.0, c401 0.', 'narrative_kind': 'template', 'discriminator': 'GET /api/studio/events', 'level': 'medium', 'told_level': None, 'episode_at': 1794792974, 'chat_thread': None, 'kind_label': '4xx spike (403/401/404…)', 'group': 'Errors', 'age_s': 417046, 'stale_s': 978}
- 2026-11-16 02:51:52 UTCreopened x-ray · severity=warning was=warning title=studiosvc GET /api/studio/events: 4xx — 25 of 25 calls (100 %; other 4xx×25) in the last 15 min — 18× its usual 1.4 level=medium was_level=medium finding={'id': 17063, 'fingerprint': '001ce0a5dd0a7ac21dd0', 'kind': 'endpoint.4xx_spike', 'severity': 'warning', 'cluster': 'test26', 'subject': 'studiosvc', 'title': 'studiosvc GET /api/studio/events: 4xx — 25 of 25 calls (100 %; other 4xx×25) in the last 15 min — 18× its usual 1.4', 'facts': {'endpoint': 'GET /api/studio/events', 'service': 'studiosvc', 'calls': 25, 'c4xx': 25, 'share_pct': 100.0, 'c401': 0, 'c403': 0, 'other_4xx': 25, 'rejecting': True, 'usual_per_window': 1.4, 'baseline_days': 7, 'window': [1794796200, 1794797100], 'window_min': 15, 'recent_deploys': [], 'discriminator': 'GET /api/studio/events'}, 'evidence': [{'label': 'The 4xx calls (status, caller, path)', 'href': '/api-calls?cluster=test26&from=1794796200&to=1794797100&service=studiosvc&status=4xx'}, {'label': 'studiosvc service page', 'href': '/service/studiosvc?cluster=test26&from=1794796200&to=1794797100'}], 'provenance': {'source': 'rollup', 'collected_at': 1794797512, 'age_seconds': 6, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 20.7, 'status': 'open', 'first_seen': 1794378931, 'last_seen': 1794797512, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 35, 'narrative': '4xx spike (403/401/404…): studiosvc GET /api/studio/events: 4xx — 56 of 56 calls (100 %; other 4xx×56) in the last 15 min — 45× its usual 1.3. Facts: endpoint GET /api/studio/events, service studiosvc, calls 56, c4xx 56, share_pct 100.0, c401 0.', 'narrative_kind': 'template', 'discriminator': 'GET /api/studio/events', 'level': 'medium', 'told_level': None, 'episode_at': 1794797512, 'chat_thread': None, 'kind_label': '4xx spike (403/401/404…)', 'group': 'Errors', 'age_s': 418586, 'stale_s': 5}
- 2026-11-16 08:15:41 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 17063, 'fingerprint': '001ce0a5dd0a7ac21dd0', 'kind': 'endpoint.4xx_spike', 'severity': 'warning', 'cluster': 'test26', 'subject': 'studiosvc', 'title': 'studiosvc GET /api/studio/events: 4xx — 66 of 66 calls (100 %; other 4xx×66) in the last 15 min — 25× its usual 2.6', 'facts': {'endpoint': 'GET /api/studio/events', 'service': 'studiosvc', 'calls': 66, 'c4xx': 66, 'share_pct': 100.0, 'c401': 0, 'c403': 0, 'other_4xx': 66, 'rejecting': True, 'usual_per_window': 2.62, 'baseline_days': 7, 'window': [1794814200, 1794815100], 'window_min': 15, 'recent_deploys': [], 'discriminator': 'GET /api/studio/events'}, 'evidence': [{'label': 'The 4xx calls (status, caller, path)', 'href': '/api-calls?cluster=test26&from=1794814200&to=1794815100&service=studiosvc&status=4xx'}, {'label': 'studiosvc service page', 'href': '/service/studiosvc?cluster=test26&from=1794814200&to=1794815100'}], 'provenance': {'source': 'rollup', 'collected_at': 1794815516, 'age_seconds': 7, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 36.3, 'status': 'resolved', 'first_seen': 1794378931, 'last_seen': 1794815516, 'resolved_at': 1794816941, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 54, 'narrative': '4xx spike (403/401/404…): studiosvc GET /api/studio/events: 4xx — 66 of 66 calls (100 %; other 4xx×66) in the last 15 min — 25× its usual 2.6. Facts: endpoint GET /api/studio/events, service studiosvc, calls 66, c4xx 66, share_pct 100.0, c401 0.', 'narrative_kind': 'template', 'discriminator': 'GET /api/studio/events', 'level': 'medium', 'told_level': None, 'episode_at': 1794797512, 'chat_thread': None, 'kind_label': '4xx spike (403/401/404…)', 'group': 'Errors', 'age_s': 438018, 'stale_s': 1433}
- 2026-11-16 08:28:23 UTCreopened x-ray · severity=warning was=warning title=studiosvc GET /api/studio/events: 4xx — 20 of 20 calls (100 %; other 4xx×20) in the last 15 min — 7× its usual 2.9 level=medium was_level=medium finding={'id': 17063, 'fingerprint': '001ce0a5dd0a7ac21dd0', 'kind': 'endpoint.4xx_spike', 'severity': 'warning', 'cluster': 'test26', 'subject': 'studiosvc', 'title': 'studiosvc GET /api/studio/events: 4xx — 20 of 20 calls (100 %; other 4xx×20) in the last 15 min — 7× its usual 2.9', 'facts': {'endpoint': 'GET /api/studio/events', 'service': 'studiosvc', 'calls': 20, 'c4xx': 20, 'share_pct': 100.0, 'c401': 0, 'c403': 0, 'other_4xx': 20, 'rejecting': True, 'usual_per_window': 2.93, 'baseline_days': 7, 'window': [1794816300, 1794817200], 'window_min': 15, 'recent_deploys': [], 'discriminator': 'GET /api/studio/events'}, 'evidence': [{'label': 'The 4xx calls (status, caller, path)', 'href': '/api-calls?cluster=test26&from=1794816300&to=1794817200&service=studiosvc&status=4xx'}, {'label': 'studiosvc service page', 'href': '/service/studiosvc?cluster=test26&from=1794816300&to=1794817200'}], 'provenance': {'source': 'rollup', 'collected_at': 1794817703, 'age_seconds': 7, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 30.5, 'status': 'open', 'first_seen': 1794378931, 'last_seen': 1794817703, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 55, 'narrative': '4xx spike (403/401/404…): studiosvc GET /api/studio/events: 4xx — 66 of 66 calls (100 %; other 4xx×66) in the last 15 min — 25× its usual 2.6. Facts: endpoint GET /api/studio/events, service studiosvc, calls 66, c4xx 66, share_pct 100.0, c401 0.', 'narrative_kind': 'template', 'discriminator': 'GET /api/studio/events', 'level': 'medium', 'told_level': None, 'episode_at': 1794817703, 'chat_thread': None, 'kind_label': '4xx spike (403/401/404…)', 'group': 'Errors', 'age_s': 438778, 'stale_s': 6}
- 2026-11-16 08:44:24 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 17063, 'fingerprint': '001ce0a5dd0a7ac21dd0', 'kind': 'endpoint.4xx_spike', 'severity': 'warning', 'cluster': 'test26', 'subject': 'studiosvc', 'title': 'studiosvc GET /api/studio/events: 4xx — 20 of 20 calls (100 %; other 4xx×20) in the last 15 min — 7× its usual 2.9', 'facts': {'endpoint': 'GET /api/studio/events', 'service': 'studiosvc', 'calls': 20, 'c4xx': 20, 'share_pct': 100.0, 'c401': 0, 'c403': 0, 'other_4xx': 20, 'rejecting': True, 'usual_per_window': 2.93, 'baseline_days': 7, 'window': [1794816300, 1794817200], 'window_min': 15, 'recent_deploys': [], 'discriminator': 'GET /api/studio/events'}, 'evidence': [{'label': 'The 4xx calls (status, caller, path)', 'href': '/api-calls?cluster=test26&from=1794816300&to=1794817200&service=studiosvc&status=4xx'}, {'label': 'studiosvc service page', 'href': '/service/studiosvc?cluster=test26&from=1794816300&to=1794817200'}], 'provenance': {'source': 'rollup', 'collected_at': 1794817703, 'age_seconds': 7, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 35.5, 'status': 'resolved', 'first_seen': 1794378931, 'last_seen': 1794817703, 'resolved_at': 1794818664, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 55, 'narrative': '4xx spike (403/401/404…): studiosvc GET /api/studio/events: 4xx — 20 of 20 calls (100 %; other 4xx×20) in the last 15 min — 7× its usual 2.9. Facts: endpoint GET /api/studio/events, service studiosvc, calls 20, c4xx 20, share_pct 100.0, c401 0.', 'narrative_kind': 'template', 'discriminator': 'GET /api/studio/events', 'level': 'medium', 'told_level': None, 'episode_at': 1794817703, 'chat_thread': None, 'kind_label': '4xx spike (403/401/404…)', 'group': 'Errors', 'age_s': 439739, 'stale_s': 967}
- 2026-11-16 12:49:12 UTCreopened x-ray · severity=warning was=warning title=studiosvc GET /api/studio/events: 4xx — 118 of 118 calls (100 %; 403×11, other 4xx×107) in the last 15 min — 39× its usual 3.0 level=medium was_level=medium finding={'id': 17063, 'fingerprint': '001ce0a5dd0a7ac21dd0', 'kind': 'endpoint.4xx_spike', 'severity': 'warning', 'cluster': 'test26', 'subject': 'studiosvc', 'title': 'studiosvc GET /api/studio/events: 4xx — 118 of 118 calls (100 %; 403×11, other 4xx×107) in the last 15 min — 39× its usual 3.0', 'facts': {'endpoint': 'GET /api/studio/events', 'service': 'studiosvc', 'calls': 118, 'c4xx': 118, 'share_pct': 100.0, 'c401': 0, 'c403': 11, 'other_4xx': 107, 'rejecting': True, 'usual_per_window': 3.02, 'baseline_days': 7, 'window': [1794831900, 1794832800], 'window_min': 15, 'recent_deploys': [], 'discriminator': 'GET /api/studio/events'}, 'evidence': [{'label': 'The 4xx calls (status, caller, path)', 'href': '/api-calls?cluster=test26&from=1794831900&to=1794832800&service=studiosvc&status=4xx'}, {'label': 'studiosvc service page', 'href': '/service/studiosvc?cluster=test26&from=1794831900&to=1794832800'}], 'provenance': {'source': 'rollup', 'collected_at': 1794833351, 'age_seconds': 12, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 27.8, 'status': 'open', 'first_seen': 1794378931, 'last_seen': 1794833352, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 56, 'narrative': '4xx spike (403/401/404…): studiosvc GET /api/studio/events: 4xx — 20 of 20 calls (100 %; other 4xx×20) in the last 15 min — 7× its usual 2.9. Facts: endpoint GET /api/studio/events, service studiosvc, calls 20, c4xx 20, share_pct 100.0, c401 0.', 'narrative_kind': 'template', 'discriminator': 'GET /api/studio/events', 'level': 'medium', 'told_level': None, 'episode_at': 1794833352, 'chat_thread': None, 'kind_label': '4xx spike (403/401/404…)', 'group': 'Errors', 'age_s': 454431, 'stale_s': 10}
Check it yourself
Every link carries the finding's own time window, so it shows tomorrow what it shows now.
Actions
Sign in (IAP) to acknowledge, snooze or resolve.
Where the numbers come from
| source | rollup |
| collected_at | 1794912040 |
| age_seconds | 13 |
| stale | False |
| shown | None |
| total | None |
| truncated | False |
| coverage_from | None |
| retain_days | None |
| note | None |
| detector | endpoint_4xx_spike |