feedsvc GET /studies/:id/feed: permission denied — 108 of 108 calls (100 %; 403×108) in the last 15 min — it rejects nearly every call
critical
4xx spike (403/401/404…)
master
resolved
impact 67
opened 5d ago · last observed 1d ago · observed 59× · #17146 · 076348fd8c5135dc5d1e
What happened template
4xx spike (403/401/404…): feedsvc GET /studies/:id/feed: permission denied — 108 of 108 calls (100 %; 403×108) in the last 15 min — it rejects nearly every call. Facts: endpoint GET /studies/:id/feed, service feedsvc, calls 108, c4xx 108, share_pct 100.0, c401 0.
An endpoint answering far more 4xx than its own usual rate — a permissions change (403), expired or broken auth (401), a missing route or contract break (404). Its usual rate is its last seven days.
First look — investigated before anyone looked
Cause not clear from logs and recent changes; feedsvc GET /studies/:id/feed returned 403 on 872 of 874 calls (99.8 %) in 15 min, 329x its usual 2.7.
Onset 07:10 UTC on 2026-11-16; the first 403 in the request log is at 07:31 UTC. GET /studies/:id/feed/week shows the same spike (580 of 580 are 403).
All 8 sampled 403s are fast (17-22 ms) and come from org "Tilda CRO", so a caller is hitting the feed many times without permission.
This is a recurring finding: 5 earlier episodes since 2026-11-12, all cleared on their own.
Checked: no feedsvc error lines stored, and the only 2 deploys were config-only autosyncs, 352 and 521 min before onset, with no link to the 403s.
Next step: open a trace and see which user or client in "Tilda CRO" is calling, then check its access to those studies.
low confidence
claude-opus-5-5 · 2d ago · $0.04 — every link above is evidence it read; nothing else was claimed.
The facts
| endpoint | GET /studies/:id/feed |
| service | feedsvc |
| calls | 108 |
| c4xx | 108 |
| share_pct | 100.00 |
| c401 | 0 |
| c403 | 108 |
| other_4xx | 0 |
| rejecting | yes |
| usual_per_window | 70.93 |
| baseline_days | 7 |
| window_min | 15 |
| discriminator | GET /studies/:id/feed |
| window | 2026-11-17 02:05:00 UTC → 2026-11-17 02:20:00 UTC |
The story — No feedsvc deploy or hand change is recorded in the hour before this opened. Everything below is feedsvc only.
- 2026-11-12 18:03:36 UTCfinding · critical feedsvc GET /studies/:id/feed/week: permission denied — 23 of 24 calls (96 %; 403×23) in the last 15 min — it rejects nearly every call
- 2026-11-12 18:03:36 UTCthis finding feedsvc GET /studies/:id/feed: permission denied — 108 of 108 calls (100 %; 403×108) in the last 15 min — it rejects nearly every call
- 2026-11-13 19:37:11 UTCfinding · info feedsvc request rate 27.4 req/min vs a normal 0.0 req/min ± 5.0 req/min for Tue 10:00 UTC (3 of the last 4 checks)
- 2026-11-15 22:28:32 UTCdeploy automated deployed feedsvc
- 2026-11-16 01:17:17 UTCdeploy automated deployed feedsvc
- 2026-11-16 07:43:55 UTCfinding · info feedsvc Neo4j load 83 q/min vs a normal 0 q/min ± 20 q/min for Mon 18:00 UTC (4 of the last 4 checks)
- 2026-11-16 11:17:28 UTCdeploy automated deployed feedsvc
- 2026-11-16 18:04:02 UTCdeploy automated deployed feedsvc
- 2026-11-16 22:26:02 UTCdeploy automated deployed feedsvc
- 2026-11-17 02:53:50 UTCdeploy automated deployed feedsvc
History
- 2026-11-15 07:28:58 UTCreopened x-ray · severity=critical was=critical title=feedsvc GET /studies/:id/feed: permission denied — 22 of 25 calls (88 %; 403×21, other 4xx×1) in the last 15 min — 8× its usual 2.6 level=high was_level=high finding={'id': 17146, 'fingerprint': '076348fd8c5135dc5d1e', 'kind': 'endpoint.4xx_spike', 'severity': 'critical', 'cluster': 'master', 'subject': 'feedsvc', 'title': 'feedsvc GET /studies/:id/feed: permission denied — 22 of 25 calls (88 %; 403×21, other 4xx×1) in the last 15 min — 8× its usual 2.6', 'facts': {'endpoint': 'GET /studies/:id/feed', 'service': 'feedsvc', 'calls': 25, 'c4xx': 22, 'share_pct': 88.0, 'c401': 0, 'c403': 21, 'other_4xx': 1, 'rejecting': False, 'usual_per_window': 2.61, 'baseline_days': 7, 'window': [1794726000, 1794726900], 'window_min': 15, 'recent_deploys': [], 'discriminator': 'GET /studies/:id/feed'}, 'evidence': [{'label': 'The 4xx calls (status, caller, path)', 'href': '/api-calls?cluster=master&from=1794726000&to=1794726900&service=feedsvc&status=4xx'}, {'label': 'feedsvc service page', 'href': '/service/feedsvc?cluster=master&from=1794726000&to=1794726900'}], 'provenance': {'source': 'rollup', 'collected_at': 1794727406, 'age_seconds': 338, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 57.7, 'status': 'open', 'first_seen': 1794506616, 'last_seen': 1794727738, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 16, 'narrative': '4xx spike (403/401/404…): feedsvc GET /studies/:id/feed: permission denied — 83 of 84 calls (99 %; 403×83) in the last 15 min — 34× its usual 2.4. Facts: endpoint GET /studies/:id/feed, service feedsvc, calls 84, c4xx 83, share_pct 98.80952380952381, c401 0.', 'narrative_kind': 'template', 'discriminator': 'GET /studies/:id/feed', 'level': 'high', 'told_level': None, 'episode_at': 1794727738, 'chat_thread': None, 'kind_label': '4xx spike (403/401/404…)', 'group': 'Errors', 'age_s': 221127, 'stale_s': 5}
- 2026-11-15 07:55:18 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 17146, 'fingerprint': '076348fd8c5135dc5d1e', 'kind': 'endpoint.4xx_spike', 'severity': 'critical', 'cluster': 'master', 'subject': 'feedsvc', 'title': 'feedsvc GET /studies/:id/feed: permission denied — 23 of 25 calls (92 %; 403×21, other 4xx×2) in the last 15 min — 9× its usual 2.6', 'facts': {'endpoint': 'GET /studies/:id/feed', 'service': 'feedsvc', 'calls': 25, 'c4xx': 23, 'share_pct': 92.0, 'c401': 0, 'c403': 21, 'other_4xx': 2, 'rejecting': False, 'usual_per_window': 2.61, 'baseline_days': 7, 'window': [1794726600, 1794727500], 'window_min': 15, 'recent_deploys': [], 'discriminator': 'GET /studies/:id/feed'}, 'evidence': [{'label': 'The 4xx calls (status, caller, path)', 'href': '/api-calls?cluster=master&from=1794726600&to=1794727500&service=feedsvc&status=4xx'}, {'label': 'feedsvc service page', 'href': '/service/feedsvc?cluster=master&from=1794726600&to=1794727500'}], 'provenance': {'source': 'rollup', 'collected_at': 1794728108, 'age_seconds': 236, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 61.7, 'status': 'resolved', 'first_seen': 1794506616, 'last_seen': 1794728338, 'resolved_at': 1794729318, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 17, 'narrative': '4xx spike (403/401/404…): feedsvc GET /studies/:id/feed: permission denied — 23 of 25 calls (92 %; 403×21, other 4xx×2) in the last 15 min — 9× its usual 2.6. Facts: endpoint GET /studies/:id/feed, service feedsvc, calls 25, c4xx 23, share_pct 92.0, c401 0.', 'narrative_kind': 'template', 'discriminator': 'GET /studies/:id/feed', 'level': 'high', 'told_level': None, 'episode_at': 1794727738, 'chat_thread': None, 'kind_label': '4xx spike (403/401/404…)', 'group': 'Errors', 'age_s': 222708, 'stale_s': 986}
- 2026-11-16 07:34:54 UTCreopened x-ray · severity=critical was=critical title=feedsvc GET /studies/:id/feed: permission denied — 872 of 874 calls (100 %; 403×872) in the last 15 min — 329× its usual 2.7 level=high was_level=high finding={'id': 17146, 'fingerprint': '076348fd8c5135dc5d1e', 'kind': 'endpoint.4xx_spike', 'severity': 'critical', 'cluster': 'master', 'subject': 'feedsvc', 'title': 'feedsvc GET /studies/:id/feed: permission denied — 872 of 874 calls (100 %; 403×872) in the last 15 min — 329× its usual 2.7', 'facts': {'endpoint': 'GET /studies/:id/feed', 'service': 'feedsvc', 'calls': 874, 'c4xx': 872, 'share_pct': 99.77116704805492, 'c401': 0, 'c403': 872, 'other_4xx': 0, 'rejecting': True, 'usual_per_window': 2.65, 'baseline_days': 7, 'window': [1794813000, 1794813900], 'window_min': 15, 'recent_deploys': [], 'discriminator': 'GET /studies/:id/feed'}, 'evidence': [{'label': 'The 4xx calls (status, caller, path)', 'href': '/api-calls?cluster=master&from=1794813000&to=1794813900&service=feedsvc&status=4xx'}, {'label': 'feedsvc service page', 'href': '/service/feedsvc?cluster=master&from=1794813000&to=1794813900'}], 'provenance': {'source': 'rollup', 'collected_at': 1794814423, 'age_seconds': 78, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 66.8, 'status': 'open', 'first_seen': 1794506616, 'last_seen': 1794814494, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 18, 'narrative': '4xx spike (403/401/404…): feedsvc GET /studies/:id/feed: permission denied — 23 of 25 calls (92 %; 403×21, other 4xx×2) in the last 15 min — 9× its usual 2.6. Facts: endpoint GET /studies/:id/feed, service feedsvc, calls 25, c4xx 23, share_pct 92.0, c401 0.', 'narrative_kind': 'template', 'discriminator': 'GET /studies/:id/feed', 'level': 'high', 'told_level': None, 'episode_at': 1794814494, 'chat_thread': None, 'kind_label': '4xx spike (403/401/404…)', 'group': 'Errors', 'age_s': 307885, 'stale_s': 7}
- 2026-11-16 11:24:39 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 17146, 'fingerprint': '076348fd8c5135dc5d1e', 'kind': 'endpoint.4xx_spike', 'severity': 'critical', 'cluster': 'master', 'subject': 'feedsvc', 'title': 'feedsvc GET /studies/:id/feed: permission denied — 1,241 of 1,245 calls (100 %; 403×1,241) in the last 15 min — 199× its usual 6.2', 'facts': {'endpoint': 'GET /studies/:id/feed', 'service': 'feedsvc', 'calls': 1245, 'c4xx': 1241, 'share_pct': 99.67871485943775, 'c401': 0, 'c403': 1241, 'other_4xx': 0, 'rejecting': True, 'usual_per_window': 6.24, 'baseline_days': 7, 'window': [1794821700, 1794822600], 'window_min': 15, 'recent_deploys': [], 'discriminator': 'GET /studies/:id/feed'}, 'evidence': [{'label': 'The 4xx calls (status, caller, path)', 'href': '/api-calls?cluster=master&from=1794821700&to=1794822600&service=feedsvc&status=4xx'}, {'label': 'feedsvc service page', 'href': '/service/feedsvc?cluster=master&from=1794821700&to=1794822600'}], 'provenance': {'source': 'rollup', 'collected_at': 1794823281, 'age_seconds': 1108, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 74.2, 'status': 'resolved', 'first_seen': 1794506616, 'last_seen': 1794824378, 'resolved_at': 1794828279, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 25, 'narrative': '4xx spike (403/401/404…): feedsvc GET /studies/:id/feed: permission denied — 1,241 of 1,245 calls (100 %; 403×1,241) in the last 15 min — 199× its usual 6.2. Facts: endpoint GET /studies/:id/feed, service feedsvc, calls 1245, c4xx 1241, share_pct 99.67871485943775, c401 0.', 'narrative_kind': 'template', 'discriminator': 'GET /studies/:id/feed', 'level': 'high', 'told_level': None, 'episode_at': 1794814494, 'chat_thread': None, 'kind_label': '4xx spike (403/401/404…)', 'group': 'Errors', 'age_s': 321671, 'stale_s': 3909}
- 2026-11-16 12:06:52 UTCreopened x-ray · severity=critical was=critical title=feedsvc GET /studies/:id/feed: permission denied — 79 of 81 calls (98 %; 403×79) in the last 15 min — 5× its usual 15.3 level=high was_level=high finding={'id': 17146, 'fingerprint': '076348fd8c5135dc5d1e', 'kind': 'endpoint.4xx_spike', 'severity': 'critical', 'cluster': 'master', 'subject': 'feedsvc', 'title': 'feedsvc GET /studies/:id/feed: permission denied — 79 of 81 calls (98 %; 403×79) in the last 15 min — 5× its usual 15.3', 'facts': {'endpoint': 'GET /studies/:id/feed', 'service': 'feedsvc', 'calls': 81, 'c4xx': 79, 'share_pct': 97.53086419753086, 'c401': 0, 'c403': 79, 'other_4xx': 0, 'rejecting': True, 'usual_per_window': 15.26, 'baseline_days': 7, 'window': [1794828000, 1794828900], 'window_min': 15, 'recent_deploys': [], 'discriminator': 'GET /studies/:id/feed'}, 'evidence': [{'label': 'The 4xx calls (status, caller, path)', 'href': '/api-calls?cluster=master&from=1794828000&to=1794828900&service=feedsvc&status=4xx'}, {'label': 'feedsvc service page', 'href': '/service/feedsvc?cluster=master&from=1794828000&to=1794828900'}], 'provenance': {'source': 'rollup', 'collected_at': 1794829599, 'age_seconds': 1222, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 65.5, 'status': 'open', 'first_seen': 1794506616, 'last_seen': 1794830812, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 26, 'narrative': '4xx spike (403/401/404…): feedsvc GET /studies/:id/feed: permission denied — 1,241 of 1,245 calls (100 %; 403×1,241) in the last 15 min — 199× its usual 6.2. Facts: endpoint GET /studies/:id/feed, service feedsvc, calls 1245, c4xx 1241, share_pct 99.67871485943775, c401 0.', 'narrative_kind': 'template', 'discriminator': 'GET /studies/:id/feed', 'level': 'high', 'told_level': None, 'episode_at': 1794830812, 'chat_thread': None, 'kind_label': '4xx spike (403/401/404…)', 'group': 'Errors', 'age_s': 324205, 'stale_s': 9}
- 2026-11-16 19:23:44 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 17146, 'fingerprint': '076348fd8c5135dc5d1e', 'kind': 'endpoint.4xx_spike', 'severity': 'critical', 'cluster': 'master', 'subject': 'feedsvc', 'title': 'feedsvc GET /studies/:id/feed: permission denied — 485 of 489 calls (99 %; 403×485) in the last 15 min — 8× its usual 62.1', 'facts': {'endpoint': 'GET /studies/:id/feed', 'service': 'feedsvc', 'calls': 489, 'c4xx': 485, 'share_pct': 99.18200408997956, 'c401': 0, 'c403': 485, 'other_4xx': 0, 'rejecting': True, 'usual_per_window': 62.09, 'baseline_days': 7, 'window': [1794854400, 1794855300], 'window_min': 15, 'recent_deploys': [], 'discriminator': 'GET /studies/:id/feed'}, 'evidence': [{'label': 'The 4xx calls (status, caller, path)', 'href': '/api-calls?cluster=master&from=1794854400&to=1794855300&service=feedsvc&status=4xx'}, {'label': 'feedsvc service page', 'href': '/service/feedsvc?cluster=master&from=1794854400&to=1794855300'}], 'provenance': {'source': 'rollup', 'collected_at': 1794855670, 'age_seconds': 182, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 75.1, 'status': 'resolved', 'first_seen': 1794506616, 'last_seen': 1794855844, 'resolved_at': 1794857024, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 58, 'narrative': '4xx spike (403/401/404…): feedsvc GET /studies/:id/feed: permission denied — 485 of 489 calls (99 %; 403×485) in the last 15 min — 8× its usual 62.1. Facts: endpoint GET /studies/:id/feed, service feedsvc, calls 489, c4xx 485, share_pct 99.18200408997956, c401 0.', 'narrative_kind': 'template', 'discriminator': 'GET /studies/:id/feed', 'level': 'high', 'told_level': None, 'episode_at': 1794830812, 'chat_thread': None, 'kind_label': '4xx spike (403/401/404…)', 'group': 'Errors', 'age_s': 350415, 'stale_s': 1187}
- 2026-11-17 02:35:20 UTCreopened x-ray · severity=critical was=critical title=feedsvc GET /studies/:id/feed: permission denied — 108 of 108 calls (100 %; 403×108) in the last 15 min — it rejects nearly every call level=high was_level=high finding={'id': 17146, 'fingerprint': '076348fd8c5135dc5d1e', 'kind': 'endpoint.4xx_spike', 'severity': 'critical', 'cluster': 'master', 'subject': 'feedsvc', 'title': 'feedsvc GET /studies/:id/feed: permission denied — 108 of 108 calls (100 %; 403×108) in the last 15 min — it rejects nearly every call', 'facts': {'endpoint': 'GET /studies/:id/feed', 'service': 'feedsvc', 'calls': 108, 'c4xx': 108, 'share_pct': 100.0, 'c401': 0, 'c403': 108, 'other_4xx': 0, 'rejecting': True, 'usual_per_window': 70.93, 'baseline_days': 7, 'window': [1794881100, 1794882000], 'window_min': 15, 'recent_deploys': [], 'discriminator': 'GET /studies/:id/feed'}, 'evidence': [{'label': 'The 4xx calls (status, caller, path)', 'href': '/api-calls?cluster=master&from=1794881100&to=1794882000&service=feedsvc&status=4xx'}, {'label': 'feedsvc service page', 'href': '/service/feedsvc?cluster=master&from=1794881100&to=1794882000'}], 'provenance': {'source': 'rollup', 'collected_at': 1794882595, 'age_seconds': 333, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 61.8, 'status': 'open', 'first_seen': 1794506616, 'last_seen': 1794882920, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 59, 'narrative': '4xx spike (403/401/404…): feedsvc GET /studies/:id/feed: permission denied — 485 of 489 calls (99 %; 403×485) in the last 15 min — 8× its usual 62.1. Facts: endpoint GET /studies/:id/feed, service feedsvc, calls 489, c4xx 485, share_pct 99.18200408997956, c401 0.', 'narrative_kind': 'template', 'discriminator': 'GET /studies/:id/feed', 'level': 'high', 'told_level': None, 'episode_at': 1794882920, 'chat_thread': None, 'kind_label': '4xx spike (403/401/404…)', 'group': 'Errors', 'age_s': 376312, 'stale_s': 8}
- 2026-11-17 03:35:58 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 17146, 'fingerprint': '076348fd8c5135dc5d1e', 'kind': 'endpoint.4xx_spike', 'severity': 'critical', 'cluster': 'master', 'subject': 'feedsvc', 'title': 'feedsvc GET /studies/:id/feed: permission denied — 108 of 108 calls (100 %; 403×108) in the last 15 min — it rejects nearly every call', 'facts': {'endpoint': 'GET /studies/:id/feed', 'service': 'feedsvc', 'calls': 108, 'c4xx': 108, 'share_pct': 100.0, 'c401': 0, 'c403': 108, 'other_4xx': 0, 'rejecting': True, 'usual_per_window': 70.93, 'baseline_days': 7, 'window': [1794881100, 1794882000], 'window_min': 15, 'recent_deploys': [], 'discriminator': 'GET /studies/:id/feed'}, 'evidence': [{'label': 'The 4xx calls (status, caller, path)', 'href': '/api-calls?cluster=master&from=1794881100&to=1794882000&service=feedsvc&status=4xx'}, {'label': 'feedsvc service page', 'href': '/service/feedsvc?cluster=master&from=1794881100&to=1794882000'}], 'provenance': {'source': 'rollup', 'collected_at': 1794882595, 'age_seconds': 333, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 66.8, 'status': 'resolved', 'first_seen': 1794506616, 'last_seen': 1794882920, 'resolved_at': 1794886558, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 59, 'narrative': '4xx spike (403/401/404…): feedsvc GET /studies/:id/feed: permission denied — 108 of 108 calls (100 %; 403×108) in the last 15 min — it rejects nearly every call. Facts: endpoint GET /studies/:id/feed, service feedsvc, calls 108, c4xx 108, share_pct 100.0, c401 0.', 'narrative_kind': 'template', 'discriminator': 'GET /studies/:id/feed', 'level': 'high', 'told_level': None, 'episode_at': 1794882920, 'chat_thread': None, 'kind_label': '4xx spike (403/401/404…)', 'group': 'Errors', 'age_s': 379950, 'stale_s': 3646}
Check it yourself
Every link carries the finding's own time window, so it shows tomorrow what it shows now.
Actions
Sign in (IAP) to acknowledge, snooze or resolve.
Where the numbers come from
| source | rollup |
| collected_at | 1794882595 |
| age_seconds | 333 |
| stale | False |
| shown | None |
| total | None |
| truncated | False |
| coverage_from | None |
| retain_days | None |
| note | None |
| detector | endpoint_4xx_spike |