docsvc on master has served nothing for 1.7 h; at this hour it normally serves about 144 calls/hour
warning
Service gone silent
master
resolved
impact 39
Owner: Prasad Limaye (backup: Harish Kumar)
opened 76d ago · last observed 2d ago · observed 108× · #650 · 91510f161fff0411bcac
What happened template
docsvc on master has served no API calls for 100 minutes (last call 2026-11-16 17:00 UTC); at this hour of the week its baseline — 4 weeks of history — expects about 144 calls an hour, 72 in the window checked. It carried 10.1% of the cluster's calls over the last week. A deploy landed just before the silence began — see the facts. Silence is not measured downtime: the pods may be up and unused, or crashed, or cut off upstream — Reliability and Pods say which. It resolves on its own once the service serves again.
A service that normally serves requests at this hour has served none for the sustain window — down, or cut off.
The facts
| service | docsvc |
| silent_minutes | 100 |
| silent_since | 1,794,848,700 |
| last_call_at | 1,794,848,400 |
| expected_calls_in_window | 72.00 |
| expected_per_hour | 144.00 |
| baseline_weeks | 4 |
| buckets_checked | 6 |
| share_of_api_calls_7d | 0.10 |
| discriminator | |
| window | 2026-11-16 18:15:00 UTC → 2026-11-16 18:45:00 UTC |
Adjacent deploys — adjacency, not proof
| automated | docsvc · 57 min before |
What is failing in docsvc — the bad events, decomposed
“Bad events” are HTTP 5xx answers from docsvc. In this finding's window (2026-11-16 18:15:00 UTC → 2026-11-16 18:45:00 UTC) there were 0; every line links to the grouped errors behind it.
The error shapes behind them
| 23× | Request completed — 500 · could not update document metadata for doc id 0211da3b-b92d-4f58-a742-62624434f054: could not … since 2026-09-30 16:47:17 UTC |
| 7× | Request completed — 500 · could not update document metadata for doc id 5e8b6a30-080e-4d35-9878-e57e75108c3d: could not … since 2026-10-08 19:24:13 UTC |
| 7× | could not update document metadata for doc id b59d0e97-516c-41b7-a96b-2fc113fc1bbe: could not update the document: Eithe… since 2026-11-13 04:59:59 UTC |
| 3× | could not update document metadata for doc id 3b6cc256-1e7a-4e3d-af36-34264f3800fc: could not update document for catego… since 2026-11-13 04:59:57 UTC |
| 1× | message was never published because transaction was never committed since 2026-11-17 08:36:31 UTC |
The story — No docsvc deploy or hand change is recorded in the hour before this opened. Everything below is docsvc only.
- 2026-09-03 05:20:10 UTCthis finding docsvc on master has served nothing for 1.7 h; at this hour it normally serves about 144 calls/hour
- 2026-10-20 11:52:56 UTCfinding · critical docsvc POST /api/documents/folders: 8 of 8 calls failed (100 %) in the last 15 min
- 2026-10-23 14:04:30 UTCfinding · info docsvc p95 1.69 s vs a normal 494 ms ± 213 ms for Mon 17:00 UTC (3 of the last 4 checks)
- 2026-11-14 15:17:18 UTCdeploy automated deployed docsvc
- 2026-11-15 00:59:17 UTCdeploy automated deployed docsvc
- 2026-11-15 01:27:34 UTCdeploy automated deployed docsvc
- 2026-11-16 00:00:00 UTCerrors docsvc 5xx peaked at 12 in one five-minute bucket (median 0 in this window)
- 2026-11-16 00:00:00 UTClatency docsvc p95 peaked at 1.58 s
- 2026-11-16 01:17:48 UTCdeploy automated deployed docsvc
- 2026-11-16 01:27:36 UTCdeploy automated deployed docsvc
- 2026-11-16 05:19:11 UTCdeploy automated deployed docsvc
- 2026-11-16 07:03:21 UTCdeploy automated deployed docsvc
- 2026-11-16 07:33:37 UTCdeploy automated deployed docsvc
- 2026-11-16 08:03:15 UTCdeploy automated deployed docsvc
- 2026-11-16 08:05:38 UTCdeploy automated deployed docsvc
- 2026-11-16 11:15:14 UTCdeploy automated deployed docsvc
- 2026-11-16 11:29:29 UTCdeploy automated deployed docsvc
- 2026-11-16 14:24:26 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/docsvc by hand (kubectl/v1.35.1 (darwin/arm64) kubernetes/8fea90b,gzip(gfe))
- 2026-11-16 14:58:58 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/docsvc by hand (kubectl/v1.35.1 (darwin/arm64) kubernetes/8fea90b,gzip(gfe))
- 2026-11-16 15:39:37 UTCdeploy automated deployed docsvc
- 2026-11-16 16:07:42 UTCdeploy automated deployed docsvc
- 2026-11-16 18:21:41 UTCdeploy automated deployed docsvc
History
- 2026-11-16 18:32:09 UTCreopened x-ray · severity=warning was=warning title=docsvc on master has served nothing for 1.3 h; at this hour it normally serves about 129 calls/hour level=medium was_level=medium finding={'id': 650, 'fingerprint': '91510f161fff0411bcac', 'kind': 'service.silent', 'severity': 'warning', 'cluster': 'master', 'subject': 'docsvc', 'title': 'docsvc on master has served nothing for 1.3 h; at this hour it normally serves about 129 calls/hour', 'facts': {'service': 'docsvc', 'silent_minutes': 80, 'silent_since': 1794848700, 'last_call_at': 1794848400, 'expected_calls_in_window': 64.5, 'expected_per_hour': 129.0, 'baseline_weeks': 4, 'buckets_checked': 6, 'share_of_api_calls_7d': 0.1017, 'recent_deploys': [{'who': 'automated', 'resource': 'docsvc', 'image': None, 'revision': '34ce8df3df2b2274f56808cd961b860ac4434500', 'ts': 1794845262, 'minutes_before': 57}], 'window': [1794851700, 1794853500], 'discriminator': ''}, 'evidence': [{'label': 'docsvc overview', 'href': '/service/docsvc?cluster=master&range=24h'}, {'label': 'Reliability', 'href': '/reliability?cluster=master&range=24h'}, {'label': 'docsvc logs', 'href': '/logs?cluster=master&service=docsvc&range=24h'}], 'provenance': {'source': 'rollup', 'collected_at': 1794853851, 'age_seconds': 105, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 34.0, 'status': 'open', 'first_seen': 1788412810, 'last_seen': 1794853929, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 105, 'narrative': "docsvc on master has served no API calls for 75 minutes (last call 2026-11-09 17:00 UTC); at this hour of the week its baseline — 4 weeks of history — expects about 212 calls an hour, 106 in the window checked. It carried 7.5% of the cluster's calls over the last week. Silence is not measured downtime: the pods may be up and unused, or crashed, or cut off upstream — Reliability and Pods say which. It resolves on its own once the service serves again.", 'narrative_kind': 'template', 'discriminator': '', 'level': 'medium', 'told_level': None, 'episode_at': 1794853929, 'chat_thread': None, 'kind_label': 'Service gone silent', 'group': 'Reliability', 'age_s': 6441145, 'stale_s': 26}
- 2026-11-16 19:14:41 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 650, 'fingerprint': '91510f161fff0411bcac', 'kind': 'service.silent', 'severity': 'warning', 'cluster': 'master', 'subject': 'docsvc', 'title': 'docsvc on master has served nothing for 1.7 h; at this hour it normally serves about 144 calls/hour', 'facts': {'service': 'docsvc', 'silent_minutes': 100, 'silent_since': 1794848700, 'last_call_at': 1794848400, 'expected_calls_in_window': 72.0, 'expected_per_hour': 144.0, 'baseline_weeks': 4, 'buckets_checked': 6, 'share_of_api_calls_7d': 0.1006, 'recent_deploys': [{'who': 'automated', 'resource': 'docsvc', 'image': None, 'revision': '34ce8df3df2b2274f56808cd961b860ac4434500', 'ts': 1794845262, 'minutes_before': 57}], 'window': [1794852900, 1794854700], 'discriminator': ''}, 'evidence': [{'label': 'docsvc overview', 'href': '/service/docsvc?cluster=master&range=24h'}, {'label': 'Reliability', 'href': '/reliability?cluster=master&range=24h'}, {'label': 'docsvc logs', 'href': '/logs?cluster=master&service=docsvc&range=24h'}], 'provenance': {'source': 'rollup', 'collected_at': 1794855211, 'age_seconds': 115, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 39.0, 'status': 'resolved', 'first_seen': 1788412810, 'last_seen': 1794855299, 'resolved_at': 1794856481, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 108, 'narrative': "docsvc on master has served no API calls for 100 minutes (last call 2026-11-16 17:00 UTC); at this hour of the week its baseline — 4 weeks of history — expects about 144 calls an hour, 72 in the window checked. It carried 10.1% of the cluster's calls over the last week. A deploy landed just before the silence began — see the facts. Silence is not measured downtime: the pods may be up and unused, or crashed, or cut off upstream — Reliability and Pods say which. It resolves on its own once the service serves again.", 'narrative_kind': 'template', 'discriminator': '', 'level': 'medium', 'told_level': None, 'episode_at': 1794853929, 'chat_thread': None, 'kind_label': 'Service gone silent', 'group': 'Reliability', 'age_s': 6443699, 'stale_s': 1210}
Check it yourself
Every link carries the finding's own time window, so it shows tomorrow what it shows now.
Actions
Sign in (IAP) to acknowledge, snooze or resolve.
Where the numbers come from
| source | rollup |
| collected_at | 1794855211 |
| age_seconds | 115 |
| stale | False |
| shown | None |
| total | None |
| truncated | False |
| coverage_from | None |
| retain_days | None |
| note | None |
| detector | service_silent |