← findings
HorizontalPodAutoscaler/docproc-hpa (tilda) cannot read its scaling metric — 15× in 2 h
warning
Autoscaler blind
test27
resolved
impact 20
opened 27d ago · last observed 2d ago · observed 1071× · #5461 · ad33433138d283cb6ec4
What happened template
Autoscaler blind: HorizontalPodAutoscaler/docproc-hpa (tilda) cannot read its scaling metric — 15× in 2 h. Facts: namespace tilda, object HorizontalPodAutoscaler/docproc-hpa, failures_2h 15, last_message unable to get metric custom.googleapis.com|docproc_memory_demand_bytes: no metrics returned from custom metrics API.
A KEDA scaler or HPA that cannot read its metric — it will not scale when load changes.
The facts
| namespace | tilda |
| object | HorizontalPodAutoscaler/docproc-hpa |
| failures_2h | 15 |
| last_message | unable to get metric custom.googleapis.com|docproc_memory_demand_bytes: no metrics returned from custom metrics API |
| discriminator |
The story — No deploy or hand change is recorded in the hour before this opened.
- 2026-10-23 12:03:15 UTCcypher ontologysvc shape 00000000 p95 went 47 ms → 775 ms across the window (357 runs)
- 2026-10-23 13:03:15 UTCfinding · warning HorizontalPodAutoscaler/tmfsync-hpa (tilda) cannot read its scaling metric — 7× in 2 h
- 2026-10-23 13:03:15 UTCfinding · warning onboardingsvc is burning its error budget at 7.3× (1 h) / 7.4× (6 h) against a 99.00% objective
- 2026-10-23 13:03:15 UTCthis finding HorizontalPodAutoscaler/docproc-hpa (tilda) cannot read its scaling metric — 15× in 2 h
- 2026-10-23 13:12:51 UTCfinding · info docproc Neo4j load 87 q/min vs a normal 11 q/min ± 1 q/min for Mon 19:00 UTC (3 of the last 4 checks)
- 2026-10-23 13:33:46 UTCfinding · warning pubsub.dashboards-test27-dashsub: 5 failed runs in the last 6 h — SyncForAIRun: merge recommendations: MergeRecommendations: failed to fetch existing IDs: failed to q
- 2026-10-23 13:42:32 UTCfinding · info docproc: Neo4j queries per request 53.3 — 9.2× the fleet's usual 5.8
- 2026-10-23 13:42:32 UTCfinding · info enrollsvc: Neo4j queries per request 42.0 — 7.1× the fleet's usual 5.9 (also p95 latency (ms) 1.98 s)
- 2026-10-23 13:49:17 UTCfinding · info docsvc: Neo4j queries per request 107.5 — 18.6× the fleet's usual 5.8 (also p95 latency (ms) 2.77 s)
- 2026-10-23 13:49:17 UTCfinding · info execsvc: p95 latency (ms) 3.05 s — 9.7× the fleet's usual 315 ms
- 2026-11-14 12:00:44 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/studiosvc by hand (kubectl/v1.35.9 (darwin/arm64) kubernetes/d834de3,gzip(gfe))
- 2026-11-14 12:01:11 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/execsvc by hand (kubectl/v1.35.9 (darwin/arm64) kubernetes/d834de3,gzip(gfe))
- 2026-11-14 14:26:23 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/execsvc by hand (kubectl/v1.35.9 (darwin/arm64) kubernetes/d834de3,gzip(gfe))
- 2026-11-14 14:26:39 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/studiosvc by hand (kubectl/v1.35.9 (darwin/arm64) kubernetes/d834de3,gzip(gfe))
- 2026-11-14 17:58:01 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/docproc by hand (kubectl/v1.35.9 (darwin/arm64) kubernetes/d834de3,gzip(gfe))
- 2026-11-15 06:12:43 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/tmfsync by hand (kubectl/v1.34.1 (darwin/arm64) kubernetes/93248f9,gzip(gfe))
- 2026-11-15 06:40:00 UTCerrors cluster 5xx peaked at 34 in one five-minute bucket (median 0 in this window)
- 2026-11-15 06:52:31 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/tmfsync by hand (kubectl/v1.34.1 (darwin/arm64) kubernetes/93248f9,gzip(gfe))
- 2026-11-15 15:15:12 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/enrollsvc by hand (kubectl/v1.32.2 (darwin/amd64) kubernetes/67a30c0,gzip(gfe))
- 2026-11-15 15:15:14 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/chat-agent by hand (kubectl/v1.32.2 (darwin/amd64) kubernetes/67a30c0,gzip(gfe))
- 2026-11-15 16:03:03 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/studiosvc by hand (kubectl/v1.32.2 (darwin/amd64) kubernetes/67a30c0,gzip(gfe))
- 2026-11-15 16:32:04 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/chat-agent by hand (kubectl/v1.32.2 (darwin/amd64) kubernetes/67a30c0,gzip(gfe))
- 2026-11-15 16:35:21 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/chat-agent by hand (kubectl/v1.32.2 (darwin/amd64) kubernetes/67a30c0,gzip(gfe))
- 2026-11-15 17:14:46 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/chat-agent by hand (kubectl/v1.32.2 (darwin/amd64) kubernetes/67a30c0,gzip(gfe))
- 2026-11-15 17:25:50 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/chat-agent by hand (kubectl/v1.32.2 (darwin/amd64) kubernetes/67a30c0,gzip(gfe))
- 2026-11-15 17:31:26 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/chat-agent by hand (kubectl/v1.32.2 (darwin/amd64) kubernetes/67a30c0,gzip(gfe))
- 2026-11-15 19:09:20 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/execsvc by hand (kubectl/v1.35.9 (darwin/arm64) kubernetes/d834de3,gzip(gfe))
- 2026-11-15 19:16:56 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/execsvc by hand (kubectl/v1.35.9 (darwin/arm64) kubernetes/d834de3,gzip(gfe))
- 2026-11-15 19:17:23 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/execsvc by hand (kubectl/v1.35.9 (darwin/arm64) kubernetes/d834de3,gzip(gfe))
- 2026-11-15 20:26:13 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/chat-agent by hand (kubectl/v1.32.2 (darwin/amd64) kubernetes/67a30c0,gzip(gfe))
- 2026-11-15 20:39:53 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/chat-agent by hand (kubectl/v1.32.2 (darwin/amd64) kubernetes/67a30c0,gzip(gfe))
- 2026-11-15 21:16:01 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/execsvc by hand (kubectl/v1.35.9 (darwin/arm64) kubernetes/d834de3,gzip(gfe))
- 2026-11-15 22:15:52 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/execsvc by hand (kubectl/v1.32.2 (darwin/amd64) kubernetes/67a30c0,gzip(gfe))
- 2026-11-15 22:15:53 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/chat-agent by hand (kubectl/v1.32.2 (darwin/amd64) kubernetes/67a30c0,gzip(gfe))
- 2026-11-16 01:14:19 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/chat-agent by hand (kubectl/v1.32.2 (darwin/amd64) kubernetes/67a30c0,gzip(gfe))
- 2026-11-16 04:26:06 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/docsvc by hand (kubectl/v1.35.1 (darwin/arm64) kubernetes/8fea90b,gzip(gfe))
- 2026-11-16 05:10:12 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/onboardingsvc by hand (kubectl/v1.35.1 (darwin/arm64) kubernetes/8fea90b,gzip(gfe))
- 2026-11-16 06:42:17 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/tmfsync by hand (kubectl/v1.34.1 (darwin/arm64) kubernetes/93248f9,gzip(gfe))
- 2026-11-16 06:54:26 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/tmfsync by hand (kubectl/v1.34.1 (darwin/arm64) kubernetes/93248f9,gzip(gfe))
- 2026-11-16 06:55:52 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/onboardingsvc by hand (kubectl/v1.35.1 (darwin/arm64) kubernetes/8fea90b,gzip(gfe))
- 2026-11-16 07:02:15 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/tmfsync by hand (kubectl/v1.34.1 (darwin/arm64) kubernetes/93248f9,gzip(gfe))
- 2026-11-16 07:10:55 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/docsvc by hand (kubectl/v1.34.1 (darwin/arm64) kubernetes/93248f9,gzip(gfe))
- 2026-11-16 07:15:20 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/docsvc by hand (kubectl/v1.34.1 (darwin/arm64) kubernetes/93248f9,gzip(gfe))
- 2026-11-16 07:50:00 UTClatency cluster p95 peaked at 1.5 min
- 2026-11-16 08:20:06 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/chat-agent by hand (kubectl/v1.36.4 (darwin/arm64) kubernetes/bb826b1,gzip(gfe))
- 2026-11-16 08:30:42 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/chat-agent by hand (kubectl/v1.36.4 (darwin/arm64) kubernetes/bb826b1,gzip(gfe))
- 2026-11-16 10:14:14 UTChand change [email protected] changed core/v1/namespaces/tilda/secrets/sheet-edit-svc-syncfusion by hand (kubectl/v1.36.4 (darwin/arm64) kubernetes/bb826b1,gzip(gfe))
- 2026-11-16 10:14:59 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/chat-agent by hand (kubectl/v1.36.4 (darwin/arm64) kubernetes/bb826b1,gzip(gfe))
- 2026-11-16 10:15:07 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/chat-agent by hand (kubectl/v1.36.4 (darwin/arm64) kubernetes/bb826b1,gzip(gfe))
- 2026-11-16 10:16:16 UTChand change [email protected] changed core/v1/namespaces/tilda/services/sheet-edit-svc by hand (kubectl/v1.36.4 (darwin/arm64) kubernetes/bb826b1,gzip(gfe))
- 2026-11-16 10:16:19 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/sheet-edit-svc by hand (kubectl/v1.36.4 (darwin/arm64) kubernetes/bb826b1,gzip(gfe))
- 2026-11-16 10:18:07 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/docsvc by hand (kubectl/v1.35.1 (darwin/arm64) kubernetes/8fea90b,gzip(gfe))
- 2026-11-16 10:23:54 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/chat-agent by hand (kubectl/v1.36.4 (darwin/arm64) kubernetes/bb826b1,gzip(gfe))
- 2026-11-16 10:24:04 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/chat-agent by hand (kubectl/v1.36.4 (darwin/arm64) kubernetes/bb826b1,gzip(gfe))
- 2026-11-16 10:25:00 UTChand change [email protected] changed core/v1/namespaces/tilda/services/sheet-edit-svc by hand (kubectl/v1.36.4 (darwin/arm64) kubernetes/bb826b1,gzip(gfe))
- 2026-11-16 10:25:00 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/sheet-edit-svc by hand (kubectl/v1.36.4 (darwin/arm64) kubernetes/bb826b1,gzip(gfe))
- 2026-11-16 10:25:07 UTChand change [email protected] changed core/v1/namespaces/tilda/secrets/sheet-edit-svc-syncfusion by hand (kubectl/v1.36.4 (darwin/arm64) kubernetes/bb826b1,gzip(gfe))
- 2026-11-16 15:00:56 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/chat-agent by hand (kubectl/v1.32.2 (darwin/amd64) kubernetes/67a30c0,gzip(gfe))
- 2026-11-16 15:18:43 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/virtual-assistant by hand (kubectl/v1.33.4 (darwin/arm64) kubernetes/74cdb42,gzip(gfe))
- 2026-11-16 15:27:04 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/chat-agent by hand (kubectl/v1.32.2 (darwin/amd64) kubernetes/67a30c0,gzip(gfe))
- 2026-11-16 16:28:29 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/studiosvc by hand (kubectl/v1.33.3 (darwin/amd64) kubernetes/80779bd,gzip(gfe))
- 2026-11-16 16:38:14 UTChand change [email protected] changed core/v1/namespaces/tilda/secrets/sheet-edit-svc-syncfusion by hand (kubectl/v1.36.4 (darwin/arm64) kubernetes/bb826b1,gzip(gfe))
- 2026-11-16 16:38:27 UTChand change [email protected] changed core/v1/namespaces/tilda/services/sheet-edit-svc by hand (kubectl/v1.36.4 (darwin/arm64) kubernetes/bb826b1,gzip(gfe))
- 2026-11-16 16:38:30 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/sheet-edit-svc by hand (kubectl/v1.36.4 (darwin/arm64) kubernetes/bb826b1,gzip(gfe))
- 2026-11-16 16:41:05 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/tmfsync by hand (kubectl/v1.34.1 (darwin/arm64) kubernetes/93248f9,gzip(gfe))
- 2026-11-16 16:41:18 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/chat-agent by hand (kubectl/v1.36.4 (darwin/arm64) kubernetes/bb826b1,gzip(gfe))
- 2026-11-16 16:41:30 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/chat-agent by hand (kubectl/v1.36.4 (darwin/arm64) kubernetes/bb826b1,gzip(gfe))
- 2026-11-16 16:46:16 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/docsvc by hand (kubectl/v1.34.1 (darwin/arm64) kubernetes/93248f9,gzip(gfe))
- 2026-11-16 16:51:21 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/chat-agent by hand (kubectl/v1.36.4 (darwin/arm64) kubernetes/bb826b1,gzip(gfe))
- 2026-11-16 16:51:32 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/chat-agent by hand (kubectl/v1.36.4 (darwin/arm64) kubernetes/bb826b1,gzip(gfe))
- 2026-11-16 16:51:51 UTChand change [email protected] changed core/v1/namespaces/tilda/services/sheet-edit-svc by hand (kubectl/v1.36.4 (darwin/arm64) kubernetes/bb826b1,gzip(gfe))
- 2026-11-16 16:51:51 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/sheet-edit-svc by hand (kubectl/v1.36.4 (darwin/arm64) kubernetes/bb826b1,gzip(gfe))
- 2026-11-16 16:51:59 UTChand change [email protected] changed core/v1/namespaces/tilda/secrets/sheet-edit-svc-syncfusion by hand (kubectl/v1.36.4 (darwin/arm64) kubernetes/bb826b1,gzip(gfe))
- 2026-11-16 17:01:20 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/tmfsync by hand (kubectl/v1.34.1 (darwin/arm64) kubernetes/93248f9,gzip(gfe))
- 2026-11-16 17:53:39 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/studiosvc by hand (kubectl/v1.33.3 (darwin/amd64) kubernetes/80779bd,gzip(gfe))
- 2026-11-16 18:04:21 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/studiosvc by hand (kubectl/v1.32.2 (darwin/amd64) kubernetes/67a30c0,gzip(gfe))
- 2026-11-16 18:30:32 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/docsvc by hand (kubectl/v1.30.5 (darwin/arm64) kubernetes/633f2e9,gzip(gfe))
- 2026-11-16 18:30:33 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/docsvc by hand (kubectl/v1.30.5 (darwin/arm64) kubernetes/633f2e9,gzip(gfe))
- 2026-11-16 18:30:34 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/docproc by hand (kubectl/v1.30.5 (darwin/arm64) kubernetes/633f2e9,gzip(gfe))
- 2026-11-16 18:30:36 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/docproc by hand (kubectl/v1.30.5 (darwin/arm64) kubernetes/633f2e9,gzip(gfe))
- 2026-11-16 18:40:42 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/tmfsync by hand (kubectl/v1.34.1 (darwin/arm64) kubernetes/93248f9,gzip(gfe))
- 2026-11-16 18:41:58 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/docproc by hand (kubectl/v1.30.5 (darwin/arm64) kubernetes/633f2e9,gzip(gfe))
- 2026-11-16 18:48:43 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/studiosvc by hand (kubectl/v1.33.3 (darwin/amd64) kubernetes/80779bd,gzip(gfe))
- 2026-11-16 18:53:59 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/tmfsync by hand (kubectl/v1.34.1 (darwin/arm64) kubernetes/93248f9,gzip(gfe))
- 2026-11-16 18:54:20 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/docproc by hand (kubectl/v1.30.5 (darwin/arm64) kubernetes/633f2e9,gzip(gfe))
- 2026-11-16 18:54:22 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/docproc by hand (kubectl/v1.30.5 (darwin/arm64) kubernetes/633f2e9,gzip(gfe))
- 2026-11-16 18:54:23 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/docsvc by hand (kubectl/v1.30.5 (darwin/arm64) kubernetes/633f2e9,gzip(gfe))
- 2026-11-16 18:54:24 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/docsvc by hand (kubectl/v1.30.5 (darwin/arm64) kubernetes/633f2e9,gzip(gfe))
- 2026-11-16 18:58:41 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/tmfsync by hand (kubectl/v1.34.1 (darwin/arm64) kubernetes/93248f9,gzip(gfe))
- 2026-11-16 19:09:53 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/studiosvc by hand (kubectl/v1.33.3 (darwin/amd64) kubernetes/80779bd,gzip(gfe))
- 2026-11-16 19:30:41 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/studiosvc by hand (kubectl/v1.33.3 (darwin/amd64) kubernetes/80779bd,gzip(gfe))
- 2026-11-16 19:53:58 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/studiosvc by hand (kubectl/v1.33.3 (darwin/amd64) kubernetes/80779bd,gzip(gfe))
- 2026-11-16 20:24:14 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/tmfsync by hand (kubectl/v1.34.1 (darwin/arm64) kubernetes/93248f9,gzip(gfe))
- 2026-11-16 20:37:25 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/tmfsync by hand (kubectl/v1.34.1 (darwin/arm64) kubernetes/93248f9,gzip(gfe))
- 2026-11-16 20:50:35 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/tmfsync by hand (kubectl/v1.34.1 (darwin/arm64) kubernetes/93248f9,gzip(gfe))
- 2026-11-16 21:30:24 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/docsvc by hand (kubectl/v1.32.2 (darwin/amd64) kubernetes/67a30c0,gzip(gfe))
- 2026-11-16 21:30:25 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/workplansvc by hand (kubectl/v1.32.2 (darwin/amd64) kubernetes/67a30c0,gzip(gfe))
- 2026-11-16 21:30:27 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/docsvc by hand (kubectl/v1.32.2 (darwin/amd64) kubernetes/67a30c0,gzip(gfe))
- 2026-11-16 21:30:27 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/workplansvc by hand (kubectl/v1.32.2 (darwin/amd64) kubernetes/67a30c0,gzip(gfe))
- 2026-11-16 21:31:28 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/tmfsync by hand (kubectl/v1.34.1 (darwin/arm64) kubernetes/93248f9,gzip(gfe))
- 2026-11-16 21:35:58 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/tmfsync by hand (kubectl/v1.34.1 (darwin/arm64) kubernetes/93248f9,gzip(gfe))
- 2026-11-16 21:46:25 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/tmfsync by hand (kubectl/v1.34.1 (darwin/arm64) kubernetes/93248f9,gzip(gfe))
- 2026-11-16 22:14:55 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/tmfsync by hand (kubectl/v1.34.1 (darwin/arm64) kubernetes/93248f9,gzip(gfe))
- 2026-11-16 22:33:12 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/docproc by hand (kubectl/v1.30.5 (darwin/arm64) kubernetes/633f2e9,gzip(gfe))
- 2026-11-16 22:38:11 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/docproc by hand (kubectl/v1.30.5 (darwin/arm64) kubernetes/633f2e9,gzip(gfe))
- 2026-11-16 22:41:00 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/tmfsync by hand (kubectl/v1.34.1 (darwin/arm64) kubernetes/93248f9,gzip(gfe))
- 2026-11-16 22:42:47 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/docproc by hand (kubectl/v1.30.5 (darwin/arm64) kubernetes/633f2e9,gzip(gfe))
History
- 2026-11-14 18:01:23 UTCreopened x-ray · severity=warning was=warning title=HorizontalPodAutoscaler/docproc-hpa (tilda) cannot read its scaling metric — 9× in 2 h level=medium was_level=medium finding={'id': 5461, 'fingerprint': 'ad33433138d283cb6ec4', 'kind': 'scaler.failing', 'severity': 'warning', 'cluster': 'test27', 'subject': 'tilda/HorizontalPodAutoscaler/docproc-hpa', 'title': 'HorizontalPodAutoscaler/docproc-hpa (tilda) cannot read its scaling metric — 9× in 2 h', 'facts': {'namespace': 'tilda', 'object': 'HorizontalPodAutoscaler/docproc-hpa', 'failures_2h': 9, 'last_message': 'unable to get metric custom.googleapis.com|docproc_memory_demand_bytes: no metrics returned from custom metrics API', 'discriminator': ''}, 'evidence': [{'label': 'Kubernetes events', 'href': '/logs?cluster=test27&from=1794675683&to=1794679283&source=events&q=HorizontalPodAutoscaler%2Fdocproc-hpa'}, {'label': 'Pods', 'href': '/pods?cluster=test27'}], 'provenance': {'source': 'x-ray', 'note': 'Kubernetes events (logs/events in BigQuery), last 45 min'}, 'impact': 14.9, 'status': 'open', 'first_seen': 1792760595, 'last_seen': 1794679283, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 1026, 'narrative': 'Autoscaler blind: HorizontalPodAutoscaler/docproc-hpa (tilda) cannot read its scaling metric — 10× in 2 h. Facts: namespace tilda, object HorizontalPodAutoscaler/docproc-hpa, failures_2h 10, last_message No recommendation.', 'narrative_kind': 'template', 'discriminator': '', 'level': 'medium', 'told_level': None, 'episode_at': 1794679283, 'chat_thread': None, 'kind_label': 'Autoscaler blind', 'group': 'Platform', 'age_s': 1918705, 'stale_s': 17}
- 2026-11-14 20:13:09 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 5461, 'fingerprint': 'ad33433138d283cb6ec4', 'kind': 'scaler.failing', 'severity': 'warning', 'cluster': 'test27', 'subject': 'tilda/HorizontalPodAutoscaler/docproc-hpa', 'title': 'HorizontalPodAutoscaler/docproc-hpa (tilda) cannot read its scaling metric — 9× in 2 h', 'facts': {'namespace': 'tilda', 'object': 'HorizontalPodAutoscaler/docproc-hpa', 'failures_2h': 9, 'last_message': 'unable to get metric custom.googleapis.com|docproc_memory_demand_bytes: no metrics returned from custom metrics API', 'discriminator': ''}, 'evidence': [{'label': 'Kubernetes events', 'href': '/logs?cluster=test27&from=1794682393&to=1794685993&source=events&q=HorizontalPodAutoscaler%2Fdocproc-hpa'}, {'label': 'Pods', 'href': '/pods?cluster=test27'}], 'provenance': {'source': 'x-ray', 'note': 'Kubernetes events (logs/events in BigQuery), last 45 min'}, 'impact': 19.9, 'status': 'resolved', 'first_seen': 1792760595, 'last_seen': 1794685993, 'resolved_at': 1794687189, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 1040, 'narrative': 'Autoscaler blind: HorizontalPodAutoscaler/docproc-hpa (tilda) cannot read its scaling metric — 9× in 2 h. Facts: namespace tilda, object HorizontalPodAutoscaler/docproc-hpa, failures_2h 9, last_message unable to get metric custom.googleapis.com|docproc_memory_demand_bytes: no metrics returned from custom metrics API.', 'narrative_kind': 'template', 'discriminator': '', 'level': 'medium', 'told_level': None, 'episode_at': 1794679283, 'chat_thread': None, 'kind_label': 'Autoscaler blind', 'group': 'Platform', 'age_s': 1926634, 'stale_s': 1236}
- 2026-11-16 18:32:09 UTCreopened x-ray · severity=warning was=warning title=HorizontalPodAutoscaler/docproc-hpa (tilda) cannot read its scaling metric — 10× in 2 h level=medium was_level=medium finding={'id': 5461, 'fingerprint': 'ad33433138d283cb6ec4', 'kind': 'scaler.failing', 'severity': 'warning', 'cluster': 'test27', 'subject': 'tilda/HorizontalPodAutoscaler/docproc-hpa', 'title': 'HorizontalPodAutoscaler/docproc-hpa (tilda) cannot read its scaling metric — 10× in 2 h', 'facts': {'namespace': 'tilda', 'object': 'HorizontalPodAutoscaler/docproc-hpa', 'failures_2h': 10, 'last_message': 'unable to get metric custom.googleapis.com|docproc_memory_demand_bytes: no metrics returned from custom metrics API', 'discriminator': ''}, 'evidence': [{'label': 'Kubernetes events', 'href': '/logs?cluster=test27&from=1794850329&to=1794853929&source=events&q=HorizontalPodAutoscaler%2Fdocproc-hpa'}, {'label': 'Pods', 'href': '/pods?cluster=test27'}], 'provenance': {'source': 'x-ray', 'note': 'Kubernetes events (logs/events in BigQuery), last 45 min'}, 'impact': 14.9, 'status': 'open', 'first_seen': 1792760595, 'last_seen': 1794853929, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 1041, 'narrative': 'Autoscaler blind: HorizontalPodAutoscaler/docproc-hpa (tilda) cannot read its scaling metric — 9× in 2 h. Facts: namespace tilda, object HorizontalPodAutoscaler/docproc-hpa, failures_2h 9, last_message unable to get metric custom.googleapis.com|docproc_memory_demand_bytes: no metrics returned from custom metrics API.', 'narrative_kind': 'template', 'discriminator': '', 'level': 'medium', 'told_level': None, 'episode_at': 1794853929, 'chat_thread': None, 'kind_label': 'Autoscaler blind', 'group': 'Platform', 'age_s': 2093359, 'stale_s': 25}
- 2026-11-16 21:08:58 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 5461, 'fingerprint': 'ad33433138d283cb6ec4', 'kind': 'scaler.failing', 'severity': 'warning', 'cluster': 'test27', 'subject': 'tilda/HorizontalPodAutoscaler/docproc-hpa', 'title': 'HorizontalPodAutoscaler/docproc-hpa (tilda) cannot read its scaling metric — 11× in 2 h', 'facts': {'namespace': 'tilda', 'object': 'HorizontalPodAutoscaler/docproc-hpa', 'failures_2h': 11, 'last_message': 'unable to get metric custom.googleapis.com|docproc_memory_demand_bytes: no metrics returned from custom metrics API', 'discriminator': ''}, 'evidence': [{'label': 'Kubernetes events', 'href': '/logs?cluster=test27&from=1794858836&to=1794862436&source=events&q=HorizontalPodAutoscaler%2Fdocproc-hpa'}, {'label': 'Pods', 'href': '/pods?cluster=test27'}], 'provenance': {'source': 'x-ray', 'note': 'Kubernetes events (logs/events in BigQuery), last 45 min'}, 'impact': 19.9, 'status': 'resolved', 'first_seen': 1792760595, 'last_seen': 1794862436, 'resolved_at': 1794863338, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 1055, 'narrative': 'Autoscaler blind: HorizontalPodAutoscaler/docproc-hpa (tilda) cannot read its scaling metric — 11× in 2 h. Facts: namespace tilda, object HorizontalPodAutoscaler/docproc-hpa, failures_2h 11, last_message unable to get metric custom.googleapis.com|docproc_memory_demand_bytes: no metrics returned from custom metrics API.', 'narrative_kind': 'template', 'discriminator': '', 'level': 'medium', 'told_level': None, 'episode_at': 1794853929, 'chat_thread': None, 'kind_label': 'Autoscaler blind', 'group': 'Platform', 'age_s': 2102766, 'stale_s': 925}
- 2026-11-16 22:41:29 UTCreopened x-ray · severity=warning was=warning title=HorizontalPodAutoscaler/docproc-hpa (tilda) cannot read its scaling metric — 20× in 2 h level=medium was_level=medium finding={'id': 5461, 'fingerprint': 'ad33433138d283cb6ec4', 'kind': 'scaler.failing', 'severity': 'warning', 'cluster': 'test27', 'subject': 'tilda/HorizontalPodAutoscaler/docproc-hpa', 'title': 'HorizontalPodAutoscaler/docproc-hpa (tilda) cannot read its scaling metric — 20× in 2 h', 'facts': {'namespace': 'tilda', 'object': 'HorizontalPodAutoscaler/docproc-hpa', 'failures_2h': 20, 'last_message': 'unable to get metric custom.googleapis.com|docproc_memory_demand_bytes: no metrics returned from custom metrics API', 'discriminator': ''}, 'evidence': [{'label': 'Kubernetes events', 'href': '/logs?cluster=test27&from=1794865289&to=1794868889&source=events&q=HorizontalPodAutoscaler%2Fdocproc-hpa'}, {'label': 'Pods', 'href': '/pods?cluster=test27'}], 'provenance': {'source': 'x-ray', 'note': 'Kubernetes events (logs/events in BigQuery), last 45 min'}, 'impact': 14.9, 'status': 'open', 'first_seen': 1792760595, 'last_seen': 1794868889, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 1056, 'narrative': 'Autoscaler blind: HorizontalPodAutoscaler/docproc-hpa (tilda) cannot read its scaling metric — 11× in 2 h. Facts: namespace tilda, object HorizontalPodAutoscaler/docproc-hpa, failures_2h 11, last_message unable to get metric custom.googleapis.com|docproc_memory_demand_bytes: no metrics returned from custom metrics API.', 'narrative_kind': 'template', 'discriminator': '', 'level': 'medium', 'told_level': None, 'episode_at': 1794868889, 'chat_thread': None, 'kind_label': 'Autoscaler blind', 'group': 'Platform', 'age_s': 2108323, 'stale_s': 29}
- 2026-11-17 00:58:16 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 5461, 'fingerprint': 'ad33433138d283cb6ec4', 'kind': 'scaler.failing', 'severity': 'warning', 'cluster': 'test27', 'subject': 'tilda/HorizontalPodAutoscaler/docproc-hpa', 'title': 'HorizontalPodAutoscaler/docproc-hpa (tilda) cannot read its scaling metric — 15× in 2 h', 'facts': {'namespace': 'tilda', 'object': 'HorizontalPodAutoscaler/docproc-hpa', 'failures_2h': 15, 'last_message': 'unable to get metric custom.googleapis.com|docproc_memory_demand_bytes: no metrics returned from custom metrics API', 'discriminator': ''}, 'evidence': [{'label': 'Kubernetes events', 'href': '/logs?cluster=test27&from=1794872242&to=1794875842&source=events&q=HorizontalPodAutoscaler%2Fdocproc-hpa'}, {'label': 'Pods', 'href': '/pods?cluster=test27'}], 'provenance': {'source': 'x-ray', 'note': 'Kubernetes events (logs/events in BigQuery), last 45 min'}, 'impact': 19.9, 'status': 'resolved', 'first_seen': 1792760595, 'last_seen': 1794875842, 'resolved_at': 1794877096, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 1071, 'narrative': 'Autoscaler blind: HorizontalPodAutoscaler/docproc-hpa (tilda) cannot read its scaling metric — 15× in 2 h. Facts: namespace tilda, object HorizontalPodAutoscaler/docproc-hpa, failures_2h 15, last_message unable to get metric custom.googleapis.com|docproc_memory_demand_bytes: no metrics returned from custom metrics API.', 'narrative_kind': 'template', 'discriminator': '', 'level': 'medium', 'told_level': None, 'episode_at': 1794868889, 'chat_thread': None, 'kind_label': 'Autoscaler blind', 'group': 'Platform', 'age_s': 2116528, 'stale_s': 1281}
Check it yourself
Every link carries the finding's own time window, so it shows tomorrow what it shows now.
Actions
Sign in (IAP) to acknowledge, snooze or resolve.
Where the numbers come from
| source | x-ray |
| note | Kubernetes events (logs/events in BigQuery), last 45 min |
| detector | k8s_events |