pubsub.dashboards-test27-dashsub: 5 failed runs in the last 6 h — SyncForAIRun: merge recommendations: MergeRecommendations: failed to fetch existing IDs: failed to q
warning
Background job failing
test27
resolved
impact 25
opened 27d ago · last observed 2d ago · observed 63× · #5488 · e9118750daae5f5f9020
What happened template
The background job pubsub.dashboards-test27-dashsub on test27 failed 5 runs recently (1,143 failure lines; lines within one run count once), detected via its ERROR-severity log lines. A job that logs its failures below ERROR severity — a WARNING carrying an error field — is invisible to the novel-shape detector and to the page's own failed count; this detector reads the error field itself, which is why the finding now appears. The sample failure is in the facts.
A named background job whose runs keep failing (error-severity bursts) in the last hour — volume, not novelty; the novel-shape detector only fires the first time.
The facts
| job | pubsub.dashboards-test27-dashsub |
| service | dashrsvc |
| failed_runs | 5 |
| error_lines | 1,143 |
| sample | SyncForAIRun: merge recommendations: MergeRecommendations: failed to fetch existing IDs: failed to query active recommendation IDs for document 6d376f15-837e-472b-bf6d-7b213b51aa11: googleapi: Error 403: Access Denied: Project tilda-trial-dev: User does not have bigquery.jobs.create permission in pr |
| error_signature | SyncForAIRun: merge recommendations: MergeRecommendations: failed to fetch existing IDs: failed to query active recommen |
| failing_at | pubsublib/googlepubsub.go:697 |
| trace | projects/tilda-trial-dev/traces/d2ee718db080ebca49cb97740684ac12 |
| detected_via | ERROR-severity log lines |
| note | a run is a burst of lines that truly look like failures — an error/exception field, an error-worded message, or a failure status; lines within one burst count as one run. Lines a job dumps at ERROR that are neither (the steps that SUCCEEDED, banner rules) are ignored, so a service that logs its whole context on failure is not miscounted. `failing_at` is the call site those failures came from — for a coarse job (one set_job wrapping a pipeline) more specific than the job name. |
| discriminator | |
| window | 2026-11-16 17:44:07 UTC → 2026-11-16 23:44:07 UTC |
The story — No deploy or hand change is recorded in the hour before this opened.
- 2026-10-23 12:33:46 UTCcypher ontologysvc shape 00000000 p95 went 47 ms → 775 ms across the window (357 runs)
- 2026-10-23 13:03:15 UTCfinding · warning HorizontalPodAutoscaler/tmfsync-hpa (tilda) cannot read its scaling metric — 7× in 2 h
- 2026-10-23 13:03:15 UTCfinding · warning HorizontalPodAutoscaler/docproc-hpa (tilda) cannot read its scaling metric — 15× in 2 h
- 2026-10-23 13:03:15 UTCfinding · warning onboardingsvc is burning its error budget at 7.3× (1 h) / 7.4× (6 h) against a 99.00% objective
- 2026-10-23 13:12:51 UTCfinding · info docproc Neo4j load 87 q/min vs a normal 11 q/min ± 1 q/min for Mon 19:00 UTC (3 of the last 4 checks)
- 2026-10-23 13:33:46 UTCthis finding pubsub.dashboards-test27-dashsub: 5 failed runs in the last 6 h — SyncForAIRun: merge recommendations: MergeRecommendations: failed to fetch existing IDs: failed to q
- 2026-10-23 13:42:32 UTCfinding · info docproc: Neo4j queries per request 53.3 — 9.2× the fleet's usual 5.8
- 2026-10-23 13:42:32 UTCfinding · info enrollsvc: Neo4j queries per request 42.0 — 7.1× the fleet's usual 5.9 (also p95 latency (ms) 1.98 s)
- 2026-10-23 13:49:17 UTCfinding · info docsvc: Neo4j queries per request 107.5 — 18.6× the fleet's usual 5.8 (also p95 latency (ms) 2.77 s)
- 2026-10-23 13:49:17 UTCfinding · info execsvc: p95 latency (ms) 3.05 s — 9.7× the fleet's usual 315 ms
- 2026-11-14 12:00:44 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/studiosvc by hand (kubectl/v1.35.9 (darwin/arm64) kubernetes/d834de3,gzip(gfe))
- 2026-11-14 12:01:11 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/execsvc by hand (kubectl/v1.35.9 (darwin/arm64) kubernetes/d834de3,gzip(gfe))
- 2026-11-14 14:26:23 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/execsvc by hand (kubectl/v1.35.9 (darwin/arm64) kubernetes/d834de3,gzip(gfe))
- 2026-11-14 14:26:39 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/studiosvc by hand (kubectl/v1.35.9 (darwin/arm64) kubernetes/d834de3,gzip(gfe))
- 2026-11-14 17:58:01 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/docproc by hand (kubectl/v1.35.9 (darwin/arm64) kubernetes/d834de3,gzip(gfe))
- 2026-11-15 06:12:43 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/tmfsync by hand (kubectl/v1.34.1 (darwin/arm64) kubernetes/93248f9,gzip(gfe))
- 2026-11-15 06:40:00 UTCerrors cluster 5xx peaked at 34 in one five-minute bucket (median 0 in this window)
- 2026-11-15 06:52:31 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/tmfsync by hand (kubectl/v1.34.1 (darwin/arm64) kubernetes/93248f9,gzip(gfe))
- 2026-11-15 15:15:12 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/enrollsvc by hand (kubectl/v1.32.2 (darwin/amd64) kubernetes/67a30c0,gzip(gfe))
- 2026-11-15 15:15:14 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/chat-agent by hand (kubectl/v1.32.2 (darwin/amd64) kubernetes/67a30c0,gzip(gfe))
- 2026-11-15 16:03:03 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/studiosvc by hand (kubectl/v1.32.2 (darwin/amd64) kubernetes/67a30c0,gzip(gfe))
- 2026-11-15 16:32:04 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/chat-agent by hand (kubectl/v1.32.2 (darwin/amd64) kubernetes/67a30c0,gzip(gfe))
- 2026-11-15 16:35:21 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/chat-agent by hand (kubectl/v1.32.2 (darwin/amd64) kubernetes/67a30c0,gzip(gfe))
- 2026-11-15 17:14:46 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/chat-agent by hand (kubectl/v1.32.2 (darwin/amd64) kubernetes/67a30c0,gzip(gfe))
- 2026-11-15 17:25:50 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/chat-agent by hand (kubectl/v1.32.2 (darwin/amd64) kubernetes/67a30c0,gzip(gfe))
- 2026-11-15 17:31:26 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/chat-agent by hand (kubectl/v1.32.2 (darwin/amd64) kubernetes/67a30c0,gzip(gfe))
- 2026-11-15 19:09:20 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/execsvc by hand (kubectl/v1.35.9 (darwin/arm64) kubernetes/d834de3,gzip(gfe))
- 2026-11-15 19:16:56 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/execsvc by hand (kubectl/v1.35.9 (darwin/arm64) kubernetes/d834de3,gzip(gfe))
- 2026-11-15 19:17:23 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/execsvc by hand (kubectl/v1.35.9 (darwin/arm64) kubernetes/d834de3,gzip(gfe))
- 2026-11-15 20:26:13 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/chat-agent by hand (kubectl/v1.32.2 (darwin/amd64) kubernetes/67a30c0,gzip(gfe))
- 2026-11-15 20:39:53 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/chat-agent by hand (kubectl/v1.32.2 (darwin/amd64) kubernetes/67a30c0,gzip(gfe))
- 2026-11-15 21:16:01 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/execsvc by hand (kubectl/v1.35.9 (darwin/arm64) kubernetes/d834de3,gzip(gfe))
- 2026-11-15 22:15:52 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/execsvc by hand (kubectl/v1.32.2 (darwin/amd64) kubernetes/67a30c0,gzip(gfe))
- 2026-11-15 22:15:53 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/chat-agent by hand (kubectl/v1.32.2 (darwin/amd64) kubernetes/67a30c0,gzip(gfe))
- 2026-11-16 01:14:19 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/chat-agent by hand (kubectl/v1.32.2 (darwin/amd64) kubernetes/67a30c0,gzip(gfe))
- 2026-11-16 04:26:06 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/docsvc by hand (kubectl/v1.35.1 (darwin/arm64) kubernetes/8fea90b,gzip(gfe))
- 2026-11-16 05:10:12 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/onboardingsvc by hand (kubectl/v1.35.1 (darwin/arm64) kubernetes/8fea90b,gzip(gfe))
- 2026-11-16 06:42:17 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/tmfsync by hand (kubectl/v1.34.1 (darwin/arm64) kubernetes/93248f9,gzip(gfe))
- 2026-11-16 06:54:26 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/tmfsync by hand (kubectl/v1.34.1 (darwin/arm64) kubernetes/93248f9,gzip(gfe))
- 2026-11-16 06:55:52 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/onboardingsvc by hand (kubectl/v1.35.1 (darwin/arm64) kubernetes/8fea90b,gzip(gfe))
- 2026-11-16 07:02:15 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/tmfsync by hand (kubectl/v1.34.1 (darwin/arm64) kubernetes/93248f9,gzip(gfe))
- 2026-11-16 07:10:55 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/docsvc by hand (kubectl/v1.34.1 (darwin/arm64) kubernetes/93248f9,gzip(gfe))
- 2026-11-16 07:15:20 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/docsvc by hand (kubectl/v1.34.1 (darwin/arm64) kubernetes/93248f9,gzip(gfe))
- 2026-11-16 07:50:00 UTClatency cluster p95 peaked at 1.5 min
- 2026-11-16 08:20:06 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/chat-agent by hand (kubectl/v1.36.4 (darwin/arm64) kubernetes/bb826b1,gzip(gfe))
- 2026-11-16 08:30:42 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/chat-agent by hand (kubectl/v1.36.4 (darwin/arm64) kubernetes/bb826b1,gzip(gfe))
- 2026-11-16 10:14:14 UTChand change [email protected] changed core/v1/namespaces/tilda/secrets/sheet-edit-svc-syncfusion by hand (kubectl/v1.36.4 (darwin/arm64) kubernetes/bb826b1,gzip(gfe))
- 2026-11-16 10:14:59 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/chat-agent by hand (kubectl/v1.36.4 (darwin/arm64) kubernetes/bb826b1,gzip(gfe))
- 2026-11-16 10:15:07 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/chat-agent by hand (kubectl/v1.36.4 (darwin/arm64) kubernetes/bb826b1,gzip(gfe))
- 2026-11-16 10:16:16 UTChand change [email protected] changed core/v1/namespaces/tilda/services/sheet-edit-svc by hand (kubectl/v1.36.4 (darwin/arm64) kubernetes/bb826b1,gzip(gfe))
- 2026-11-16 10:16:19 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/sheet-edit-svc by hand (kubectl/v1.36.4 (darwin/arm64) kubernetes/bb826b1,gzip(gfe))
- 2026-11-16 10:18:07 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/docsvc by hand (kubectl/v1.35.1 (darwin/arm64) kubernetes/8fea90b,gzip(gfe))
- 2026-11-16 10:23:54 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/chat-agent by hand (kubectl/v1.36.4 (darwin/arm64) kubernetes/bb826b1,gzip(gfe))
- 2026-11-16 10:24:04 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/chat-agent by hand (kubectl/v1.36.4 (darwin/arm64) kubernetes/bb826b1,gzip(gfe))
- 2026-11-16 10:25:00 UTChand change [email protected] changed core/v1/namespaces/tilda/services/sheet-edit-svc by hand (kubectl/v1.36.4 (darwin/arm64) kubernetes/bb826b1,gzip(gfe))
- 2026-11-16 10:25:00 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/sheet-edit-svc by hand (kubectl/v1.36.4 (darwin/arm64) kubernetes/bb826b1,gzip(gfe))
- 2026-11-16 10:25:07 UTChand change [email protected] changed core/v1/namespaces/tilda/secrets/sheet-edit-svc-syncfusion by hand (kubectl/v1.36.4 (darwin/arm64) kubernetes/bb826b1,gzip(gfe))
- 2026-11-16 15:00:56 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/chat-agent by hand (kubectl/v1.32.2 (darwin/amd64) kubernetes/67a30c0,gzip(gfe))
- 2026-11-16 15:18:43 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/virtual-assistant by hand (kubectl/v1.33.4 (darwin/arm64) kubernetes/74cdb42,gzip(gfe))
- 2026-11-16 15:27:04 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/chat-agent by hand (kubectl/v1.32.2 (darwin/amd64) kubernetes/67a30c0,gzip(gfe))
- 2026-11-16 16:28:29 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/studiosvc by hand (kubectl/v1.33.3 (darwin/amd64) kubernetes/80779bd,gzip(gfe))
- 2026-11-16 16:38:14 UTChand change [email protected] changed core/v1/namespaces/tilda/secrets/sheet-edit-svc-syncfusion by hand (kubectl/v1.36.4 (darwin/arm64) kubernetes/bb826b1,gzip(gfe))
- 2026-11-16 16:38:27 UTChand change [email protected] changed core/v1/namespaces/tilda/services/sheet-edit-svc by hand (kubectl/v1.36.4 (darwin/arm64) kubernetes/bb826b1,gzip(gfe))
- 2026-11-16 16:38:30 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/sheet-edit-svc by hand (kubectl/v1.36.4 (darwin/arm64) kubernetes/bb826b1,gzip(gfe))
- 2026-11-16 16:41:05 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/tmfsync by hand (kubectl/v1.34.1 (darwin/arm64) kubernetes/93248f9,gzip(gfe))
- 2026-11-16 16:41:18 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/chat-agent by hand (kubectl/v1.36.4 (darwin/arm64) kubernetes/bb826b1,gzip(gfe))
- 2026-11-16 16:41:30 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/chat-agent by hand (kubectl/v1.36.4 (darwin/arm64) kubernetes/bb826b1,gzip(gfe))
- 2026-11-16 16:46:16 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/docsvc by hand (kubectl/v1.34.1 (darwin/arm64) kubernetes/93248f9,gzip(gfe))
- 2026-11-16 16:51:21 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/chat-agent by hand (kubectl/v1.36.4 (darwin/arm64) kubernetes/bb826b1,gzip(gfe))
- 2026-11-16 16:51:32 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/chat-agent by hand (kubectl/v1.36.4 (darwin/arm64) kubernetes/bb826b1,gzip(gfe))
- 2026-11-16 16:51:51 UTChand change [email protected] changed core/v1/namespaces/tilda/services/sheet-edit-svc by hand (kubectl/v1.36.4 (darwin/arm64) kubernetes/bb826b1,gzip(gfe))
- 2026-11-16 16:51:51 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/sheet-edit-svc by hand (kubectl/v1.36.4 (darwin/arm64) kubernetes/bb826b1,gzip(gfe))
- 2026-11-16 16:51:59 UTChand change [email protected] changed core/v1/namespaces/tilda/secrets/sheet-edit-svc-syncfusion by hand (kubectl/v1.36.4 (darwin/arm64) kubernetes/bb826b1,gzip(gfe))
- 2026-11-16 17:01:20 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/tmfsync by hand (kubectl/v1.34.1 (darwin/arm64) kubernetes/93248f9,gzip(gfe))
- 2026-11-16 17:53:39 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/studiosvc by hand (kubectl/v1.33.3 (darwin/amd64) kubernetes/80779bd,gzip(gfe))
- 2026-11-16 18:04:21 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/studiosvc by hand (kubectl/v1.32.2 (darwin/amd64) kubernetes/67a30c0,gzip(gfe))
- 2026-11-16 18:30:32 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/docsvc by hand (kubectl/v1.30.5 (darwin/arm64) kubernetes/633f2e9,gzip(gfe))
- 2026-11-16 18:30:33 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/docsvc by hand (kubectl/v1.30.5 (darwin/arm64) kubernetes/633f2e9,gzip(gfe))
- 2026-11-16 18:30:34 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/docproc by hand (kubectl/v1.30.5 (darwin/arm64) kubernetes/633f2e9,gzip(gfe))
- 2026-11-16 18:30:36 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/docproc by hand (kubectl/v1.30.5 (darwin/arm64) kubernetes/633f2e9,gzip(gfe))
- 2026-11-16 18:40:42 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/tmfsync by hand (kubectl/v1.34.1 (darwin/arm64) kubernetes/93248f9,gzip(gfe))
- 2026-11-16 18:41:58 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/docproc by hand (kubectl/v1.30.5 (darwin/arm64) kubernetes/633f2e9,gzip(gfe))
- 2026-11-16 18:48:43 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/studiosvc by hand (kubectl/v1.33.3 (darwin/amd64) kubernetes/80779bd,gzip(gfe))
- 2026-11-16 18:53:59 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/tmfsync by hand (kubectl/v1.34.1 (darwin/arm64) kubernetes/93248f9,gzip(gfe))
- 2026-11-16 18:54:20 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/docproc by hand (kubectl/v1.30.5 (darwin/arm64) kubernetes/633f2e9,gzip(gfe))
- 2026-11-16 18:54:22 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/docproc by hand (kubectl/v1.30.5 (darwin/arm64) kubernetes/633f2e9,gzip(gfe))
- 2026-11-16 18:54:23 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/docsvc by hand (kubectl/v1.30.5 (darwin/arm64) kubernetes/633f2e9,gzip(gfe))
- 2026-11-16 18:54:24 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/docsvc by hand (kubectl/v1.30.5 (darwin/arm64) kubernetes/633f2e9,gzip(gfe))
- 2026-11-16 18:58:41 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/tmfsync by hand (kubectl/v1.34.1 (darwin/arm64) kubernetes/93248f9,gzip(gfe))
- 2026-11-16 19:09:53 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/studiosvc by hand (kubectl/v1.33.3 (darwin/amd64) kubernetes/80779bd,gzip(gfe))
- 2026-11-16 19:30:41 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/studiosvc by hand (kubectl/v1.33.3 (darwin/amd64) kubernetes/80779bd,gzip(gfe))
- 2026-11-16 19:53:58 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/studiosvc by hand (kubectl/v1.33.3 (darwin/amd64) kubernetes/80779bd,gzip(gfe))
- 2026-11-16 20:24:14 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/tmfsync by hand (kubectl/v1.34.1 (darwin/arm64) kubernetes/93248f9,gzip(gfe))
- 2026-11-16 20:37:25 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/tmfsync by hand (kubectl/v1.34.1 (darwin/arm64) kubernetes/93248f9,gzip(gfe))
- 2026-11-16 20:50:35 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/tmfsync by hand (kubectl/v1.34.1 (darwin/arm64) kubernetes/93248f9,gzip(gfe))
- 2026-11-16 21:30:24 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/docsvc by hand (kubectl/v1.32.2 (darwin/amd64) kubernetes/67a30c0,gzip(gfe))
- 2026-11-16 21:30:25 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/workplansvc by hand (kubectl/v1.32.2 (darwin/amd64) kubernetes/67a30c0,gzip(gfe))
- 2026-11-16 21:30:27 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/docsvc by hand (kubectl/v1.32.2 (darwin/amd64) kubernetes/67a30c0,gzip(gfe))
- 2026-11-16 21:30:27 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/workplansvc by hand (kubectl/v1.32.2 (darwin/amd64) kubernetes/67a30c0,gzip(gfe))
- 2026-11-16 21:31:28 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/tmfsync by hand (kubectl/v1.34.1 (darwin/arm64) kubernetes/93248f9,gzip(gfe))
- 2026-11-16 21:35:58 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/tmfsync by hand (kubectl/v1.34.1 (darwin/arm64) kubernetes/93248f9,gzip(gfe))
- 2026-11-16 21:46:25 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/tmfsync by hand (kubectl/v1.34.1 (darwin/arm64) kubernetes/93248f9,gzip(gfe))
- 2026-11-16 22:14:55 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/tmfsync by hand (kubectl/v1.34.1 (darwin/arm64) kubernetes/93248f9,gzip(gfe))
- 2026-11-16 22:33:12 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/docproc by hand (kubectl/v1.30.5 (darwin/arm64) kubernetes/633f2e9,gzip(gfe))
- 2026-11-16 22:38:11 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/docproc by hand (kubectl/v1.30.5 (darwin/arm64) kubernetes/633f2e9,gzip(gfe))
- 2026-11-16 22:41:00 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/tmfsync by hand (kubectl/v1.34.1 (darwin/arm64) kubernetes/93248f9,gzip(gfe))
- 2026-11-16 22:42:47 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/docproc by hand (kubectl/v1.30.5 (darwin/arm64) kubernetes/633f2e9,gzip(gfe))
History
- 2026-11-16 08:28:23 UTCreopened x-ray · severity=warning was=warning title=pubsub.dashboards-test27-dashsub: 5 failed runs in the last 6 h — failed to query active recommendation IDs for document ?: googleapi: Error ?: Access Denied: Project level=medium was_level=medium finding={'id': 5488, 'fingerprint': 'e9118750daae5f5f9020', 'kind': 'job.failing', 'severity': 'warning', 'cluster': 'test27', 'subject': 'pubsub.dashboards-test27-dashsub', 'title': 'pubsub.dashboards-test27-dashsub: 5 failed runs in the last 6 h — failed to query active recommendation IDs for document ?: googleapi: Error ?: Access Denied: Project', 'facts': {'job': 'pubsub.dashboards-test27-dashsub', 'service': 'dashrsvc', 'failed_runs': 5, 'error_lines': 387, 'sample': 'failed to query active recommendation IDs for document 612b056a-24cd-46e7-92b9-482339a640e2: googleapi: Error 403: Access Denied: Project tilda-trial-dev: User does not have bigquery.jobs.create permission in project tilda-trial-dev., accessDenied', 'error_signature': 'failed to query active recommendation IDs for document ?: googleapi: Error ?: Access Denied: Project tilda-trial-dev: Us', 'failing_at': 'pubsublib/googlepubsub.go:697', 'trace': 'projects/tilda-trial-dev/traces/967a2bd8b2f384890d2e5eeb57f64059', 'detected_via': 'structured error/exception field (below ERROR severity)', 'window': [1794796103, 1794817703], 'note': 'a run is a burst of lines that truly look like failures — an error/exception field, an error-worded message, or a failure status; lines within one burst count as one run. Lines a job dumps at ERROR that are neither (the steps that SUCCEEDED, banner rules) are ignored, so a service that logs its whole context on failure is not miscounted. `failing_at` is the call site those failures came from — for a coarse job (one set_job wrapping a pipeline) more specific than the job name.', 'discriminator': ''}, 'evidence': [{'label': "This job's page", 'href': '/jobs/pubsub.dashboards-test27-dashsub?cluster=test27&from=1794796103&to=1794817703'}, {'label': 'Background jobs', 'href': '/jobs?cluster=test27&from=1794796103&to=1794817703'}, {'label': 'Its error logs', 'href': '/logs?cluster=test27&from=1794796103&to=1794817703&severity=ERROR&q=pubsub.dashboards-test27-dashsub'}, {'label': 'Example trace', 'href': '/trace/projects/tilda-trial-dev/traces/967a2bd8b2f384890d2e5eeb57f64059?cluster=test27'}], 'provenance': {'source': 'live', 'collected_at': None, 'age_seconds': None, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': 'error-severity job lines from BigQuery, last hour, burst-grouped per job'}, 'impact': 20.0, 'status': 'open', 'first_seen': 1792762426, 'last_seen': 1794817703, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 61, 'narrative': "The background job pubsub.dashboards-test27-dashsub on test27 failed 5 runs recently (719 failure lines; lines within one run count once), detected via its structured error/exception field (below ERROR severity). A job that logs its failures below ERROR severity — a WARNING carrying an error field — is invisible to the novel-shape detector and to the page's own failed count; this detector reads the error field itself, which is why the finding now appears. The sample failure is in the facts.", 'narrative_kind': 'template', 'discriminator': '', 'level': 'medium', 'told_level': None, 'episode_at': 1794817703, 'chat_thread': None, 'kind_label': 'Background job failing', 'group': 'Jobs', 'age_s': 2055331, 'stale_s': 54}
- 2026-11-16 11:36:11 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 5488, 'fingerprint': 'e9118750daae5f5f9020', 'kind': 'job.failing', 'severity': 'warning', 'cluster': 'test27', 'subject': 'pubsub.dashboards-test27-dashsub', 'title': 'pubsub.dashboards-test27-dashsub: 5 failed runs in the last 6 h — no data found for label RecommendationByDocument, id ?', 'facts': {'job': 'pubsub.dashboards-test27-dashsub', 'service': 'dashrsvc', 'failed_runs': 5, 'error_lines': 387, 'sample': 'no data found for label RecommendationByDocument, id 612b056a-24cd-46e7-92b9-482339a640e2', 'error_signature': 'no data found for label RecommendationByDocument, id ?', 'failing_at': 'pubsublib/googlepubsub.go:697', 'trace': 'projects/tilda-trial-dev/traces/967a2bd8b2f384890d2e5eeb57f64059', 'detected_via': 'structured error/exception field (below ERROR severity)', 'window': [1794801012, 1794822612], 'note': 'a run is a burst of lines that truly look like failures — an error/exception field, an error-worded message, or a failure status; lines within one burst count as one run. Lines a job dumps at ERROR that are neither (the steps that SUCCEEDED, banner rules) are ignored, so a service that logs its whole context on failure is not miscounted. `failing_at` is the call site those failures came from — for a coarse job (one set_job wrapping a pipeline) more specific than the job name.', 'discriminator': ''}, 'evidence': [{'label': "This job's page", 'href': '/jobs/pubsub.dashboards-test27-dashsub?cluster=test27&from=1794801012&to=1794822612'}, {'label': 'Background jobs', 'href': '/jobs?cluster=test27&from=1794801012&to=1794822612'}, {'label': 'Its error logs', 'href': '/logs?cluster=test27&from=1794801012&to=1794822612&severity=ERROR&q=pubsub.dashboards-test27-dashsub'}, {'label': 'Example trace', 'href': '/trace/projects/tilda-trial-dev/traces/967a2bd8b2f384890d2e5eeb57f64059?cluster=test27'}], 'provenance': {'source': 'live', 'collected_at': None, 'age_seconds': None, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': 'error-severity job lines from BigQuery, last hour, burst-grouped per job'}, 'impact': 25.0, 'status': 'resolved', 'first_seen': 1792762426, 'last_seen': 1794822612, 'resolved_at': 1794828971, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 62, 'narrative': "The background job pubsub.dashboards-test27-dashsub on test27 failed 5 runs recently (387 failure lines; lines within one run count once), detected via its structured error/exception field (below ERROR severity). A job that logs its failures below ERROR severity — a WARNING carrying an error field — is invisible to the novel-shape detector and to the page's own failed count; this detector reads the error field itself, which is why the finding now appears. The sample failure is in the facts.", 'narrative_kind': 'template', 'discriminator': '', 'level': 'medium', 'told_level': None, 'episode_at': 1794817703, 'chat_thread': None, 'kind_label': 'Background job failing', 'group': 'Jobs', 'age_s': 2066711, 'stale_s': 6525}
- 2026-11-16 23:44:07 UTCreopened x-ray · severity=warning was=warning title=pubsub.dashboards-test27-dashsub: 5 failed runs in the last 6 h — SyncForAIRun: merge recommendations: MergeRecommendations: failed to fetch existing IDs: failed to q level=medium was_level=medium finding={'id': 5488, 'fingerprint': 'e9118750daae5f5f9020', 'kind': 'job.failing', 'severity': 'warning', 'cluster': 'test27', 'subject': 'pubsub.dashboards-test27-dashsub', 'title': 'pubsub.dashboards-test27-dashsub: 5 failed runs in the last 6 h — SyncForAIRun: merge recommendations: MergeRecommendations: failed to fetch existing IDs: failed to q', 'facts': {'job': 'pubsub.dashboards-test27-dashsub', 'service': 'dashrsvc', 'failed_runs': 5, 'error_lines': 1143, 'sample': 'SyncForAIRun: merge recommendations: MergeRecommendations: failed to fetch existing IDs: failed to query active recommendation IDs for document 6d376f15-837e-472b-bf6d-7b213b51aa11: googleapi: Error 403: Access Denied: Project tilda-trial-dev: User does not have bigquery.jobs.create permission in pr', 'error_signature': 'SyncForAIRun: merge recommendations: MergeRecommendations: failed to fetch existing IDs: failed to query active recommen', 'failing_at': 'pubsublib/googlepubsub.go:697', 'trace': 'projects/tilda-trial-dev/traces/d2ee718db080ebca49cb97740684ac12', 'detected_via': 'ERROR-severity log lines', 'window': [1794851047, 1794872647], 'note': 'a run is a burst of lines that truly look like failures — an error/exception field, an error-worded message, or a failure status; lines within one burst count as one run. Lines a job dumps at ERROR that are neither (the steps that SUCCEEDED, banner rules) are ignored, so a service that logs its whole context on failure is not miscounted. `failing_at` is the call site those failures came from — for a coarse job (one set_job wrapping a pipeline) more specific than the job name.', 'discriminator': ''}, 'evidence': [{'label': "This job's page", 'href': '/jobs/pubsub.dashboards-test27-dashsub?cluster=test27&from=1794851047&to=1794872647'}, {'label': 'Background jobs', 'href': '/jobs?cluster=test27&from=1794851047&to=1794872647'}, {'label': 'Its error logs', 'href': '/logs?cluster=test27&from=1794851047&to=1794872647&severity=ERROR&q=pubsub.dashboards-test27-dashsub'}, {'label': 'Example trace', 'href': '/trace/projects/tilda-trial-dev/traces/d2ee718db080ebca49cb97740684ac12?cluster=test27'}], 'provenance': {'source': 'live', 'collected_at': None, 'age_seconds': None, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': 'error-severity job lines from BigQuery, last hour, burst-grouped per job'}, 'impact': 20.0, 'status': 'open', 'first_seen': 1792762426, 'last_seen': 1794872647, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 63, 'narrative': "The background job pubsub.dashboards-test27-dashsub on test27 failed 5 runs recently (387 failure lines; lines within one run count once), detected via its structured error/exception field (below ERROR severity). A job that logs its failures below ERROR severity — a WARNING carrying an error field — is invisible to the novel-shape detector and to the page's own failed count; this detector reads the error field itself, which is why the finding now appears. The sample failure is in the facts.", 'narrative_kind': 'template', 'discriminator': '', 'level': 'medium', 'told_level': None, 'episode_at': 1794872647, 'chat_thread': None, 'kind_label': 'Background job failing', 'group': 'Jobs', 'age_s': 2110255, 'stale_s': 34}
- 2026-11-17 00:48:04 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 5488, 'fingerprint': 'e9118750daae5f5f9020', 'kind': 'job.failing', 'severity': 'warning', 'cluster': 'test27', 'subject': 'pubsub.dashboards-test27-dashsub', 'title': 'pubsub.dashboards-test27-dashsub: 5 failed runs in the last 6 h — SyncForAIRun: merge recommendations: MergeRecommendations: failed to fetch existing IDs: failed to q', 'facts': {'job': 'pubsub.dashboards-test27-dashsub', 'service': 'dashrsvc', 'failed_runs': 5, 'error_lines': 1143, 'sample': 'SyncForAIRun: merge recommendations: MergeRecommendations: failed to fetch existing IDs: failed to query active recommendation IDs for document 6d376f15-837e-472b-bf6d-7b213b51aa11: googleapi: Error 403: Access Denied: Project tilda-trial-dev: User does not have bigquery.jobs.create permission in pr', 'error_signature': 'SyncForAIRun: merge recommendations: MergeRecommendations: failed to fetch existing IDs: failed to query active recommen', 'failing_at': 'pubsublib/googlepubsub.go:697', 'trace': 'projects/tilda-trial-dev/traces/d2ee718db080ebca49cb97740684ac12', 'detected_via': 'ERROR-severity log lines', 'window': [1794851047, 1794872647], 'note': 'a run is a burst of lines that truly look like failures — an error/exception field, an error-worded message, or a failure status; lines within one burst count as one run. Lines a job dumps at ERROR that are neither (the steps that SUCCEEDED, banner rules) are ignored, so a service that logs its whole context on failure is not miscounted. `failing_at` is the call site those failures came from — for a coarse job (one set_job wrapping a pipeline) more specific than the job name.', 'discriminator': ''}, 'evidence': [{'label': "This job's page", 'href': '/jobs/pubsub.dashboards-test27-dashsub?cluster=test27&from=1794851047&to=1794872647'}, {'label': 'Background jobs', 'href': '/jobs?cluster=test27&from=1794851047&to=1794872647'}, {'label': 'Its error logs', 'href': '/logs?cluster=test27&from=1794851047&to=1794872647&severity=ERROR&q=pubsub.dashboards-test27-dashsub'}, {'label': 'Example trace', 'href': '/trace/projects/tilda-trial-dev/traces/d2ee718db080ebca49cb97740684ac12?cluster=test27'}], 'provenance': {'source': 'live', 'collected_at': None, 'age_seconds': None, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': 'error-severity job lines from BigQuery, last hour, burst-grouped per job'}, 'impact': 25.0, 'status': 'resolved', 'first_seen': 1792762426, 'last_seen': 1794872647, 'resolved_at': 1794876484, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 63, 'narrative': "The background job pubsub.dashboards-test27-dashsub on test27 failed 5 runs recently (1,143 failure lines; lines within one run count once), detected via its ERROR-severity log lines. A job that logs its failures below ERROR severity — a WARNING carrying an error field — is invisible to the novel-shape detector and to the page's own failed count; this detector reads the error field itself, which is why the finding now appears. The sample failure is in the facts.", 'narrative_kind': 'template', 'discriminator': '', 'level': 'medium', 'told_level': None, 'episode_at': 1794872647, 'chat_thread': None, 'kind_label': 'Background job failing', 'group': 'Jobs', 'age_s': 2114098, 'stale_s': 3877}
Check it yourself
Every link carries the finding's own time window, so it shows tomorrow what it shows now.
Actions
Sign in (IAP) to acknowledge, snooze or resolve.
Where the numbers come from
| source | live |
| collected_at | None |
| age_seconds | None |
| stale | False |
| shown | None |
| total | None |
| truncated | False |
| coverage_from | None |
| retain_days | None |
| note | error-severity job lines from BigQuery, last hour, burst-grouped per job |
| detector | job_failing |