feedsvc shape 00000000: p95 4.85 s now vs 28 ms earlier (172.7×, 3,394 runs)
warning
Query shape degrading
test26
resolved
impact 30
opened 20d ago · last observed 1d ago · observed 14× · #8769 · fd21d23b22050dcbf624
What happened template
A feedsvc query shape on test26 has p95 4.8 s in the second half of the window against 28 ms in the first (172.7×, 3,394 executions). The shape reads: “RETURN EXISTS { MATCH (cfg2:WorkplanCfg) WHERE cfg2.progressPending = true AND coalesce(cfg2.lastViewedAtMs, ?) > (times…”. Slower can mean more data, a lost index, or lock contention — the shape page has the CPU/wait split. Shapes whose p95 stays under a second are never reported: below that nobody notices, whatever the ratio.
A Cypher shape whose p95 is worsening across the window and is at least 1 s now; spikes under a second are ignored.
The facts
| service | feedsvc |
| plan_hash | 00000000 |
| queries | 3,394 |
| floor_ms | 1,000 |
| p95_early | 28.09 |
| p95_late | 4,850 |
| ratio | 172.69 |
| avg_ms | 9.58 |
| sample | RETURN EXISTS { MATCH (cfg2:WorkplanCfg) WHERE cfg2.progressPending = true AND coalesce(cfg2.lastViewedAtMs, ?) > (timestamp() - ?) AND (timestamp() - coalesce(cfg2.lastProgressComputedAtMs, ?)) > coalesce(cfg2.progressComputeIntervalMs, ?) } AS fleetProgressPending, EXISTS { MATCH (item:WorkplanInstanceMilestoneItem) WHERE item.itemType = ? AND coalesce(item.workOrderId, ?) <> ? AND coalesce(item |
| rw | write |
| severity_capped | critical → warning: test26 is a test-tier cluster |
| discriminator | 00000000 |
| window | 2026-11-17 03:59:37 UTC → 2026-11-17 09:59:37 UTC |
The story — No feedsvc deploy or hand change is recorded in the hour before this opened. Everything below is feedsvc only.
- 2026-10-28 18:05:55 UTCthis finding feedsvc shape 00000000: p95 4.85 s now vs 28 ms earlier (172.7×, 3,394 runs)
- 2026-11-14 20:00:18 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/feedsvc by hand (kubectl/v1.36.1 (darwin/arm64) kubernetes/7569396,gzip(gfe))
- 2026-11-14 23:49:52 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/feedsvc by hand (kubectl/v1.36.1 (darwin/arm64) kubernetes/7569396,gzip(gfe))
- 2026-11-16 17:45:31 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/feedsvc by hand (kubectl/v1.36.1 (darwin/arm64) kubernetes/7569396,gzip(gfe))
- 2026-11-16 19:49:16 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/feedsvc by hand (kubectl/v1.36.1 (darwin/arm64) kubernetes/7569396,gzip(gfe))
- 2026-11-17 00:15:40 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/feedsvc by hand (kubectl/v1.36.1 (darwin/arm64) kubernetes/7569396,gzip(gfe))
- 2026-11-17 00:30:51 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/feedsvc by hand (kubectl/v1.36.1 (darwin/arm64) kubernetes/7569396,gzip(gfe))
- 2026-11-17 00:34:22 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/feedsvc by hand (kubectl/v1.36.1 (darwin/arm64) kubernetes/7569396,gzip(gfe))
- 2026-11-17 00:53:22 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/feedsvc by hand (kubectl/v1.36.1 (darwin/arm64) kubernetes/7569396,gzip(gfe))
- 2026-11-17 01:11:14 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/feedsvc by hand (kubectl/v1.36.1 (darwin/arm64) kubernetes/7569396,gzip(gfe))
- 2026-11-17 02:15:09 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/feedsvc by hand (kubectl/v1.36.1 (darwin/arm64) kubernetes/7569396,gzip(gfe))
History
- 2026-11-17 09:22:05 UTCreopened x-ray · severity=warning was=warning title=feedsvc shape 00000000: p95 4.85 s now vs 28 ms earlier (173.5×, 4,656 runs) level=low was_level=low finding={'id': 8769, 'fingerprint': 'fd21d23b22050dcbf624', 'kind': 'shape.degrading', 'severity': 'warning', 'cluster': 'test26', 'subject': 'feedsvc', 'title': 'feedsvc shape 00000000: p95 4.85 s now vs 28 ms earlier (173.5×, 4,656 runs)', 'facts': {'service': 'feedsvc', 'plan_hash': '00000000', 'queries': 4656, 'floor_ms': 1000, 'p95_early': 27.95731707317073, 'p95_late': 4850.0, 'ratio': 173.47873500545256, 'avg_ms': 8.210051546391753, 'sample': 'MATCH (currentUser:User{deltimestamp:?,deltimestamp:?,email:$email}) WHERE (currentUser.isDummy IS NULL OR currentUser.isDummy <> true) OPTIONAL MATCH (supportUser:User{email:$supportUserEmail}) OPTIONAL MATCH (targetOrg:Organization{delpending:false, alias:$orgAlias}) // A deactivated (Inactive) org actor is not a membership (ENG-?). OPTIONAL MATCH (currentUser)-[:FOR_USER]-(orgActor:Actor{delpen', 'rw': 'write', 'window': [1794885725, 1794907325], 'recent_deploys': [], 'severity_capped': 'critical → warning: test26 is a test-tier cluster', 'discriminator': '00000000'}, 'evidence': [{'label': 'Query shape', 'href': '/neo/shape/00000000?cluster=test26&from=1794885725&to=1794907325'}, {'label': 'Degrading shapes', 'href': '/neo/queries?cluster=test26&from=1794885725&to=1794907325&order=degrading'}], 'provenance': {'source': 'rollup', 'collected_at': 1794907321, 'age_seconds': 13, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 24.7, 'status': 'open', 'first_seen': 1793210755, 'last_seen': 1794907325, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 12, 'narrative': 'A feedsvc query shape on test26 has p95 1.8 s in the second half of the window against 9 ms in the first (198.3×, 280 executions). The shape reads: “MATCH (task:Task) WHERE task.task_number = ? AND EXISTS { (task)-[:BELONGS_TO]->(:Inbox{type:?})-[:MANAGED_BY]->(:Organi…”. Slower can mean more data, a lost index, or lock contention — the shape page has the CPU/wait split. Shapes whose p95 stays under a second are never reported: below that nobody notices, whatever the ratio.', 'narrative_kind': 'template', 'discriminator': '00000000', 'level': 'low', 'told_level': None, 'episode_at': 1794907325, 'chat_thread': None, 'kind_label': 'Query shape degrading', 'group': 'Neo4j', 'age_s': 1696578, 'stale_s': 8}
- 2026-11-17 10:21:45 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 8769, 'fingerprint': 'fd21d23b22050dcbf624', 'kind': 'shape.degrading', 'severity': 'warning', 'cluster': 'test26', 'subject': 'feedsvc', 'title': 'feedsvc shape 00000000: p95 4.85 s now vs 28 ms earlier (172.7×, 3,394 runs)', 'facts': {'service': 'feedsvc', 'plan_hash': '00000000', 'queries': 3394, 'floor_ms': 1000, 'p95_early': 28.085561497326147, 'p95_late': 4850.0, 'ratio': 172.68659558263553, 'avg_ms': 9.579846788450206, 'sample': 'RETURN EXISTS { MATCH (cfg2:WorkplanCfg) WHERE cfg2.progressPending = true AND coalesce(cfg2.lastViewedAtMs, ?) > (timestamp() - ?) AND (timestamp() - coalesce(cfg2.lastProgressComputedAtMs, ?)) > coalesce(cfg2.progressComputeIntervalMs, ?) } AS fleetProgressPending, EXISTS { MATCH (item:WorkplanInstanceMilestoneItem) WHERE item.itemType = ? AND coalesce(item.workOrderId, ?) <> ? AND coalesce(item', 'rw': 'write', 'window': [1794887977, 1794909577], 'recent_deploys': [], 'severity_capped': 'critical → warning: test26 is a test-tier cluster', 'discriminator': '00000000'}, 'evidence': [{'label': 'Query shape', 'href': '/neo/shape/00000000?cluster=test26&from=1794887977&to=1794909577'}, {'label': 'Degrading shapes', 'href': '/neo/queries?cluster=test26&from=1794887977&to=1794909577&order=degrading'}], 'provenance': {'source': 'rollup', 'collected_at': 1794909577, 'age_seconds': 9, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 29.7, 'status': 'resolved', 'first_seen': 1793210755, 'last_seen': 1794909577, 'resolved_at': 1794910905, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 14, 'narrative': 'A feedsvc query shape on test26 has p95 4.8 s in the second half of the window against 28 ms in the first (172.7×, 3,394 executions). The shape reads: “RETURN EXISTS { MATCH (cfg2:WorkplanCfg) WHERE cfg2.progressPending = true AND coalesce(cfg2.lastViewedAtMs, ?) > (times…”. Slower can mean more data, a lost index, or lock contention — the shape page has the CPU/wait split. Shapes whose p95 stays under a second are never reported: below that nobody notices, whatever the ratio.', 'narrative_kind': 'template', 'discriminator': '00000000', 'level': 'low', 'told_level': None, 'episode_at': 1794907325, 'chat_thread': None, 'kind_label': 'Query shape degrading', 'group': 'Neo4j', 'age_s': 1700158, 'stale_s': 1336}
Check it yourself
Every link carries the finding's own time window, so it shows tomorrow what it shows now.
Actions
Sign in (IAP) to acknowledge, snooze or resolve.
Where the numbers come from
| source | rollup |
| collected_at | 1794909577 |
| age_seconds | 9 |
| stale | False |
| shown | None |
| total | None |
| truncated | False |
| coverage_from | None |
| retain_days | None |
| note | None |
| detector | shapes |