docsvc shape 00000000: p95 1.29 s now vs 55 ms earlier (23.2×, 941 runs)
warning
Query shape degrading
test26
resolved
impact 30
Owner: Prasad Limaye (backup: Harish Kumar)
opened 75d ago · last observed 2d ago · observed 168× · #755 · 5fe21ac7f57bab582719
What happened template
A docsvc query shape on test26 has p95 1.3 s in the second half of the window against 55 ms in the first (23.2×, 941 executions). The shape reads: “MATCH (currentUser:User{deltimestamp:?,deltimestamp:?,email:$email}) WHERE (currentUser.isDummy IS NULL OR currentUser.i…”. Slower can mean more data, a lost index, or lock contention — the shape page has the CPU/wait split. Shapes whose p95 stays under a second are never reported: below that nobody notices, whatever the ratio.
A Cypher shape whose p95 is worsening across the window and is at least 1 s now; spikes under a second are ignored.
The facts
| service | docsvc |
| plan_hash | 00000000 |
| queries | 941 |
| floor_ms | 1,000 |
| p95_early | 55.42 |
| p95_late | 1,286 |
| ratio | 23.20 |
| avg_ms | 48.27 |
| sample | MATCH (currentUser:User{deltimestamp:?,deltimestamp:?,email:$email}) WHERE (currentUser.isDummy IS NULL OR currentUser.isDummy <> true) OPTIONAL MATCH (supportUser:User{email:$supportUserEmail}) OPTIONAL MATCH (targetOrg:Organization{delpending:false, alias:$orgAlias}) // A deactivated (Inactive) org actor is not a membership (ENG-?). OPTIONAL MATCH (currentUser)-[:FOR_USER]-(orgActor:Actor{delpen |
| rw | write |
| severity_capped | critical → warning: test26 is a test-tier cluster |
| discriminator | 00000000 |
| window | 2026-11-16 10:55:04 UTC → 2026-11-16 16:55:04 UTC |
What is failing in docsvc — the bad events, decomposed
“Bad events” are HTTP 5xx answers from docsvc. In this finding's window (2026-11-16 10:55:04 UTC → 2026-11-16 16:55:04 UTC) there were 0; every line links to the grouped errors behind it.
The error shapes behind them
| 156× | template backfill: could not move document — document has no primary version, cannot move to Templates since 2026-11-16 13:27:23 UTC |
The story — No docsvc deploy or hand change is recorded in the hour before this opened. Everything below is docsvc only.
- 2026-09-03 21:27:59 UTCthis finding docsvc shape 00000000: p95 1.29 s now vs 55 ms earlier (23.2×, 941 runs)
- 2026-10-29 01:01:22 UTCfinding · info docsvc request rate 28.8 req/min vs a normal 0.0 req/min ± 5.0 req/min for Mon 06:00 UTC (3 of the last 4 checks)
- 2026-11-02 05:09:15 UTCfinding · warning docsvc GET /api/documents/:id: 4xx — 123 of 130 calls (95 %; other 4xx×123) in the last 15 min — it usually has none
- 2026-11-14 20:00:17 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/docsvc by hand (kubectl/v1.36.1 (darwin/arm64) kubernetes/7569396,gzip(gfe))
- 2026-11-15 00:00:00 UTClatency docsvc p95 peaked at 2.55 s
- 2026-11-16 00:00:00 UTCerrors docsvc 5xx peaked at 240 in one five-minute bucket (median 0 in this window)
- 2026-11-16 02:26:12 UTCfinding · warning docsvc: a new error shape, 11× in the last hour, never seen in 90 days (Request completed — ? · document cannot be promoted to TMF: document m)
- 2026-11-16 02:26:12 UTCfinding · warning docsvc PUT /api/documents/promote: 16 of 16 calls failed (100 %) in the last 15 min
- 2026-11-16 06:01:45 UTCfinding · warning docsvc: a new error shape, 41× in the last hour, never seen in 90 days (Request completed — ? · ?/? documents promoted to TMF and ? failed)
- 2026-11-16 13:26:38 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/docsvc by hand (kubectl/v1.35.1 (darwin/arm64) kubernetes/8fea90b,gzip(gfe))
- 2026-11-16 13:29:19 UTCfinding · warning docsvc: a new error shape, 156× in the last hour, never seen in 90 days (template backfill: could not move document — document has no primary v)
- 2026-11-16 14:01:33 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/docsvc by hand (kubectl/v1.30.2 (darwin/arm64) kubernetes/3968350,gzip(gfe))
History
- 2026-11-15 05:09:36 UTCreopened x-ray · severity=warning was=warning title=docsvc shape 00000000: p95 1.70 s now vs 42 ms earlier (40.8×, 1,999 runs) level=low was_level=low finding={'id': 755, 'fingerprint': '5fe21ac7f57bab582719', 'kind': 'shape.degrading', 'severity': 'warning', 'cluster': 'test26', 'subject': 'docsvc', 'title': 'docsvc shape 00000000: p95 1.70 s now vs 42 ms earlier (40.8×, 1,999 runs)', 'facts': {'service': 'docsvc', 'plan_hash': '00000000', 'queries': 1999, 'floor_ms': 1000, 'p95_early': 41.66923076923069, 'p95_late': 1699.9999999999982, 'ratio': 40.797489385268634, 'avg_ms': 36.731865932966485, 'sample': 'MATCH (m:TildaDataMigration {id: $id}) RETURN m.status = $applied', 'rw': 'write', 'window': [1794697776, 1794719376], 'recent_deploys': [], 'severity_capped': 'critical → warning: test26 is a test-tier cluster', 'discriminator': '00000000'}, 'evidence': [{'label': 'Query shape', 'href': '/neo/shape/00000000?cluster=test26&from=1794697776&to=1794719376'}, {'label': 'Degrading shapes', 'href': '/neo/queries?cluster=test26&from=1794697776&to=1794719376&order=degrading'}], 'provenance': {'source': 'rollup', 'collected_at': 1794719376, 'age_seconds': 5, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 25.9, 'status': 'open', 'first_seen': 1788470879, 'last_seen': 1794719376, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 161, 'narrative': 'A docsvc query shape on test26 has p95 1.2 s in the second half of the window against 80 ms in the first (15.0×, 216 executions). The shape reads: “MATCH (task:Task) WHERE task.task_number = ? AND EXISTS { (task)-[:BELONGS_TO]->(:Inbox{type:?})-[:MANAGED_BY]->(:Organi…”. Slower can mean more data, a lost index, or lock contention — the shape page has the CPU/wait split. Shapes whose p95 stays under a second are never reported: below that nobody notices, whatever the ratio.', 'narrative_kind': 'template', 'discriminator': '00000000', 'level': 'low', 'told_level': None, 'episode_at': 1794719376, 'chat_thread': None, 'kind_label': 'Query shape degrading', 'group': 'Neo4j', 'age_s': 6248501, 'stale_s': 4}
- 2026-11-15 06:07:50 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 755, 'fingerprint': '5fe21ac7f57bab582719', 'kind': 'shape.degrading', 'severity': 'warning', 'cluster': 'test26', 'subject': 'docsvc', 'title': 'docsvc shape 00000000: p95 1.02 s now vs 42 ms earlier (24.4×, 2,067 runs)', 'facts': {'service': 'docsvc', 'plan_hash': '00000000', 'queries': 2067, 'floor_ms': 1000, 'p95_early': 41.94615384615382, 'p95_late': 1024.9999999999986, 'ratio': 24.43609022556389, 'avg_ms': 39.355587808417994, 'sample': 'MATCH (e:Event {tildaid: $eventId}) OPTIONAL MATCH (e)<-[:EVENTS]-(:EventPlan)-[:PLAN_CONTEXT]->(ctxCandidate) WITH e, head(collect(ctxCandidate)) AS ctx OPTIONAL MATCH (orgCandidate:Organization {tildaid: e.orgId}) WITH e, ctx, head(collect(orgCandidate)) AS org OPTIONAL MATCH (e)-[:HAS_SYSTEM_RULE]->(rule:SystemEventRule) RETURN e.type AS eventType, e.name AS eventName, coalesce(e.status, ?) AS ', 'rw': 'write', 'window': [1794700098, 1794721698], 'recent_deploys': [], 'severity_capped': 'critical → warning: test26 is a test-tier cluster', 'discriminator': '00000000'}, 'evidence': [{'label': 'Query shape', 'href': '/neo/shape/00000000?cluster=test26&from=1794700098&to=1794721698'}, {'label': 'Degrading shapes', 'href': '/neo/queries?cluster=test26&from=1794700098&to=1794721698&order=degrading'}], 'provenance': {'source': 'rollup', 'collected_at': 1794721698, 'age_seconds': 5, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 30.7, 'status': 'resolved', 'first_seen': 1788470879, 'last_seen': 1794721698, 'resolved_at': 1794722870, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 163, 'narrative': 'A docsvc query shape on test26 has p95 1.0 s in the second half of the window against 42 ms in the first (24.4×, 2,067 executions). The shape reads: “MATCH (e:Event {tildaid: $eventId}) OPTIONAL MATCH (e)<-[:EVENTS]-(:EventPlan)-[:PLAN_CONTEXT]->(ctxCandidate) WITH e, h…”. Slower can mean more data, a lost index, or lock contention — the shape page has the CPU/wait split. Shapes whose p95 stays under a second are never reported: below that nobody notices, whatever the ratio.', 'narrative_kind': 'template', 'discriminator': '00000000', 'level': 'low', 'told_level': None, 'episode_at': 1794719376, 'chat_thread': None, 'kind_label': 'Query shape degrading', 'group': 'Neo4j', 'age_s': 6251996, 'stale_s': 1177}
- 2026-11-16 09:06:49 UTCreopened x-ray · severity=warning was=warning title=docsvc shape 00000000: p95 1.50 s now vs 17 ms earlier (86.9×, 30,363 runs) level=low was_level=low finding={'id': 755, 'fingerprint': '5fe21ac7f57bab582719', 'kind': 'shape.degrading', 'severity': 'warning', 'cluster': 'test26', 'subject': 'docsvc', 'title': 'docsvc shape 00000000: p95 1.50 s now vs 17 ms earlier (86.9×, 30,363 runs)', 'facts': {'service': 'docsvc', 'plan_hash': '00000000', 'queries': 30363, 'floor_ms': 1000, 'p95_early': 17.260882352941167, 'p95_late': 1500.0, 'ratio': 86.90169884301469, 'avg_ms': 12.32183908045977, 'sample': 'MATCH (task:Task) WHERE task.task_number = ? AND EXISTS { (task)-[:BELONGS_TO]->(:Inbox{type:?})-[:MANAGED_BY]->(:Organization) } RETURN task.tildaid LIMIT ?', 'rw': 'write', 'window': [1794798409, 1794820009], 'recent_deploys': [], 'severity_capped': 'critical → warning: test26 is a test-tier cluster', 'discriminator': '00000000'}, 'evidence': [{'label': 'Query shape', 'href': '/neo/shape/00000000?cluster=test26&from=1794798409&to=1794820009'}, {'label': 'Degrading shapes', 'href': '/neo/queries?cluster=test26&from=1794798409&to=1794820009&order=degrading'}], 'provenance': {'source': 'rollup', 'collected_at': 1794820004, 'age_seconds': 11, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 30.0, 'status': 'open', 'first_seen': 1788470879, 'last_seen': 1794820009, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 164, 'narrative': 'A docsvc query shape on test26 has p95 1.0 s in the second half of the window against 42 ms in the first (24.4×, 2,067 executions). The shape reads: “MATCH (e:Event {tildaid: $eventId}) OPTIONAL MATCH (e)<-[:EVENTS]-(:EventPlan)-[:PLAN_CONTEXT]->(ctxCandidate) WITH e, h…”. Slower can mean more data, a lost index, or lock contention — the shape page has the CPU/wait split. Shapes whose p95 stays under a second are never reported: below that nobody notices, whatever the ratio.', 'narrative_kind': 'template', 'discriminator': '00000000', 'level': 'low', 'told_level': None, 'episode_at': 1794820009, 'chat_thread': None, 'kind_label': 'Query shape degrading', 'group': 'Neo4j', 'age_s': 6349136, 'stale_s': 6}
- 2026-11-16 13:29:19 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 755, 'fingerprint': '5fe21ac7f57bab582719', 'kind': 'shape.degrading', 'severity': 'warning', 'cluster': 'test26', 'subject': 'docsvc', 'title': 'docsvc shape 00000000: p95 1.22 s now vs 17 ms earlier (70.5×, 30,362 runs)', 'facts': {'service': 'docsvc', 'plan_hash': '00000000', 'queries': 30362, 'floor_ms': 1000, 'p95_early': 17.26029411764706, 'p95_late': 1216.6666666666638, 'ratio': 70.4893357189513, 'avg_ms': 12.318094987154996, 'sample': 'MATCH(a:Study{tildaid:$a_tildaid}) RETURN a', 'rw': 'write', 'window': [1794801012, 1794822612], 'recent_deploys': [], 'severity_capped': 'critical → warning: test26 is a test-tier cluster', 'discriminator': '00000000'}, 'evidence': [{'label': 'Query shape', 'href': '/neo/shape/00000000?cluster=test26&from=1794801012&to=1794822612'}, {'label': 'Degrading shapes', 'href': '/neo/queries?cluster=test26&from=1794801012&to=1794822612&order=degrading'}], 'provenance': {'source': 'rollup', 'collected_at': 1794822605, 'age_seconds': 13, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 35.0, 'status': 'resolved', 'first_seen': 1788470879, 'last_seen': 1794822612, 'resolved_at': 1794835759, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 165, 'narrative': 'A docsvc query shape on test26 has p95 1.2 s in the second half of the window against 17 ms in the first (70.5×, 30,362 executions). The shape reads: “MATCH(a:Study{tildaid:$a_tildaid}) RETURN a…”. Slower can mean more data, a lost index, or lock contention — the shape page has the CPU/wait split. Shapes whose p95 stays under a second are never reported: below that nobody notices, whatever the ratio.', 'narrative_kind': 'template', 'discriminator': '00000000', 'level': 'low', 'told_level': None, 'episode_at': 1794820009, 'chat_thread': None, 'kind_label': 'Query shape degrading', 'group': 'Neo4j', 'age_s': 6364888, 'stale_s': 13155}
- 2026-11-16 16:14:38 UTCreopened x-ray · severity=warning was=warning title=docsvc shape 00000000: p95 1.42 s now vs 54 ms earlier (26.4×, 934 runs) level=low was_level=low finding={'id': 755, 'fingerprint': '5fe21ac7f57bab582719', 'kind': 'shape.degrading', 'severity': 'warning', 'cluster': 'test26', 'subject': 'docsvc', 'title': 'docsvc shape 00000000: p95 1.42 s now vs 54 ms earlier (26.4×, 934 runs)', 'facts': {'service': 'docsvc', 'plan_hash': '00000000', 'queries': 934, 'floor_ms': 1000, 'p95_early': 53.749999999999744, 'p95_late': 1421.4285714285718, 'ratio': 26.445182724252625, 'avg_ms': 48.442184154175585, 'sample': 'MATCH (task:Task) WHERE task.task_number = ? AND EXISTS { (task)-[:BELONGS_TO]->(:Inbox{type:?})-[:MANAGED_BY]->(:Organization) } RETURN task.tildaid LIMIT ?', 'rw': 'write', 'window': [1794824078, 1794845678], 'recent_deploys': [], 'severity_capped': 'critical → warning: test26 is a test-tier cluster', 'discriminator': '00000000'}, 'evidence': [{'label': 'Query shape', 'href': '/neo/shape/00000000?cluster=test26&from=1794824078&to=1794845678'}, {'label': 'Degrading shapes', 'href': '/neo/queries?cluster=test26&from=1794824078&to=1794845678&order=degrading'}], 'provenance': {'source': 'rollup', 'collected_at': 1794845678, 'age_seconds': 8, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 24.6, 'status': 'open', 'first_seen': 1788470879, 'last_seen': 1794845678, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 166, 'narrative': 'A docsvc query shape on test26 has p95 1.2 s in the second half of the window against 17 ms in the first (70.5×, 30,362 executions). The shape reads: “MATCH(a:Study{tildaid:$a_tildaid}) RETURN a…”. Slower can mean more data, a lost index, or lock contention — the shape page has the CPU/wait split. Shapes whose p95 stays under a second are never reported: below that nobody notices, whatever the ratio.', 'narrative_kind': 'template', 'discriminator': '00000000', 'level': 'low', 'told_level': None, 'episode_at': 1794845678, 'chat_thread': None, 'kind_label': 'Query shape degrading', 'group': 'Neo4j', 'age_s': 6374806, 'stale_s': 7}
- 2026-11-16 17:51:15 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 755, 'fingerprint': '5fe21ac7f57bab582719', 'kind': 'shape.degrading', 'severity': 'warning', 'cluster': 'test26', 'subject': 'docsvc', 'title': 'docsvc shape 00000000: p95 1.29 s now vs 55 ms earlier (23.2×, 941 runs)', 'facts': {'service': 'docsvc', 'plan_hash': '00000000', 'queries': 941, 'floor_ms': 1000, 'p95_early': 55.41666666666648, 'p95_late': 1285.7142857142858, 'ratio': 23.200859291084935, 'avg_ms': 48.272051009564294, 'sample': 'MATCH (currentUser:User{deltimestamp:?,deltimestamp:?,email:$email}) WHERE (currentUser.isDummy IS NULL OR currentUser.isDummy <> true) OPTIONAL MATCH (supportUser:User{email:$supportUserEmail}) OPTIONAL MATCH (targetOrg:Organization{delpending:false, alias:$orgAlias}) // A deactivated (Inactive) org actor is not a membership (ENG-?). OPTIONAL MATCH (currentUser)-[:FOR_USER]-(orgActor:Actor{delpen', 'rw': 'write', 'window': [1794826504, 1794848104], 'recent_deploys': [], 'severity_capped': 'critical → warning: test26 is a test-tier cluster', 'discriminator': '00000000'}, 'evidence': [{'label': 'Query shape', 'href': '/neo/shape/00000000?cluster=test26&from=1794826504&to=1794848104'}, {'label': 'Degrading shapes', 'href': '/neo/queries?cluster=test26&from=1794826504&to=1794848104&order=degrading'}], 'provenance': {'source': 'rollup', 'collected_at': 1794848102, 'age_seconds': 10, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 29.6, 'status': 'resolved', 'first_seen': 1788470879, 'last_seen': 1794848104, 'resolved_at': 1794851475, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 168, 'narrative': 'A docsvc query shape on test26 has p95 1.3 s in the second half of the window against 55 ms in the first (23.2×, 941 executions). The shape reads: “MATCH (currentUser:User{deltimestamp:?,deltimestamp:?,email:$email}) WHERE (currentUser.isDummy IS NULL OR currentUser.i…”. Slower can mean more data, a lost index, or lock contention — the shape page has the CPU/wait split. Shapes whose p95 stays under a second are never reported: below that nobody notices, whatever the ratio.', 'narrative_kind': 'template', 'discriminator': '00000000', 'level': 'low', 'told_level': None, 'episode_at': 1794845678, 'chat_thread': None, 'kind_label': 'Query shape degrading', 'group': 'Neo4j', 'age_s': 6380603, 'stale_s': 3378}
Check it yourself
Every link carries the finding's own time window, so it shows tomorrow what it shows now.
Actions
Sign in (IAP) to acknowledge, snooze or resolve.
Where the numbers come from
| source | rollup |
| collected_at | 1794848102 |
| age_seconds | 10 |
| stale | False |
| shown | None |
| total | None |
| truncated | False |
| coverage_from | None |
| retain_days | None |
| note | None |
| detector | shapes |