Xray
daily rollups 1h 6h 24h 48h 3d 5d 7d 14d 30d ‹
Last 5 days▾
Ends now. Anything up to this cluster's 30-day retention.
Start and end times
Cancel
Interpreted in UTC (change it at the right of this bar). Earliest available: 2026-10-19 22:49:18.
› →|
updated 1d ago
Sample data. This database was generated by scripts/seed_dev_db.py — every service, deploy, person and figure on this page is synthetic. Names are sample.*@seed.invalid and image tags are prefixed SEED-. Nothing here happened.
← findings

studiosvc shape 00000000: p95 2.15 s now vs 128 ms earlier (16.8×, 865 runs)

warning Query shape degrading test27 resolved impact 30 Owner: Suresh Boddapati (backup: Madhava Mukkamala) opened 13d ago · last observed 4d ago · observed 8× · #16285 · 4a6d5a9ad3ac4020bcc5

What happened template

A studiosvc query shape on test27 has p95 2.2 s in the second half of the window against 128 ms in the first (16.8×, 865 executions). The shape reads: “MATCH (task:Task) WHERE task.task_number = ? AND EXISTS { (task)-[:BELONGS_TO]->(:Inbox{type:?})-[:MANAGED_BY]->(:Organi…”. Slower can mean more data, a lost index, or lock contention — the shape page has the CPU/wait split. Shapes whose p95 stays under a second are never reported: below that nobody notices, whatever the ratio.

A Cypher shape whose p95 is worsening across the window and is at least 1 s now; spikes under a second are ignored.

The facts

servicestudiosvc
plan_hash00000000
queries865
floor_ms1,000
p95_early127.71
p95_late2,150
ratio16.83
avg_ms44.40
sampleMATCH (task:Task) WHERE task.task_number = ? AND EXISTS { (task)-[:BELONGS_TO]->(:Inbox{type:?})-[:MANAGED_BY]->(:Organization) } RETURN task.tildaid LIMIT ?
rwwrite
severity_cappedcritical → warning: test27 is a test-tier cluster
discriminator00000000
window2026-11-14 10:49:02 UTC → 2026-11-14 16:49:02 UTC

What is failing in studiosvc — the bad events, decomposed

“Bad events” are HTTP 5xx answers from studiosvc. In this finding's window (2026-11-14 10:49:02 UTC → 2026-11-14 16:49:02 UTC) there were 0; every line links to the grouped errors behind it.

The error shapes behind them

13006× error on pubsub receive — rpc error: code = NotFound desc = Resource not found (resource=study-site-binding-test27-studi… since 2026-11-14 14:23:11 UTC
17× http error response returned to client since 2026-10-06 01:20:44 UTC
12× failed to fetch form summary — form 76909f97-e9ff-48dd-b4ff-65597c5c1ee1 not found since 2026-11-05 13:41:09 UTC
9× failed to fetch question detail — question not found since 2026-09-16 05:04:44 UTC
8× Request completed — 500 · graph action pre failed for FormV2:object is being used for checking conditions in one or more… since 2026-10-18 02:42:24 UTC

All studiosvc errors, grouped → · service page · its logs

The story — No studiosvc deploy or hand change is recorded in the hour before this opened. Everything below is studiosvc only.

History

  • 2026-11-14 16:08:09 UTCreopened x-ray · severity=warning was=warning title=studiosvc shape 00000000: p95 2.15 s now vs 128 ms earlier (16.8×, 865 runs) level=low was_level=low finding={'id': 16285, 'fingerprint': '4a6d5a9ad3ac4020bcc5', 'kind': 'shape.degrading', 'severity': 'warning', 'cluster': 'test27', 'subject': 'studiosvc', 'title': 'studiosvc shape 00000000: p95 2.15 s now vs 128 ms earlier (16.8×, 865 runs)', 'facts': {'service': 'studiosvc', 'plan_hash': '00000000', 'queries': 865, 'floor_ms': 1000, 'p95_early': 127.71428571428552, 'p95_late': 2150.0000000000023, 'ratio': 16.834451901566037, 'avg_ms': 44.39537572254335, 'sample': '// First part for StudySiteUser scope match (:StudySiteDuty)-[:STUDYSITEDUTY_TO_STUDYSITEACTOR]->(scopeUser:StudySiteActor{delpending:false}) with distinct scopeUser match (scopeUser)-[:STUDYSITEACTOR_TO_PLAN_DEFINITION]->(sitePd:DslPlanDefinition{type:?}) match (sitePd)-[:PLAN_DEFINITION_TO_ADOPTED_WORKFLOWS]->(w:WorkflowV2)<-[:TRIGGER_TO_WORKFLOW]-(t:Trigger{scope:?}) match (sitePd)-[:PLAN_DEFIN', 'rw': 'write', 'window': [1794650889, 1794672489], 'recent_deploys': [], 'severity_capped': 'critical → warning: test27 is a test-tier cluster', 'discriminator': '00000000'}, 'evidence': [{'label': 'Query shape', 'href': '/neo/shape/00000000?cluster=test27&from=1794650889&to=1794672489'}, {'label': 'Degrading shapes', 'href': '/neo/queries?cluster=test27&from=1794650889&to=1794672489&order=degrading'}], 'provenance': {'source': 'rollup', 'collected_at': 1794672487, 'age_seconds': 7, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 24.9, 'status': 'open', 'first_seen': 1793896422, 'last_seen': 1794672489, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 6, 'narrative': 'A studiosvc query shape on test27 has p95 2.6 s in the second half of the window against 11 ms in the first (231.6×, 3,642 executions). The shape reads: “MATCH (c:TMFSyncConfig) RETURN c {.*} AS c ORDER BY c.created_at DESC SKIP $offset LIMIT $limit…”. Slower can mean more data, a lost index, or lock contention — the shape page has the CPU/wait split. Shapes whose p95 stays under a second are never reported: below that nobody notices, whatever the ratio.', 'narrative_kind': 'template', 'discriminator': '00000000', 'level': 'low', 'told_level': None, 'episode_at': 1794672489, 'chat_thread': None, 'kind_label': 'Query shape degrading', 'group': 'Neo4j', 'age_s': 776071, 'stale_s': 4}
  • 2026-11-14 22:06:36 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 16285, 'fingerprint': '4a6d5a9ad3ac4020bcc5', 'kind': 'shape.degrading', 'severity': 'warning', 'cluster': 'test27', 'subject': 'studiosvc', 'title': 'studiosvc shape 00000000: p95 2.15 s now vs 128 ms earlier (16.8×, 865 runs)', 'facts': {'service': 'studiosvc', 'plan_hash': '00000000', 'queries': 865, 'floor_ms': 1000, 'p95_early': 127.71428571428552, 'p95_late': 2150.0000000000023, 'ratio': 16.834451901566037, 'avg_ms': 44.39537572254335, 'sample': 'MATCH (task:Task) WHERE task.task_number = ? AND EXISTS { (task)-[:BELONGS_TO]->(:Inbox{type:?})-[:MANAGED_BY]->(:Organization) } RETURN task.tildaid LIMIT ?', 'rw': 'write', 'window': [1794653342, 1794674942], 'recent_deploys': [], 'severity_capped': 'critical → warning: test27 is a test-tier cluster', 'discriminator': '00000000'}, 'evidence': [{'label': 'Query shape', 'href': '/neo/shape/00000000?cluster=test27&from=1794653342&to=1794674942'}, {'label': 'Degrading shapes', 'href': '/neo/queries?cluster=test27&from=1794653342&to=1794674942&order=degrading'}], 'provenance': {'source': 'rollup', 'collected_at': 1794674940, 'age_seconds': 7, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 29.9, 'status': 'resolved', 'first_seen': 1793896422, 'last_seen': 1794674942, 'resolved_at': 1794693996, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 8, 'narrative': 'A studiosvc query shape on test27 has p95 2.2 s in the second half of the window against 128 ms in the first (16.8×, 865 executions). The shape reads: “MATCH (task:Task) WHERE task.task_number = ? AND EXISTS { (task)-[:BELONGS_TO]->(:Inbox{type:?})-[:MANAGED_BY]->(:Organi…”. Slower can mean more data, a lost index, or lock contention — the shape page has the CPU/wait split. Shapes whose p95 stays under a second are never reported: below that nobody notices, whatever the ratio.', 'narrative_kind': 'template', 'discriminator': '00000000', 'level': 'low', 'told_level': None, 'episode_at': 1794672489, 'chat_thread': None, 'kind_label': 'Query shape degrading', 'group': 'Neo4j', 'age_s': 797579, 'stale_s': 19059}

Check it yourself

Every link carries the finding's own time window, so it shows tomorrow what it shows now.

Actions

Sign in (IAP) to acknowledge, snooze or resolve.

Where the numbers come from

sourcerollup
collected_at1794674940
age_seconds7
staleFalse
shownNone
totalNone
truncatedFalse
coverage_fromNone
retain_daysNone
noteNone
detectorshapes