Xray
1h 6h 24h 48h 3d 5d 7d 14d 30d ‹
Last 1 hour▾
Ends now. Anything up to this cluster's 30-day retention.
Start and end times
Cancel
Interpreted in UTC (change it at the right of this bar). Earliest available: 2026-10-19 13:01:26.
› →|
updated 1d ago
Sample data. This database was generated by scripts/seed_dev_db.py — every service, deploy, person and figure on this page is synthetic. Names are sample.*@seed.invalid and image tags are prefixed SEED-. Nothing here happened.

Neo4j schema test27

Every Go service creates the constraints and indexes for the labels it owns when it boots, by reading the live schema (main #16243). A constraint the data violates does not fail the boot: a standby _idx index stands in and the object reads DEGRADED until the rows are fixed — often by a data migration that needs a person. This page shows what each service found on its latest boot, what is blocking each degraded object, and how the clusters compare. UNEXPECTED objects (in the database, not in the model) are informational — usually an older key waiting on its two-release retirement — and never make a service unhealthy.

Last refreshed never · read every 6 h — only the minutes around each service start
Services healthy
11 / 14
3 carry a degraded, drifted or failed object
Objects not in place
6
6 degraded · 0 drift · 0 failed
Migrations waiting on a person
0
0 failed · 0 blocked · 0 pending
Objects in place
687
7 created on the latest boots · 52 undeclared (informational)

Newest boot 34 h ago. Convergence runs only at boot, so a report is what the service found when it last started — data fixed since then shows on its next boot.

Services 14

One row per owning service, unhealthy first. The bar is every object the service owns, by what convergence found.

presentcreatedretireddegradeddriftfailedunexpected
servicestateobjectsnot in place data migrationsreportedearlier boots
workplansvc unhealthy
36 objects
3 degraded 3 undeclared
2 / 2 applied · counted
2 applied on earlier boots (counted, not listed)
39 h ago
boot · pod cltkx
studiosvc unhealthy
109 objects
2 degraded 19 undeclared
10 / 10 applied · counted
10 applied on earlier boots (counted, not listed)
41 h ago
boot · pod l6x49
execsvc unhealthy
194 objects
1 degraded 7 undeclared
8 / 8 applied · counted
8 applied on earlier boots (counted, not listed)
34 h ago
boot · pod k54gl
advpsvc healthy
36 objects
— none declared 14 days ago
boot · pod j7s65
budgetsvc healthy
8 objects
— none declared 8 days ago
boot · pod zwphf
docsvc healthy
100 objects
20 undeclared
6 / 6 applied · counted
6 applied on earlier boots (counted, not listed)
39 h ago
boot · pod 22fkg
enrollsvc healthy
30 objects
— none declared 2 days ago
boot · pod fxbfc
finsvc healthy
27 objects
— none declared 14 days ago
boot · pod qrc22
notifsvc healthy
23 objects
— none declared 14 days ago
boot · pod 5k9ps
onboardingsvc healthy
80 objects
3 undeclared
3 / 3 applied · counted
3 applied on earlier boots (counted, not listed)
2 days ago
boot · pod cwwhk
patientsvc healthy
47 objects
— none declared 7 days ago
boot · pod 2p6jf
regsvc healthy
22 objects
— none declared 7 days ago
boot · pod bnpbr
schedsvc healthy
15 objects
— none declared 13 days ago
boot · pod lgzdg
tusker healthy
18 objects
—
1 / 1 applied · counted
1 applied on earlier boots (counted, not listed)
7 days ago
boot · pod w5vq8

Objects not in place 6

Each one, with how much of its data violates the key, the migration that would heal it, and whether other clusters have the same problem — degraded on production as well usually means the declared key is wrong, not the data.

Training_activityId_name_orgId_scopeId_status_deltimestamp

DEGRADEDexecsvc
:Training activityIdnameorgIdscopeIdstatusdeltimestamp
2 duplicate keys over 5 nodes — 1.5% of the label
standby index "Training_activityId_name_orgId_scopeId_status_deltimestamp_idx" still stands in for the constraint: 0 rows with a null key property (of 333), 2 duplicate keys over 5 nodes
Next: no migration needing attention targets it; 8 applied one(s) are not listed by the boot log — check whether one of those was meant to fix it. Not degraded on any other cluster with a report.

Media_archivedtimestamp_name_orgid_scopeid_smoversion_state_version_deltimestamp

DEGRADEDstudiosvc
:Media archivedtimestampnameorgidscopeidsmoversionstateversiondeltimestamp
4 duplicate keys over 8 nodes — 1.43% of the label
standby index "Media_archivedtimestamp_name_orgid_scopeid_smoversion_state_version_deltimestamp_idx" still stands in for the constraint: 0 rows with a null key property (of 560), 4 duplicate keys over 8 nodes
Next: no migration needing attention targets it; 10 applied one(s) are not listed by the boot log — check whether one of those was meant to fix it. Not degraded on any other cluster with a report.

WorkPlanInstance_name_orgid_scopeId_deltimestamp

DEGRADEDworkplansvc
:WorkPlanInstance nameorgidscopeIddeltimestamp
49 of 8,552 rows have a null key property — 0.57% of the label
standby index "WorkPlanInstance_name_orgid_scopeId_deltimestamp_idx" still stands in for the constraint: 49 rows with a null key property (of 8552), 0 duplicate keys over 0 nodes
Next: no migration needing attention targets it; 2 applied one(s) are not listed by the boot log — check whether one of those was meant to fix it. Not degraded on any other cluster with a report.

WorkplanCfg_name_orgid_deltimestamp

DEGRADEDworkplansvc
:WorkplanCfg nameorgiddeltimestamp
1 duplicate key over 6 nodes — 0.35% of the label
standby index "WorkplanCfg_name_orgid_deltimestamp_idx" still stands in for the constraint: 0 rows with a null key property (of 1713), 1 duplicate keys over 6 nodes
Next: no migration needing attention targets it; 2 applied one(s) are not listed by the boot log — check whether one of those was meant to fix it. Not degraded on any other cluster with a report.

WorkPlanType_name_orgid_deltimestamp

DEGRADEDworkplansvc
:WorkPlanType nameorgiddeltimestamp
1 duplicate key over 4 nodes — 0.34% of the label
standby index "WorkPlanType_name_orgid_deltimestamp_idx" still stands in for the constraint: 0 rows with a null key property (of 1193), 1 duplicate keys over 4 nodes
Next: no migration needing attention targets it; 2 applied one(s) are not listed by the boot log — check whether one of those was meant to fix it. Not degraded on any other cluster with a report.

ResponseSet_archivedtimestamp_name_orgid_scopeid_shared_smoversion_state_variablename_version_deltimestamp

DEGRADEDstudiosvc
:ResponseSet archivedtimestampnameorgidscopeidsharedsmoversionstatevariablenameversiondeltimestamp
15 duplicate keys over 33 nodes — 0.02% of the label
standby index "ResponseSet_archivedtimestamp_name_orgid_scopeid_shared_smoversion_state_variablename_version_deltimestamp_idx" still stands in for the constraint: 0 rows with a null key property (of 182357), 15 duplicate keys over 33 nodes
Next: no migration needing attention targets it; 10 applied one(s) are not listed by the boot log — check whether one of those was meant to fix it. Not degraded on any other cluster with a report.

Created and retired on the latest boots 7

What convergence changed. A created constraint that replaced a standby index is a degraded object that healed — its data was fixed and the boot made the real constraint. Retired objects were declared obsolete by their owner and dropped.

objectchangeservicelabelpropertieswhen
WorkplanInstanceEvent_orgId created workplansvcWorkplanInstanceEvent orgId 39 h ago
WorkplanJob_orgid created workplansvcWorkplanJob orgid 39 h ago
Report_orgid created enrollsvcReport orgid 2 days ago
Filter_orgid created enrollsvcFilter orgid 2 days ago
ChartReviewDecision_orgid created enrollsvcChartReviewDecision orgid 2 days ago
ChartReviewList_orgid created enrollsvcChartReviewList orgid 2 days ago
ChartReviewKeyword_orgid created enrollsvcChartReviewKeyword orgid 2 days ago

Every cluster 7

Objects not in place per service and cluster (green = healthy). A problem on one dev cluster is stale data; the same one on production is the model. Click a cell for that cluster.

servicemasterautomationnexusproductiondeltatest27test26
advpsvc · ✓ · · · ✓ ✓
authz · · · · · · 3
budgetsvc · · · · · ✓ ·
docsvc ✓ ✓ ✓ ✓ ✓ ✓ ✓
enrollsvc ✓ ✓ ✓ ✓ ✓ ✓ ✓
execsvc 1 ✓ ✓ ✓ ✓ 1 1
finsvc ✓ ✓ ✓ ✓ ✓ ✓ ✓
notifsvc ✓ ✓ ✓ ✓ ✓ ✓ ✓
onboardingsvc ✓ ✓ ✓ ✓ ✓ ✓ ✓
patientsvc ✓ ✓ ✓ ✓ ✓ ✓ ✓
regsvc ✓ ✓ ✓ ✓ ✓ ✓ ✓
schedsvc ✓ ✓ ✓ ✓ ✓ ✓ ✓
studiosvc ✓ ✓ ✓ ✓ ✓ 2 ✓
tusker ✓ ✓ ✓ ✓ ✓ ✓ ✓
workplansvc ✓ ✓ ✓ ✓ ✓ 3 ✓

Undeclared objects 52

In the database, absent from the data model. Never dropped automatically: each is a pending retirement (declare it in schema_retired.go with its ReplacedBy) or something created outside the model. Not unhealthy.

docsvc — 20
objectlabelproperties
Document_orgId_studyIdDocument orgIdstudyId
Document_orgId_templateDocument orgIdtemplate
Document_visitIdDocument visitId
Document_templateDocument template
Document_studyId_statusDocument studyIdstatus
Document_status_archivedDocument statusarchived
Document_needsAttentionDocument needsAttention
Document_studyId_visitIdDocument studyIdvisitId
Document_patientIdDocument patientId
Document_studyId_archivedDocument studyIdarchived
Document_orgId_archivedDocument orgIdarchived
Document_studyId_patientIdDocument studyIdpatientId
Document_systemTemplateKeyDocument systemTemplateKey
Document_statusDocument status
Document_studyId_userIdDocument studyIduserId
Document_orgId_statusDocument orgIdstatus
Document_studyId_siteIdDocument studyIdsiteId
Document_fileNameDocument fileName
Document_aiEnabled_needsAttentionDocument aiEnabledneedsAttention
Document_blindedAccessDocument blindedAccess
execsvc — 7
objectlabelproperties
Visit_name_nodeId_treeId_deltimestampVisit namenodeIdtreeIddeltimestamp
DocumentSet_name_nodeId_treeId_deltimestampDocumentSet namenodeIdtreeIddeltimestamp
Activity_name_nodeId_treeId_deltimestampActivity namenodeIdtreeIddeltimestamp
Questionnaire_name_nodeId_treeId_deltimestampQuestionnaire namenodeIdtreeIddeltimestamp
Question_name_nodeId_treeId_deltimestampQuestion namenodeIdtreeIddeltimestamp
DocOverlay_docId_name_nodeId_treeId_deltimestampDocOverlay docIdnamenodeIdtreeIddeltimestamp
ParticipantDocUpload_carePlanId_docUploadId_nodeId_deltimestampParticipantDocUpload carePlanIddocUploadIdnodeIddeltimestamp
onboardingsvc — 3
objectlabelproperties
CredentialCheckVersion_orgId_deltimestamp_idxCredentialCheckVersion orgIddeltimestamp
MemberLabelScopeIdScopeTypeIndexMemberLabel scopeIdscopeType
MemberLabelGroupScopeIdScopeTypeIndexMemberLabelGroup scopeIdscopeType
studiosvc — 19
objectlabelproperties
MediaSet_archivedtimestamp_name_orgid_scopeid_smoversion_state_type_version_deltimestampMediaSet archivedtimestampnameorgidscopeidsmoversionstatetypeversiondeltimestamp
TaskList_archivedtimestamp_name_orgid_scopeid_smoversion_state_type_version_deltimestampTaskList archivedtimestampnameorgidscopeidsmoversionstatetypeversiondeltimestamp
TodoV2_state_title_version_deltimestampTodoV2 statetitleversiondeltimestamp
DocumentSetV2_archivedtimestamp_name_orgid_scopeid_smoversion_state_type_version_deltimestampDocumentSetV2 archivedtimestampnameorgidscopeidsmoversionstatetypeversiondeltimestamp
DocReview_orgid_scopeid_state_title_version_deltimestampDocReview orgidscopeidstatetitleversiondeltimestamp
Table_archivedtimestamp_name_orgid_scopeid_smoversion_state_type_version_deltimestampTable archivedtimestampnameorgidscopeidsmoversionstatetypeversiondeltimestamp
Statement_orgid_scopeid_smoversion_state_title_version_deltimestampStatement orgidscopeidsmoversionstatetitleversiondeltimestamp
ControlV2_archivedtimestamp_name_orgid_scopeid_smoversion_state_type_version_deltimestampControlV2 archivedtimestampnameorgidscopeidsmoversionstatetypeversiondeltimestamp
DocOverlayV2_orgid_scopeid_state_title_version_deltimestampDocOverlayV2 orgidscopeidstatetitleversiondeltimestamp
ESign_archivedtimestamp_scopeid_state_title_version_deltimestampESign archivedtimestampscopeidstatetitleversiondeltimestamp
VisitV2_archivedtimestamp_name_orgid_scopeid_smoversion_state_type_version_deltimestampVisitV2 archivedtimestampnameorgidscopeidsmoversionstatetypeversiondeltimestamp
WorkflowV2_archivedtimestamp_name_orgid_scopeid_smoversion_state_type_version_deltimestampWorkflowV2 archivedtimestampnameorgidscopeidsmoversionstatetypeversiondeltimestamp
TaskConfig_orgid_scopeid_state_title_version_deltimestampTaskConfig orgidscopeidstatetitleversiondeltimestamp
CheckList_archivedtimestamp_name_orgid_scopeid_smoversion_state_type_version_deltimestampCheckList archivedtimestampnameorgidscopeidsmoversionstatetypeversiondeltimestamp
PrePostVisitV2_archivedtimestamp_name_orgid_scopeid_smoversion_state_type_version_deltimestampPrePostVisitV2 archivedtimestampnameorgidscopeidsmoversionstatetypeversiondeltimestamp
Trigger_state_version_deltimestamp_scope_operation_orgidTrigger stateversiondeltimestampscopeoperationorgid
ProtocolVersionV2_archivedtimestamp_name_orgid_scopeid_smoversion_state_type_version_deltimestampProtocolVersionV2 archivedtimestampnameorgidscopeidsmoversionstatetypeversiondeltimestamp
ScreenerV2_archivedtimestamp_name_orgid_scopeid_smoversion_state_type_version_deltimestampScreenerV2 archivedtimestampnameorgidscopeidsmoversionstatetypeversiondeltimestamp
DocUpload_orgid_scopeid_state_title_version_deltimestampDocUpload orgidscopeidstatetitleversiondeltimestamp
workplansvc — 3
objectlabelproperties
WorkplanCfg_budgetTemplateIdWorkplanCfg budgetTemplateId
WorkPlanType_budgetTemplateIdWorkPlanType budgetTemplateId
WorkPlanTemplate_budgetTemplateIdWorkPlanTemplate budgetTemplateId