Xray
daily rollups 1h 6h 24h 48h 3d 5d 7d 14d 30d ‹
Last 5 days▾
Ends now. Anything up to this cluster's 30-day retention.
Start and end times
Cancel
Interpreted in UTC (change it at the right of this bar). Earliest available: 2026-10-19 11:47:15.
› →|
updated 1d ago
Sample data. This database was generated by scripts/seed_dev_db.py — every service, deploy, person and figure on this page is synthetic. Names are sample.*@seed.invalid and image tags are prefixed SEED-. Nothing here happened.

Releases

What changed in X-ray, and which build you are looking at right now. X-ray reads live infrastructure, so when a number looks wrong the first question is whether the fleet changed or the dashboard did — this page answers the second half.

running versionv3.9.21
commitdev
built—
sourcelocal checkout

This instance is running from a source checkout, not a released image — the commit is read from git and there is no build timestamp. A deployed build stamps both at image build time.

v3.9.21 2026-11-08 you are here
X-ray no longer sends anyone a Google Chat DM: findings go to the alert channels

Changed

  • Owner alerts in Google Chat DMs are off. No first alerts, reminders, escalations, "resolved" notes, first looks or deploy notices reach anyone's DM; findings still post to the alert channels (prod-notifications, dev-notifications) as their alerts say. In test mode every DM had gone to one person: 3,263 in the last 7 days, 2,487 of them reminders.
  • Messages that were already queued when DMs were switched off are cancelled, not delivered late. OBS_DM_MODE=test or live switches them back on; unset now means off.
v3.9.20 2026-11-07
Ask answers show their evidence: charts, tables, timelines and buttons

Added

  • An Ask X-ray answer can carry a view under its prose: KPI cards, line and bar charts, donuts, heatmaps, a trace waterfall, before-and-after comparisons, sortable tables, status grids, timelines and checklists, in X-ray's colours in both themes, filling in while the answer streams on /ask and in the dock. The prose still leads and cites; numbers in a view are checked against the tools like every other number, and the PDF export reads a view as text.
  • A view's buttons only ever ask. A follow-up question or a choice is sent as your next message, in the chat for everyone to see; an approval shows the queued action's live status and takes you to its approve/reject card, which is still the only place a write is approved.
  • /ask/views draws 60 example questions exactly as Ask draws them, each with an "Ask it live" link. OBS_ASK_GENUI=false stops Ask from drawing views; answers that already have one still show it.

Fixed

  • A service's endpoint queries page (/service/<service>/queries) works again. Since 3.9.14 it failed with a server error: the coverage ledger's new per-endpoint call count had taken the name of the count that page reads.
v3.9.19 2026-11-07
The test agent also waits while the automation suites or a cluster reset run in Jenkins

Changed

  • While api-tests-master, api-tests-production, ui-tests-master, ui-tests-production, Reset To Master or Reset To Production is building or queued, the test agent reads no Ready ticket. Jenkins is asked on every poll, as the Evals tab's Jenkins bot; if it can't be read the agent waits. OBS_TEST_AGENT_GATE_JENKINS_JOBS changes the list; empty stops the check. A run already started is not stopped.
v3.9.18 2026-11-06
The test agent waits for master images on the automation cluster

Changed

  • With OBS_TEST_AGENT_GATE_K8S_URL set, the test agent reads no Ready ticket while any Tilda service on the automation cluster runs a production image, or while the cluster can't be read. Runs that ended are still reported and open pull requests still followed. Off until that setting is given.
  • The queue is oldest first, as documented: Linear lists newest first, so each new ticket jumped every older one waiting in Ready for Development.
v3.9.17 2026-11-06
The test agent reads the user guide for the feature it is testing

Changed

  • A run's GitHub token also covers tildabio/whats-up-doc; the runner clones the Academy's Markdown beside the suite and the ticket's pack names the guide page, which the agent reads before the handlers.
v3.9.16 2026-11-06
A blocked test-agent run leaves a draft pull request

Changed

  • The work and the questions of a run that stopped are a draft pull request linked on the ticket; the answered run reworks it in place and, once green, marks it ready for review.
v3.9.15 2026-11-06
A blocked test-agent run keeps its work

Changed

  • When the agent stops with questions, the runner pushes the branch (no pull request) and the ticket comment names it; the next run on that ticket resumes from the branch instead of starting over.
v3.9.14 2026-11-06
The coverage ledger ranks gaps by production traffic

Changed

  • GET /api/v2/endpoint-calls: every endpoint's request count over a window of days, from the rollups, keyed the way the coverage ledger keys a route. The nightly test-gaps CronJob reads it for cluster production, so a gap on a route customers hit 50,000 times a week outranks one nobody calls.
v3.9.13 2026-11-05
The test agent's Jobs can reach what the Jenkins test pods reach

Changed

  • A test-agent Job runs as the api-tests Google identity (Workload Identity) and mounts the Google Calendar key, so the suites' document uploads and scheduling tests work there as they do in Jenkins. Its environment is the one the Jenkins pods get, declared in ash-nazg beside the dashboard's secrets.
v3.9.12 2026-11-05
The test agent: coverage tickets become verified API and UI tests

Added

  • A test agent runs beside the docs agent. A ticket in Linear's Test Agent Automation project in Ready for Development, with the coverage ledger's yaml block or written in words, becomes one Kubernetes Job that writes the test in the suite, runs it against the automation cluster and opens the pull request only when that run is green. Questions and failures go back on the ticket.
  • The coverage ledger (tools/test-sync-agent) reads the served OpenAPI contract and the suites' coverage logs, ranks the uncovered routes and screens, and files the tickets the agent works from.

Versions are judged by effect on the reader, not by how much code moved: major when a page is gone, moved or now means something different; minor for a new page or a new answer on an existing one; patch for a fix to something the page already claimed to do. The same notes are in CHANGELOG.md, generated from the same source so the two cannot disagree.