What Leo saw at Sourcegraph, 3. The upgrade done by hand, again and again
Sourcegraph upgraded each hosted customer by hand, with a short outage every time. Leo laid out three ways forward: script it, keep each customer's settings apart, record the change first. By November 2022 all three were done.
We gave Leo Sourcegraph's public handbook exactly as it stood on 31 December 2021, and nothing later.
Sourcegraph had no part in this
We used Sourcegraph's handbook because it is public and openly licensed (Apache 2.0). Sourcegraph did not take part, and nothing here is Sourcegraph's view of Leo. Every quote below links back to Sourcegraph's own pages and commits.
The claims, and where to check them
| Claim | Where to check it |
|---|---|
| Each hosted customer instance was upgraded by hand through the same long sequence, with a short outage each time: about 8 minutes 40 seconds of downtime in one step, and about 1 minute 35 seconds of errors in another | Sourcegraph's managed instance pages, 31 Dec 2021 (the upgrade page at that commit) |
| On 1 November 2022 the upgrades became an automated workflow that runs in batches | Sourcegraph's commit 174eb5b2 |
| By 2024 a one click action could upgrade every instance, with pull requests created automatically | Sourcegraph's cloud upgrade page, 2024 |
The work as it was
Some of Sourcegraph's largest customers did not run the software themselves: Sourcegraph ran a private copy for each of them. Every release, each of those copies was upgraded by hand, through the same sequence: a banner to warn users, read only mode, a snapshot, a second instance, the upgrade, the switch, the clean up, and a pull request to record the change. Each run meant a short outage: "around 8m40s of downtime" in one step, errors "For 1m35s" in another.
What Leo saw
Leo joined the upgrade guide with the release and infrastructure pages, and saw the same hand routine repeated for every customer, every release, and each customer's settings merged by hand into the shared configuration.
The ways forward
- Improve: script the sequence. The whole routine runs as one supervised job, instead of by hand.
- Redesign: keep each customer's settings apart. An upgrade replaces the shared base without merge conflicts.
- Redesign: the record before the change. The configuration change is merged first, so nobody starts the next upgrade from the wrong state.
What happened next
- 1 November 2022. "finalize v1.1 bulk upgrades workflow." The upgrade became an automated workflow, run in batches, opening a pull request for each instance to be reviewed and merged together. Each customer's configuration now pointed at a shared base file.
- 2023 and 2024. A one click action could upgrade every instance. The hand route stayed only as a fallback.
All three of Leo's ways, in substance, were taken: the sequence scripted, the settings kept apart, the record made automatically. It took Sourcegraph 305 days from the handbook Leo read.
What this does not show
The handbook shows the routine and its replacement, not how many hours it saved, or what else the team built along the way. The count of instances on Leo's own page could not be traced to a handbook page, so it is left out here.
Leo learns how a business really runs by reading what its people write and talking with them, lays out the ways the work could go, side by side, and keeps the choice with the people who make it. When they take a path, its card records it, including one Leo did not offer.