What Leo saw at Zulip, 3. The playbook keeps up when the work changes
When Zulip's summer contributors arrived in 2023, pull request review grew three new stages. Leo's page grew them too, from what people did, in 37 minutes for US$17. Nobody edited it.
The first two posts showed Leo writing a playbook for Zulip's open source community from 13 weeks of what 20 people wrote in chat, issues and pull requests. A written playbook goes stale the day the work changes. So we kept teaching: the same people, the next two months, 24 May to 23 July 2023.
The work did change. At the end of May, Google Summer of Code began, and new contributors arrived with mentors. Pull request review grew new stages.
Zulip had no part in this
Zulip builds in the open: its chat at chat.zulip.org, its issues and pull requests on GitHub. We read only channels anyone can read without signing in, through Zulip's own public interface, and its GitHub record. Zulip did not take part, and nothing here is Zulip's view of Leo. People are taught by role and number and named nowhere. Quotes are kept to a phrase and link back to the source.
The claims, and where to check them
| Claim | Where to check it |
|---|---|
| Between 24 Feb and 23 May 2023 the "maintainer review" label was added to pull requests twice, "QA needed" once, "buddy review" and "mentor review" never | GitHub's label events for zulip/zulip, those dates |
| Between 24 May and 23 Jul 2023: "maintainer review" 107 times, "buddy review" 49, "mentor review" 34, "QA needed" 25, "priority integration review" 11, "api docs review" 8, "documentation review" 6, "help center review" 5 | The same, those dates |
| On three pull requests, buddy review to maintainer review took 27, 22 and 27 days | The label history of #25848, #25883 and #26049 |
| Zulip set up issue templates on 12 to 13 Jun 2023 | PR #26000, "github: Configure templates for filing issues" |
A contributor had asked for exactly that in #feedback on 3 May 2023 |
chat.zulip.org, #feedback, "issues link in description" |
| The two months took 37 minutes and US$16.95 to teach | Leo's own run log for this test |
What changed at Zulip
Review labels added to pull requests, from GitHub's own record:
| Label | 24 Feb to 23 May 2023 | 24 May to 23 Jul 2023 |
|---|---|---|
| integration review | 276 | 220 |
| maintainer review | 2 | 107 |
| buddy review | 0 | 49 |
| mentor review | 0 | 34 |
| QA needed | 1 | 25 |
| priority integration review | 0 | 11 |
| api docs review, documentation review, help center review | 0 | 19 |
What changed on Leo's page
Leo's page for pull request review and merge, before and after, from the brain's own history.
Before (taught to 23 May): "Review runs in two stages. A first reviewer, often a contributor who knows that area, reviews the pull request. Once it looks good, they tag it 'integration review' so a maintainer can integrate it."
After (taught to 23 July), new lines, each from the person who does it:
- "From late May 2023 a Google Summer of Code contributor's pull request goes through three review stages, each marked by a label the contributor sets by zulipbot comment command: 'buddy review' ..., then 'mentor review' ..., then 'maintainer review'."
- "From 'buddy review' to 'maintainer review' took 3 to 4 weeks on the three pull requests that got there by mid July 2023: #25848 ... (27 days), #25883 ... (22 days), #26049 ... (27 days)."
- "The 'QA needed' label was agreed in a chat.zulip.org #general thread in early June 2023. It lets anyone interested test a pull request by hand."
One contributor's flow was redrawn, from 11 steps to 14. Before, it went from "ask a named maintainer or reviewing contributor to review" to adding "integration review". After, it runs buddy review, mentor review, maintainer review, then integration review, with the time between them:
And Leo laid out a new way forward that could not have existed before: the bot could move the stage label when the stage's reviewer approves, instead of the contributor swapping labels and naming the next reviewer each time.
27 of the brain's commits in those two months touched this one page. Nobody edited it by hand.
A wish, and then the thing itself
In the first 13 weeks, Leo had kept one contributor's wish on its issue triage page: a link to file issues, with a template that asks whether the idea was vetted in chat first. Six weeks after that conversation, Zulip set up issue templates. In the two months after the cut off, Leo's page gained it:
- "On 12 June 2023 this maintainer set up issue templates for zulip/zulip (#26000). They steer people towards starting a chat.zulip.org conversation when that fits better than an issue."
- And why, in the team's own words: the templates exist "to encourage folks to ask for help on CZO instead of filing a GitHub issue".
The time it took Leo
| Two months of change | |
|---|---|
| People taught | 18 of the 20 (two were quiet) |
| Characters read | 2,070,105 |
| Time | 37 minutes |
| Cost | US$16.95 |
| Staff hours | None |
What this does not show
- Leo kept the old and the new side by side. The "two stages" line from before is still on the page, next to the three GSoC stages. Every line carries the date it was taught, not the date of the work, so a reader cannot tell at a glance which is current. The wish for issue templates still sits under "how they want it to work", although the same page now says the templates were set up.
- By then Zulip had written its own review page (24 May 2023), and people may have been following it. This post claims only that Leo's page followed what people did; it says nothing about the written one.
- The model behind Leo may have read Zulip's public docs in training. Every line above names the person whose words it came from, and the label counts are GitHub's own.
Leo learns how a business really runs by reading what its people write and talking with them, lays out the ways the work could go, side by side, and keeps the choice with the people who make it. When they take a path, its card records it, including one Leo did not offer.