What Leo saw at Zulip, 1. A review playbook from nothing, in 73 minutes
Zulip used review labels on its pull requests for three years before anyone wrote down what they meant. Leo read 20 people's chat and GitHub, and nothing else, and wrote the playbook in 73 minutes for US$29.75, with some things the written one still leaves out.
The worry we hear from people thinking about Leo is the setup: that it needs hundreds of hours of somebody explaining the business before it is any use. So we tried it on an organisation that never explained itself to Leo at all.
Zulip, the open source team chat, is built by a small core team and hundreds of volunteers. They work in three places: a public chat at chat.zulip.org, GitHub issues, and GitHub pull requests. Every pull request goes through review before it is merged, and by 2023 the review had stages, marked with labels, that everybody used and nobody had written down. Zulip wrote the page on 24 May 2023.
We gave Leo what the 20 most active people wrote in the 13 weeks before that day: their chat messages, their issues, their pull request comments and reviews, and what they did on GitHub besides writing, each person's own words, by role and number, never by name. We gave it none of Zulip's documentation.
Zulip had no part in this
Zulip builds in the open: its chat at chat.zulip.org, its issues and pull requests on GitHub. We read only channels anyone can read without signing in, through Zulip's own public interface, and its GitHub record. Zulip did not take part, and nothing here is Zulip's view of Leo. People are taught by role and number and named nowhere. Quotes are kept to a phrase and link back to the source.
The claims, and where to check them
| Claim | Where to check it |
|---|---|
| Zulip's page on the review process for pull requests first appeared on 24 May 2023 | Commit afc016c0, PR #25749 |
| In its first year it took 4 commits by 2 people, the last on 3 May 2024 | git log --follow docs/contributing/review-process.md |
| The "integration review" label was in use on pull requests by May 2020 | PR #15086 carries it; search for the label |
Before the page, CONTRIBUTING.md had a short "Stages of a pull request" section, written for the contributor, with no labels |
CONTRIBUTING.md at commit b3196e53 (23 May 2023) |
| Between 24 Feb and 23 May 2023, 738 of the 777 pull request merges on GitHub were by one maintainer | GitHub's own merge events for zulip/zulip in those dates (the timeline of each pull request) |
| "integration review" was added to pull requests 276 times in those 13 weeks, "maintainer review" twice and "QA needed" once | GitHub's label events for zulip/zulip, the same dates |
| The labels "help center review" and "api docs review", named on Zulip's page, were first used after the cut off | api docs review (June 2023), help center review (September 2023) |
| The teaching took 73 minutes and cost US$29.75; the second look US$0.58 | Leo's own run log for this test |
The time it took them, and the time it took Leo
| Zulip | Leo | |
|---|---|---|
| Review labels in use before the page | Three years at least ("integration review" by May 2020) | |
| Writing it | One pull request (205 lines added across 5 files), then 4 commits by 2 people over the next 11 months | One teaching run, 73 minutes, three channels joined |
| People's time | The maintainers who wrote and reviewed it | None. Nobody at Zulip answered a question |
| Money | US$29.75 for the whole community, which wrote 27 process pages; US$0.58 more for the ways forward |
The playbook Leo wrote
Leo's page for pull request review and merge is joined from 19 of the 20 people: 6 maintainers and 13 contributors. Each line says whose words it came from. The project lead's own flow, as Leo drew it:
- A contributor opens a pull request.
- Review is routed to someone who knows the area; a first reviewer reviews, and tags it "integration review" when happy.
- Integration review: code, and the house rules (translation tags, migrations, one idea per commit, the API changelog).
- The contributor revises.
- The maintainer fixes the small things, lint and merge conflicts, themselves, or pulls a part out.
- Changes to the interface get product review, testing by hand, or a test deploy on chat.zulip.org.
- Merge once the tests pass. A regression is reverted, and the contributor opens a new pull request.
What Leo wrote that Zulip's page later says
| Zulip's page, 24 May 2023 | Leo's page, from 24 Feb to 23 May 2023 |
|---|---|
| Code may first be reviewed by other contributors, then a maintainer reviews | "Review runs in two tiers: an experienced contributor reviews and manually tests another contributor's pull request, then adds the integration review label to hand it to a maintainer" |
| Integration review is the final round, generally done by the project lead | One maintainer merged 738 of the 777 pull requests in the window; Leo counted about 59 a week from that maintainer's own record |
| Product review: seeing a first version can change the design | One maintainer does product review by hand, in light and dark theme, at narrow widths, and comments with screenshots before integration review |
| Labels mark the stage; "maintainer review" and "QA needed" among them | "Labels route a pull request to the next reviewer", naming "integration review", "chat.zulip.org review", "maintainer review" and "QA needed" |
| Pass the tests before asking for review; post "Ready for review!" when a draft is ready | Contributors point newcomers to "CI should pass before asking for review"; drafts are marked ready with a comment listing what is still pending |
What Leo saw that the page never wrote down
- One person at the end of the line. 95% of merges went through one maintainer. Zulip's page says integration review is "generally done" by the lead; it does not say what that means for waiting. Leo's page holds the waits: a pasting fix with four reminders in seven weeks, a docs fix closed by its author 55 days after opening, a stuck pull request raised in chat three at a time.
- The chat.zulip.org review. Changes to the look of the app get their own label and go out to Zulip's own chat server to be lived with before merging. The page never names the label.
- The maintainer finishing the job. When a pull request is close, the maintainer often does the trivial rebase and the small fixes and merges, instead of sending it back.
- What reviewers say every time. Missing translation tags are the commonest comment in review, with no automatic check; commit message format comes back about four times a week from one reviewer alone.
The ways forward Leo laid out
- Redesign: a shared review queue built from the labels and how long each pull request has waited, so whoever has time picks up the oldest, instead of authors pinging a named reviewer each round.
- Improve: check commit messages and translation tags before a person reviews.
- Eliminate: the maintainer sets the version bump at merge, so contributors stop rebasing for it while they wait.
What the page has that the chat never showed
Zulip's page names two documentation review labels, "help center review" and "api docs review". Neither had been used when Leo was taught: they first appear in June and September 2023. The page also gives a rule of a week before nudging a quiet pull request. Leo saw people nudge, but no rule. A playbook drawn from what people do holds what they did, not what the team then decided they should do.
What this does not show
- Zulip's docs are public, and the model behind Leo may have read them in training. Leo was not given them, every line above names the person whose words it came from, and we checked the lines quoted here against those words. The parts Zulip never wrote down cannot have come from its docs.
- Leo drew five flows on this page, one per person who described the work, rather than one joined flow. The joining is still a reader's job.
- It shows Leo writing a playbook. It does not show anyone at Zulip using it, or that it made review faster.
Leo learns how a business really runs by reading what its people write and talking with them, lays out the ways the work could go, side by side, and keeps the choice with the people who make it. When they take a path, its card records it, including one Leo did not offer.