What Leo saw at Zulip, 2. How a report in chat becomes a tracked issue

Zulip, the open source team chat, is built by a small core team and hundreds of volunteers who talk in public at chat.zulip.org. When someone hits a bug or wants a feature, they say so in a chat thread. Some of those threads become GitHub issues. For seven and a half years nobody wrote down how.

We gave Leo what the 20 most active people in that community wrote between 24 February and 23 May 2023: their chat messages, their issues, their pull request comments and what they did on GitHub, each person's own words, by role and number, never by name. We gave it none of Zulip's documentation.

The claims, and where to check them

Claim	Where to check it
Zulip's #issues channel has run since 28 Dec 2015	chat.zulip.org, #issues, its first message
Zulip's written page on managing bug reports, and its page on suggesting features, first appeared on 12 Jun 2023	Commit 052a109b (bugs) and commit 9ee5a5a7 (features), both in PR #25998
Before that, the written word was a six point checklist of what to put in a report, and a paragraph on where to post	docs/contributing/bug-reports.md and the "Reporting issues" section of CONTRIBUTING.md, at commit b3196e53 (23 May 2023)
In their first year the two pages took 5 commits each, by 4 people	git log --follow on each page, 12 Jun 2023 to 12 Jun 2024
On 2 May 2023 the project lead wrote in #feedback that for feature requests the team prefers to file the issues itself, after discussing the feedback	chat.zulip.org, #feedback, "issues link in description"
On 13 Jun 2023 the maintainer who wrote the pages said it "ended up being quite a project", for want of documentation to point people to	The same thread (after the cut off: never taught)
On 11 Apr 2023 one contributor went through 16 stale #issues topics between 11:09 and 12:13 UTC, asking whether each needed an issue	chat.zulip.org, #issues, that day's messages
The teaching cost US$29.75 and took 73 minutes	Leo's own run log for this test

The time it took them, and the time it took Leo

	Zulip	Leo
The work running before it was written down	#issues since December 2015, #feedback since June 2017
Writing it	A chat thread on 2 May 2023, then three pull requests opened on 12 June, then 5 commits by 4 people over the following year	One teaching run, 73 minutes
People's time	The maintainer who wrote it: "quite a project"	None. Nobody at Zulip answered a question
Money		US$29.75 for the whole community, which wrote 27 process pages. This is one of them

The playbook Leo wrote

Leo's page for issue triage is joined from 17 people: 6 maintainers and 11 contributors. Its flow, drawn from one maintainer's work, runs:

1. Notice a problem, raise it in a chat thread in #issues, and settle the fix there.
2. File a GitHub issue with an example, the behaviour wanted, and a link back to the thread.
3. Add an area label, "help wanted", and a priority when it matters; post the issue number back in the thread.
4. A contributor claims it; questions are pointed to the guide and to chat.
5. A pull request, a review, a merge.

What Leo wrote that Zulip's page later says

Zulip's page of 12 June 2023 added a section called "Managing bug reports". Set side by side:

Zulip's page, 12 Jun 2023	Leo's page, from 24 Feb to 23 May 2023
Cross link the issue and the conversation	Each issue a maintainer opened "links the chat.zulip.org thread where it was first discussed"; contributors file issues "linking back to the thread"
If a bug discussed in chat is confirmed, anyone can help by filing a GitHub issue	Contributors file the issue themselves when a thread settles on a change not being done straight away
Use Zulipbot to add "bug" and at least one area label	"Contributors label issues by commenting to zulipbot", with an area label
Add "help wanted" if appropriate; sometimes not, for example if a core contributor should take it	"An issue with an unresolved blocker is not marked help wanted"
A report on GitHub that is not clear enough is sent to chat to be discussed	A support question filed as an issue was redirected to chat.zulip.org; the issue tracker "is not a support channel"

And on features, Zulip's page says the product team usually files the issue once a discussion leads to a concrete proposal. Leo had it in one line, from the project lead's own words in chat: "Feature requests are usually filed as GitHub issues by the maintainers themselves, after discussion, rather than by the users asking for them."

What Leo saw that the page never wrote down

• The sweep. On 11 April 2023 one contributor went through 16 stale #issues topics in about an hour, asking of each whether it needed an issue. Two became work within nine days. They did it again on 9 May. Zulip's page says nothing about stale threads. Leo's way forward: watch the topics continuously, and nudge a topic with no issue, no pull request and no resolved mark after a set time.
• Who holds an issue. In one contributor's twelve weeks there were at least 12 times somebody sorted out by hand who held an issue, including newcomers asking for issues whose pull request was already in review. Leo's way forward: the bot answers with who holds it and points them to "help wanted".
• The idea that started the page. In #feedback on 2 and 3 May a contributor asked for a link to file issues, with a template that asks whether the idea was vetted in chat first. Leo kept it as how they want it to work. That thread is where Zulip's pages and its issue templates came from, six weeks later.

What the page has that the chat never showed

Zulip's page tells people to report security issues by email, never in public; to email support for anything private; to give the version of the app they were using; and that for a minor bug an issue alone is enough. None of that came up in the 13 weeks Leo read, and none of it is on Leo's page. A playbook drawn from what people do only holds what they did in the weeks it was given.

What this does not show

• Zulip's docs are public, and the model behind Leo may have read them in training. Leo was not given them, and every line on its page names the person whose words it came from; we checked the lines above against those words. But a model that knows the docs may lean towards their shape. The parts Zulip never wrote down, the sweep and the claim conflicts, cannot have come from the docs.
• Some of the page existed before the cut off: where to post, and what to put in a report. We compared only what the June 2023 page added.
• It shows Leo writing the playbook. It does not show anyone at Zulip using it.

Leo learns how a business really runs by reading what its people write and talking with them, lays out the ways the work could go, side by side, and keeps the choice with the people who make it. When they take a path, its card records it, including one Leo did not offer.