What Leo saw at Kafka, 2. The red build nobody owned
In 2019 Kafka's developers filed 193 tickets for tests that fail for no reason, and the board reports never mentioned it. Leo drew the work from twelve developers' mail and JIRA. In 2024 Kafka adopted a quarantine for flaky tests, one of the ways Leo had laid out.
Every software team knows the test that fails for no reason. Your change is fine, the build goes red anyway, and before your work can merge somebody has to decide whether the failure is real. Kafka's board reports never mentioned it. Kafka's developers wrote about it almost every week.
Where this comes from
Apache Kafka's dev mailing list and its JIRA are public, kept by the Apache Software Foundation. The foundation and the Kafka project did not take part, and nothing here is their view of Leo. People are taught by role and number and named nowhere.
The claims, and where to check them
| Claim | Where to check it |
|---|---|
In 2019, 193 Kafka JIRA issues were opened for flaky tests (titled "flaky" or labelled flaky-test); 121 were still open at the end of the year |
Kafka's JIRA, issues created in 2019; Leo's own run log for this test |
| One developer opened 82 of them between 12 Feb and 21 Apr 2019, while running a release | The same JIRA, filtered by reporter and date |
| 65 of the 193 are still unresolved in Kafka's JIRA in October 2026 | The same JIRA, read on 19 Oct 2026 |
| A developer offered on 28 Feb 2019 to contribute a rerun of failed tests, and opened it as KAFKA-8164 on 27 Mar 2019 | The dev list archive for March 2019; KAFKA-8164 |
| Kafka merged a rerun of failed tests on 6 Feb 2020 and documented it in its README as "Specifying test retries" | KAFKA-8164, pull request 8019 |
| In September 2024 Kafka adopted a quarantine for flaky and newly added tests, giving its reasons | KIP-1090 Flaky Test Management, created 15 Sep 2024; KAFKA-17712 |
| Leo was taught nothing after 31 Dec 2019 | Every file taught is dated 2019; Leo's own run log for this test |
The work as it was
When a pull request build failed on a test the change had not touched, the developer looked in JIRA for a ticket about that test. If one existed, they added a comment by hand: "failed again", the build link, often the stack trace. If not, they opened a new one. In 2019 that made 193 tickets, and a great many comments. One developer opened 82 of them in ten weeks while trying to get a release out, because, in their words, approved pull requests could not merge without a green build.
The same test was often filed twice. Fixes often did not hold. One developer raised timeouts when a slow machine was to blame; another's rule was to make the test wait for the thing it depends on, "never by lengthening the timeout". Both lines sit side by side on Leo's page, from different people.
What Leo saw
Nobody at Kafka had a page for this. Leo made one from twelve people's mail and comments: how a failure is reported, how a fix is checked (one test was looped 20,000 times before its ticket was closed), when a test may be switched off ("Disabling a test should come with a blocker JIRA for the next release"), and how often a test filed as flaky turned out to be hiding a real bug. It also drew the steps as a flow, with where it hurts: "Done by hand each time: 25 comments in six weeks".
The ways forward
- Improve: report the failures by machine. When a pull request build fails, a job reads the test report, matches each failed test to its open ticket and posts the link, so nobody types "failed again".
- Redesign: take known flaky tests out of the gate. Known flaky tests run in a separate tracked job, so a red pull request build means the change, and the flaky list is worked as its own queue.
- Eliminate: nobody triages unrelated failures at all. Known flaky tests stay out of the pull request build until fixed.
What happened next
- 6 Feb 2020. Kafka made pull request builds rerun a failed test before failing, the change one of the developers had proposed in March 2019. Leo had that proposal on its page, but not among its ways forward. On the card it is recorded as a path Leo did not offer.
- 2021 to 2023. The flaky tickets kept coming. 65 of the 193 opened in 2019 are still unresolved today.
- 15 Sep 2024. Kafka's KIP-1090 put known flaky tests, and newly added tests, into a quarantine that still runs but is reported separately. Its reasons, as written: "The presence of flaky tests erodes confidence in our test results. Re-running all tests up to 3 times can mask problems. New tests are unknowns and frequently the source of flaky failures." On the card this matches Leo's redesign, seen in action, with Kafka's reason in its own words.
KIP-1090 also says something Leo had heard in 2019: "There have been several occasions where Apache Kafka developers have committed a fix for a flaky test only to find that they have simply reduced the flakiness." Leo's page, from 2019 mail: "Several first fixes did not hold and needed a second."
What this does not show
- It does not show that Leo's card would have got Kafka to a quarantine sooner than five years. Rerunning failed tests was a fair choice for 2020; it was cheap and it made builds go green. KIP-1090 names its cost.
- One of Leo's other redesigns, run new tests many times before they merge, is the approach KIP-1090 considered and set aside as "very ad-hoc and difficult to enforce".
- The counts are from Kafka's JIRA as it reads today. A ticket's title or labels can have been edited since.
Leo learns how a business really runs by reading what its people write and talking with them, lays out the ways the work could go, side by side, and keeps the choice with the people who make it. When they take a path, its card records it, including one Leo did not offer.