What Leo saw at Apache, 2. One security team, dozens of teams to chase
In 2019 Apache's Security Team chased projects by hand every month, with 7 to 13 issues over a year old at any time. Leo heard from the team and 30 project leads and laid out three ways forward. In 2023 the same chase was still running.
We gave Leo the Apache Software Foundation's 2019 board minutes, with the board, its officers and each of 211 project chairs taught to it as people of their own. This is the second thing it saw that needed many of them to see.
The Apache Software Foundation had no part in this
We used the foundation's board minutes because they are public. The ASF did not take part, and nothing here is the ASF's view of Leo. Every quote below links back to the minutes it came from.
The claims, and where to check them
| Claim | Where to check it |
|---|---|
| In 2019 the foundation's Security Team handled about 33 security reports and 25 other emails a month, and every month carried 7 to 13 issues older than a year | The Security Team's reports in the 2019 board minutes (archive) |
| One project's fixes were disclosed only after "numerous contact attempts and even a board escalation" | The 2019 minutes |
| In 2021 the team told the board unresponsive projects "can lead to users being exposed" | Minutes, 17 Feb 2021 |
| In 2023 a project resolved a security issue "open for over a year" by deciding it was not a vulnerability | Minutes, 20 Sep 2023 |
The work as it was
Apache has one Security Team for more than 200 projects. Reports of vulnerabilities come to it; it passes each one to the project concerned, and then waits for that project to fix it and tell its users.
In 2019 that was about 33 real security reports a month, plus about 25 emails to the security address that were licence questions or requests for support. And every month, between 7 and 13 issues were more than a year old. The team chased projects with monthly reminders. For one project, it took "numerous contact attempts and even a board escalation" before fixes already in its code were announced to the people using it.
What Leo saw
Leo joined the Security Team's monthly reports with what 30 project chairs said about their own security work. From the centre, a team chasing by hand. From the projects, a mix: some caught vulnerable libraries automatically in every build, most found them when a researcher wrote in.
The ways forward
- Improve: a drafted reply for the misdirected mail. If the licence and support questions repeat, answers can be drafted, taking about 25 emails a month off the team.
- Redesign: an alert that escalates by itself. Instead of monthly reminders, an open security issue that passes an agreed age raises an alert that steps up on its own, toward the board.
- Redesign: check in every build. Vulnerable dependencies are caught continuously in each project's build, as two projects already did, rather than when someone reports them.
What happened next
- 2021. The Security Team told the board the problem stood: "Where projects are unresponsive it reflects badly on the ASF and can lead to users being exposed when reporters choose to go public without a co-ordinated fix."
- 2022. A single vulnerability in one widely used library brought a flood: "This event triggered a large number of enquiries and vulnerability reports." The team worked directly with the project.
- 2023. The pattern was still there. One project resolved "a security issue that has been open for over a year by deciding it was not a vulnerability, after much sporadic back-and-forth."
On Leo's card, this one stays not decided yet. The record shows no alert, no escalation by age, and no foundation-wide checking in builds. The choice is the foundation's to make, and four years later the ways Leo laid out are still on the table.
What this does not show
The minutes record what the team reported, not everything it built or tried. A volunteer foundation cannot order projects to act; it can only ask, which may be exactly why the chasing is by hand. What the record shows is the same chase, month after month, for four years.
Leo learns how an organisation really runs by hearing from each of its people, then lays out the ways the work could go, side by side, and keeps the choice with the people who make it. When they take a path, its card records it, including one Leo did not offer.