Formed in 2009, the Archive Team (not to be confused with the archive.org Archive-It Team) is a rogue archivist collective dedicated to saving copies of rapidly dying or deleted websites for the sake of history and digital heritage. The group is 100% composed of volunteers and interested parties, and has expanded into a large amount of related projects for saving online and digital history.
A year after starting up its bug bounty program GitHub is showing how complementary bug bounties can be to DevOps practices. Essentially crowdsourcing vulnerability hunting by systematically paying independent researchers prizes for flaws they find, bug bounties are in a sense an extension of the DevOps ethos. They offer a means of continuously delivering application security assurance.
“The advantage of having a community that is just working 24/7 worldwide is it does tend to be a more continuous approach to vulnerability hunting as opposed to the traditional method of hiring out a consultancy or third party to spot check at some point in time,” explains Shawn Davenport, vice president of security for GitHub.
And according to GitHub’s security leaders, their program has become one of the most effective and cost-conscious tools they have for extending internal appsec resources. With approximately four application security experts on staff, GitHub has a more than respectable team in place for a company of its size, with an employee total of just over 250. But with so many deploys a day, the team needs air support to ensure they’re thoroughly inspecting the code base.
“We ship software 100 times a day, we update the site constantly and the security team isn’t looking at every one of those changes,” says Ben Toews, application security engineer at GitHub and the mastermind behind its bug bounty program. “The way we fit security into our software development life cycle is that when someone is getting ready to ship a feature, they’ll reach out to the security team and ask for a review if they think it’s necessary and we’ll get a review in that point in time, but then a lot of smaller changes don’t necessarily get our attention. And then there’s a lot of legacy code we’ve never looked at.”
Toews says the bug bounty fills the holes a number of ways. He estimates that for any given feature on the site, there’s probably been several hundred researchers who have tested them, “so the odds of someone looking at new code pretty quickly after a change is pretty high.” In its first year, GitHub’s bug bounty program fielded nearly 2,000 reports from bug hunters, 869 of which warranted further review.
As GitHub rounds the corner into its second year of offering bug bounties, it wants to increase those odds of good coverage with some tweaks to its program. In addition to doubling its max bounty prize to $10,000, it is also experimenting with giving bounty hunters preview access to new features before they ship.
“We think by doing that we can get even better coverage on features before potentially putting the community at risk,” says Toews, who admits it does complicate rapid release schedules. “We’ve been trying recently to be more methodical about how we ship larger features. If we’re developing a new feature, you will have that feature hidden from the general population and you can develop that and ship it many times a day and iterate on it quickly, but then we do have a little bit of a process around making that feature published and exposing it to the rest of the world.”
Of course, the efficacy of a bug bounty program depends on organizations finding a good way to marry bounty vulnerability remediation with the deployment process. It’s not good enough to find the vulnerability—you also need to be able to fix it quickly. Fortunately, the continuous delivery model makes this easier than in traditional organizations, says Toews, who reports that independent researchers are often “astounded” that his team can sometimes have problems fixed within hours of receiving a report, considering that they’re used to other organizations taking months to remediate problems.
Toews says the secret to success is rapid triage of vulnerabilities. If it is simple, he and his team of application security engineers will fix it on the spot. If it is more complex, they’ll loop in the development team. This is where it is important to have buy-in from developers and senior leadership about bug bounties and application security in general, he says.
“Our developers care a lot about security and they know that our leadership cares a lot about security, so if there’s a problem, everyone is going to jump in and try to fix it,” he says.
