Hard-earned insights from a pair of secure DevOps pros
Mike Johnson, CISO at Fastly, and Ben Kero, Senior DevOps Engineer at Brave Software, are passionate about security, having 35 years of experience between them. Both are strong believers in secure DevOps and have witnessed the many ways in which it enables their respective companies to secure their products and provide the best possible user experiences. While Mike is shepherding Fastly's vision for safety and compliance, and partnering with our product team to build developer-friendly web application security tools, Ben is driving the culture behind the web's newest secure browser, day in and day out. Mike and Ben recently chatted about the concept of secure DevOps (or DevSecOps) and how companies can make security a core part of their development pipeline. Here are some highlights:
There will be trade-offs. They will be worth it.
Implementing security measures earlier in the development process means that subsequent updates and improvements may become more labor-intensive. However, the extra time and effort are worth it when the result is a well-secured app.
The Brave team practices the principle of least privilege — only giving the minimum level of access required to execute a task. Ben says there’s a downside: if you change functionality later, you might “play whack-a-mole” trying to add a single new permission that a feature needs, but he also explains that this trade-off is well worth it for his team for the level of security it offers Brave’s applications. It’s a practice they’re willing to invest in, as it’s one that could potentially save their properties from costly breaches in the future.
Mike and the Fastly team tend to look at this theory on a sliding scale. The further down the development process timeline you are when a vulnerability is exposed, the greater the potential damage, expense, liability, and overall effort to course-correct. So, the earlier that security practices are put into place, the less archeology your team will need to perform within the code. Mike also points out that attaining “absolute security” is near impossible, and can impose a way of working that can both change a company’s culture for the worse and result in products that don’t actually meet customers’ needs.
Threat modeling sets everyone up for success.
Threat modeling — the process of identifying an application’s potential vulnerabilities, understanding the severity of those vulnerabilities, and assigning the right mitigation strategies to each — is something Fastly practices meticulously. It’s one of our most crucial security tools, and it’s not terribly difficult to create. A well-thought-out threat model enables developers to build faster, security teams to understand what’s meant to happen at runtime, and those wearing both hats to easily identify abnormalities. It’s a great blueprint for how applications should perform, and a map to a solution should a manifestation from your threat model arise.
A running inventory of your open source code can save you.
Your custom code could be flawless and protected against any attack, but the moment you bring in someone else’s code, from, say, a code library or open-source module, you introduce potential vulnerabilities to your codebase. What’s more, that vulnerability might not even exist today; it might only be introduced in later versions of the library. Coding everything from scratch isn’t the answer — code reuse and open source are foundational to modern app and website development, Fastly included. Mike recommends building and maintaining an inventory of the open source code used within your app — ideally one that automatically alerts your team when a known vulnerability arises in any piece of that code. It’s a proactive way to keep tabs on future issues.
Developer ergonomics matter.
To ensure security measures are in place earlier in the DevOps cycle, it’s important your security tools are built with developers in mind. Security control vendors, Mike says, can’t assume that their users will be security experts, but they also shouldn’t assume they’re novices. Assessing your users’ comfort level and understanding of security practices is key. Visibility is a close second to understanding the user, in Mike’s book, because — just like security professionals — developers appreciate having a clear view of what their code and product are doing. Giving developers a tool that can help strengthen both their code and their security mindset is a win-win.
Some of the features Ben and his team find valuable are changelogs for both fixes and new features, documentation for how to integrate your tool with various workflows, such as Travis or Terraform, and guidance (bonus point for screenshots!) of what can be achieved with a good configuration — something that helps developers envision the end goal. Developer-centric tools can also provide additional layers in a defense-in-depth security strategy. For instance, Fastly’s code snippets — short blocks of VCL logic that can be included directly within configs — keep Brave’s S3 buckets private.
Secure DevOps doesn’t replace post-deployment security. It strengthens it.
Applying your CI/CD security learnings to your post-deployment security efforts can help your team build new efficiencies, specifically through automation. The Brave team uses a GitHub custom action that scans deployed code and sends an alert when a vulnerability is found. It even automatically opens a pull request and initiates a review. Ben finds that the best tools are those that minimize the time between finding vulnerabilities and fixing them.
Remember the threat model? When looking at traffic, Mike’s team here at Fastly always goes back to the threat model to understand, “Is this expected behavior, and what’s the best remedy if not?” If they see any sort of drift from that initial plan, they can investigate and — if need be — mitigate. (Pro tip: when looking for remedies, your first stop should be your existing controls. Is the WAF configured appropriately for the traffic load? If it's a large amount of traffic, is DDoS mitigation being applied appropriately?) That’s the true strength of a great threat model: it will point you to the best next step, delivering a holistic one-two punch of security — within CI/CD and after deployment.
An engaged community can be your secret ingredient.
Brave leverages communities to improve both their CI/CD workflow as well as their own products. To streamline their pipeline, Ben’s team looks to their security vendors to find tools built by other teams that they can repurpose. On the flip side, Brave’s Bug Bounty program rewards users and security researchers for finding and reporting bugs within their software — a feedback loop that not only results in a more secure application for their users but also leans into Brave’s value of trust.
At Fastly, we invest in communities and organizations that have proven mutually beneficial for the security of both our products and the internet at large. Our engineers have fostered development of security policies and fuzzing integrations to a few of the open source projects that we’ve used over the years, including H2O, Wasmtime, Knot DNS, and Tokio-rs. And our involvement in the IETF has helped enable the evolution and release of TLS 1.3. On the flip side, the Bytecode Alliance is providing key technology for our serverless compute environment, Compute@Edge, where the use of WebAssembly makes it possible to provide especially strong isolation guarantees.
To dive into more technical detail around each of these insights, watch the full conversation below.
