Leading online education app builds site resilience and simplifies security with Fastly

The challenge

Duolingo's 113 million monthly active learners count on the app to make studying a language easy and fun without downtime that interrupts learning or breaks up a learning streak. DDoS attacks were frequent and expensive, as Matt Brandman, Senior Engineering Manager, Platform Security at Duolingo found out . “My first two weeks were spent in a constant firefight. In addition to the impact on customers, the cost of engineering hours added up—a DDoS attack could spawn five different issues and consume dozens of hours of engineering time,” he says.

Brandman’s approach to security at Duolingo focuses on systemic-level solutions and team education, so he wanted to put in place an edge security solution that made life easier for developers and achieved site resilience. “As an engineering leader, I care deeply about the interplay between the developer experience and the security experience. Building trust with developers is easier with Fastly.”

After negative experiences with other vendors, including some who deprecated services unexpectedly, Duolingo wanted a security partner with a history of supporting its customers long-term, convenient and responsive communication channels, and security experts his team could trust when the stakes were high.

The solution

Fastly provides the developer productivity, reliability, and resilience that Duolingo needs. “The Next-Gen WAF was live in production and blocking enough traffic that it made a real dent in the DDoS attacks in less than a week,” Brandman says.

With platform resiliency comes the flexibility to innovate

Developers at Duolingo deploy services faster knowing they are protected behind Next-Gen WAF on the Fastly edge. “They can put up a server sooner, knowing they can rate limit it, if needed,” Brandman says. “And, because Fastly does its job and protects the services, they don’t get woken up at night by incidents.

With a DevOps culture that runs hundreds of experiments every quarter, Duolingo has even found the Next-Gen WAF provides resilience against developer experiments gone haywire. “A few developers thought we rate-limited them. It turned out their experiment was sending us tens of thousands of requests, so we actually prevented an internal mistake from bringing the site down.”

Visibility saves hours of engineering time

Another key problem Fastly solves for Duolingo is visibility, which enables Duolingo to react to security incidents fast. “Fastly dashboards give us the visibility to see that a service just got 5 million requests at once and we immediately know that’s why it went down,” Brandman says. “The biggest cost for engineering companies is development time—the cost of tools is almost negligible compared to 30 engineers trying to troubleshoot an issue for three hours. With Fastly, we spend two to three engineering hours at most to resolve an issue—one or two engineers jump on it, make a change, and push it to production. The cost savings are tangible and one of the biggest benefits of Fastly,” he says.

Granular controls fine-tune security with convenient inheritance

Different paths require different security approaches, so Brandman’s team appreciates the ability to customize rules with precision. A login path warrants more protection, he says. “You don’t want to allow 2,000 bad logins per second, because you are going to end up with enumeration attempts. With Next-Gen WAF, we can be highly specific with what behavior we prevent. For example, it’s probably not a real user if they've failed to log in 30 times in the last second, so we put them in a penalty box for a few minutes. Given how often IPs cycle and how many proxies exist, a penalty box is a much better approach than a ban.”

Brandman goes on to say that it is also really useful to be able to put the Next-Gen WAF into monitoring, see if it matches the traffic, and make sure it doesn’t block all the traffic before turning it on in blocking mode. “We can even do that path-by-path, based on a header-based user agent. Blocking a specific user agent on a path can give us the time to implement a larger fix.”

Even more powerful, signals can be combined to create larger rules. “One of my favorite features of Next-Gen WAF is how composable it is,” Brandman says. “Over time, we've built up foundational signals, like old user agents. By itself, that might not be a warning signal, but if we see an old user agent with ten log-in failures coming from an old version of an app, we may enforce a longer timeout than on a user agent on a more recent version of the app. Fastly makes it easy to explore, easy to understand, and easy to make reviewed changes to these type of rules.”

The benefits of security automation plus the flexibility of hands-on management

Duolingo enjoys the benefits of automated security tools as well as Fastly security experts' knowledge built by protecting customers dealing with a variety of attacks: what other customers have experienced, what they've observed in specific verticals, and which attack vectors are on the rise. Fastly regularly reviews Duolingo's logs to identify trends that automation doesn’t—before they reach emergency levels. As a result, Duolingo prevents incidents. And when the in-house teams need a hand, Response Security Service steps up to help. “Early on, when implementing the Next-Gen WAF, we engaged the cybersecurity operations center (CSOC) and the implementation team at the same time. They worked in lockstep as we figured it out,” Brandman says.

Brandman says every interaction is a great experience, because Fastly takes the time to understand the customer. “Fastly has a technical team that really knows what they’re talking about, and they really understand our tech stack. When we open a ticket with the CSOC, response time is normally a few minutes. And they don’t just look at a dashboard, they write scripts and analyze traffic. They ask good questions and give valuable advice on what to try, specific to our stack. And they provide the extra hands you need in an incident when everyone else is occupied.”

Key takeaway

Duolingo found a security partner for the long term that supports both resilience and innovation. A strong partnership and expert approach lead to stronger protection, peace of mind, operational focus, and developer innovation. “When considering a security and CDN partner, really look at the quality of the team you’re working with, and make sure that they're going to be able to keep up with your organization over time. They're the backbone and your first line defense. A strong relationship will pay dividends for years. That's what led us to Fastly,” concludes Brandman.