New York Media on surviving DDoS and building a better web

At Altitude 2016 in San Francisco, we heard from industry leaders and Fastly engineers about the future of the edge.

In this post, I’ll recap a talk given by Larry Chevres, CTO of New York Media, the parent company for a wide range of online and print properties, including New York Magazine, Vulture, The Cut, and Science of Us. New York Magazine covers the news, culture, entertainment, lifestyle, fashion, and personalities that drive New York City. Founded in April 1968, it reaches 1.8 million readers each week.

In his talk, Larry discussed a massive DDoS attack and the steps they took to mitigate so they could quickly deliver the news to readers across the world. We’ll also take a peek inside New York Media’s stack — looking at the technology they use to ensure fast, dynamic experiences on their websites. Larry discussed how New York Media is building a better web, working towards creating a symbiotic relationship between readers, publishers, and advertisers to ensure great experiences for everyone.

DDoS’ing the news & quick mitigation

On July 26, 2015, New York Magazine published one of their most ambitious journalistic pieces to date, covering the 35 women who came forward to tell their stories about being assaulted by Bill Cosby. The piece had an equally ambitious counterpart on nymag.com, with rich, interactive experiences, video, and images.

Source: nymag.com

The high-profile story immediately garnered the attention of “nefarious types” and nymag.com was hit with a DDoS attack shortly after the link went live. The attack consumed all bandwidth on both pipes, bringing down their DNS servers, taking all the memory on routers and firewalls — rendering their site dark and totally inaccessible to readers. The attack, which was a combination of HTTP/SSDP UDP flood, was a highly distributed botnet that targeted multiple origin server IPs; at its peak it consumed a then-sizable 5 Gbps of origin bandwidth (since then, attacks have evolved both in method and scale).

At that point in time, New York Media was only serving static content with Fastly’s edge cloud platform — after a few hours of attempted mitigation and false pauses in the attack, Larry reached out to Fastly at 11:40 AM ET. Five minutes later he heard back, and the Fastly team “mobilized amazingly quickly” to establish a chat room. From noon to 1:45 PM the Fastly team worked rapidly to provision new services for nymag.com, moving the entirety of New York Media’s properties over to Fastly. The sites were up across the board by 2:00 PM ET, though Larry noted they would’ve been up in 20-30 minutes if not for internal damage to New York Media’s system, which forced them to reboot database and application servers in a sequence to get everything online properly. “On the Fastly side, they were ready,” Larry said. (You can check out the full timeline of events and mitigation details here.)

Although he’d never want to go through a large-scale DDoS attack again, Larry “couldn’t be happier” with the experience in terms of getting things turned around. New York Magazine was able to deliver the story to readers in spite of attacks, and that day turned out to be biggest traffic day of the summer. While DDoS attacks do take a significant emotional toll, it’s important to remember not to panic — you can prepare by being aware of the weak spots in your infrastructure, running drills, and having partners you trust.

Inside New York Media’s stack

The tech behind New York Media’s infrastructure has changed “dramatically” since Larry joined in 2011. He worked quickly to “sweep aside” the various legacy systems that had been around since 2002, working with cloud-based technologies and drawing from tools like Docker, Node.js, Kinesis Firehose, AWS, and Elastic (and many more).

Clay CMS

New York Media’s custom-built CMS is named after Clay Felker, who founded the magazine in 1968. A Node.js/Express-based CMS, Clay is a collection of lightweight frameworks. Based on DRY principles, maintainability, and rapid development, Clay CMS is “made to be fast.” Its core engine is Amphora, which composes components into renderable pages and is architected to provide non-blocking delivery of content. It also provides a full RESTful API for managing instances of components, URLs, and pages. Larry dubbed it an “Anti CMS;” there’s no application or form entry. Editors make changes to the site as it appears in production, hitting “publish” when they’re ready for edits to go live. And with Fastly, “changes are live in seconds.”

“Like Fastly,” Larry noted, “New York Media is passionate about open source,” and they’ve open-sourced Clay CMS here: github.com/nymag.

Fastly’s CDN

In addition to providing an “extra layer of security,” Fastly’s CDN serves “most of New York Media’s images,” and Larry finds that cache hit ratio “rarely dips below 99.3%.” Their Fastly setup includes 14 services, 36 production domains, seven QA domains, and full API integration, and New York Media saw a “40% cost reduction” after switching to Fastly (their prior vendor “never moved the needle”).

“Fastly complements the way we work,” Larry said, by enabling:

• Rapid prototyping/testing.

• Iterative development — “Fastly is so forgiving;” one of Larry’s “favorite things” is how their team can “put in a config and wind it back or put in a fix, and the whole thing will be up in seconds.”

• Instant feedback with real-time data, streaming from the edge. “Fastly tools make endless possibilities.”

• Extensible, open, and flexible APIs.

Results: the definition of fast

After modernizing their stack, New York Media saw performance improvements across desktop and mobile. According to PageSpeed Insights, their properties score 91/100 on desktop and 87/100 on mobile, meeting "the definition of fast” by responding in under 200ms, prioritizing above-the-fold content, using cache-control headers to use the browser cache appropriately, and optimizing images.

New York Media also saw a 65% reduction in page load time, a 50% drop in time to start render, and 85% in CSS/JS (a key benefit of Clay CMS, which is based on component architecture, avoiding CSS and JS bloat over time) — culminating in a 100% improvement in PageSpeed score.

Building a better web

Media sites rely on ad revenue, but the ads themselves can come at a cost to performance and the reader experience, “and of course there’s adblock.” Larry acknowledged that ads in their current state can impact performance, and forcing ads when they’re not wanted creates an adversarial situation with the user. New York Media’s approach to solving this conundrum is not by forcing ads on users but rather fixing the user experience.

In February 2016, New York Media announced their latest policies designed to build trust between readers, advertisers, and publishers. Larry’s team solves for performance, fostering a performant environment for premium content. As part of these policies, New York Media guarantees 100% human traffic and 100% ad viewability, but in return ads must load within 500ms of the original ad call. All parties fulfilling this contract will create a user experience that benefits publishers, readers, and advertisers.

Larry’s team aims to have their pages readable in less than 1.5 seconds, relying on the filmstrip method for measuring pagespeed, which he prefers to waterfall charts because it shows what users are experiencing. As a whole, New York Media properties perform well, except it when it comes to a lack of control over third-party provider performance — “all the social networks, ads, and other third-party services that content providers must have on their sites these days.” Larry stated that the industry needs to find another solution in the form of a partnership with all the players (publishers, advertisers, and readers) in the media space.

According to Larry, while there are other solutions in place to address the disconnect between publishers, advertisers, and readers — such as Facebook Instant Articles, Apple News, and Google AMP — these tend to function as their own, self-contained ecosystem and not part of an open web (creating added complexity for publishers, developers, and designers). As part of New York Media’s overarching goal, Larry believes we can address these issues in an open internet — providing great online experiences while continuing to innovate.

What’s next?

In their effort to move the web forward, New York Media will: continue turning to partners “like Fastly who obsess about speed” to foster richer, faster online experiences; leverage accepted standards (such as HTML5, CSS3, and HTTP/2); and own their data to leverage intelligence at the edge and personalize reader experiences. We can’t wait to see where it takes them.

Watch the full video of Larry’s talk below, and stay tuned — we’ll continue to report on how our customers are helping move the web forward.

Want more Altitude? Join us in San Francisco June 28-29, 2017 — save your place.