Fastly and the Fediverse, pt.1
Unless you've been living under an Internet Rock, then the topic of the Fediverse is going to have recently crossed your radar.
There are a plethora of implementations of different decentralized social networking protocols, but the majority of folks are experimenting with Mastodon, the decentralized social network based on the ActivityPub standard. While Mastodon has been around for more than six years, their servers have recently seen an influx of users and various sysadmins are having to learn how to rapidly scale their systems. Beyond Mastodon and other implementations of ActivityPub, as well as other decentralized protocols such as Scuttlebutt, we’re seeing an unprecedented flourishing of new tools and technologies that are tapping into the Fediverse explosion.
We care deeply about all things open source and open standards, and we’re excited to see how the Fediverse continues to grow in the coming months. Today, I want to explain how it works and how we’re supporting it. And in a follow-up post, what comes next for Fastly and the Fediverse.
How federated networks — the Fediverse — work
First, it's useful to understand what the underlying architecture is. There are many, many, guides out there, so we're not going to go into too much detail. Here are the crucial points:
There are countless servers in the federation. Some are run by individuals, and some by organizations or businesses. Some have a single or a few users, while the bigger instances have thousands of users.
Users on different instances can follow, interact, and share content with each other. When a user on one instance publishes content, it is federated (hence 'Fediverse') to the instance of anyone who follows them.
Users can choose to migrate between servers if they find another one they want to join instead, or if they want to leave their existing server for any reason. Users can take all their followers and personal data with them.
Mastodon is an example of one federated network, but like I said, there are many. And possibly the coolest thing about decentralized social networking protocols is that any platform with these protocols implemented can communicate with any other platform that has done the same. Well—as long as it’s the same protocol.
In some ways it's a little like email – a federated address like '@simon@example.com' has two parts: the user, 'simon,' and the instance where they're hosted, 'example.com.' In other ways, it’s a bit like podcasting — everyone can use different tools and systems for creating and accessing content, but it all works together in a pretty surprising and creative way.
Why is this important?
Similar to how other social networks often have problems with scaling as they get popular, so too do a lot of Fediverse servers, like Mastodon instances. (Especially since few instances have experienced this kind of sudden increase in active users!) As more users join your instance, you have to federate content from the instances that they follow – but also, as you get more users, you get lots of other instances trying to federate from you. And if you have one super popular user, then you might just get absolutely hammered by inbound requests. It’s hard to scale centralized social platforms, let alone thousands of decentralized ones.
This is made worse by the fact that a lot of instances are run by hobbyists, and there aren't a lot of useful guides out there for scaling Fediverse servers in the same way there are for other types of web apps.
What does this have to do with Fastly?
We want to help the open web and see the Fediverse thrive, and we believe that Fastly can become a valuable and valued part of the community. And we have a clear point of view on how we'll do that: we want to help the users and creators of the Fediverse to succeed on their own terms, on their own platforms, at the scale that they want to reach. We don't want to make the Fediverse more proprietary, more centralized, or more dependent on any one vendor — it's special specifically because it brings us back to what we love most about the open web. Some of that has looked like Fastly team members reaching out to their friends in the Fediverse to offer expertise or insight into scaling.
There’s also a vibrant community of Glitch creators making tools and toys for the Fediverse, including the indispensable Fedifinder from Luca Hammer, which countless thousands of Mastodon users have used to help bring their existing social networks to the new platform. Indeed, there are so many Fediverse tools on Glitch, especially for developers, that the team has curated the Fediverse of Madness playlist just to highlight a couple of standouts. And our Fast Forward program for supporting open source is already working with key participants in the Fediverse ecosystem (more on that soon ✨). (If you’re not already in the loop, get in touch with us and we’d be eager to connect!)
Over time, we expect the greatest value that Fastly may offer is in being a good place to help with scaling Fediverse servers in a way that maintains decentralization by allowing you to host your instance wherever you choose, but scale to any size you wish.
For a start, we have out-of-the-box caching that can immediately tame inbound requests. This on its own helps with load on the server—especially with popular users since the content presented to their many followers is exactly the same. And since our caches are distributed all over the world, it doesn't matter where the follower lives–they get the new posts as fast as possible.
However, this would be of no use if your users couldn't post more content. Traditional caching is based on time-based expiration (so called TTL, or Time To Live), which assumes that new content comes along at predictable intervals. However, right from its inception 11 years ago Fastly was built to be based around Event Driven Expiration. Fastly's Instant Purge can invalidate and update new content in, on average, 150ms worldwide – so there's no lag between a new post and followers being able to see it.
All this is great, until you as the instance owner need to know what's going on. Which is where our Real Time Logs come in: Even if a request is served from our caches, we can stream logs to you immediately to any number of different locations and services and in whatever format you want, just as if they were coming from one of your servers.
And speaking of servers, Fastly can transparently load balance between different servers making up your instance. Or it can act as an even smarter load balancer failing over between different servers if one goes down, or even between different clouds. Or doing geographic routing for performance or data privacy compliance reasons.
Here's a list of Fastly employees that you can follow on Mastodon (to be clear, these are personal accounts, so content may not be about work (or even work-safe)! but we wanted to share to show the broad enthusiasm from our team:
@glitchdotcom@mastodon.social
@mikevj@infosec.exchange
@anildash@me.dm
@keith@cute.is
@chrispoole@mastodon.social
@casey@sharetron.com
@haubles@fosstodon.org
@daxtens@ozlabs.house
@orangeacme@fosstodon.org
@freeformz@hachyderm.io
@castillar@infosec.exchange
Over the coming weeks, we’ll be sharing how Fastly is already engaging with the Fediverse here on our blog, on Glitch’s blog, and of course, around our corners of the Fediverse!
You can check out part 2 of this series here.
