The team at Fastly paid a lot of attention to the AI Action Summit that happened in Paris this week. Representatives from 80 countries including world leaders, tech bosses, academics, and other experts, gathered at the 125-year-old Grand Palais for a two-day global summit to discuss the current state of AI as well as future goals for the hottest technology around.
The emergence of the new Deepseek R1 Large Language Model out of China a few weeks ago caused huge waves in the industry since it seemed to shift the epicenter of innovation away from Silicon Valley and they did it by training a model for a far, far lower price than its competitors - a claim of a relatively frugal $5M against a rumored $500M that OpenAI spent on training O1 (and $1 billion they are reportedly spending on GPT-5, up from $80-100 million to train GPT-4).
These events highlighted a number of concerns that many people have regarding topics ranging from the ethics of the AI Industry training its models on other people’s content, often without consent, (with the added twist of industry giant OpenAI accusing Deepseek of using its models as a basis for its own training) to geopolitical power struggles, the need for regulation (both the US and the UK refused to sign the International AI Declaration claiming that it would stifle innovation) and, not least, the concerns about the enormous amounts of power needed to train and query Large Language Models. No frontier matters more in the sustainability conversation right now than AI, alongside worries about content consent and its impact on workers.
The activist group, Beyond Fossil Fuels put out a report which, amongst other things, claims that:
"New data centres alone could create up to 121 million tons CO2e more emissions [...] and could take up to 20% of the renewable energy planned to be built in Europe by 2030 [...] almost the same as what the entire transport sector will need to decarbonise itself" - Jill McArdle, PhD
Sustainability at Fastly
Sustainability is a priority for Fastly. We believe that efficiency is a critical enabler of sustainability – by innovating data caching and delivery, we minimize computational waste, improve performance, and lessen environmental impact. We applied the lessons of almost 15 years of smart caching to bringing caching to AI, with sustainability as a first-class concern (we’ll talk more about this later).
We continue to prioritize sustainability with launches like our Sustainability Dashboard (currently in beta - and we’re working to get this to GA ASAP, with valuable feedback from the community getting us there even faster!) which allows our customers to know how much energy we are using for their workloads - allowing them to fine-tune their delivery chain to reduce energy consumption.
Additionally, at WebSummit in Lisbon last year I gave a keynote talk about Sustainability and the Web - in it, I talked about various techniques site owners could use (and how Fastly can make your website more sustainable and give it a speed boost)to reduce the amount of work done and therefore the amount of carbon generated. The elephant in the room is, of course, AI. But I argued that some of the same techniques that can be used for regular websites and traditional database-driven applications (which are cheap and well understood) could be used for LLMs too.
Caching is Still King
The best way to not burn electricity is not to do any work. In the web world, we do this by caching the results of queries. You can even use various sophisticated techniques to be able to combine smaller queries offsetting cache misses by augmenting the data with cached fragments or, in the best case scenario, being able to build an entire response from those same fragments reducing the amount of work needed to zero.
This is Fastly's bread and butter and what we were originally built for - through several innovative methods we are able to cache many more types of content than our competitors and get higher cache ratios. The interesting thing is that even small increases in cache hit ratio (CHR) can have dramatic results - increasing from 90% to 95% for example may seem small but in actuality, it HALVES the amount of traffic going back to the database - this gets the results to the user faster, saves energy AND protects the origin from traffic surges. In the long term, our customers need less hardware, with reduced environmental impact right across the lifecycle, from material extraction to end of life.
Last year we bought that innovation to bear on Large Language Models (LLMs). Conventional wisdom suggests that queries written in Natural Language are too free-form and variable to cache. The two phrases "Where can I get good coffee in San Francisco?" and "Where's good coffee in SF?" mean the same thing but look different enough that they wouldn't cache the same. To do that you would need to analyze the grammar and extract the semantic concepts out of the query.
Our innovation was realizing that this was exactly what LLMs were already doing. They turn text into tokens and plot those tokens in a vector space - similar concepts, even if they are phrased differently, will be clustered together in the same N-dimensional space. If we took the coordinates in that space and used those to make a cache key then we could not only cache the two phrases above to the same answer but also even more dissimilar phrases asking the same question like "Can you recommend a coffee shop in San Francisco, California that serves great coffee?".
This turned into our AI Accelerator product which has been hugely popular with our customers and we're already working hard on other technologies that can help further and looking at other ways we can be part of the "Frugal AI" movement.
The Art of Doing Things Differently
And we're not the only ones thinking outside the box.
Beyond the work done by Deepseek, we're also seeing frugal innovations from small companies who are forced to innovate because they lack the huge war chest of the big players like OpenAI and hyperscalers like Microsoft, Google, and Meta. In "The Short Case for Nvidia" Jeffrey Emanuel lists a whole host of companies like Cerebras and Groq who are taking radically different approaches to chip design to provide performance that far exceeds that of Nvidia's flagship H100s.
Ultimately, this is a young industry that's still finding its feet. We hope that further leftfield inventions will reduce the massive Energy Budgets required for AI and ML.
