Hello and welcome back to our regular feature where we write a little bit about some of the games we've been playing. This week it's not about what we've been playing, but what we hope to be playing on Christmas Day (you're reading this now, but we wrote this back in June or something as we plan ahead).
Last week, Warframe’s hotly anticipated Old Peace update launched, kicking off a saga that digs into the very foundations of the nearly 13-year-old game’s lore. Of course, like clockwork, servers immediately took a tumble, resulting in crashes, outages, and chat issues. But why has this pattern become so predictable with online games? Especially when developers are well aware that a storm of their own making is on the horizon?
During a Game Awards-adjacent event celebrating The Old Peace’s launch last week, I asked creative director Rebb Ford.
"You've gotta spin up capacity,” she told Aftermath, referring to the practice of paying money to a distribution partner for additional servers ahead of or during moments when many players will be trying to access content. “You're allowing so many connections. We're an always-online game, right, so every time a player does something, there's a server call. There's something that needs to be verified server and client side. … Login, mission complete, anything that needs to talk to us to say 'You did this, you did that’ – it happens to us at a volume level that's very hard to account for."
In The Old Peace’s case, Digital Extremes was ready for a stampede the moment it opened the gates, but not quite ready enough.
"We actually didn't fall over as much as I thought we would,” said Ford. “That's when we realized 'Oh, we didn't think this was gonna be bigger than TennoCon [Warframe’s annual convention that often drives record player numbers].' We spun up IRC servers, we spun up things just to deal with volume. But sometimes you just cannot be prepared enough when you didn't predict it to be the third-best day in the history of the game. That was an error on our part, but it's not so much a tech error; it was an anticipation error. We fixed it very quickly."
The ability to quickly rectify server issues is also the result of preparation – in some cases years of preparation.
"One of our most important things to do is make sure people can get the content as fast as possible,” said Ford. “With Warframe, when we have the build or the update, we release it to our distributing partners, and we do something called a pre-heat of our CDN, or content delivery network – which is basically us saying 'People shouldn't all be fetching the game data from one node.' Because that will take forever. It'll get congested. So we distribute it, and this is through years of network partner shopping, working with really good network partners. We have content servers in 16 or 17 central population hubs."
The pre-heat, Ford explained, ensures that the whole network doesn’t hinge on a single point of potential failure.
“So sometimes you'll be going in the Philippines instead of being routed to our deploying headquarters, which is Ontario,” she said. “We pre-heated our server structure across the globe so that people can fetch [new content] quicker, and that takes a lot of load off.”
But that’s only one stair in what Ford characterized as a winding staircase of individual, overlapping needs.
"So that's the first point of failure: Can you download the game at all?” she said. “Second point of failure is: Can you login at all? When that happens, that all comes to us through login capacity. That one, you just have to spin up more capacity. Then you have the question of 'Can people play missions?' So you can kind of see the staircase: Can you download the game? Can you login to the game? Can you play the game? And each one of those is a slightly different sector of stability."
In Warframe’s case, elements can function independently, but if they’re not all working in conjunction, players quickly begin to see the seams.
“A lot of people can be logged into the game, and that's cool, but if you can't play, [then there's a problem],” said Ford. “It's like Maslow's Hierarchy of Needs. Can you chat? You don't need chat to play the game, but chat servers run independently. Those were hit the hardest [on Old Peace launch day], and we fixed it fairly fast by spinning up more capacity."
Games are complicated, as is the process of distributing them to millions of different computers with as many hardware configurations as there are stars in the sky. You will not be surprised to learn, then, that many other things can also go wrong.
“We have issues where we release new code in this build, and then maybe one piece of code fires every second on a heartbeat,” Ford said. “And sometimes we find these heartbeats, and we're like 'What is pinging the servers every second on the second,' and we're like 'It's the new title system we put in,' for example. ‘It's checking against server-client to issue you a title, but it's doing it in a way where we were unsure because it's checking all this indexed stuff.’"
Warframe has been around for over a decade and regularly pulls in tens – or in Old Peace launch day’s case, hundreds – of thousands of concurrent players. Nonetheless, said Ford, Digital Extremes still frets about The Ramifications as though it were a much smaller company.
"We still feel very young and scrappy, and we're like 'Can we even afford $600 more per month in capacity?'” she said. “That's the kind of question we ask ourselves on launch day. And then we're like 'Just do it! Just do it!'"
Warframe’s servers weren’t quite able to withstand the sheer weight of years’ worth of anticipation on launch day, but Ford was relieved that they didn’t go down for “hours and hours,” which would’ve necessitated a suitably less jubilant speech at the launch event in LA.
"It's exciting. It's thrilling. Everyone did an amazing job,” she said. “We asked our team to do the impossible with this update, so even though those little hiccups happened, we had two speeches prepared – funeral or the celebration – and we undoubtedly got to do the celebration."
AftermathLuke Plunkett
In the past year alone, I've plowed well over 50 hours into Warframe. Whether it's farming materials, decorating my Orbiter, or going back and forth on what color really looks the best on Oraxia, it's very easy to find yourself awake at 3am bopping Grineer and watching the damage numbers soar. But, as a newer player, I found the initial grind to be quite the hurdle - while things pick up with the Natah quest, the initial Awakening missions feel like a series of disparate parts that have no real rhyme or reason to them. As someone who has been desperate to get to the Warframe 1999 content and is invested in the MMO's world, I'm happy to put up with the grind, but I understand why some others would play a few hours and call it a day.
Read the full story on PCGamesN: Warframe's The Old Peace skip is "sh*tty" for some, but it's Digital Extremes' "survival strategy" in a busy crowd
"War is hell," Warframe's creative director Rebecca Ford says as community director Megan Everett tears through the hordes in the new Perita Rebellion mode. It's a phrase the duo continue to repeat during our preview of The Old Peace, the latest chapter in the MMO's saga. In many ways, it's fitting: while the 12-minute war that Everett finds herself embroiled in certainly is hellish, the quest's adjacent plotline, The Devil's Triad, adds the new 'devilframe' Uriel and Descendia mode, both of which are inspired by classical depictions of Dante's Inferno and other religious iconography. "It's not a coincidence that there's a devil Frame, and that an Orokin has an unfortunate relationship with that devil Frame," Ford says with a smile. "None of this is an accident."
Read the full story on PCGamesN: Warframe 1999 was the "furthest stretch of the rubber band," but The Old Peace offers "the most soulslike thing" DE has ever done
