How 2XKO Handles Online Play

Hey, everyone! Tony Cannon here, Senior Principal Software Engineer on 2XKO. In this post, we’re pulling back the curtain to show you what we’re doing to build the best online play experience possible.

We’ve got a lot of tech to talk through, so here’s the TLDR if you need it:

• We really care about making online play as secure and lag-free as possible. We’re using a ton of different systems to make that true, including client-server architecture, rollback netcode, and something we call Fair Play mode.
• We take anti-cheat seriously. This means we’re using Riot Vanguard, which will help ensure that your opponents are actual humans who aren’t scripting.
• We’re using a lot of new tools to make our around-game systems work. 
  • We use something called the Game Validation Server to run a copy of every game in the cloud, which powers anti-cheat, skill rating, and spectating.
  • Matchmaking can now search for games across lobbies, so you’ll find an opponent quickly even if your lobby isn’t crowded.

All right. Let’s get into it.

Building for Online First

From the very beginning of 2XKO’s development, we’ve put a lot of attention on online play. Our very first priority when building the core tech was to integrate GGPO and move all playtests online. Even though at the time the game was peer-to-peer, we built and ran a packet proxy in Riot’s Las Vegas datacenter so all daily playtests would take place over the real internet. It’s super critical to stand this up in the design phase. Each character's moves, mixups, animations, and effects were built for and tested for online first, to ensure online play feels just as good as offline.

Client-Server Architecture

As development progressed, we transitioned from peer-to-peer to a client-server architecture, a shift driven both by the needs of four-player matches and by the desire for more consistent performance and scalability. In a four-player game, each player sends and receives about one-third of the traffic in the client server than they would in a peer-to-peer topology, which reduces bandwidth requirements, lowers the chance of dropped packets, and helps ensure smoother performance even on less reliable connections. There are many other reasons to use client-server architecture as well.

Privacy & IP Protection

First and most important: by using client-server architecture, we never expose your IP address to other players in the game. This is one of the biggest hazards of peer-to-peer. Just by playing against another player, you give them an attack vector to flood your network with packets, forcing you to disconnect and take that L. Even worse, exposing a player’s IP address exposes them to dox attacks. Player security is something we take very seriously, and moving to client-server is the most effective way we can protect your IP. 

Time & Input Authority

The server provides another very important role in the system: it acts as the authority for input, state, and time. As a developer, once you’ve got rollback working, the biggest factor affecting the quality of your netcode is the robustness of your algorithm for balancing the “rift.” This just means making sure players share the rollback load. If I see a three-frame rollback, you should be seeing the same. If I’m seeing five-frame rollbacks and you’re only seeing one-frame rollbacks, it’s likely because the rift isn’t balanced. 

Without getting too much into the weeds, balancing the rift means synchronizing the game clocks of every player in the game. This is already tricky to do in a two-player peer-to-peer game. How do you make sure both players are working off the same information? (Geek moment: this is actually related to the Two Generals’ problem, and is probably impossible, but I digress…) In a four player game, it’s an even bigger headache! 

This is what it means for the server to be the authority of time. With a central authority that keeps track of the clock of all players in the game, we can do a much better job of balancing the rift for everyone.

Fair Play Mode

We’re working on another feature to improve 2XKO’s online experience called Fair Play mode. We’ve all had those matches against players who are on wifi, or a tethered phone, or a computer that’s just a little bit below the minspec. What happens? The game ends up pausing and stuttering; or worse, you get these mammoth rollbacks where characters are teleporting everywhere. In a peer-to-peer game, that’s literally the best you can do. If you don’t have the inputs for all the players in the game in a timely manner, not even rollback can save you. This is especially problematic given bad actors. A malicious player can add a lag-switch to their controllers to induce packet loss on demand, or just grab the Windows title bar on a PC to stop their game from ticking. Both will cause a huge lag spike for you, making you drop your combo at just the wrong time. 

When Fair Play mode is on, the 2XKO game server will give each player’s inputs a short grace period to arrive. If they’re late, the player’s last input is replicated and the game goes on. This is a really big deal. It means no matter what your opponent does to their network, your gameplay experience shouldn’t be affected. If they hit their lag switch or their wifi drops out, you probably won’t even notice. This is a double-edged sword, though. If your network connection experiences a glitch for a few seconds, you may find yourself unexpectedly eating a big combo by the other team.

Fair Play is currently experimental, and it’ll take a lot of tuning to dial it in just right. We want to make sure it punishes cheaters who are trying to gain an advantage by dropping packets without accidentally causing problems for everyone else. We’ll let you know once it’s fully implemented so we can get your feedback on how it’s feeling.

Global Infrastructure & Protection

Of course, all of these improvements to networking wouldn’t matter without a strong global backbone to support them. That means we’ll be leveraging the full power of Riot’s network infrastructure to hold all this together. 2XKO networking traffic will ride the Riot Direct backbone to ensure your packets get to the server at faster-than internet latencies. The servers themselves are protected by the Generalized DDoS Service, so bad actors shouldn’t be able to take down the grand finals of a tournament you’re hosting in your Private Lobby. We’re using Riot’s Game Provisional Platform to deploy servers worldwide and ensure low-ping matches for all players in our launch regions. We aspire to have servers near all players globally, but it may take some time for us to work up to that. We appreciate your patience as we roll them out.

Riot Vanguard Anti-Cheat

And since fair online competition also depends on integrity, we’re pairing strong networking with equally strong cheat protection. For anti-cheat, we’ll be using Vanguard to make it much more difficult for scripters and botters. Vanguard’s been super effective in dropping the number of botters on League by over 99%. This will greatly help ensure that your opponents are actually humans, who don’t use any cheats like auto-parry or whiff punish scripts. That said, preserving competitive integrity isn’t something you solve once and walk away from—it’s an ongoing effort. This is something we’ll be keeping an eye on and adapting our approach to as needed.

Fixed Input Delay & Frame Pacing

We’re also regulating our input-delay and frame-pacing subsystems to ensure 2XKO’s gameplay feel is as consistent as possible, irrespective of whether you’re playing online or offline, or on PC or console. 

Fixed Input Delay

The first trick we use is to add a fixed three frames of delay to all inputs. This is a common technique to use in fighting games to minimize the length of rollback. In 2XKO, we apply this delay both online and offline. This means your combo timing and muscle memory built in offline practice should apply directly to online play. We decided on three frames of inputs years and years ago. Everything from the timing of a whiff punish to the fastest-hitting move in the game has been built and tested assuming three frames of input delay everywhere, so the game still feels great regardless of the delay. 

Frame Pacing

The second trick is to invest very heavily in frame-pacing. Frame pacing is how quickly and consistently you can simulate, render, and display a frame. With inconsistent frame pacing, you may notice a lot of video stuttering or that your one-frame links sometimes just don’t come out, even though you’re sure you got the timing right. We’ve focused our frame-pacing efforts both on minimizing input delay and minimizing differences between console and PC. Right now, the time between when you press a button and you see the result on screen is nearly identical on PC, PS5, and Xbox. Again, this means 2XKO should feel the same whether you play offline, online, on your PC at home, or on a console at a live event.

Game Validation Server (GVS)

I mentioned earlier that the game server was the authority for state, but didn’t explain specifically what that meant. The main benefit is for anti-cheat and anti-smurf. Every competitive match of 2XKO runs another copy of the game on the server. So in a 2v2 game, there’s actually a fifth copy of the game running in the cloud in real time. We call this piece the Game Validation Server, or GVS. 

The benefits we get from this are pretty significant. First of all, by nature of having a server at all, we can identify exactly who to punish if someone rage-quits or pulls their internet plug. GVS lets us take that one step further: If one player’s copy of the game is different from everyone else’s due to tampering, we can identify that player by comparing their checksum of gamestate to GVS’s, defeating another way to force a draw by intentionally modifying your game state. 

GVS also feeds match outcomes to our downstream data pipeline. For example, we feed detailed statistics of every game to Riot’s centralized Ranking services. The Ranking service feeds our end-of-game stats through Microsoft’s True Skill 2 algorithm to effectively narrow in on a player’s skill rating in just a handful of games. This means less suffering through lopsided matches, and we hope it’ll dramatically reduce the use of smurf accounts.

Lobbies & Spectating

Beyond core gameplay, we’ve invested a ton of effort into our around-game systems, the most substantial of which is our lobby system. Fighting games are best when played with a group of friends at home, or gathered together at a local venue for a tournament or weekly meetup. 2XKO lobbies are meant to replicate these social systems online.

Matches in Progress

One of my favorite features is how you can spectate any game in progress. Watching other players is actually one of the most important beats in offline play. It’s a good way to learn new techniques for your character, see what tools other players are using, or just cheer on your friends and root against the opponent who kicked you out of the tournament. These are all really amazing feelings that we wanted to capture online.

When a player in a 2XKO lobby gets close enough to a cabinet with a match in progress, we pop up a screen above their heads, and in just a few seconds you see the exact state of the match. Whether they’re in champion select, just starting a game, or at the very end of the last round scrapping for that final hit that’ll decide the winner, you get to see exactly what the players are seeing in near-real time—without lengthy syncing or playback delays, so you can drop right into the action as if you’d been there from the start.

How Server-Side Spectate Works

Getting this spectating system to work took quite a bit of experimentation. Fighting games that use rollback like 2XKO are fully deterministic, so you can always replay the state of the match if you only have the inputs. So if someone walks up to a cabinet and starts spectating from round one, we feed them the inputs in order and everything is great. But what if we’re already in the middle of a match? Well, you could still feed each spectator all the inputs and have them fast-forward through the whole game until they’re caught up, but that takes a lot of time. Early versions of 2XKO lobbies before Alpha Lab 1 worked exactly like this, actually. It was functional, but didn’t feel great. Having to wait between 5 and 20 seconds for the screen to catch up discouraged folks from using the feature at all.

Once GVS came online, though, we had the opportunity to do something smarter. Remember, GVS is the system that’s validating the state of the game in real time on the server. When a player starts watching a game from the middle, we don’t send all the inputs. Instead, we ask GVS for a snapshot of the game right now and start sending inputs from that point. A snapshot is a memory-efficient save state of just the deterministic parts of the system. It’s only about 50 KB, so it gets to players super fast. Instead of having to fast-forward through three minutes of inputs, we just load the 50 KB save and the player is instantly caught up. It feels like magic, at least to me.

We absolutely could not build this feature without server-side game simulation. In prototype versions of spectate, we would take the snapshot from the first player in the game and send it to the spectator, which caused all kinds of problems. First of all, taking and compressing the snapshot is relatively expensive. On low-end machines, this showed up as an FPS hitch for the player every time someone started spectating. That right there is unacceptable. Can you imagine missing your hit confirm because someone started watching your game at just the wrong time? Not only that, it’s insecure. We did not want to open up an attack vector for malicious actors, where they could manufacture a snapshot that contained some kind of buffer overflow attack or other means to compromise another player’s machine. All these problems are avoided by moving the entire source of spectate data to a trusted source on the server.

Cross-Lobby Matchmaking

One last thing I wanted to talk about is our cross-lobby matchmaking feature. 2XKO lobbies have lots of ways to get a match: You can directly challenge individual players, sit at a cabinet and wait for someone else to come up and sit next to you, put your quarter up to “get next” like the good ol’ days in the arcade, or just ask the matchmaker to find an opponent for you. 

This generally works well when the lobby is full, but has some edge cases. What if everyone else in your lobby is having a great time and you’re the only player looking for a match? 

In Alpha Lab 1, there were lots of great things about playing in a lobby—like the leaderboards, winstreaks, and walk-up spectate—but they sometimes came at a cost of time-to-match in edge cases like this. In fact, for players who exclusively wanted to get an online match as fast as possible, something like “Find a Match” from the main menu could have gotten them one faster, simply because it would have looked across the entire player base rather than just the players in the current lobby. 

In Closed Beta, we modified the matchmaker to search for matches in all online lobbies, not just the one you’re currently in. If it finds a match in your current lobby, we reserve a match at a cabinet in the lobby and send players there to play. If a match is found with a player in another lobby, the experience is roughly the same. Each player in their own lobby will get a reservation at a cabinet to play, and we’ll render a holographic depiction of their opponent to sit down at the cabinet. When the match is over, both players return to their original lobbies. Don’t worry, your win streak will be intact. 

Since all players in all lobbies shard-wide are queued into the same cross-lobby matchmaker, finding a match in a lobby should be just as fast as finding a match from the main menu in a non-lobby system. The matchmaker is tuned to give you the best match possible within a certain amount of time. If it’s peak hours and there are lots of people online, you should expect to get a fair match after just a few seconds of waiting. If it’s 3:00am and there just aren’t that many people to match against, we’ll still try to get you a match promptly—it may just have a higher skill disparity than you’d see during peak hours. That kind of slower queue experience during off-hours isn’t unique to 2XKO; it’s just how matchmaking works in any online game when the player pool is smaller. How much we prioritize time-to-match vs. skill disparity is tunable, and we’d love to hear your feedback on how the experience is working for you. We wanna get this completely dialed in as we move through Early Access.

Looking Ahead

We covered a lot in this article, so kudos if you made it this far. We’re big believers that online play should never be the second-best way to enjoy a fighting game. Every feature we’ve built—from server validation and rollback tuning to instant spectating and cross-lobby matchmaking—is designed to make online play just as fair, fast, and fun as offline. That said, we’re still tuning, testing, and listening closely to your feedback as we head into Early Access and beyond. Let us know how it’s feeling, and keep those bug reports and suggestions coming.

Thanks for reading,

Tony Cannon