The Life of a 2XKO Patch

Welcome back to the 10th edition of the Weekend Warmup.

When patch day hits, you clock the notification, download the update, and hit the lab. Today we’ll talk about the internal, months-long process that every patch goes through before it’s released to you. 

Shipping a multi-platform release to the entire world is a massive logistical puzzle, and we want to pull back the curtain on that process. Here’s how we get from a whiteboard brainstorm to your console or PC.

6+ Months Out: Introducing dev-main

Long before a patch has a number attached to it, 2XKO’s design and product teams sit down to map out the next two seasons of content. This is the big picture: champion releases, major features, skinlines, etc. Much of this takes longer to actually make, but on average we start to see the definitive shape of the patch around six months out. 

Every piece of code, every in-game asset, and every design tweak the team makes goes into a massive, central vault we call dev-main. This is a pretty standard development model—think of dev-main as a giant pool of everything currently in development for 2XKO. But we can’t release everything in that pool at once. To build a specific patch, we have to carve out exactly what we need into a narrower, safer channel called a branch.

6 Weeks Out: Cutting the Branch

About a month and a half before launch day, our team executes a branch cut, creating a dedicated workspace separate from dev-main where developers can stabilise, tune, and polish the features that go in this update. Where possible, we merge the fixes we make back into dev-main.

Once the high-level work for the branch is done, we go to feature lock. Leaders from design, art, and product look at every planned feature and ask, “is this ready?” If it is, it’s locked in to release in the upcoming patch. If a feature still needs baking, it gets bumped to a future patch. (After feature lock, there’s still lots of work to do, but it’s mostly execution and fine-tuning.)

Around the same time, we hit localisation lock. This is the absolute final deadline for a developer to add or change any text in the game. Our localisation teams need time to translate all the in-game text strings into every language we support globally, and changes after this point make that difficult. We’ll do a deeper dive into 2XKO’s localisation at a later date!

3 Weeks Out: Zero Bug Day

Three weeks from launch, we hit one of the most critical days in development: Zero Bug Day (ZBD). And good news: we never have bugs in 2XKO! 

… Okay, so that was a freakin’ lie. So why’s it called “Zero Bug Day”, then?

I’m gonna reveal something that might shock some of you: game devs knowingly ship bugs every patch. Game software is incredibly complicated and multi-layered, so there’s a list of questions we ask like:

• How impactful is this bug? (On a scale from typo to game-breaking.)

• How likely is it to come up? (A key stat for determining the severity of a bug is reproduction rate.)

• How complex is the fix? (And would it impact other parts of the game?)

• Is it happening on all machines and platforms? Is it controller-specific? Is it a dependency for a future update? Is it a new legal requirement? etc.

Every bug we identify gets scrutiny. ZBD is where we make the hard calls. We review the bug list and divide it into “Must Fixes” and “things we can live with for now.” Our goal is to hit absolute zero on that “Must Fix” list.

This date is a hard line in part because of our partners on console. Once we submit a patch to PlayStation and Xbox for evaluation, we need to freeze all changes. If we find a massive bug the day after ZBD and decide to fix it, we have to resubmit the build. That restarts the console certification timeline and risks missing our launch window (editor’s note: this is why we’re often cagey around dates).

Admittedly, sometimes critical bugs slip through. Our teams do heroic work, but our tests can’t completely cover the millions of different setups, controllers, connection speeds, and playstyles of the whole wide world. With every patch we release, we get more reps and expertise in identifying and fixing these issues, but we know we’re still a ways out from S-tier stability. Keep us posted on any bugs you see with #2XKO_BUGS on social.

Patch Day: Pressing Buttons

Once we get the green light from our partners, we get everything ready for the patch to go out.

At this point, 2XKO devs from release, QA, engineering, core gameplay, publishing, and production all get into an early-morning call which we call a war room—personally I wish we kept to arboreal metaphors (the treehouse?)—to coordinate a series of actions: posting comms, taking down matchmaking, building and deploying the executable, distributing data across global servers, upgrading databases, and adjusting season/store/matchmaking configurations. We’ve got a bunch of graphs available to monitor players in game, login attempts, and crash rates to check for outliers or weirdness.

When the patch goes out (and it becomes available in Offline Mode) our QA team goes into hyperdrive—they smoke test the live version of the game on special accounts to check that all our core flows (menus, store, matchmaking, in-game, progression, missions etc.) are working. If we get the thumbs up, we flip the switch, and we’re live!

Post-Launch

Sometimes, a patch drops and something acts up. A frequent and valid question we see often is: “you know it's broken, so why don't you just fix it right now?”

It all comes down to where the bug lives, which can be one of two places:

• Server-side: if the issue is with matchmaking, the store, or backend configs, we can often deploy instant, invisible fixes.

• Client-side: if the bug is baked into the game's code (affecting how a champion moves, visual glitches, or other complex interactions), it requires a brand new patch to fix.

We also have to go back to our bug-fixing criteria: is this something that must be fixed immediately? How painful would this be to let it rock until the next update? Examples like the Illaoi tentacle issue from 1.1.3 took a lot of internal discussion on whether we should drop everything to fix it, or whether the affected players could find routes around the issue temporarily. We ultimately never want to put players in this position, but in this case, we opted for the latter and worked on fixes for 1.1.5 to avoid a domino effect of delays.

These are the tough decisions that I am glad I don’t have to make, and I can comfortably type Weekend Warmup articles about them instead.

The final plot twist: we’re always juggling at least three patches at once. At any one time, the 2XKO team is building the content for a future patch, polishing the next patch towards feature lock and ZBD, and monitoring the patch you’re playing right now. It’s a complex, fast-moving machine, and I think it’s a miracle that any of this works at all. But when it all comes together on patch day, there’s a bit of magic in opening the gates and seeing everyone posting their new discoveries.

Locals Spotlight

We’re continuing to work on bringing more support and visibility to the IRL communities that drive the FGC.

We started testing a locals rewards program last month, and our pilot events have been going well. Look out for more on this tomorrow at PAX East! We’ll be on the Main Stage on Friday.

Also, we’re excited to say that we’re partnering with Chipotle to distribute IRL prizing to TOs, which will go to top-placing 2XKO players at locals across the US. We’ve already contacted the first wave of TOs who will receive the rewards, but if you have a local event that should be on our radar, let us know.

Local Spotlight: Nexus Knockout (NXKO)

Nexus Knockout (NXKO) is a 2XKO community based in New England. We’re headed to Boston for PAX, so you’ll see some 2XKO devs at their Friday Fight Night this weekend! (More info here.)

Direct from the organizers: “Nexus Knockout is a grassroots community focused on 2XKO as both a competitive tournament series, and also highlighting the endless creativity and fun of Doubles and mechanics. We’ve hosted 5v5 Doubles crew battles, Combo Contests, and more! Two friends fell in love with the game at EVO 2024. From week 1 of the Closed Beta, we were running weekly brackets through our FGC scene! Before we knew it, we were driving some of the most passionate Monthlies in Boston history. Through passion and drive from our growing volunteer staff, we’ve found and formed not just a fun event series, but a special and competitive community of friends and rivals.”

We had to include this bonus photo. Meeperdoodle, one of NXKO’s organizers, designed custom shirts for the whole NXKO crew, with poro-fied versions of their favorite League of Legends/2XKO champs.

Next NXKO events:

March 27 - PAX Weekend Friday Fight Night. (Featuring 2XKO devs!)
April 17 -  Nexus Knockout Weekly. (2XKO Feature stream)
May 31 - Nexus Knockout Monthly, featuring 5v5 (Duos) Crew Battles.

All events held at the Balance Patch Gaming Cafe in Boston, MA (USA).

Keep up with them at @NexusKnockout.

Game Updates

Akali + Patch 1.1.5

We’re aiming for patch 1.1.5 (and new champion, Akali) to hit on April 7 in the afternoon (Los Angeles time). More to come next week!

Upcoming Dates

• Mar 26: Crime City Megabundle and skins return

• Mar 27: Texas Showdown (2XKO Competitive Series Challenger event) begins

• Mar 28: Viennality XL (2XKO Competitive Series Challenger event) begins

(Targeting) Apr 7: Patch 1.1.5 + Akali release

Closing Thoughts

It’s a stacked weekend, with two Challenger events hitting on top of each other in Houston and Vienna. Akali and patch 1.1.5 are around the corner. We’re thrilled with the turnout for Evo Japan. 2XKO is cool, y’all. Huge props to everyone who keeps this scene running.

—@draggles