Red Post Collection: TFT Galaxies Set Intro, MSI 2020 Reschedule, Legends of Runeterra State of the Beta, & More

Additionally, after reviewing the calendar and listening to feedback from our teams and players, we have decided to sunset Rift Rivals in the remaining regions where the tournament was scheduled to take place. Rift Rivals produced some epic battles, but the challenges it created to the annual schedule were felt at both a regional and global level. We remain committed to producing globally resonant events that showcase our sport, and will work with regions to determine fresh ways to do so in the years ahead. 

As noted in my February 25 post, we want to thank our teams, partners, players, and fans who have demonstrated their support to those impacted by the virus, through charitable donations and other means. We’re eager to reunite the global League of Legends community at the Mid-Season Invitational this July. We’ll continue to assess the situation to ensure the best experience for fans and players, and update you along the way."

Incident Report: Recent Outages in Europe & Brazil

"Hello folks, Brian "Penrif" Bossé from the Technology department of League of Legends to share details about the technical issues that caused the recent outages in EUW, EUNE, and BR towards the end of February. I'm going to get a little nerdy on y'all here, but if you're curious about the details behind those outages and what we've done to fix them, come with me on a journey with computers and graphs! 

Setting the Scene 

The first symptom that came to our vigilant Network Operations Center's attention was that the number of League games starting went down drastically.

Not typical behavior

There are a lot of systems that go into a successful game creation, from matchmaking to load distribution to the game server itself, and it's not immediately clear from a symptom like this where the breakdown is. When we brought the experts in each of those systems into triage to make that determination, they all indicated that their service's health looked good, but very little traffic was coming in to them. A matchmaker with few people coming in to request a match doesn't have much to do. 

So, all of that indicated that we're dealing with a systemic issue in getting player traffic into our backend. Metrics showed us some major problems in that neighborhood: 

The horizontal red line all the way at the bottom is where sirens go off

What we're looking at here is the number of inbound connections to one of our generic container hosts. Those are highly capable single computers that run a bunch of smaller applications - in shop-talk, containers - that comprise the overall system that runs League. Two of those hosts are getting vastly more connections than is reasonable. To understand why, we need to talk about a particular kind of container, one performing an "edge" function. 

Living on the Edge 

Edge processes are tasked with taking in traffic from the internet, filtering it, and routing it to the appropriate backend service. They take the garbage pile that is the public internet and leave only nice, clean stream of bytes for everyone else to process in comfort. As you can imagine, edge processes go through a lot of traffic, but really shouldn't regularly see the kind of spike we saw during these events. There are three factors that all lined up to create this situation; I'll give them each a section then pull it all together. 

So It Begins 

First up, the spark that started the cascade - an unreasonable amount of requests to one service. For a few months we've noticed an increase in the volatility of call frequency targeted towards that service, but it didn't appear to have any impact and all of the upstream systems had been handling it fine. Diagnosing something that wasn't hurting anyone didn't rank as a high priority for active effort, but we kept it in the back of our minds. We've diagnosed it now though; there was an error in how the request was being made which, in certain cases, would cause it to constantly fail - and constantly be retried. 

Containers Sprung a Leak 

We had a known issue involving an interaction of our container system with the operating system version in use. The problem involved a leak of memory inside the bowels of the operating system, which given enough time to build up, could halt critical system functions. Prior to this event we had never seen that manifest, but we had already executed upgrades on about 60% of Riot's container fleet anyway. Unfortunately, the update to Europe and Latin America's clusters was still in progress. 

Luck Matters 

And lastly, we got unlucky. We utilize a piece of software to group up the containers into bunches that can fit on a single host computer. It has constraints programmed into it for keeping these network-heavy edge services separate from each other within the same shard's set of containers. However, we couldn't control if edge services from different shards could land on the same computer, so edge services from EUW could end up sitting right next to edge services from EUNE. That multiplication of load onto a single machine was the magnifier that threw the prior two problems into a major incident. 

On all of the outages, edge containers from at least three shards were put on to a single host. With a spark of traffic from the malformed requests, amplified by having multiple shards worth of that traffic hitting the same host, then the operating system memory leak pushing that machine into a non-functional state, we experienced a breakdown of the integrity of the shard, and significant outage. 

Impact and Resolution 

It generally takes significant effort to untangle this kind of problem and find isolated causes. When we needed to make the call on whether to go forward with Clash on top of a potentially unstable cluster, we had suspicions that the container problems were a key part of the picture, but did not yet have a handle on what the traffic was exactly. Even though the issue was not at all caused by Clash, we made the decision to delay a week in order to be sure that the Clash experience would be protected. I sincerely apologize for the disruption these events caused, and want to assure you that we have fixes for every aspect that played into them. 

The code causing broken requests to be sent has been corrected, and just in case we have similar problems in the future, we've changed how the retry mechanism works to avoid them from causing large spikes. The container software upgrade has been completed on all of our shards. We have concrete plans for moving the edge services to a balancing system that can accommodate spreading them out cross-shard. Until those plans are executed, we have alerting in place so someone's phone gets turned into a screaming angerbox until the load is manually redistributed. 

We feel confident that with all of these different causes behind us, this particular issue will not happen again. That said, we're continually evolving the game and its supporting systems, and situations like this are always a possibility. We’re committed to restoring service as quickly as we can when they do. Thanks for making it to the end here; if this sort of content sparks your curiosity, you can catch more in-depth League technology articles over at our TechBlog. Either way, I'll see you all on the Rift."

Legends of Runeterra: State of the Beta

"It’s been a little over a month since Legends of Runeterra entered Open Beta, and we wanted to share thoughts on the state of our game: reflect on what we’ve seen, recap some feedback we’ve heard, and finally, share a bit about our priorities this year. 

Open Beta 

We released LoR into Open Beta early because we believed that was the best thing we could do for players. We launched with the intent to see what you discovered, found delightful, and found painful, and couldn’t wait to start the process of improving the game with the community. While we’ve learned a lot along the way, the overall response has been extremely positive, and we’re super glad we kicked off the journey with you this January! 

Many of the thoughts we have about Legends of Runeterra are informed by more than the numbers—they come directly from players in communities across forums, social media, streams, and even DMs. It’s impossible to walk the floor at Riot without seeing a LoR stream on a monitor. We appreciate the early support and hope you see this article and continuous improvements to the game as a signal that we’re listening and committed to improving the game with you. 

Now, prepare for a deep dive—let’s get into it!  

The Game 

Our goal is to bring deep, interactive strategy to the genre through a combination of back-and-forth gameplay, indirect attacking, “skillshot” spells that provide reaction windows, and exciting champions that level up through smart play. All this, optimized for PC and mobile. 

We’ve heard tons of positive feedback in this area, but also see some opportunities to do better. 

The good: 

• Champions: Champion level-ups are slick and champs often feel like their League counterpart, even down to obnoxiously-effective ones like Teemo and Ezreal. With the update to displaying champions at match start and whenever their level-up makes progress, champions are even more present and active throughout games.
• Interactive Strategy: The back and forth gameplay combined with fast and slow spells allow for lots of strategy and outplay moments; we’ll probably never get tired of screenshots like this: 

The stuff we’re improving:

• Clarity: So far, we’ve improved clarity via card text and other updates, like the action log now including discarded cards. Given the huge combinatorial nature of the game, we suspect we’ll never be “done” here, and intend to continue improving clarity over time.
• In addition to card clarity, UI and Deckbuilder improvements are also top of mind. We’ve received a ton of feedback on the client, deckbuilder and collection management in particular, and we hear you about how annoying certain parts can be. We’ve already made some improvements (like updating the way deckbuilder shows card counts), and you can look forward to similar improvements for other parts of the general out-of-game experience over the course of the year, such as currency management and crafting cards and decks.
• The Timer: Getting timed-out round over round or due to a barrage of burst spells sucks. You can look forward to timer adjustments for burst spells in one of the next two patches.
• FPS Lock: Some players were disappointed when they discovered our FPS options. This was a feature we misjudged the value of, and a great example of the type of feedback we were excited to get by jumping into Open Beta earlier. We added several FPS options in 0.9.0, so now players everywhere can warm themselves with the heat of their rigs.
• Animation speed and input blocking: Finally, In 0.9.1, we began reducing both animation time and input block-time for common actions like playing units, drawing, and start of round. Our priority is to make a game that is as much fun to interact with as it is to look at. Once this baseline is good, we’ll continue on to the more egregious single-card examples, while updating our internal guidelines to ensure these lessons get applied to new content too—no more multi-second chained axes from Draven.

The Meta

We think that great card games offer endlessly interesting and interactive puzzles—chief among them a healthy metagame waiting to be cracked. From the beginning, we believed bringing the Riot “live service” mentality to bear would help keep LoR fresh and ever-evolving. Our strategies to accomplish this include:

• No bad cards, so even a modest initial card pool could yield lots of interesting decks.
• Monthly card updates that empower the developers to keep the meta healthy (and awesome to play) between sets.
• New cards and content to exponentially expand the possibilities of our mix-and-match deckbuilding.
• And finally, slow and steady progression, so the meta isn’t solved day one. 

How We’re Doing 

Overall, we’ve been blown away by the variety of decks we’ve seen at all levels of play. It’s felt like there was a new “OP deck” meme each week: Dawnspeakers, Elusives, Fearsome... Elnuks? 

But we also see room to improve. We focus on a few key indicators to inform our overall understanding and where we should look to make changes: 

• Card play rate, especially Champions: As RubinZoo mentioned in the Patch 0.9.0 notes, our aspiration is that every card has a role and every champion’s dream is realizable for every player. It’s a never-ending task in general, but simply put, some of our champions aren’t hitting the bar here currently.
• Card win rate: As opposed to our champion play rate metrics, for card win rate we look specifically at Platinum and above ranked matches. We do look at all levels of ranked play, but our hard metrics for “must-change” cards are indicated by higher ranked matches. Here, we have thresholds for how high of a win rate any given card can have before warranting a change. We specifically look at a card’s ideal region combination for these metrics, as opposed to overall win rate (for instance, Shen has a much higher win rate in a Demacia/Ionia deck than in other region combinations).
• Deck win rate: Deck win rate is folded into our card win rate metrics. While we’re looking at how cards perform in their best region combinations, we get to see how card clusters, archetypes, or specific decks are performing. This helps us isolate problems with certain matchups and determine what changes should be made (if any). 

These metrics combine to provide other ways to look at health as well, such as region prevalence. With these metrics in mind, here are some key areas we’re looking to improve: 

• Healthy patterns: While the core gameplay provides a ton of opportunities for interaction, players have been able to push the boundaries of minimal interactivity a lot with certain decks. We’ve mentioned Fearsome and Elusive decks before, but we’re keeping an eye on things like Ezreal Frostbite decks as well. Here we need to walk a fine line between satisfaction, interaction, and games actually ending. Certain regions will be better or worse at different aspects of interaction, but we want players to have tools available to combat certain strategies if need be. We’ll continue looking at ways to provide those tools in addition to keeping low interaction with decks in check.
• RNG: We’ve always held the opinion that games should feel decided by players and not cards. RNG has a place, but it shouldn’t take the lead, and good RNG design creates novel circumstances that players can adapt to. We’re always watching how these designs are received (looking at you, Elnuks), and will make tweaks where needed to bring them in line.
• Shadow Isles: Shadow Isles has been performing as the best region holistically by a decent margin. This is partially due to Shadow Isles having a ton of outright power, but also because it’s incredibly strong at doing what it’s good at compared to other regions and their respective strengths. And on the flipside, Shadow Isles’ intended weaknesses—such as frail units and the inability to protect its units—are too easily mitigated by cards like Mark of the Isles and Frenzied Skitterer. While we’re making adjustments over time to address these issues, the biggest change is from a development standpoint: we’re going to focus on ensuring regions have clear, distinct weaknesses (and therefore challenges for you to build around) earlier in the design process.
• Champion Updates: Some of our Champions such as Vlad, Shen and Kalista have low play\rates that we want to improve. Some of the lowest playrate champions have very narrow archetypes (Vladimir, Shen). For these champions, we look to make their narrow decks a bit more meta or make tweaks to their supporting cast of followers before we outright change the champion itself. Others, like Kalista, not only don’t have a clear home but struggle by simply not being effective or having an unrealizable gameplay dream. For this type of problem, we generally look more at updating the champion itself. And then there are what we call “consumptive” champions, whose level-up dreams involve… destroying your opponents’ dreams (looking at you, Yasuo and Ezreal). While we aren’t planning to rework either of those champs in the near future, we’re being careful about how we design champs like that going forward, and are keeping a close eye on existing champs that fall into this category. 

The Economy and Progression

From the inception of LoR, we knew we wanted a progression system that proved to players why card games could be awesome: imaginative new decks, endless tinkering, and fierce competition. We anticipated having to get creative to accomplish this, even if that meant throwing away the traditional pack model. 

And for the most part, we feel very good about the bets we’ve made so far. We’ve seen a bunch of anecdotal evidence, but we don’t want to rely on just what we see on various forums (or god forbid, social media comments), so we’ve also been conducting surveys to get a more data-driven indication of how we’re doing. So far, 75% of Open Beta players are saying they’re satisfied with the progression system, while 18% are dissatisfied (7% neutral). That’s pretty solid, but we think there are some key areas where LoR can do better: 

Collection agency: While collections are building up at a great rate on average, that’s not the only way we measure success. We think a critical measure is how well players can achieve their shorter term goals, like completing a particular deck. Looking at that data again, we see about 1/3 players saying the decks they want to play feel unattainable, likely from getting left behind due to poor RNG in their Vaults (particularly for Champions). So, we’re planning to update reward distributions to tilt toward more agency and less randomness before launch. 

Experimentation: We also see that some players are uneasy experimenting because wildcards feel too precious and it’s too unpredictable when you’ll get more. Instead of experimenting, we’re seeing stockpiling, which is definitely the less fun thing to do (and sort of defeats some of the “progression” you would otherwise have if you were using them). Before launch, we’re planning updates to our progression system to increase predictability and encourage using wildcards, so players are more excited to experiment. There’s no way to capture the exact same experience as Open Beta and the first set of cards, but we’re thinking hard about how to make the next set (and every future set) an equally exciting experience. 

Welcome to All Players: At the end of the day, we want LoR to be a welcoming game for anyone interested in its strategic core gameplay. The results we’ve seen so far in Open Beta have shown us we can do better to accommodate all the kinds of players who want to engage with LoR: those who can’t play much outside weekends, those who want to grind all night, new players just joining the community, and would-be players who might have passed on LoR initially because of the higher time barrier to entry. We want everyone to feel like they have multiple options for picking up the cards and decks they want, and we think we can make the game more accessible to a broader swath of CCG fans. 

Personalization 

Finally, personalization. We’ve been delighted by the response of the initial batch of guardians and boards (T-Hex FTW), but we’re just getting started. We’re investigating ways to add more features to guardians and boards, but we also can’t wait to expand into new cosmetic lines as well, such as card backs, emotes, and card styles—expect to hear more details about these after launch. 

On the Horizon 

Besides these updates, in the short term the team is focused on mobile, global launch, and the next set of cards. We think it's critical for us to hit the commitments we’ve already made and establish our baseline of new content and updates. 

But what’s next after that? 

We’ve already talked about some of our plans to increase player expression in the form of new cosmetic lines, boards, and pets. But there’s something else we haven’t mentioned yet: modes and formats. 

One thing that has always kept us coming back to card games is the infinite number of ways they can be played; many of our favorite formats were even made by the community, in lieu of the original designers pursuing them. We believe that it’s when tons of cards meet an equally wide variety of modes that our game will truly shine, and we’re excited to start down that path this year. You can look forward to new ways to play with your collection, compete, and be recognized. 

Thanks for reading this long article on the state of the Open Beta. As we head towards launch, we’ll be back with more info about ongoing work, a date for the official launch of LoR (both PC and mobile!)—and maybe even sneak peeks at the next set of cards."

Miscellaneous

• This month's Twitch Prime loot is up, including a free mystery skin permanent for LoL! Redeem your reward here!

• The trade paperback of the Zed comic is now available for pre-order in the Riot Games merch store!

"He aims to protect his homeland of Ionia... even if he must spill innocent blood! The man known as Zed, formerly of the honorable Kinkou Order, is now the leader of a band of assassins. When the magically gifted psychopath known as Khada Jhin returns to kill again, Zed must confront his troubled past in more ways than one - including a reuinion with his former friend Shen, son of the master that Zed killed! Experience a tale of magic, revenge, and honor lost and regained. 

Includes Zed Issues #1-6
Written by Odin Austin Shafer
Artwork by Edgar Salaza"