Update on League of Legends service in NA

Riot has released a statement about the continuing problems plaguing the NA servers.

Summoners,

Lance, aka CaptSturm here. For those of you who don’t know me, I’m charged with maintaining the health of the League of Legends live service around the world. As you know, it’s been a rough week here in North America.

We are painfully aware of the problems many of you have been experiencing:

-Extended outages: The North American service has been unavailable for significant stretches of time
-Long queue times: We’ve had to throttle logins and concurrent players due to these issues
-We’ve also confirmed that some players are missing content: runes, rune pages, champions, etc.

This is frustrating for you, and we feel that frustration deeply. This isn’t the service you’ve come to expect and it isn’t what we want to deliver. It’s been code red for engineering and support Rioters, and we’ve been working literally around the clock to identify and resolve these problems.

Many of you have asked for more information on what’s causing these problems, and we understand that desire – I’m going to try my best to give you some insight into what we’re working on, but please understand that our first priority is to put out the fire rather than exhaustively document it.

-This patch is like an iceberg – what you see on the surface is small in comparison to the behind-the-scenes infrastructure changes it includes.
-The problems we’re experiencing are database-related – specifically, it’s a data throughput issue. Our infrastructure handles a massive volume of data and requests every day, and the backend changes in this patch introduced a new bottleneck.
-We put every patch through the wringer including load testing and PBE player testing, and this patch passed those tests. The problems didn’t crop up until we pushed it to live. This has exposed some load testing gaps that we’re going to address (simulating millions of concurrent players is truly challenging).
-Over the last 48 hours, we’ve made substantial hardware configuration changes that we believe will address the problems. We’re actively monitoring the service, but we’re still not out of the woods.

I know the thing you want to hear most is when everything’s going to be back to normal. I’d love to give you a timeframe, but we want to be absolutely confident in our solution before signaling the all-clear. Our number one priority is restoration of a stable game experience.

Thanks for your patience. I’m going to continue keeping you guys up-to-date as we work to fully restore League of Legends to the service you’ve come to expect.

Salute,
'Sturm

- CaptSturm, Sr. Producer, Live Service, via the official forums.