EUW maintenance to address recent server and chat problems

There will be a server maintenance for EUW tonight, aimed at fixing the recent problems plaguing the server.

Here's the Riot NOC announcement:

"Summoners, 

Starting at 02:30 GMT+1, ranked queues will be disabled in preparation for maintenance at 04:00 GMT+1. At 04:00 GMT+1, the servers will be shut down and all games currently in progress will end in a draw and stats will not be recorded. We estimate the battlegrounds will be unavailable for 5 hours. 

Thank you for your patience."

Which Chager elaborated as being:

"We are having a proactive maintenance early this morning to address some of the service issues that have been happening in EUW. I wanted to raise some transparency on what specifically we’re doing tonight, and what these actions will get us in terms of stability. "

Continue reading for more information on this EUW maintenance

Here's Chager with the full details on what this maintenance is hoping to address:

"Hello Summoners,

We are having a proactive maintenance early this morning to address some of the service issues that have been happening in EUW. I wanted to raise some transparency on what specifically we’re doing tonight, and what these actions will get us in terms of stability. 

This morning’s maintenance will provide: 

• Code update for our servers: This will provide upgrade to specific servers and reduce internal network traffic.
• Upgrade DB: One of our DBs is running on an older software version. We are upgrading this to make sure it isn’t a bottleneck for the chat issues you have experienced.
• Increase & Add Monitoring: We’re adding and adjusting several monitoring tools tonight. The intent of this is to gather more data, both from different systems and more in depth than what we normally gather. This will help us diagnose any issues that may happen.
• Power Cycle Server Hardware: Power cycling the hardware gives us confidence that it's functioning correctly.
• New Additional Chat Hardware: We added chat hardware late last week and were able to prep additional hardware that we’ll be putting in tonight. This helps us by increasing the number of max chat connections we can handle, and also distribute the load across more servers. 

All of this is intended to eliminate potential problems that we can’t see, and also help reinforce the infrastructure we already have. We are not 100% confident that the above changes will eliminate our problems, but they will eliminate symptoms and allow us to narrow in on the actual root cause and fix it.

We’ll keep updating everyone as we have more information. 

Thanks,
-Chager"

Elaborating on the data looking to be gathered, he noted:

"Internal systems data for how specific pieces of hardware, software and network traffic are operating. Several of the systems have additional debug monitoring that provide more granular info that we don't normally have on, but in this case we want to make sure we capture."

In a separate thread, Chager discussed the ongoing chat issues that lead to the maint.:

"Hello Summoners, 

I wanted to stop and give you an update on the ongoing chat issues we're seeing in EUW, what we've done to date, and what we're in the process of doing to get things back to the quality of service you deserve. 

Last week, we experienced multiple chat drops in EUW during the evening (heavy load times). At that point, it looked like a potential hardware issue, so we dramatically upgraded the chat database, and also added in additional hardware to the chat infrastructure to reinforce the system. While we believe this helped, there were still issues that needed to be fixed. After that we also found a bug in the 3.12 client code that magnified the number of chat connections on the platform. A hotfix for that went out on Friday just before the weekend.

We had no chat issues on Friday or Saturday, but saw a new issue Sunday evening and reoccurring today. Based on the symptoms we saw yesterday and today, we believe we are looking at a different issue than earlier in the week. At this point, we are looking at every connection, configuration, and hardware/network infrastructure around chat to determine the root cause. We are making some manual configuration changes right now, and will continue to dig and monitor. 

We are truly sorry for the pain this has caused over the weekend, and will keep pushing to make the EUW experience rock again. 

Thanks,
-Chager"

 He continued:

"Regarding compensation; at this point we are working to get chat in a healthy state and to get ranked back on so everyone can play.

Once we are stable, then the decisions about compensation will start to be made. We know there's been consistent player pain in EUW and we want to make sure we address that, but first let's get everything stable and then we'll go from there."