- 6/11 Minor PBE Update - Futbol Ward Store Image, Up to Date Nidalee Changelist
- Summoner's Rift Dev Map Tour + Q&A - English, French, and Spanish Streams
European Service Update 11th June - Amsterdam datacenter
Following yesterday's server issues, Errigal, managing director of EU, has posted a European service update and confirmed yesterday's maintenance was to move the platform to the new Amsterdam data center."Summoners,
EUW is recovering from nearly 24 hours of downtime and I want to explain what happened.
The reason for the original maintenance was to move the platform to our new datacentre in Amsterdam. As mentioned in service updates between October and now, this migration is just one part of our commitment to get you the service you deserve. Aside from this, we’ve built an EU focused tech team, worked to improve service messaging, beefed up our ability to track service quality and set up Riot Direct in order to mitigate pain from ISP routing issues. The new datacentre is an important step, but it’s not the solution to all problems or the end of our commitment to giving you quality service.
Now onto what happened: We took the platform offline for migration maintenance at 3am GMT+1/4am CET and everything looked good until EUW went live again at 11am. This is when we (players and Rioters) saw that 3 days of match history, LP changes and purchases were missing. To figure out the problem we took the platform back down at 12pm.
Our options at this stage were:
- Restart the platform and acknowledge the loss of 3 days data
- Roll back to the old datacentre, figure out the problem and try the launch again in a couple of days
- Fix the problem on the new datacentre so it was live as quickly as possible
Anyone who finally won a promo series during those days will understand why option 1 was a last resort. We picked option 3 as it was the shortest route to the best long term state. Also, rolling back meant just as much downtime and would lead to another round of disruption when we went to launch again a few days later.
The amount of information involved was huge, so the assessment of what was missing, data transfer, validation, platform restart and QA took up until 3am GMT+1/4am CET today. In fact, as I write this, Leagues data is still being copied over and consequently ranked queues are disabled. If you want more details on any of the steps taken, Tmx has been keeping you posted with details of the data transfer issue here, and all the updates from 3am yesterday onward are shown in the official service status post here.
You can see from current platform health that even though the new datacentre was built with the latest hardware and thoroughly stress tested, adding millions of us into the equation can cause hiccups. The problems the team are focusing on now are:
- Getting the remaining Leagues data copied over so ranked can be re-enabled
- Fixing the slow population of friends lists when you log in that some of you are seeing and chat instability
- Re-enabling server transfers
- Getting the “codes” section of the store back online again
Looking back, we were overly confident in our ability to transition quickly and we should have shared that this was the datacentre migration so you guys could have prepared for possible issues. Aside from the issues mentioned above, teams are working on even better Internet connectivity, minimizing downtimes (planned and unplanned), and more easily expanding every part of the service.
In closing, we’re committed to keeping League of Legends available whenever you want to play. Long term this may mean we change specific features, hardware specifications or data centres, but our overall commitment to you is to provide you the quality service you deserve."
Riot tmx also entered the fray to answer a question about ping change EUW summoner's may be experiencing:
"From what we can see the majority of players actually get lower latency on the new servers. Sure, the distance still plays a major role (so part of Germany and players from Eastern and Southern Europe might have lower ping to FRA servers), but another aspect is how your local ISP peers with us. By peering I mean the exchange of a traffic between various networks, so if your ISP routes the traffic to us suboptimally, this might be the main culprit of higher latency. But this is also something we can get fixed: the beauty of us being plugged into Amsterdam Internet Exchange is that we can now peer with all the major ISPs in Europe. We launched 1 day ago and are already peered with hundreds of major companies and backbone ISPs, but this is still work in progress.
Additionally, we're constantly working on analyzing our players' traceroutes. If we notice that traffic of given network is incorrectly routed (eg from Spain to Netherlands through United States) we can speak to the companies to adjust the routing or go through another Level 1 provider. Currently we use all of the major Tier 1 carriers, including: Level 3, TeliaSonera, GTT or Verizon, so in worst case we can speak to your ISP support and find the optimal solution.
It's a wide topic, but to keep it short: if your ping is too high (this is again very subjective) we can analyze it through our Network Diagnostic page (it will be announced tomorrow in GD forums) and take proper actions. We will not do that if your ping increased from 35 to 60 though, because this is something completely normal."
When asked why they didn't share the maintenance involved the new datacenter, Riot Dieselnoi commented:
"Good point and well made. A number of others have brought this up too. To paraphrase: why didn't we tell you it was a server migration before the downtime happened?
The answer is we had a couple of reasons that were based around minimising technical risks, but we should not have put those reasons above keeping all of you informed.He continued:
Trusting you guys with as much information as possible is something we've been working really hard to be better at in EU and we were so disappointed in ourselves when we realised we messed up on what is an important step towards giving you the top quality service you deserve. You can consider this lesson re-learned and as we move on with whatever updates/service changes/issues come up next, we will include you in the conversation."
"We absolutely apologise for not telling you this was a migration ahead of time. We messed up by not doing so"
Riot tmx also commented on when the "codes" section of the store should be back up:
"Codes and Server Transfers are related to the same Store code change that our teams currently work on. It should be 1-2 business days until this is resolved."
Riot Natural20 also noted that ranked decay ( losing LP passively for not playing any games over a set amount of time ) has temporarily been disabled in the wake of the recent issues:
"You are right and we feel the same way. Ranked decay is off now to prevent that and we will message before we turn it back on."
Riot Draggles also confirmed that EUW is currently the largest server:
"EUW is the largest server! TAKE THAT, AreksF's friend! :D"
[ Update ] Looks like ranked queues have been re-enabled on EUW!
Here's Riot Nautral20 with the announcement and a little joke:
"Hi all,
This is Riot Natural20 using Riot TMX's account while he sleeps, and I get mine set as a redname.
- We have finished the migration of all the leagues data and Ranked queues are re-enabled. If you see any issues, let us know.
- ADDED: Ranked decay is off so as to no punish you all for our mistake and the long downtime. We will post again before it is turned back on
- In addition, we reset everyone to Bronze to level the playing field in the new datacenter.
Riot Nautral20 also left a reminder that the last three days of match history are recorded but WILL NOT show up in a players match history:JUST KIDDING about the Bronze thing!
Ranked is up!
-Riot Natural20"
"Yeah it is a known issue. the last 3 days of match history are gone and we cannot restore them. This ONLY affects the display of the games played. Any LP gained (or lost), and any IP gained is correct on your account.
Sorry about that."
An "Issues in progress" thread, tracking several issues the team is still working to fix on EUW, can be found here.
The Pentakill Q&A Rocks out on June 12th
"The team behind Pentakill has agreed to unplug their guitars and set down their hitty-sticks for a Q&A session on 6.12! They’ve agreed to bless your firstborns, scribble out a few guitar tabs and chat away about all things Pentakill related*, so come armed with your questions and neck braces because things are about to get HEAVY.
In the meantime, crank things up to 11 and check out the Pentakill website here!
*They will do at least one of these things. Possibly two. "While not specified in the main post, the article preview notes it will start at 2:25 PM PDT.
New Match History Alpha Live on BR
From the announcement yesterday:
".After taking into account the size of each region, the available architecture and hardware, and the current player experience, we’re kicking off alpha-testing in Brazil. We’ll be testing server load along with how players use the new match history as we refine the online experience. Our wider beta release will follow in other regions, using the info we get from the alpha-test. "While it is only available for BR summoners atm, feel free to preview the new options for yourself!
No comments
Post a Comment