Hello! Thank you for your patience during our extended downtime. All indications are that things are on schedule. Thank you for being here!
LOTRO Twitter, March 6, 8:35am
Famous last words?
Standing Stone Games had been reminding everybody via various sources, including Twitter, that both Lord of the Rings Online and Dungeons & Dragons Online would be down pretty much all of Wednesday, March 6th, for what was described as “data center work.” Time to muck with the servers I guess. And, as the quote above indicates, hours into the work things looked good.
And then things began to unravel and the time to complete started to creep forward. Midnight of the first day moved to 5am of the next. That became 11am, and then 5pm, and soon the time had moved into Friday and began to threaten weekend play time.
These things happen. If you have a server farm and bring all the machines down… machines that have been up and running sometimes for years at a stretch… some small percentage are going to fail when you go to bring them back up. And that leaves out any changes you might intend to make to them or the infrastructure while things are down.
A computer looks almost like an appliance, nicely packed away in a box. But it is not. It isn’t even close. An appliance is a serious of parts… and even software… designed specifically to work together and do a very specific set of tasks.
Your computer is disguised chaos, a thousand points of failure flying in tight formation, hacked together from a bunch of components that mostly work together. I’ve worked on the hardware end of things and I’ve had various suppliers show up to tell us that the part we use is being discontinued, but they have a replacement that is pin compatible and functionally the same… and then had that part stop production because it turned out that the old part was actually out of spec or our design only worked due to some unintended behavior of the part of some such and now we have to hire somebody to sit at the end of the line and hand solder a surface mount resistor on one of the pins on every single unit with the new part while we work on a redesign. True story.
So I feel for those members of the team trying to get things back up and running.
And come Saturday morning there seemed to be light at the end of the tunnel. At about 5am Eastern or 2am Pacific time, the announcement went out that the servers were up again.
The game worlds have reopened. Thank you for your patience as we worked through these various data center issues in recent days.
LOTRO Twitter, March 9, 2:06am
But we were not done yet.
UPDATE: We are bringing back down the game worlds while we troubleshoot an issue with characters not receiving their proper account unlocks.
LOTRO Twitter, March 9, 2:36am
That persisted for a while, but about four hours later the servers were opened up again.
The game worlds have reopened. We appreciate your patience during this downtime, and thank you for being here.
LOTRO Twitter, March 9, 6:56am
That last one seemed to do it… but I held onto this post for another day because I wouldn’t be surprised if they found yet another glitch they needed to take things down to fix.
And, having gone more than two days past their planned outage, SSG has some offers for people to make them feel better about having been kept offline. It is in the form of a FAQ.
Sales will be extended, everybody will get a bonus pack of various boosters, and VIP accounts that have logged in over the last 30 days will get 250 LOTRO Points.
DDO players will get something similar and have their own FAQ.
And so it goes. They fared better that SOE did back in May 2011 when they were down almost 13 days, though that was due to a hacking incident. And, of course, there is Alganon, which has been down since November 2017.
Now hopefully things will settle down a bit and SSG will get around to telling us when Moria is going to unlock on the LOTRO Legendary server.