Return of the Ring – LOTRO Up Again

Hello! Thank you for your patience during our extended downtime. All indications are that things are on schedule. Thank you for being here!

LOTRO Twitter, March 6, 8:35am

Famous last words?

Standing Stone Games had been reminding everybody via various sources, including Twitter, that both Lord of the Rings Online and Dungeons & Dragons Online would be down pretty much all of Wednesday, March 6th, for what was described as “data center work.”  Time to muck with the servers I guess.  And, as the quote above indicates, hours into the work things looked good.

And then things began to unravel and the time to complete started to creep forward.  Midnight of the first day moved to 5am of the next.  That became 11am, and then 5pm, and soon the time had moved into Friday and began to threaten weekend play time.

These things happen.  If you have a server farm and bring all the machines down… machines that have been up and running sometimes for years at a stretch… some small percentage are going to fail when you go to bring them back up.  And that leaves out any changes you might intend to make to them or the infrastructure while things are down.

A computer looks almost like an appliance, nicely packed away in a box.  But it is not.  It isn’t even close.  An appliance is a serious of parts… and even software… designed specifically to work together and do a very specific set of tasks.

Your computer is disguised chaos, a thousand points of failure flying in tight formation, hacked together from a bunch of components that mostly work together.  I’ve worked on the hardware end of things and I’ve had various suppliers show up to tell us that the part we use is being discontinued, but they have a replacement that is pin compatible and functionally the same… and then had that part stop production because it turned out that the old part was actually out of spec or our design only worked due to some unintended behavior of the part of some such and now we have to hire somebody to sit at the end of the line and hand solder a surface mount resistor on one of the pins on every single unit with the new part while we work on a redesign.  True story.

So I feel for those members of the team trying to get things back up and running.

And come Saturday morning there seemed to be light at the end of the tunnel.  At about 5am Eastern or 2am Pacific time, the announcement went out that the servers were up again.

The game worlds have reopened. Thank you for your patience as we worked through these various data center issues in recent days.

LOTRO Twitter, March 9, 2:06am

But we were not done yet.

UPDATE: We are bringing back down the game worlds while we troubleshoot an issue with characters not receiving their proper account unlocks.

LOTRO Twitter, March 9, 2:36am

That persisted for a while, but about four hours later the servers were opened up again.

The game worlds have reopened. We appreciate your patience during this downtime, and thank you for being here.

LOTRO Twitter, March 9, 6:56am

That last one seemed to do it… but I held onto this post for another day because I wouldn’t be surprised if they found yet another glitch they needed to take things down to fix.

And, having gone more than two days past their planned outage, SSG has some offers for people to make them feel better about having been kept offline.  It is in the form of a FAQ.

Sales will be extended, everybody will get a bonus pack of various boosters, and VIP accounts that have logged in over the last 30 days will get 250 LOTRO Points.

DDO players will get something similar and have their own FAQ.

And so it goes.  They fared better that SOE did back in May 2011 when they were down almost 13 days, though that was due to a hacking incident.  And, of course, there is Alganon, which has been down since November 2017.

Now hopefully things will settle down a bit and SSG will get around to telling us when Moria is going to unlock on the LOTRO Legendary server.

6 thoughts on “Return of the Ring – LOTRO Up Again

  1. Knifesedge Games

    Thanks for this. I tried to log in for the first time in a while but it wouldn’t load. Had no idea what was going on and would probably have been put off trying again for a while.

    Like

  2. flosch

    Ouch. I feel with them, that week / weekend must’ve been the kind of crunch that you don’t even dare wishing onto your worst enemy.

    Regarding Alganon, I can only repeat what a very smart person said about the game:

    “I think the fact that I only just now read that the game has been down for over a year is probably worse sign than the fact that it has been down for over a year.”

    Like

  3. flosch

    I’m pretty sure I hadn’t heard about it until then. Not a good sign either. Then again, not sure I would’ve played it either way, so maybe it’s not such a huge problem anyway…

    Like

  4. Redbeard

    Hmmm…. I wonder if they changed data structures and performed a major OS upgrade. The former based on the clue of users not having their unlocks set properly, and the latter based on having dealt with Red Hat 6.5 to Red Hat 7.x upgrades. Could also be fixes for Spectre/Meltdown too.

    Like

  5. Pendan

    On Friday night when I could not get on I went to the official Standing Stone Games Lord of the Rings message board. The only post by a developer was about the extended Wednesday down time. No updates were posted on the official message board about the status after Wednesday.

    Like

Voice your opinion... but be nice about it...

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s