The Night the Lights Went Out in Norrath

A memory of the Great December Downtime in EverQuest II

It was just about ten years ago.

EverQuest II had be live for a little over a month.  There were troubles.  After having a couple weeks to itself in the market, World of Warcraft launched and the harsh comparisons began.  It wasn’t that EQII didn’t have some better features than WoW… for example, I have always felt that EQII’s version of in-game maps was superior… but in a market that, up until that moment, had been dominated by EverQuest, it was something of a fight to see which of the two would become EQ’s true successor.  After all, EQ was more than five years old at that point, and who plays a five year old game?  It was practically on death’s door, waiting to hand off to a new generation.

And in that fight, EverQuest II was not faring well.  Some people I knew who came from EverQuest had either gone back or moved on to WoW at that point.  EQII was down, but not out.  The game was still growing, this still being the age of the slow ramp rather than the sudden spike.

SOE was trying to fix things that were becoming a hindrance to players.  We were destined to get floating quest markers over NPCs and changes to the woefully inadequate quest log and the first of many revamps to the crafting system.  SOE knew they had to adapt.  They could see WoW.

In our guild, a mash-up of players from the EverQuest guild Knights of Force, the TorilMUD guild Shades of Twilight, and a few fellow travelers from the Old Gaming Veterans clan, things were holding on.  A few players had dropped out of the game, though they were mostly the non-MMO players from OGV who went back to playing Desert Combat.  But for the most part we were holding in there, grouping up to run through zones or crafting away.

On voice coms we mocked those who ran off to Blizzard’s cartoon MMO, though there was a feeling that maybe EQII wasn’t the true successor to EQ.  The early buzz around Brad McQuaid and Vanguard had started.  That was going to be the real deal.  But for now, EQII was the best we had, so we put up with locked encounters and experience debt and system requirements that burnt out more than a couple nVidia 6800 GT cards in our guild. (I was running with a 6600 GT card, which meant I had to keep the graphic settings modest, but I also didn’t need to replace the damn thing… or my power supply… over and over like some.  There is probably a post in “video cards I have run” some day.)

We were coming up to a good stretch of game play.  The holiday’s were coming.  Like many people in our guild, I had a stretch of time off and was looking forward to some good, solid chunks of game play time.

Then, as we were headed to that first weekend, SOE applied some updates and restarted the servers.

And they did not come back up.

Here is where the details get a bit vague.  I recall the game, or at least our server, being down pretty much Friday night through Sunday, a huge patch of premium gaming time washed away.

But concrete details are not easy to come by.

The SOE forum posts, all the status updates and such, have long since been washed away by changes to the forum software.  The conspiracy nut in me suspects that they change the forums every few years just to dump bad memories and excess baggage.

I mentioned that Massive Magazine did an article about the incident in their first issue.  That was just about two years after the event, when memories of the whole thing were sharper.  I think I still have a copy stuffed away in a box.  But we packed up and moved houses since then, so if it is in a box somewhere, it appears well hidden.

Digging around the web, I found some references to what happened.  Terra Nova mentions the event, but links the SOE forum thread, long since gone, and a site called MMORPGDOT, also a distant memory. (And looking at the internet archive only shows them making a very brief mention of the event.)

Likewise, there is a mention of the even happening at Slashdot, written by Michael Zenke, which links to a few sources, including the SOE forums, all of which are long gone save the Terra Nova post mentioned above.

Other news sites that cover MMO don’t go back that far (Massively) or went through changes or otherwise appear to have purged their archives beyond a certain point.

This is one of those points when I wish I had started blogging sooner.  Two years earlier and I would have written something about this, as I wrote about the great Sony hacking of 2011 which brought down both the PlayStation Network and SOE. (Not to be confused with the great Sony hacking of 2014.)

PSNDownSo I started nosing around at various blogs just to see what people were writing about when the downtime occurred.  A lot of the self-hosted blogs from that era have disappeared, or have had database problems, but a few still linger. (My Great Survey of Linking Blogs post helped out.  I will have to do another of those at some point.)

However, it did not seem to garner much attention.  The event coincided with Raph Koster’s book, A Theory of Fun, hitting the shelves.  There was a discussion of niche games in the MMO market, which still seems relevant today, and something about what WoW would mean to Dark Age of Camelot. (Or something of a contrary view.)

The only real mention I could find amongst the few blogs remaining from the time was by Tobold, for whom the server down time meant moving to WoW ahead of his initial plan. (Poking around also got me to this then-so-current WoW vs. EQ2 post at GameSpy.)

So here I sit, vague memories swirling, wondering how big of a deal the whole thing really was at the time.  Certainly evidence of the event has faded from the internet and worse things have happened.  Didn’t Arche Age just have a similar incident.

I think our own guild was emotionally entrenched in EQII at the time, so we just carried on once things were up again.

Do you remember the Great December Downtime of ten years ago?

Can you find anything else about it on the net?  If you find something I’ll add a link to the end of the post.

7 thoughts on “The Night the Lights Went Out in Norrath

  1. bhagpuss

    Hmm. I have literally no memory of that whatsoever. And given that both I and Mrs Bhagpuss were playing EQ2 and pretty much only EQ2 from September 2004 (beta) until somewhere around May2005, usually clocking up 40+ hours a week, I really would expect to remember it.

    I wonder if it was specific servers rather than the whole game? Our server at launch was Steamfont (later merged with Oasis), which was always one of the smaller servers. I can recall a couple of times in EQ when servers I didn’t play on went down for prolonged periods – I seem to remember one being down for three days once – but I can’t remember it ever happening to a server I was playing on at the time.

    Liked by 1 person

  2. gnomenecro

    I checked our old private guild forums and saw that this unscheduled downtime happened on December 17th after a patch the day before. One of the guildies said that the servers were brought down to address an issue where people were losing quest logs, call of Qeynos/Freeport abilities, and spells/combat arts after zoning into Commonlands. It was a roll-back. One guildie reported losing 2 levels (21 back down to 19), about 6 hours worth of questing progress,and he lost status on 15 quests which were just wiped out of his quest log after the smoke settled. He logged off in disgust and didn’t check to see if he had lost his newly-upgraded adept-level spells. After that there is a significant drop-off in posts concerning EQ2. Several other old guildies dropped in to say come on over to WoW, it runs on older video cards and the game rocks.

    My first kid was born the second week of December 2004 and as newbie parents our MMO career went on hold before this great patch/rollbaåck fiasco so I missed out. When I did resume playing MMOs I went to WoW until Vanguard, then after Vanguard imploded I went to EQ2 where I have been off and on over the years.

    Like

  3. gnomenecro

    I forgot to say that the first report of the damage done from the rollback was on December 18th around 8:30 p.m. EST best I can tell. So it looked to be a bit over 24 hours. Our server was and is Guk. No idea how the other servers fared, but I assume similarly. I suppose the whole thing could have been brought down again. As TAGN says, the record is a bit incomplete.

    Like

  4. C. T. Murphy

    I love comparing these two games, if only because that’s what the magazines love doing. In hindsight, it is crazy to actually put these two next to one another since WoW has been such a behemoth success and EQII … well, hasn’t been much of anything. There are certainly things I love and still love about EQII, but WoW just seemed so much cleaner and more modern from the start. Well, except for the instanced world maps – that took a little while for Blizzard!

    Like

  5. Shintar

    That WoW vs. EQ2 article was nostalgia gold. I especially loved this quote: “Players can gain their first level as fast as 15 minutes after they begin playing.” Only 15 minutes to level 2? No way!

    Like

  6. gwjanimej

    I never played EQ2 in any great amount of time, but went digging through GWJ’s MMORPG forums to see what I could find. What I found seemingly corroborates the mid Dec timeline, with it being on or around the 17th. Downtime seems to have run up to 72 hours in some cases. Smedley published a letter to the community, which was copypasta’d to the thread on the downtime, which I’ll copy again here:

    An Important Message from John Smedley 12/18/2004
    Hello Everyone,
    We sincerely apologize for the downtime this weekend. We had some hardware issues that caused us to have to tread very carefully in bringing the game back up. We wanted to triple check everything to make sure there wouldn’t be any major problems, and it’s taken a while to both fix the problem (we had to fly out engineers from one of our hardware vendors) and to bring each of the servers back online because of the careful validation process we need to go through. We have to do this very carefully and we hope you can understand our desire to do this right. We”re trying to balance that with getting the game back up as quickly as possible. So far most servers are back up and the rest should be up in the next 4-6 hours, hopefully less. I realize that ETA”s that come and go can frustrate people, so we do our best to give you the most realistic estimates possible.

    Out of respect for our players, we’re doing the following:

    1) Every player will receive 3 free days.

    2) Early this week (Monday or Tuesday) we will have a 24 hour period where we will have accelerated XP gain. Details are still being worked out and we’ll communicate this to you as we firm up the plans.

    We are committed to providing you with excellent service. Our team has been here for over 40 hours straight, and I can tell you our dedication to doing right by our players is the thing that’s foremost on everyone’s mind here at SOE.

    Thank You,

    John Smedley

    So there’s that. Here’s a link to the thread in question with some discussion on it as well:

    http://www.gamerswithjobs.com/node/12238

    Like

  7. Brian 'Psychochild' Green

    I don’t think many people wrote about it for two reasons: for developers, we have an unwritten rule that we don’t tear into other games too savagely. And, I think pointing out this pain would have been kicking SOE when it was down. It was hardly news after it had happened, and there really wasn’t much commentary to write about to dig in behind the scenes.

    And, most of us developers just pray this never happens to us….

    Like

Voice your opinion... but be nice about it...

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s