Regarding Great Lakes' Revert

[Wallenstein]

So most of us probably know Great Lakes was reverted today to fix an issue that caused almost all locked down and secure items in Luna city homes to vanish. But, how can we be sure this won't happen tomorrow at server-up? As far as I know the issue took place somewhere between early this morning and 10:00am ET, or right around the time of the maintenance period.

Regardless, backups of the servers should be made every 6 hours or so instead of 24. In an MMORPG thousands of players accomplish things in a 24-hour period, and to revert an entire day is a big loss to some. A 6-hour revert is nowhere near as much of a hardship. Why isn't this possible? Why must all reverts go back 24 hours?

Just food for thought..

 

[Mapper]

When each server goes down each day isnt that it backing up? Surely it couldnt backup every 6 hours unless it went down every 6 hours.

 

[Viper09]

Well from what Chrissay had already said, the issue has been resolved. Surely they would want to fix whatever bug caused it so they don't have to revert the shard again.

No on is asking the important question. Did they fix the problem or just revert it in the meanwhile.

The issue has been resolved.

 

[Devil_Woman]

From EA Support in answer to when shards go down for maintenance:

The Ultima Online servers are constantly in the process of "backing up" the game. It takes about 45 to 90 minutes for each cycle to complete, though the length of time this takes depends greatly on the amount of data that is stored in each world.

So, obviously back ups are run of each server several times each day. The problem may be that whatever issue caused the problem was imbedded in the later back ups therefore the revert has to go back to a known "clean" back up.

And all reverts are not 24 hours. Every time the shards are taken down in the middle of the day for a publish, and the last time for that was a week or two ago, they "revert" to the most recent back up. When was the last time you recall a publish in the middle of the day reverting you to the day before?

 

[KrazyFrog]

This conversation brings back really old memories of days past. It use to be that you would actually get a message every few hours when the server was performing its backup. Of course, back then there was not nearly as much to back up and they didn't take hours to complete.

Which things brings back the memories of old when there would be "server down wars". It use to be you would get a message informing you that the server would be going down "soon" for its maintenance and all chaos would break loose all over because you knew none of it was going to be saved (except that one time...).

Fun days those were... back in '97/'98

 

[Beastmaster]

The part that I find most upsetting is that we are told only "the issue was resolved". Well DUH! In EA's eyes that could simply mean a)something went wrong and stuff was lost, b)we reverted the shard, c)issue resolved. In otherwords the items were returned and thus they resolved the issue of missing items. It does not speak to what caused the items to go missing in the first place.

I'm sure some of you will say, well it was a bug, glitch, etc., and EA fixed it because they are a huge gaming company and surely they want to make things right.

Others will say, EA figures it was a glitch but has no clue what it really was so they just crossed their fingers and reverted.

Personally, I think it was somewhere in the middle. Like they found a likely cause, but still aren't sure it actually was the cause and cannot reproduce it. So we continue on and hope it doesn't rear its ugly head again.

 

[JC the Builder]

You have to go to a backup where the issue doesn't exist. If you go to the 1 hour backup there are items missing. The 6 hour backup there are items missing. The issue began at server up, so they had to revert to that.

If you ever login after a server crash, you usually don't lose more than an hour.

 

[Lady Michelle]

So most of us probably know Great Lakes was reverted today to fix an issue that caused almost all locked down and secure items in Luna city homes to vanish. But, how can we be sure this won't happen tomorrow at server-up? As far as I know the issue took place somewhere between early this morning and 10:00am ET, or right around the time of the maintenance period.

Regardless, backups of the servers should be made every 6 hours or so instead of 24. In an MMORPG thousands of players accomplish things in a 24-hour period, and to revert an entire day is a big loss to some. A 6-hour revert is nowhere near as much of a hardship. Why isn't this possible? Why must all reverts go back 24 hours?

Just food for thought..

I wasnt happy about it, but I would rather lose 24 hours than see anyone lose years of things they have worked for, or collected.I am just glad they fixed the issue, and hope it doesnt happen again.

 

[Basara]

IF people would just check the knowledge base, Stratics, or any of the other sources (trying to remember if JC's wiki includes the information) -

It has been stated repeatedly that UO backs up every 45 MINUTES, in a staggered pattern for the specific server farm. That's why, for example, shards go down for morning maintenance, in usually the same order, as the game completes the 45 minutes for each shard before dropping it. So, for example, a couple weeks ago, Lake Austin would go down first, Pacific & Origin last, of the Western US servers. Shortly after the latter two dropped, LA would come back up, and the time between it coming back up and Pacific coming up was typically the same time difference between them dropping.

If it's a simple system failure, reverts are typically back up to 45-90 minutes, if it's known that the failure didn't corrupt the save data, or wasn't caused by something in the software that a restore would reinstitute.

Things like what happened to GL, get complicated, especially if they occur when the server guys and programmers aren't at work. consider how many 45 minute backups occurred between "The Vanishing" and someone finding out about it - especially if the GMs are going on about "can't replace lost items" off their script, and don't realize that something major has FUBARed with the lockdown/decay system for an entire subserver (and, that would probably be partly the pagers' fault, if they didn't phrase the correspondence properly to the effect of "all the houses in Luna are losing their contents!" as opposed to "My stuff is gone!").

 

[Tinsil]

Shouldn't they be able to back up certain things easily though?

Like character skills, etc then revert?

 

[Experimental]

Is it possible that this is what happened to Hemi??? Just a thought.

 

[Tinsil]

Is it possible that this is what happened to Hemi??? Just a thought.

Naw. His supposedly wasn't even owned by him anymore.

 

[blueturtle]

just a thought what if some1 xsftered a char to gl yesterday, guess it just got deleted for good?

 

[JC the Builder]

just a thought what if some1 xsftered a char to gl yesterday, guess it just got deleted for good?

You need to contact support for EA to restore the character. They have the logs of everything transferred.

 

[Tinsil]

You need to contact support for EA to restore the character. They have the logs of everything transferred.

Ouch. Transfers weren't backed up?

That's goooing to suuuuuck for my few buddies that transferred to GL yesterday. They get to deal with EA support. lol

 

[Godiva_DF]

On the other hand if you transfered stuff FROM GL to another Shard did you legally dupe the items?

 

[Tinsil]

On the other hand if you transfered stuff FROM GL to another Shard did you legally dupe the items?

Was wondering that myself also. Of course I don't know that EA would admit if this was the case.

 

[Fink]

Regardless, backups of the servers should be made every 6 hours or so instead of 24. In an MMORPG thousands of players accomplish things in a 24-hour period, and to revert an entire day is a big loss to some. A 6-hour revert is nowhere near as much of a hardship. Why isn't this possible? Why must all reverts go back 24 hours?

Perhaps if you're playing 6-12 hours a day, and need 6-hourly backups, you should simply appreciate your good fortune at having so much free time to play. Unless you're scripting 24/7 I can't see that a day's setback is too big a deal for most people.

It shouldn't take you too long to recoup your losses. I understand it, a few hours' gain lost to a revert would be minimal compared to the losses some would have sustained with their homes torn open and their possessions wiped out.

 

[Wallenstein]

You have to go to a backup where the issue doesn't exist. If you go to the 1 hour backup there are items missing. The 6 hour backup there are items missing. The issue began at server up, so they had to revert to that.

If you ever login after a server crash, you usually don't lose more than an hour.

I understand that, but the server had no issues for the entire day prior to the shard revert. So, to have the only backup available be the one that was made 24 hours earlier during the previous day's shard backup means that there were either; (A) no backups made for that entire 24-hour period prior to the shard revert, or (B) corrupt shard backups made for that entire 24-hour period prior to the shard revert. Either way, an issue somewhere needs to be addressed.

This, of course, based on the posting that shard backups are made "multiple times per day". Apparently this isn't so.

 

[Shamus Turlough]

Ever stop to think that maybe there was a problem in the code that was only triggered a few hours after the Monday morning server up? If they needed to revert that code (or whatever) then they would have had to go back to the latest instance of the "good version". Sometimes problems are not immediately visible!

 

[Cetric]

I understand that, but the server had no issues for the entire day prior to the shard revert. So, to have the only backup available be the one that was made 24 hours earlier during the previous day's shard backup means that there were either; (A) no backups made for that entire 24-hour period prior to the shard revert, or (B) corrupt shard backups made for that entire 24-hour period prior to the shard revert. Either way, an issue somewhere needs to be addressed.

This, of course, based on the posting that shard backups are made "multiple times per day". Apparently this isn't so.

UO is constantly being backed up, the problem probably showed its head early monday, and the backups between then and the revert had the problem on them. That is why in the IT world you keep backups for different intervals of time, and the past shows that this is something they have done effectively

 

[Chrissay_]

We reverted to the last verifiable point prior to the issue starting. We didn't want to take any chances with some players getting restored while others weren't.

 

[Xalan Dementia]

Yes reverts are an annoyance, but not devastating. Everything lost can be regained, its not like someone received a one of a kind item then lost it due to the revert. (cloaks, crafted items, loot, it all can be regained.)