Extended Server Maintenance (Monday 4/29/2019)

[MattScott 15245]

Hi all,

 

I am looking at the data issue with my team, and there is no easy answer.

Right now we have two choices.

 

1) Leave it the way it is and work on fixing the accounts that lost paid items. We have records on the payment side, so it should be easy to restore those. The team is exhausted, but over time we can also try restoring some of the new data in a separate area which would allow us to verify some of the bigger lost progression items / in-game items in order to grant those back to players.

or

2) Take the servers down and try another round of restoring data from a different set of possibly newer backups. We already know one of the databases from this newer set of backups is significantly out of sync and older. So there is risk that we will introduce a whole bunch of problems with incompatible data.

Personally, as painful as it is, I'm going to recommend that we plow forward and escalate the support tickets for the players who lost real money purchases.
Apologies for the issues.

 

Sorry,
Matt

[ChrisKrypton 15]

1 minute ago, tiago0182 said:

I can not continue like this

As I mentioned Stand by they may roll up the server and you will get everything back ..... hopefully.

[ChrisKrypton 15]

10 minutes ago, MattScott said:

I am looking at the data issue with my team, and there is no easy answer.

 

Well as the server stands right now, on what is currently loaded with the 4/23 data. Can you save this as it stands so you can always go back to this worse comes to worse no matter what.

 

Try taking a look at the files that you have that are possibly more updated and try throwing them in and see what happens. I will throw this as to my own ignorance but, a backup file..... is it not a file compiled of many files, can't you just look at the day it was compiled?

 

I guess im confused on how old data gets mixed in with new data, I am guessing you are saying that you plan to patch in new data on top of whats already there rather than wiping completley clean whats there, and placing other data in it's place.

Edited May 4, 2019 by ChrisKrypton

[Sooney 7]

ty Matt, I undertand that trying messing again wih the server could bring us back again to relive these past days again. Now you have the server going, let's keep it that way and face the new problems aside...

 

I wonder if there would be a way of mergin the real last account data, from some other database in those backups, with the actual one we are seeing now in the server.

It's what i asked in my support ticket, idk if it's a viable solution as we aren't so many players...

 

But take a rest whenever you can, please, your team deserves it. I'll wait longer as i've been doing all this time.

Cheer up.

[Sang 66]

Losing game-play time and gains is a non-issue for me. I lost no real money so for me, there isn't anything worth more downtime for. 

I got my wastelands back and that's all that matters.

[dougman10 0]

Great, I lost 8 lvls and items, so much waiting for that

[StubbyClaps 74]

Just now, Sang said:

Losing game-play time and gains is a non-issue for me. I lost no real money so for me, there isn't anything worth more downtime for. 

I got my wastelands back and that's all that matters.

Not everyone has been end game for years like some of us so i see their issue

[Grelko 33]

32 minutes ago, MattScott said:

Hi all,

 

I am looking at the data issue with my team, and there is no easy answer.

Right now we have two choices.

 

1) Leave it the way it is and work on fixing the accounts that lost paid items. We have records on the payment side, so it should be easy to restore those. The team is exhausted, but over time we can also try restoring some of the new data in a separate area which would allow us to verify some of the bigger lost progression items / in-game items in order to grant those back to players.

or

2) Take the servers down and try another round of restoring data from a different set of possibly newer backups. We already know one of the databases from this newer set of backups is significantly out of sync and older. So there is risk that we will introduce a whole bunch of problems with incompatible data.

Personally, as painful as it is, I'm going to recommend that we plow forward and escalate the support tickets for the players who lost real money purchases.
Apologies for the issues.

 

Sorry,
Matt

I lost a few levels, items, my "first purchase" items, about 50 quests, 30 in a bunch of trade skills, gear, weapons, mail items,  etc etc.

 

I still say take door number ONE.  If you take door number two, there's always the chance of a full world wipe, or another very long downtime. Especially if you say the data is out of sync.

I vote to leave it how it is.

Edited May 4, 2019 by Grelko

[tiago0182 5]

33 minutes ago, MattScott said:

2) Take the servers down and try another round of restoring data from a different set of possibly newer backups. We already know one of the databases from this newer set of backups is significantly out of sync and older. So there is risk that we will introduce a whole bunch of problems with incompatible data.
 

better take a rest and try again next week
i cant play like this, I lost many hours of gameplay
i hope you guys can fix
good luck

[MattScott 15245]

Hi all,

 

We are going to try getting our secondary environment up and running.

Then we can test a new set of data. This wont be a super fast process, and many of my team of recuperating this weekend.

Bear with us.

 

My advice for current players would be to not go crazy. If we can confirm a solid way to restore the 2 weeks of missing progress we will.

But that will mean losing anything since the servers came back online.

Thanks,
Matt

[ChrisKrypton 15]

I understand that there are going to be mixed feelings about this, and I guess whatever happens is whatever happens .... this too shall pass.

 

4 minutes ago, Grelko said:

I still say take door number ONE.  If you take door number two, there's always the chance of a full world wipe.

This just does'nt make sense to me. If we patch in the most up to date data the whole world is going to possibly get wiped, where did that idea come from. You can back up what you are literally staring at and save it so nothing can touch it. Then take another set of data and load it in. I will take another day of downtime to recover days and days of crafting levels and work over having to redo it all.

2 minutes ago, MattScott said:

Hi all,

 

We are going to try getting our secondary environment up and running.

Then we can test a new set of data. This wont be a super fast process, and many of my team of recuperating this weekend.

Bear with us.

 

My advice for current players would be to not go crazy. If we can confirm a solid way to restore the 2 weeks of missing progress we will.

But that will mean losing anything since the servers came back online.

Thanks,
Matt

Thank you Matt,

 

I understand this is adding more pain to what the patootie has already endured this week, but it just makes sense, and consider this a continuation of the move. Again I understand there are going to be mixed feelings about this according to what each person has lost, if not for nothing to the vets, think about your newer population.

[Sang 66]

Good lord no ...please

[Grelko 33]

50 minutes ago, MattScott said:

So there is risk that we will introduce a whole bunch of problems with incompatible data.

 

 

6 minutes ago, ChrisKrypton said:

This just does'nt make sense to me. If we patch in the most up to date data the whole world is going to possibly get wiped, where did that idea come from. You can back up what you are literally staring at and save it so nothing can touch it.

Removing 1 section of data "could" corrupt another block, so that when you remove the older version to replace it with a new version, that gets corrupted also which could bug out the entire system.

I'm not saying it will, but there's always that chance.

 

Now, if they made a completely separate backup of what we have now, they could continuously go back to where we're at.

 

The easiest thing, would be to put up a test server with all of the data and see how it responds, before integrating it to what's already live, even though it's rolled back a bit.

[tiago0182 5]

18 minutes ago, MattScott said:

Bear with us.

we are
hope you guys can do it 
good luck and thanks for all

[Grelko 33]

Just a thought.

 

What about all the players that don't come to the forums?

 

They'll just be playing the game and after another "maintenance" which is the (roll forward), they'll complain about losing a week or 2 worth of time in the game also.

[Boundsoul 9]

23 minutes ago, MattScott said:

Hi all,

We are going to try getting our secondary environment up and running.

Thanks,
Matt

 

Just roll with it, a week isn't bad

[ChrisKrypton 15]

6 minutes ago, Grelko said:

Now, if they made a completely separate backup of what we have now, they could continuously go back to where we're at.

The easiest thing, would be to put up a test server with all of the data and see how it responds, before integrating it to what's already live, even though it's rolled back a bit.

Test server is an even better idea!

Data corruption can happen even when bits are just sitting idle on disk, they also can happen when the server autonomously copies it self at regular intervals, and to give you credit it's true you can mess it up just by moving it but if hardware is going to fail in that manner and it fails by copying it today then its going to fail by copying it tomorrow...... catch 22. Anyways whatever happens happens, it just sucks but the right thing to do is hit the ground with a 4/29 save. I still am lost as to how OLD data gets mixed in with NEW data.

[Grelko 33]

Just now, ChrisKrypton said:

Test server is an even better idea!

Data corruption can happen even when bits are just sitting idle on disk, they also can happen when the server autonomously copies it self at regular intervals, and to give you credit it's true you can mess it up just by moving it but if hardware is going to fail in that manner and it fails by copying it today then its going to fail by copying it tomorrow...... catch 22. Anyways whatever happens happens, it just sucks but the right thing to do is hit the ground with a 4/29 save. I still am lost as to how OLD data gets mixed in with NEW data.

What about leaving it the way it is now since it works. Then compensate for the lost items etc.

 

Most of the people should be ok with that and also the people who don't use the forums won't have anything lost after the roll forward?

 

 

Idk though, it's not my decision.

[miketam 30]

Back up "current data" take server down restore data etc..not sure how they managed to lose ....is it 2 weeks? of peoples progess if they backed up data just before server shutdown

My personal view is to leave as is...any guys who lost progress just have to suck it up...i know i have had to over the years numerous times on roll backs.

[Micwilbur 4]

So, I can't log in.  When I "Launch", it goes to a screen that wants my password and says that I am already logged in and that I can force a disconnect with my password.  It goes to another screen that ultimately forces me to disconnect and try again.  I have done this with various permutations over and over again, but it won't go to the game.  Suggestions?

[Grelko 33]

2 minutes ago, Micwilbur said:

So, I can't log in.  When I "Launch", it goes to a screen that wants my password and says that I am already logged in and that I can force a disconnect with my password.  It goes to another screen that ultimately forces me to disconnect and try again.  I have done this with various permutations over and over again, but it won't go to the game.  Suggestions?

I had the same problem. Try putting in your password to "force disconnect", then completely shut down the game back to desktop. Then get back in the game and it should let you in through the updater.

[ChrisKrypton 15]

I am not trying to be the spokesperson about how to proceed here, I am one opinion of many.

I just was'nt understanding some of the concerns. The right thing to do is to hit the ground with data from 4/29 so no one loses anything, that's just my 2 cents. Naturally people are going to base they're opinion on what they're losses are.

Whatever happens is whatever happens, I will refrain from playing untill an official stance on this is taken.

If they can decide that they are able to restore the game within 1 - 2 days,  I am sure most people will appreciate gaining 6 days worth of work while losing 2.

ya wastelanders!

[miketam 30]

Takes 2 or 3 enter password steps,,then u shud be on alec masters screen...and hit play from there.

[Micwilbur 4]

Just now, Grelko said:

I had the same problem. Try putting in your password to "force disconnect", then completely shut down the game back to desktop. Then get back in the game and it should let you in through the updater.

Seems I've done that, but I'll try again.  Thanks.  I'll report back.

 

Just now, miketam said:

Takes 2 or 3 enter password steps,,then u shud be on alec masters screen...and hit play from there.

I can't get to the Alec Masters screen.

[Rubbermade 36]

9 minutes ago, miketam said:

Back up "current data" take server down restore data etc..not sure how they managed to lose ....is it 2 weeks? of peoples progess if they backed up data just before server shutdown

My personal view is to leave as is...any guys who lost progress just have to suck it up...i know i have had to over the years numerous times on roll backs.

my guess is corrupted files