5/9/2019 Maintenance Update

[MattScott 15245]

 

Hi all,

 

We spent some time yesterday looking at the character login issues and found that 6 tables across our various databases needed heavy re-indexing after they got restored during the move. The tables are all over 5GB in size with the largest being 550MM rows at 55GB. This largest table contains all character attributes for the more than 2.5MM characters made in Fallen Earth to date.

 

The short term fix is to re-index and stabilize things.

The long term fix will be to move old character data to an archive database where it can be restored on the fly if those older players choose to log back in.

 

Since 1am we spent a couple hours backing everything up, and then about 8 hours re-indexing. We are down to the last large table. It appears that it will take about 5 hours more and then we need about another hour to turn on servers and run through a quick QA test. That puts our earliest ETA for letting players back in at ~6pm Pacific time. With a little luck we should have things running much better for the weekend.

 

I'll keep monitoring this and updating the community.

 

EDIT: We hit a serious snag and ran out of drive space on the volume while doing the repair. The team is working around it, but we are going to have to extend the maintenance.

EDIT: 5/10 1am Pacific. Nearly done restoring and adding more space for the repair. No ETA yet.

EDIT: 5/10 3:20am Pacific. The server dumped. We've decided to scrap it and replace it due to ongoing issues with it. The team is working on replacing it now.

EDIT: 5/10 12:00pm Pacific. New hardware is online. Reinstalling everything and then we'll start the database restore / repair process.

EDIT: 5/10 3:20pm Pacific. Current estimate is 9:30pm for the restore to finish. Then the repair will start.

EDIT: 5/11 3:30am Pacific. Still waiting on various parts to finish.

EDIT: 5/11 12pm Pacific. Finishing one last task and then we'll be bringing servers online for QA internal testing.

 

To be clear, backups are intact. No rollback is expected.

 

Thanks,
Matt

[Spoiled 23]

I hope the community is appreciative for all the hard work LO has been doing since they took over. Thanks Matt and team. 

[Grelko 33]

Do what you need to do and take the time to do it properly.

 

If it takes a couple more hours to get rid of the problems we've been having lately, that'll be great.

 

[CommanderDeacon 4]

I want to thank you all for playing this game. I want to show that I am grateful for a full return soon. and your work is not un appreciated. most of us have a lot of passion to play. I feel like returning to fallen earth and you are all the reason I can still create my friends and roam the vast wasteland. 

[FakyouArtemiSix 9]

1 hour ago, MattScott said:

The long term fix will be to move old character data to an archive database where it can be restored on the fly if those older players choose to log back in.

the sooner the better, gl LO, how many estimated weeks till its done ?

and about the software rewriting in order to upgrade hardware, how many estimated months ?

 

seems like the previous team did a crappy job to let 2.5M users on the table without doing what you plan to do...dusting

Edited May 9, 2019 by zefcool

[Lukaso 72]

1 hour ago, Spoiled said:

I hope the community is appreciative for all the hard work LO has been doing since they took over. Thanks Matt and team. 

I fully agree with that +1

Edited May 9, 2019 by Lukasocz

[Anneliesse 27]

Even more so I appreciate the detailed explanation as to what is going on

[ColeTrane 2]

not everyday you get an explained thread from an admin on the issues at hand... thanks and good luck.

[Hereogar 7]

These comms from the dev/admins really help in keeping the faith.

 

Thank you.

[ChrisKrypton 15]

3 hours ago, MattScott said:

Since 1am we spent a couple hours backing everything up, and then about 8 hours re-indexing. We are down to the last large table. It appears that it will take about 5 hours more and then we need about another hour to turn on servers and run through a quick QA test. That puts our earliest ETA for letting players back in at ~6pm Pacific time. With a little luck we should have things running much better for the weekend.

Thanks Matt, I really just want to express how much I and perhaps we appreciate the honesty with the player community in all of the work you guys are doing. It is a quality not found in very many places anymore and is refreshing to see.

[Mcgeek 1]

Y'all are pretty cool, this is the most initiative I've ever seen since I started playing, LET LO, LETS GO *clap* *clap* *clap*

[Eekaspydar 1]

That is refreshing news, by my estimate that was posted 4 hours ago which means, roughly 2 to go. ha, ha, ha,

 

[MattScott 15245]

Hi all,

 

Small update. I have updated the main post, but we hit a serious snag and are working around it now.

We are going to have to unfortunately extend the maintenance window.

Thanks,
Matt

[WreckageExplorer 20]

Don't forget to give your team a break every now and then, too. 

[NIMPO 6]

well truely i m dead inside i want to play so bad

[miketam 30]

+ Defrag the hampsters..i.e indexes.

Guess im not as daft / crazy as i thought.

Edited May 10, 2019 by miketam

[Eekaspydar 1]

Nimpo, i wanna play so bad i can't get into anything else. LOL

[purplepills3 0]

Will we get to play tonight at least?

[Eekaspydar 1]

32 minutes ago, miketam said:

+ Defrag the hampsters..i.e indexes.

Guess im not as daft / crazy as i thought.

Or change internet connection to Fiber, Dialup is out dated guys!

[ChellyBean 200]

Hey all, Im from the APB community, but I just wanna say that I appreciate Matt's transparency about everything 

We've had similar issues on our game, and LO has been equally transparent about it.

Its just nice to know what's going on 

[Eekaspydar 1]

1 minute ago, purplepills3 said:

Will we get to play tonight at least?

just noticed a post 46mins ago, Extended Maintenance again. i'll try tomorrow after work, going to bed in an hour

 

[Killenkittens 3]

Is there a rough time estimate

[Eekaspydar 1]

try tomorrow

 

[CommanderDeacon 4]

I hate it when the drive goes empty....

I am a disliked social pariah and this is all I have the capacity to do. im not even mad cause this is what happens every time I want to do something. we always just work around it...

 

[CommanderDeacon 4]

I think my workaround tonight will be a bottle of jack and I wont stop drinking till it complements me.