Jump to content
MattScott

5/9/2019 Maintenance Update

Recommended Posts

 

Hi all,

 

We spent some time yesterday looking at the character login issues and found that 6 tables across our various databases needed heavy re-indexing after they got restored during the move. The tables are all over 5GB in size with the largest being 550MM rows at 55GB. This largest table contains all character attributes for the more than 2.5MM characters made in Fallen Earth to date.

 

The short term fix is to re-index and stabilize things.

The long term fix will be to move old character data to an archive database where it can be restored on the fly if those older players choose to log back in.

 

Since 1am we spent a couple hours backing everything up, and then about 8 hours re-indexing. We are down to the last large table. It appears that it will take about 5 hours more and then we need about another hour to turn on servers and run through a quick QA test. That puts our earliest ETA for letting players back in at ~6pm Pacific time. With a little luck we should have things running much better for the weekend.

 

I'll keep monitoring this and updating the community.

 

EDIT: We hit a serious snag and ran out of drive space on the volume while doing the repair. The team is working around it, but we are going to have to extend the maintenance.

EDIT: 5/10 1am Pacific. Nearly done restoring and adding more space for the repair. No ETA yet.

EDIT: 5/10 3:20am Pacific. The server dumped. We've decided to scrap it and replace it due to ongoing issues with it. The team is working on replacing it now.

EDIT: 5/10 12:00pm Pacific. New hardware is online. Reinstalling everything and then we'll start the database restore / repair process.

EDIT: 5/10 3:20pm Pacific. Current estimate is 9:30pm for the restore to finish. Then the repair will start.

EDIT: 5/11 3:30am Pacific. Still waiting on various parts to finish.

EDIT: 5/11 12pm Pacific. Finishing one last task and then we'll be bringing servers online for QA internal testing.

 

To be clear, backups are intact. No rollback is expected.

 

Thanks,
Matt

  • Like 10
  • Thanks 10
  • Dislike 1

Share this post


Link to post
Share on other sites

I hope the community is appreciative for all the hard work LO has been doing since they took over. Thanks Matt and team. 

  • Like 4
  • Dislike 1

Share this post


Link to post
Share on other sites

Do what you need to do and take the time to do it properly.

 

If it takes a couple more hours to get rid of the problems we've been having lately, that'll be great.

 

  • Like 2
  • Dislike 1

Share this post


Link to post
Share on other sites

I want to thank you all for playing this game. I want to show that I am grateful for a full return soon. and your work is not un appreciated. most of us have a lot of passion to play. I feel like returning to fallen earth and you are all the reason I can still create my friends and roam the vast wasteland. 

  • Like 1
  • Dislike 1

Share this post


Link to post
Share on other sites
Posted (edited)
1 hour ago, MattScott said:

The long term fix will be to move old character data to an archive database where it can be restored on the fly if those older players choose to log back in.

the sooner the better, gl LO, how many estimated weeks till its done ?

and about the software rewriting in order to upgrade hardware, how many estimated months ?

 

seems like the previous team did a crappy job to let 2.5M users on the table without doing what you plan to do...dusting

Edited by zefcool
  • Dislike 1

Share this post


Link to post
Share on other sites
Posted (edited)
1 hour ago, Spoiled said:

I hope the community is appreciative for all the hard work LO has been doing since they took over. Thanks Matt and team. 

I fully agree with that +1

Edited by Lukasocz
  • Like 3
  • Dislike 1

Share this post


Link to post
Share on other sites

Even more so I appreciate the detailed explanation as to what is going on

  • Like 2
  • Dislike 1

Share this post


Link to post
Share on other sites

not everyday you get an explained thread from an admin on the issues at hand... thanks and good luck.

  • Like 1
  • Dislike 1

Share this post


Link to post
Share on other sites

These comms from the dev/admins really help in keeping the faith.

 

Thank you.

Share this post


Link to post
Share on other sites
3 hours ago, MattScott said:

Since 1am we spent a couple hours backing everything up, and then about 8 hours re-indexing. We are down to the last large table. It appears that it will take about 5 hours more and then we need about another hour to turn on servers and run through a quick QA test. That puts our earliest ETA for letting players back in at ~6pm Pacific time. With a little luck we should have things running much better for the weekend.

Thanks Matt, I really just want to express how much I and perhaps we appreciate the honesty with the player community in all of the work you guys are doing. It is a quality not found in very many places anymore and is refreshing to see.

  • Like 1

Share this post


Link to post
Share on other sites

Y'all are pretty cool, this is the most initiative I've ever seen since I started playing, LET LO, LETS GO *clap* *clap* *clap*

Share this post


Link to post
Share on other sites

That is refreshing news, by my estimate that was posted 4 hours ago which means, roughly 2 to go. ha, ha, ha,

 

Share this post


Link to post
Share on other sites

Hi all,

 

Small update. I have updated the main post, but we hit a serious snag and are working around it now.

We are going to have to unfortunately extend the maintenance window.


Thanks,
Matt

  • Thanks 5

Share this post


Link to post
Share on other sites

well truely i m dead inside i want to play so bad

Share this post


Link to post
Share on other sites
Posted (edited)

+ Defrag the hampsters..i.e indexes.

Guess im not as daft / crazy as i thought.

Edited by miketam

Share this post


Link to post
Share on other sites
32 minutes ago, miketam said:

+ Defrag the hampsters..i.e indexes.

Guess im not as daft / crazy as i thought.

Or change internet connection to Fiber, Dialup is out dated guys!

Share this post


Link to post
Share on other sites

Hey all, Im from the APB community, but I just wanna say that I appreciate Matt's transparency about everything 

We've had similar issues on our game, and LO has been equally transparent about it.

Its just nice to know what's going on 

  • Like 2

Share this post


Link to post
Share on other sites
1 minute ago, purplepills3 said:

Will we get to play tonight at least?

just noticed a post 46mins ago, Extended Maintenance again. i'll try tomorrow after work, going to bed in an hour

 

Share this post


Link to post
Share on other sites

I hate it when the drive goes empty....

I am a disliked social pariah and this is all I have the capacity to do. im not even mad cause this is what happens every time I want to do something. we always just work around it...

 

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.


  • Recently Browsing   0 members

    No registered users viewing this page.

×
×
  • Create New...