MattScott 15242 Posted May 9, 2019 Hi all, We spent some time yesterday looking at the character login issues and found that 6 tables across our various databases needed heavy re-indexing after they got restored during the move. The tables are all over 5GB in size with the largest being 550MM rows at 55GB. This largest table contains all character attributes for the more than 2.5MM characters made in Fallen Earth to date. The short term fix is to re-index and stabilize things. The long term fix will be to move old character data to an archive database where it can be restored on the fly if those older players choose to log back in. Since 1am we spent a couple hours backing everything up, and then about 8 hours re-indexing. We are down to the last large table. It appears that it will take about 5 hours more and then we need about another hour to turn on servers and run through a quick QA test. That puts our earliest ETA for letting players back in at ~6pm Pacific time. With a little luck we should have things running much better for the weekend. I'll keep monitoring this and updating the community. EDIT: We hit a serious snag and ran out of drive space on the volume while doing the repair. The team is working around it, but we are going to have to extend the maintenance. EDIT: 5/10 1am Pacific. Nearly done restoring and adding more space for the repair. No ETA yet. EDIT: 5/10 3:20am Pacific. The server dumped. We've decided to scrap it and replace it due to ongoing issues with it. The team is working on replacing it now. EDIT: 5/10 12:00pm Pacific. New hardware is online. Reinstalling everything and then we'll start the database restore / repair process. EDIT: 5/10 3:20pm Pacific. Current estimate is 9:30pm for the restore to finish. Then the repair will start. EDIT: 5/11 3:30am Pacific. Still waiting on various parts to finish. EDIT: 5/11 12pm Pacific. Finishing one last task and then we'll be bringing servers online for QA internal testing. To be clear, backups are intact. No rollback is expected. Thanks, Matt 10 10 Share this post Link to post Share on other sites
Spoiled 23 Posted May 9, 2019 I hope the community is appreciative for all the hard work LO has been doing since they took over. Thanks Matt and team. 4 Share this post Link to post Share on other sites
Grelko 33 Posted May 9, 2019 Do what you need to do and take the time to do it properly. If it takes a couple more hours to get rid of the problems we've been having lately, that'll be great. 2 Share this post Link to post Share on other sites
CommanderDeacon 4 Posted May 9, 2019 I want to thank you all for playing this game. I want to show that I am grateful for a full return soon. and your work is not un appreciated. most of us have a lot of passion to play. I feel like returning to fallen earth and you are all the reason I can still create my friends and roam the vast wasteland. 1 Share this post Link to post Share on other sites
FakyouArtemiSix 9 Posted May 9, 2019 (edited) 1 hour ago, MattScott said: The long term fix will be to move old character data to an archive database where it can be restored on the fly if those older players choose to log back in. the sooner the better, gl LO, how many estimated weeks till its done ? and about the software rewriting in order to upgrade hardware, how many estimated months ? seems like the previous team did a crappy job to let 2.5M users on the table without doing what you plan to do...dusting Edited May 9, 2019 by zefcool Share this post Link to post Share on other sites
Lukaso 72 Posted May 9, 2019 (edited) 1 hour ago, Spoiled said: I hope the community is appreciative for all the hard work LO has been doing since they took over. Thanks Matt and team. I fully agree with that +1 Edited May 9, 2019 by Lukasocz 3 Share this post Link to post Share on other sites
Anneliesse 27 Posted May 9, 2019 Even more so I appreciate the detailed explanation as to what is going on 2 Share this post Link to post Share on other sites
ColeTrane 2 Posted May 9, 2019 not everyday you get an explained thread from an admin on the issues at hand... thanks and good luck. 1 Share this post Link to post Share on other sites
Hereogar 7 Posted May 9, 2019 These comms from the dev/admins really help in keeping the faith. Thank you. Share this post Link to post Share on other sites
ChrisKrypton 15 Posted May 9, 2019 3 hours ago, MattScott said: Since 1am we spent a couple hours backing everything up, and then about 8 hours re-indexing. We are down to the last large table. It appears that it will take about 5 hours more and then we need about another hour to turn on servers and run through a quick QA test. That puts our earliest ETA for letting players back in at ~6pm Pacific time. With a little luck we should have things running much better for the weekend. Thanks Matt, I really just want to express how much I and perhaps we appreciate the honesty with the player community in all of the work you guys are doing. It is a quality not found in very many places anymore and is refreshing to see. 1 Share this post Link to post Share on other sites
Mcgeek 1 Posted May 9, 2019 Y'all are pretty cool, this is the most initiative I've ever seen since I started playing, LET LO, LETS GO *clap* *clap* *clap* Share this post Link to post Share on other sites
Eekaspydar 1 Posted May 9, 2019 That is refreshing news, by my estimate that was posted 4 hours ago which means, roughly 2 to go. ha, ha, ha, Share this post Link to post Share on other sites
MattScott 15242 Posted May 10, 2019 Hi all, Small update. I have updated the main post, but we hit a serious snag and are working around it now. We are going to have to unfortunately extend the maintenance window. Thanks, Matt 5 Share this post Link to post Share on other sites
WreckageExplorer 20 Posted May 10, 2019 Don't forget to give your team a break every now and then, too. Share this post Link to post Share on other sites
NIMPO 6 Posted May 10, 2019 well truely i m dead inside i want to play so bad Share this post Link to post Share on other sites
miketam 30 Posted May 10, 2019 (edited) + Defrag the hampsters..i.e indexes. Guess im not as daft / crazy as i thought. Edited May 10, 2019 by miketam Share this post Link to post Share on other sites
Eekaspydar 1 Posted May 10, 2019 Nimpo, i wanna play so bad i can't get into anything else. LOL Share this post Link to post Share on other sites
purplepills3 0 Posted May 10, 2019 Will we get to play tonight at least? Share this post Link to post Share on other sites
Eekaspydar 1 Posted May 10, 2019 32 minutes ago, miketam said: + Defrag the hampsters..i.e indexes. Guess im not as daft / crazy as i thought. Or change internet connection to Fiber, Dialup is out dated guys! Share this post Link to post Share on other sites
ChellyBean 200 Posted May 10, 2019 Hey all, Im from the APB community, but I just wanna say that I appreciate Matt's transparency about everything We've had similar issues on our game, and LO has been equally transparent about it. Its just nice to know what's going on 2 Share this post Link to post Share on other sites
Eekaspydar 1 Posted May 10, 2019 1 minute ago, purplepills3 said: Will we get to play tonight at least? just noticed a post 46mins ago, Extended Maintenance again. i'll try tomorrow after work, going to bed in an hour Share this post Link to post Share on other sites
Killenkittens 3 Posted May 10, 2019 Is there a rough time estimate Share this post Link to post Share on other sites
CommanderDeacon 4 Posted May 10, 2019 I hate it when the drive goes empty.... I am a disliked social pariah and this is all I have the capacity to do. im not even mad cause this is what happens every time I want to do something. we always just work around it... Share this post Link to post Share on other sites
CommanderDeacon 4 Posted May 10, 2019 I think my workaround tonight will be a bottle of jack and I wont stop drinking till it complements me. 2 Share this post Link to post Share on other sites