MattScott

Hi there, All Premium has now gone out. The easiest way to check your account is to log into the store: https://www.gamersfirst.com/marketplace/ingame/index.php?gameID=20 If you logged into the game within the last 60 days, and you still don't have it, then please open a support ticket at http://support.gamersfirst.com. Thanks, Matt

Hi everyone, I just checked with the team, and it looks like all players who logged in within the last 60 days should now have 2 weeks of Premium added to their account. This includes players on all worlds, on PC, XB1, and PS4. Several players asked me why this took so long, and why we didn't just issue a code like last time. I would have preferred to issue a code, because it's much easier to do, but console players can't redeem them. There was one other way, but it would have involved waiting till Wednesday's patch to implement. The easiest way to check for Premium is to log into the APB Marketplace here: https://www.gamersfirst.com/marketplace/ingame/index.php?gameID=20 If you logged in during the last 60 days, but for some reason your account didn't get the compensation, please open a Support ticket, and we will do our best to escalate getting it out to you. Thanks, Matt

Yes. it's a requirement.

Hi all, I just posted a topic about this issue. Some of the team was out this weekend attempting to recoup from the last two weeks of overtime, and based on a couple issues, we didn't realize the XB1 districts were down. Apologies, Matt

Hi all, I just posted a response to this. The TL;DR is that we didn't know XB1 was down till a couple hours ago, and we're working with our ISP to bring them back. Thanks, Matt

Hi all, It looks like several things happened to make this situation fairly bad. First, our XB1 monitoring got screwed up during the data center move. So it appears there were several DDoS attacks that none of the team were aware of. Second, the DDoS attacks were so large that our ISP decided to block all traffic to the Xbox district for a period of time. Third, we're short handed this weekend, because many of my engineers have been working 7 day weeks since the end of April. We're working on the situation now with our ISP to get better notifications from them when they block IPs. And we're rebuilding our monitoring for XB1. Sorry, Matt

Hi all, We didn't do a code, because the console players can't redeem codes, and we promised 2 weeks of Premium to everyone. For those that don't have it, we are still distributing the compensation. Thanks, Matt

Hi there, I recommend you put in a support ticket, so they can get your account looked at. Thanks, Matt

Hi all, Finishing one last (short) task and then we'll be bringing servers back online to test internally. Thanks, Matt

Hi all, We're still working on it. Thanks, Matt

Hi all, Just confirming that the play test is happening. We just got final signoff from QA that the build looks good and has many of the changes we wanted. Thanks, Matt

Hi all, We just started awarding the Premium (4pm Friday Pacific time). There are a lot of players to gift, so we're doing this in waves. To start we will hit everyone who has logged in within the last 30 days sorted by most recent login first. Then we'll go back 30 days, etc. Your Premium wont start till it is awarded to the account. I'll keep you posted. Thanks, Matt

I can appreciate your position, but I've already gone on record to say we are not wiping data. We can and will get the server back up. The requirements are just a little tricky to nail down for something like this. Having a server up for years collecting and indexing data over time is a different spec from starting a server empty and loading it with years of data. Once we get stabilized then we have a different way of moving old data out of the active tables dynamically without wiping anything. This just requires some coding and testing in our PTS environment, which we don't have time for right now. Thanks, Matt

We can't change the software OS, the size or structure of the structure of the database. There is no way to avoid some of the "dirty tricks" to keep the game online till we re-write the backend. However we are running on new hardware across the board. The issue we are solving right now relates to the problematic database server (used for pulling all character data). Originally we got everything restored, but clearly players were having trouble retrieving characters on login. At this point, we have determined that this specific database is so large (and poorly structured) that we can no longer simply restore it. Loading that much data results in unusable table indexes which generates the problems we were seeing before we took it down. It takes 6 hours each attempt to restore and then 7 hours each attempt to repair. We have tried 3 different ways to get the database setup, and all have yielded the same result. We are shifting tactics and moving to a different set of expanded hardware. Then we'll give it a 4th attempt. Everyone is working the weekend to get this back online asap. Thanks, Matt

Hi all, This is my last update for the evening. I'm managing the hand off between engineers and then going to call it a night. We got entirely through the restore and most of the repair and then the database errored. It cost us hours of work. Instead of proceeding on a shaky server, I'm having the team tear it down and rebuild it, so we know its stable. Unfortunately there is a lot more work to do. I don't have a good ETA, but likely 12 more hours. I'll update everyone where we are at in the morning. Sorry, Matt

Hi all, We're nearly done with the partial restore to get back to a position to repair things. The backup was done right after we took the servers offline, so there should be no rollback. Restore should be done around 1am, and then I'll have a better estimate for the repair time. Unfortunately we wont be back online at 1am like I had hoped. Sorry, Matt

Sure thing. I can see the confusion. It was both. The contract was up, and we had already put our notice in with the old hosting facility. However at the very end there was a delay with the new hardware arriving and getting setup on-time at the new facility. My preference would have been to get 2 more weeks at the old facility, but they denied our request to stay a bit longer.

Hi all, Small update. I have updated the main post, but we hit a serious snag and are working around it now. We are going to have to unfortunately extend the maintenance window. Thanks, Matt

Hi all, We spent some time yesterday looking at the character login issues and found that 6 tables across our various databases needed heavy re-indexing after they got restored during the move. The tables are all over 5GB in size with the largest being 550MM rows at 55GB. This largest table contains all character attributes for the more than 2.5MM characters made in Fallen Earth to date. The short term fix is to re-index and stabilize things. The long term fix will be to move old character data to an archive database where it can be restored on the fly if those older players choose to log back in. Since 1am we spent a couple hours backing everything up, and then about 8 hours re-indexing. We are down to the last large table. It appears that it will take about 5 hours more and then we need about another hour to turn on servers and run through a quick QA test. That puts our earliest ETA for letting players back in at ~6pm Pacific time. With a little luck we should have things running much better for the weekend. I'll keep monitoring this and updating the community. EDIT: We hit a serious snag and ran out of drive space on the volume while doing the repair. The team is working around it, but we are going to have to extend the maintenance. EDIT: 5/10 1am Pacific. Nearly done restoring and adding more space for the repair. No ETA yet. EDIT: 5/10 3:20am Pacific. The server dumped. We've decided to scrap it and replace it due to ongoing issues with it. The team is working on replacing it now. EDIT: 5/10 12:00pm Pacific. New hardware is online. Reinstalling everything and then we'll start the database restore / repair process. EDIT: 5/10 3:20pm Pacific. Current estimate is 9:30pm for the restore to finish. Then the repair will start. EDIT: 5/11 3:30am Pacific. Still waiting on various parts to finish. EDIT: 5/11 12pm Pacific. Finishing one last task and then we'll be bringing servers online for QA internal testing. To be clear, backups are intact. No rollback is expected. Thanks, Matt

Out of curiosity, did you create your account in April?

There is no intention to change the core mechanics to the game. Specifically we are not turning Fallen Earth into a first person game. Most of this effort is just to upgrade the tech so it is more supportable.

Hmm. Did you lose all your achievements? According to my understanding, we only lost about 2 weeks of progression. So the large majority of your 8.5 year progress should be preserved.

Hi all, This post went up today. It's been a while since I posted engine update progress for @FallenEarth. One big blocker has been figuring out the animation structure and porting them over to match the new characters and creatures. Looks like we have that solved now. #ThisOldGame Thanks, Matt

Hi all, The team has organized how to make up for the lost progression, and we will be starting 2 weeks of Commander for all players on Friday 5/10. I have edited the original post to reflect this. Thanks, Matt

Hi there, We recently had an issue with our primary database. The situation is fixed, and no data was lost. But we are still restoring the old Hoplon data that used by the migration system. You should be able to migrate your account by Friday. Thanks, Matt