MattScott

Hi all, We have activated a 4 week Commander code as compensation to the players in the hopes of helping them catch up on lost progress. The code is: FallenNot4gotten I'll be editing this original post and making a separate announcement on the forums, Twitter and Facebook as well. Thanks, Matt

Hi all, The server did crash around 1am, so we started maintenance early. Thanks, Matt

We are back online early! Servers are up.

Hi all, One of the game servers crashed before 4am, so the team started early. My original plan had been to limit the scope of optimizations to reduce downtime. The goal was to do a short window this time and then more smaller scheduled fixed each couple of days till we got everything finished. The bad news is that once we got started, the shorter maintenance wasn’t possible. So instead they decided to do everything in one go. My community team was asleep, so we weren’t able to get a post up. The good news is that the work is done and servers are coming online now. The game should be up around 9:30am Pacific. Thanks, Matt

Hi all, At approximately 4pm, our Xbox servers got hit with a 40Gbps DDoS attack. Unfortunately when we get hit that hard, we have to shutdown access to the box so that the attack doesn't affect other servers. We have shifted resources to other systems in the meantime, and at least 1 instance of each district is online. We will put the other districts back online as soon as possible. Thanks, Matt

Hi all, We'll be going down around 11:30am UTC time / 4:30am Pacific for regular maintenance. There is no large database restore or repair planned, so this should be well within the normal maintenance window. We have made some progress on optimizations that should help performance. The hope is to clean up the various character loading and gameplay issues. EDIT: Servers are back online as of 9:10am Pacific. Thanks, Matt

Hi all, This is taking a bit longer than I expected. We had to build some custom tools for our Support team to award these items. Thanks, Matt

My original intention had been to award Premium to everyone. However once we figured out that we couldn't release a code due to problems with consoles players, that meant we had individually award the Premium. Even doing this programmatically took an obscene amount of time. So at that point, I had to change the plan and limit the compensation to just players who have logged in within 60 days.

Due to some of our tools being down from the network migration, we're running behind on new ARMAS features. However, we're hoping the next set will get rotated in by Friday.

Hi there, All Premium has now gone out. The easiest way to check your account is to log into the store: https://www.gamersfirst.com/marketplace/ingame/index.php?gameID=20 If you logged into the game within the last 60 days, and you still don't have it, then please open a support ticket at http://support.gamersfirst.com. Thanks, Matt

Hi everyone, I just checked with the team, and it looks like all players who logged in within the last 60 days should now have 2 weeks of Premium added to their account. This includes players on all worlds, on PC, XB1, and PS4. Several players asked me why this took so long, and why we didn't just issue a code like last time. I would have preferred to issue a code, because it's much easier to do, but console players can't redeem them. There was one other way, but it would have involved waiting till Wednesday's patch to implement. The easiest way to check for Premium is to log into the APB Marketplace here: https://www.gamersfirst.com/marketplace/ingame/index.php?gameID=20 If you logged in during the last 60 days, but for some reason your account didn't get the compensation, please open a Support ticket, and we will do our best to escalate getting it out to you. Thanks, Matt

Yes. it's a requirement.

Hi all, I just posted a topic about this issue. Some of the team was out this weekend attempting to recoup from the last two weeks of overtime, and based on a couple issues, we didn't realize the XB1 districts were down. Apologies, Matt

Hi all, I just posted a response to this. The TL;DR is that we didn't know XB1 was down till a couple hours ago, and we're working with our ISP to bring them back. Thanks, Matt

Hi all, It looks like several things happened to make this situation fairly bad. First, our XB1 monitoring got screwed up during the data center move. So it appears there were several DDoS attacks that none of the team were aware of. Second, the DDoS attacks were so large that our ISP decided to block all traffic to the Xbox district for a period of time. Third, we're short handed this weekend, because many of my engineers have been working 7 day weeks since the end of April. We're working on the situation now with our ISP to get better notifications from them when they block IPs. And we're rebuilding our monitoring for XB1. Sorry, Matt

Hi all, We didn't do a code, because the console players can't redeem codes, and we promised 2 weeks of Premium to everyone. For those that don't have it, we are still distributing the compensation. Thanks, Matt

Hi there, I recommend you put in a support ticket, so they can get your account looked at. Thanks, Matt

Hi all, Finishing one last (short) task and then we'll be bringing servers back online to test internally. Thanks, Matt

Hi all, We're still working on it. Thanks, Matt

Hi all, Just confirming that the play test is happening. We just got final signoff from QA that the build looks good and has many of the changes we wanted. Thanks, Matt

Hi all, We just started awarding the Premium (4pm Friday Pacific time). There are a lot of players to gift, so we're doing this in waves. To start we will hit everyone who has logged in within the last 30 days sorted by most recent login first. Then we'll go back 30 days, etc. Your Premium wont start till it is awarded to the account. I'll keep you posted. Thanks, Matt

I can appreciate your position, but I've already gone on record to say we are not wiping data. We can and will get the server back up. The requirements are just a little tricky to nail down for something like this. Having a server up for years collecting and indexing data over time is a different spec from starting a server empty and loading it with years of data. Once we get stabilized then we have a different way of moving old data out of the active tables dynamically without wiping anything. This just requires some coding and testing in our PTS environment, which we don't have time for right now. Thanks, Matt

We can't change the software OS, the size or structure of the structure of the database. There is no way to avoid some of the "dirty tricks" to keep the game online till we re-write the backend. However we are running on new hardware across the board. The issue we are solving right now relates to the problematic database server (used for pulling all character data). Originally we got everything restored, but clearly players were having trouble retrieving characters on login. At this point, we have determined that this specific database is so large (and poorly structured) that we can no longer simply restore it. Loading that much data results in unusable table indexes which generates the problems we were seeing before we took it down. It takes 6 hours each attempt to restore and then 7 hours each attempt to repair. We have tried 3 different ways to get the database setup, and all have yielded the same result. We are shifting tactics and moving to a different set of expanded hardware. Then we'll give it a 4th attempt. Everyone is working the weekend to get this back online asap. Thanks, Matt

Hi all, This is my last update for the evening. I'm managing the hand off between engineers and then going to call it a night. We got entirely through the restore and most of the repair and then the database errored. It cost us hours of work. Instead of proceeding on a shaky server, I'm having the team tear it down and rebuild it, so we know its stable. Unfortunately there is a lot more work to do. I don't have a good ETA, but likely 12 more hours. I'll update everyone where we are at in the morning. Sorry, Matt

Hi all, We're nearly done with the partial restore to get back to a position to repair things. The backup was done right after we took the servers offline, so there should be no rollback. Restore should be done around 1am, and then I'll have a better estimate for the repair time. Unfortunately we wont be back online at 1am like I had hoped. Sorry, Matt