MattScott

Hi all, Servers are back online with the Valentine's Day patch. This maintenance was extra long because we were moving databases and more resources into the Europe data center. This also means we finally have enough resources to turn back on threat for both worlds. Thanks, Matt

Hi all, We shipped a couple APB 2.1 builds to SPCT this week. All of them were focused on performance. The engineers were able to fix the issue that was identified last week, and the game is running a lot better. We also fixed several of the top game play bugs including an issue with destructible props and shoot windows. The biggest blockers are the memory issue on Xbox and a frame rate drop when looking around. The team is currently memory mapping the game profiling builds to identify the fixes. Thanks, Matt

Hi there, For the record, we don't ever account wipe. More than likely your account got caught up in the password reset from January 2018 that forced everyone to either reset their password or contact support to regain access. Thanks, Matt

Hi all, We're now very close to finally being completed. The remaining Europe servers will be delivered on Monday, and it is my hope to have them up and running by the end of next week. As soon as all servers are online, then we can re-activate threat. Thanks, Matt

Hi all, Another big push this week to try and get caught up. As of today we have: 180 unanswered/new tickets 339 total tickets We are now responding to tickets from 1/21 which puts us at 10 days to respond. This is the lowest response time we have ever had. The goal is to continue bringing it down until we reach industry standards. Thanks, Matt

Hi all, About a week ago we swapped in a bunch of more powerful CPUs for the district servers. However it looks like some important configuration settings were missed. The plan is to take them down and make adjustments during maintenance. Thanks, Matt

Hi all, This week we made significant (exciting) progress, but it's going to be a bit complicated to discuss, so bear with me. In July, we had a build that was missing correct lighting, had a bunch of visual issues, and most importantly was missing all the content and latest code changes from Live. However, the frame rate from this build when tested by SPCT was within 1ms frame time (5-10FPS) of Live. We felt based on testing that this was an acceptable place to start testing on OTW while we finished optimizing the performance. In September, we got our first look at a candidate build with all the new code and content migrated over, all action districts cleaned up from duplicate objects in the scene, and correct lighting implemented. Unfortunately the frame time had plummeted. We were 5ms (~40FPS) slower than Live. I'm sure you can imagine the disappointment experienced by the team and our SPCT testers. Since then, the team has been hammering away at optimizations and critical visual issues such as building LODs and features. It has been a game of inches. Sometimes lowering 0.2 ms at a time. Several systems in APB 1.20 had to be completely abandoned and rebuilt. A week ago, based on new systems and many optimizations that we have added, we were 3ms (~20FPS) slower than Live. As of last night, we finally identified a massive 2ms bottleneck that got introduced back in August. The issue happens as part of our build process, which skews our ability to debug and log it making it virtually impossible to track down. To be clear, we know what the issue is, but the engineers haven't yet proposed a solution. This will be the #1 area we focus on next week to try and finally get the game close to Live's FPS. During the last 2 weeks, our internal QA did a full regression on our current APB 2.1 build. We have 59 active bugs in the database including console submission bugs. All platforms (PC, XB1, and PS4) are using the same exact code. Performance on both consoles is better than the console builds we have in Live, however our APB 2.1 build for XB1 runs out of memory after running around action districts for a while. The issue was introduced during the rewrite of 2 other major systems, so we need to tread very carefully. I think this will be the biggest remaining issue after we solve the primary performance bottleneck. Moving forward, I'll countdown the bugs in the database. Just know that as QA finds more bugs and the number will shift around a little. Thanks, Matt

Hi all, Good momentum this week. The team put in a lot of hard work. As of today we have: 240 unanswered/new tickets 386 total tickets (!!) We are 2 tickets away from 1/8 which puts us at 16 days to respond. Thanks, Matt

Looks like in our rush, we left a restriction on that booted regular players but not our admin who tested that everything was up. Everything should be working properly now.

Having the team take another look.

This was completed on 1/23. So no maintenance on 1/24.

We are back online. We are still short servers to put threat back in. I’m anticipating we’ll have those online by Friday the 31st.

Hi all, I am sorry. I have just been notified by my network engineers that our Europe provider misunderstood and scheduled our hardware upgrades for today instead of tomorrow. Europe districts are offline, so we are going to do the maintenance now. Thanks, Matt

Hi there, I can't speak to your specific issue, but we are well past December 15th. The team is running ~18-21 days right now. You may want to log back into the Support portal and check the status. Often players miss our response, and after a couple unanswered replies, we close the ticket. If you find no response came from Support, then PM your ticket # and I'll take a look. Thanks, Matt

Hi all, ‪We are anticipating a batch of replacement CPUs for the Europe servers tomorrow. Rather than do maintenance and then take servers down again, we are postponing the maintenance till 1am PST on 1/24.‬ Thanks, Matt

Hi all, Quick update. We have replacement CPUs coming in for the Europe servers which should address the lag issues. We're hoping those will get swapped out during tonight's maintenance window. More new Europe servers are coming in on Thursday. They were due today, but some components failed testing before shipping which delay them by 2 days. As soon as we get the next batch of Europe servers up, I think we can re-enable threat across all worlds. We are working on a patch that will help players identify the fastest / closest servers much easier. We are also spinning up more East Coast servers to help those players get better latency. Thanks, Matt

Hi all, I missed Saturday's post. Just got busy with my weekend and was out of town. I did put up an early update for PS4 players on my Twitter feed and then linked it here on Thursday. We're working on a short list of major tasks, and then sorting through the bug list. The big items are: Performance continues to be sub par. This is very frustrating because some scenes are identical to what we had mid last year such as the 3D login scene. Last year that scene was running way above Live's framerate. Now it's running terrible, and nothing in the scene has changed. So during the many many code and asset merges between Live and 3.5, I think we introduced a performance bug that is slowing everything down. The team is profiling this scene now to figure it out. Raycasting issues shooting through windows and other see thru objects. This was recently reported by SPCT. Distance culling is an issue for props. Should be an easy fix. Culling values are just missing. Code optimization. We found something that pops up in the code quite a bit which is an expensive way of doing things. This is an alternate very fast method, so we are switching those calls. XB1 performance is atrocious (sub 15FPS). We are using the Login scene as a test. This scene is identical to what XB1 is running in live. The XB1 render system is identical to what is running on XB1 in live. Yet our current version performs worse. Solving this will likely shed light on the performance issues for PC as well. Interestingly, PS4 uses an entirely different render system, and it seems to be performing the same as PS4 Live. NOTE: We solved the XB1 issue and it currently runs at 28FPS which only 2FPS slower than PS4. We're having some trouble getting districts to connect to that environment, but I'll try to post a video of us running around as soon as I can. QA has made 1 full regression pass on APB 2.1, and they are in the middle of a second one. All bugs have been prioritized and passed to the dev team, and we have a couple guys working down that list, while the rest of the team focuses on the big stuff. Currently we have 94 open bugs. The estimate is that 20 or so can be closed of this second round of internal testing. Thanks, Matt

Yep. Got busy this weekend and missed it. Working on it now.

Hi all, As of today we have: 317 unanswered tickets 511 total tickets We are answering tickets from 12/30 which puts us at 18 days to respond. Thanks, Matt

I try to read the forums every day. Lately, I've been knee deep working on finishing the network move and getting new servers in place so we can finally put threat back in, so I'm behind. (Small note: my inbox is a mess and I'm only now responding to Nov 21) We still have a long way to go, but the kind words are much appreciated. I'll pass them along to the team. They do all the heavy lifting. Thanks, Matt

Hi all, This is just a small early update for PS4 players. I made a tweet yesterday that I had seen a very good looking PS4 build, and a couple players private messaged me asking for video. So today I tweeted that video. You can see it here on my personal twitter. The main @apb_reloaded on Twitter retweets anything I post about the game. Thanks, Matt

In our rush to get servers up, we missed a couple BIOS settings which are causing lag when the servers get populated. We have to take the servers down to change the BIOS.

Hi all, We are bringing up the world now along with Social, and then we'll bring up each server right after as they are rebooted. Thanks, Matt

Hi all, We are making some adjustments to the Citadel EU servers to help performance. Shouldn't take long. Apologies for the lack of notice. Thanks, Matt

Thanks for the heads up. We started looking at this about an hour ago.