MattScott

Hi all, Very good Customer Support week! As of today we have: 106 new tickets 233 total tickets We are now responding to tickets from 2/17, which puts us at 11 days to respond. Thanks, Matt

Hi all, We've made some good strides with Culling. There is no silver bullet, but it runs better than it did. There's a big slowdown related to audio that we're hunting down. After one of the content merges we left 6000+ audio emitters in Financial. They are all legitimate places where audio can play, but we need to do a better job managing them so they don't cause other parts of the code that search the scene to hit those objects too. During the week, we fixed several lighting bugs. On Friday we found a series of bugs that need to be addressed related to memory crashes. We also found a bug in the new Building Feature code that was preventing that system from working correctly. I'm going to try and post some benchmarks next week to illustrate where we are related to past benchmarks. Thanks, Matt

Hi all, As of today we have: 185 new tickets 295 total tickets (I think this is the lowest we have ever been) We are now responding to tickets from 2/9 (almost 2/10), which puts us at 12 days to respond. Thanks, Matt

Hi there, I can appreciate this comment, and I agree that we need to get the Engine Upgrade out. However, that’s not as simple as you think. I make the effort to post on a weekly basis to talk about our progress. You can read it here: Thanks, Matt

HI all, I missed yesterday's update, but the TL;DR is that this week was disappointing. In an effort to explain, I'm going to talk a little bit about the innerworkings of the APB engine. I'll start by sharing performance from scenes that have no actual gameplay in them. These are small contained scenes with not a lot of objects in them. For the Login scene on my test machine, APB 1.20 (Live) runs at an average of 194FPS and APB 2.1 (Upgrade) runs at an average of 221FPS. We tested a number of other non-gameplay scenes, and clearly our new multithreaded renderer is faster - even with the more complex lighting and more objects overall. However, that is not true in Mission districts. These are much more expansive scenes where it's not feasible to render everything. As an example, in Financial there are 30,000+ objects. There are also a lot of special cases where task objects can be enabled/disabled, buildings can show various levels of detail, NPCs animate and move, and lighting is combination of Time of Day + lightmaps. All of that requires special case code. As mentioned previously, the Engine itself has two main threads - one for game play and one for rendering. The rendering thread performs all the CPU functions necessary to throw geometry to the graphics card (GPU), which then splits those draw calls into more processes. The trick to getting the game to run fast is to balance the tasks in each of these three areas so that from the beginning of a frame to the end, they take the same time to run. To help optimize and balance performance, we have a number of tools that do performance profiling and tell us general areas, shaders, or functions that are taking a lot of time. This part can get very tricky because we have various versions of builds we can make such as Debug, Release, and Final builds. Debug is the slowest, but has the most diagnostics information we can see. Final is the build we ship, but it provides the least diagnostic information. Also running the game at a slower speed can wreak havoc with the balance of threads, which massively skews profiling results. With all of that in mind, a couple weeks ago we zeroed in on a performance bottleneck in the render thread where we do Culling (and its associated functions). This area of the code got reworked when we implemented multithreaded rendering, and clearly it is now under performing. Culling is the process of eliminating as many items as possible from being sent to the graphics card for rendering. Even though GPUs are very fast, it still takes time to copy data between the CPU and GPU, and we take a hit every time we switch materials/shaders between rendering each object. Since APB is so old, our version of Unreal has no "batching" or "instancing", which is what allows modern engines to push so many objects. In Financial, each frame we end up culling those 30,000+ objects down to 800-1000 objects that get processed for rendering to the GPU. So Culling is very important, and we use a bunch of different techniques to cull including render distance, object state, and clipping against the view frustum. All of that starts with accessing the scene objects in an Octree, a 3 dimensional grid that sub divides each cell until only a few objects are present. Each frame we start by looping down the Octree to eliminate objects that are behind or to the side of the player and out of view. The Octree is both our best way to cull, but also it's the hardest thing to multithread. This last week, we took a stab at multithreading that. This was a massive effort, and it ultimately didn't work. We got the code working, but on Thursday night when we started testing, we noticed that some objects in the scene didn't render. By Friday night, we figured out that the original devs on APB store scene data for objects in two places, the Octree and a main list of objects in the scene. For various reasons, those objects don't match. So we need to rework the approach and try again this week. Thanks, Matt

Hi all, Lots of support requests this week. While there are a couple straggler Keys to the City tickets asking for back grants, we stayed on top of things pretty well. As of today we have: 188 new tickets (which is lower than last week) 328 total tickets (which is lower than last week) We are now responding to tickets from 1/31 (except for a couple KTTC tickets) which puts us at 15 days to respond. Thanks, Matt

Hi all, Not much to report this week. Work was finished a number of bugs, but the focus has been on performance. Fortunately it looks like the team may have found the largest performance bottleneck on Friday. Looking forward to sharing the results. Thanks, Matt

Hi all, We had a bit of a rough week that required us to split time in Customer Service between older tickets and immediate item delivery issues on GamersFirst.com caused by moving the Citadel database. Everything has smoothed out now, but we fell behind a little overall. As of today we have: 195 unanswered/new tickets 336 total tickets We are now responding to tickets from 1/25 (almost 1/26) which puts us at 13 days to respond. Thanks, Matt

Yes. We are prioritizing those above others, but there really isn't a need to open a ticket now. We're working proactively through about 100 Citadel accounts that were affected. Some of these were accounts that missed the delivery of an item going back as far as November. We got about half of them sorted out today, and we'll hit the rest over the next 2 days.

Hi there, You should have received an email receipt for the purchase. You can just paste the details of the receipt to Support. Otherwise, you can let Support know what you purchased, and they can look up your account and find missing items, but that might take a bit longer. Thanks, Matt

Hi everyone, We had an issue with G1 purchases this morning when Citadel came back online. For a period of about 3 hours, item purchases weren't being delivered to players on that world. This was due to an issue caused by moving around the database during maintenance. The issue has been solved, and all purchase should work now. However, for anyone who is still missing weapons, you'll need to open a ticket at http://support.gamersfirst.com. Give them the transaction id from your purchase, and we'll escalate delivery as fast as possible. Thanks, Matt

You may want to experiment with different action districts. Some are West Coast, and some are East Coast. The team is nearly done putting in a region ping/latency indicator. That will show up on the Advanced tab to show which servers are better for you.

The team is nearly done with this. Hoping it will go into the next major patch.

Hi all, Can you test again? We made some adjustments today. Thanks, Matt

Hi there, This could be a case of selecting a New York server or a Los Angeles server. We have action districts in both. The team is nearly done with a new patch that will show players a ping/latency indicator to the closest servers. If you get in and out of districts, you can see your latency go up and down depending on what is closer. Thanks, Matt

Hi there, We've fixed the problem, so new purchases will work correctly. However, you'll need to open a ticket at http://support.gamersfirst.com to get the items delivered. Give them the transaction id from your purchase, and we'll escalate delivery as fast as possible. Thanks, Matt

Hi all, Servers are back online with the Valentine's Day patch. This maintenance was extra long because we were moving databases and more resources into the Europe data center. This also means we finally have enough resources to turn back on threat for both worlds. Thanks, Matt

Hi all, We shipped a couple APB 2.1 builds to SPCT this week. All of them were focused on performance. The engineers were able to fix the issue that was identified last week, and the game is running a lot better. We also fixed several of the top game play bugs including an issue with destructible props and shoot windows. The biggest blockers are the memory issue on Xbox and a frame rate drop when looking around. The team is currently memory mapping the game profiling builds to identify the fixes. Thanks, Matt

Hi there, For the record, we don't ever account wipe. More than likely your account got caught up in the password reset from January 2018 that forced everyone to either reset their password or contact support to regain access. Thanks, Matt

Hi all, We're now very close to finally being completed. The remaining Europe servers will be delivered on Monday, and it is my hope to have them up and running by the end of next week. As soon as all servers are online, then we can re-activate threat. Thanks, Matt

Hi all, Another big push this week to try and get caught up. As of today we have: 180 unanswered/new tickets 339 total tickets We are now responding to tickets from 1/21 which puts us at 10 days to respond. This is the lowest response time we have ever had. The goal is to continue bringing it down until we reach industry standards. Thanks, Matt

Hi all, About a week ago we swapped in a bunch of more powerful CPUs for the district servers. However it looks like some important configuration settings were missed. The plan is to take them down and make adjustments during maintenance. Thanks, Matt

Hi all, This week we made significant (exciting) progress, but it's going to be a bit complicated to discuss, so bear with me. In July, we had a build that was missing correct lighting, had a bunch of visual issues, and most importantly was missing all the content and latest code changes from Live. However, the frame rate from this build when tested by SPCT was within 1ms frame time (5-10FPS) of Live. We felt based on testing that this was an acceptable place to start testing on OTW while we finished optimizing the performance. In September, we got our first look at a candidate build with all the new code and content migrated over, all action districts cleaned up from duplicate objects in the scene, and correct lighting implemented. Unfortunately the frame time had plummeted. We were 5ms (~40FPS) slower than Live. I'm sure you can imagine the disappointment experienced by the team and our SPCT testers. Since then, the team has been hammering away at optimizations and critical visual issues such as building LODs and features. It has been a game of inches. Sometimes lowering 0.2 ms at a time. Several systems in APB 1.20 had to be completely abandoned and rebuilt. A week ago, based on new systems and many optimizations that we have added, we were 3ms (~20FPS) slower than Live. As of last night, we finally identified a massive 2ms bottleneck that got introduced back in August. The issue happens as part of our build process, which skews our ability to debug and log it making it virtually impossible to track down. To be clear, we know what the issue is, but the engineers haven't yet proposed a solution. This will be the #1 area we focus on next week to try and finally get the game close to Live's FPS. During the last 2 weeks, our internal QA did a full regression on our current APB 2.1 build. We have 59 active bugs in the database including console submission bugs. All platforms (PC, XB1, and PS4) are using the same exact code. Performance on both consoles is better than the console builds we have in Live, however our APB 2.1 build for XB1 runs out of memory after running around action districts for a while. The issue was introduced during the rewrite of 2 other major systems, so we need to tread very carefully. I think this will be the biggest remaining issue after we solve the primary performance bottleneck. Moving forward, I'll countdown the bugs in the database. Just know that as QA finds more bugs and the number will shift around a little. Thanks, Matt

Hi all, Good momentum this week. The team put in a lot of hard work. As of today we have: 240 unanswered/new tickets 386 total tickets (!!) We are 2 tickets away from 1/8 which puts us at 16 days to respond. Thanks, Matt

Looks like in our rush, we left a restriction on that booted regular players but not our admin who tested that everything was up. Everything should be working properly now.