MattScott

Hi everyone, The last two updates have been pretty sparse, so this week I'll take a little more time to explain where we are at. First, a portion of the team has now moved back to bugs, which is good. As much as we want the performance to get better, we also need to fix actual gameplay issues too. Right now we are tracking 20 bugs identified by SPCT or QA. 2 of which were determined to be duplicates. 4 of which were checked in as fixed by Friday, which leaves 14 unaddressed bugs. Second, there are two overall issues we are still working on - Lighting and Performance. For lighting, we've narrowed down the lighting issue to a system in Unreal 3.5 that handles "Indirect" lights. This is a process that gets pre-processed and packaged when we ship the game. It looks like because Reloaded never turned on Day/Night, they also never tuned indirect lights that are largely only visible at night. Part of the team has now started diving into these tools to work through all our maps to fix this. For performance, I've started to realize there is no silver bullets - something that we'll find and fix that will make the game suddenly run great. However, the good news is that we've tested Asylum across a number of systems, and APB 2.1 runs that district better or at the same frame rate as APB 1.20 (Live). On one of my higher end systems with an i7 with an RTX 2080, here is the comparison with both running 1920x1080: APB 1.20 Asylum on Maximum (ignoring micro stutters) 1% Min = 6.5ms 153FPS Average = 4.85ms 206FPS 1% Max = 3.15ms 316FPS APB 2.1 Asylum on Very High (which is equivalent to Maximum) 1% Min = 4.68ms 213FPS Average = 3.71 269FPS 1% Max = 3.02ms 330FPS I ran a test on a lower end laptop with an i7 and a GTX 1050. Amazingly APB ran at an average of 100FPS on APB 2.1 which was about 10FPS faster that APB 1.20. We have spent months getting the engine to take advantage of more cores. Here is what APB 1.20 Asylum looks like on that same i7 with 12 logical processors. And here is what APB 2.1 Asylum looks like on the same i7. Other districts are more hit or miss right now, but using Asylum as a benchmark, we started looking at differences between the other districts to help us narrow down the remaining performance issues. We started to recognize very specific areas where frame rates went down significantly, and that didn't happen in Asylum. Unfortunately there are so many areas in this engine that simply weren't finished sometimes I feel like we're looking for a needle in a haystack. Or as it turns out this week: a blade of grass in a park For those playing at home, I'll give you a little sample of what we deal with on a weekly basis. Here are two screenshots. If you look specifically at the foliage, can you spot a potential issue? APB 1.20: APB 2.1: The foliage in the top shot (APB 1.20) is set to only render very close in. You can barely see it from where the screenshot is taken, and that is on Maximum settings. The foliage in the bottom shot (APB 2.1) can be seen all the way across the district. In fact it has no draw distance limit set. That means 2.1 is currently rendering thousands of tiny polygons that each cast shadows on each other while the scene elements cast shadows onto them. The team started working on this Friday, and will be cleaning it up this week. Thanks, Matt EDITED: Added frame rate stats and cleaned up grammar.

Hi all, As of today we have: 154 new tickets 252 total tickets While those numbers are higher, we are now responding to tickets from 2/28, which puts us at 1 week to respond. Thanks, Matt

Hi all, I wish I had more visuals to show for this week. Those are always more fun to post than the bland technical write-ups. We fixed several memory crashes, but the Xbox One still has an issue where we run out of memory entirely. There are three big blocks that we're looking into (shaders, textures, and audio). We'll need to investigate ways to cut those down. The crash doesn't show up right away. You have to run around a bit in Financial. We believe it's only barely over the limit based on new content that was added since the last update for that platform. Audio work is ongoing. That code is a mess. We've addressed several areas, but there are a bunch more where audio arbitrarily loops through every emitter in the scene -- every frame. The team was in the middle of designing a caching mechanism so that the audio system doesn't have to do that unless anything moves, and even then most of the cache would still be valid. We got secondary shadows and the sky lighting working correctly.. but on Friday QA found areas in Financial where the odd self shadowing issues reappear that we fixed months ago. This is the kind of thing that drives me crazy with the APB code base. 2 steps forward and 1 step back. We'll be looking into that this week. I opted out of benchmarks this week, because there was too much work in progress. Thanks, Matt

Hi all, Very good Customer Support week! As of today we have: 106 new tickets 233 total tickets We are now responding to tickets from 2/17, which puts us at 11 days to respond. Thanks, Matt

Hi all, We've made some good strides with Culling. There is no silver bullet, but it runs better than it did. There's a big slowdown related to audio that we're hunting down. After one of the content merges we left 6000+ audio emitters in Financial. They are all legitimate places where audio can play, but we need to do a better job managing them so they don't cause other parts of the code that search the scene to hit those objects too. During the week, we fixed several lighting bugs. On Friday we found a series of bugs that need to be addressed related to memory crashes. We also found a bug in the new Building Feature code that was preventing that system from working correctly. I'm going to try and post some benchmarks next week to illustrate where we are related to past benchmarks. Thanks, Matt

Hi all, As of today we have: 185 new tickets 295 total tickets (I think this is the lowest we have ever been) We are now responding to tickets from 2/9 (almost 2/10), which puts us at 12 days to respond. Thanks, Matt

Hi there, I can appreciate this comment, and I agree that we need to get the Engine Upgrade out. However, that’s not as simple as you think. I make the effort to post on a weekly basis to talk about our progress. You can read it here: Thanks, Matt

HI all, I missed yesterday's update, but the TL;DR is that this week was disappointing. In an effort to explain, I'm going to talk a little bit about the innerworkings of the APB engine. I'll start by sharing performance from scenes that have no actual gameplay in them. These are small contained scenes with not a lot of objects in them. For the Login scene on my test machine, APB 1.20 (Live) runs at an average of 194FPS and APB 2.1 (Upgrade) runs at an average of 221FPS. We tested a number of other non-gameplay scenes, and clearly our new multithreaded renderer is faster - even with the more complex lighting and more objects overall. However, that is not true in Mission districts. These are much more expansive scenes where it's not feasible to render everything. As an example, in Financial there are 30,000+ objects. There are also a lot of special cases where task objects can be enabled/disabled, buildings can show various levels of detail, NPCs animate and move, and lighting is combination of Time of Day + lightmaps. All of that requires special case code. As mentioned previously, the Engine itself has two main threads - one for game play and one for rendering. The rendering thread performs all the CPU functions necessary to throw geometry to the graphics card (GPU), which then splits those draw calls into more processes. The trick to getting the game to run fast is to balance the tasks in each of these three areas so that from the beginning of a frame to the end, they take the same time to run. To help optimize and balance performance, we have a number of tools that do performance profiling and tell us general areas, shaders, or functions that are taking a lot of time. This part can get very tricky because we have various versions of builds we can make such as Debug, Release, and Final builds. Debug is the slowest, but has the most diagnostics information we can see. Final is the build we ship, but it provides the least diagnostic information. Also running the game at a slower speed can wreak havoc with the balance of threads, which massively skews profiling results. With all of that in mind, a couple weeks ago we zeroed in on a performance bottleneck in the render thread where we do Culling (and its associated functions). This area of the code got reworked when we implemented multithreaded rendering, and clearly it is now under performing. Culling is the process of eliminating as many items as possible from being sent to the graphics card for rendering. Even though GPUs are very fast, it still takes time to copy data between the CPU and GPU, and we take a hit every time we switch materials/shaders between rendering each object. Since APB is so old, our version of Unreal has no "batching" or "instancing", which is what allows modern engines to push so many objects. In Financial, each frame we end up culling those 30,000+ objects down to 800-1000 objects that get processed for rendering to the GPU. So Culling is very important, and we use a bunch of different techniques to cull including render distance, object state, and clipping against the view frustum. All of that starts with accessing the scene objects in an Octree, a 3 dimensional grid that sub divides each cell until only a few objects are present. Each frame we start by looping down the Octree to eliminate objects that are behind or to the side of the player and out of view. The Octree is both our best way to cull, but also it's the hardest thing to multithread. This last week, we took a stab at multithreading that. This was a massive effort, and it ultimately didn't work. We got the code working, but on Thursday night when we started testing, we noticed that some objects in the scene didn't render. By Friday night, we figured out that the original devs on APB store scene data for objects in two places, the Octree and a main list of objects in the scene. For various reasons, those objects don't match. So we need to rework the approach and try again this week. Thanks, Matt

Hi all, Lots of support requests this week. While there are a couple straggler Keys to the City tickets asking for back grants, we stayed on top of things pretty well. As of today we have: 188 new tickets (which is lower than last week) 328 total tickets (which is lower than last week) We are now responding to tickets from 1/31 (except for a couple KTTC tickets) which puts us at 15 days to respond. Thanks, Matt

Hi all, Not much to report this week. Work was finished a number of bugs, but the focus has been on performance. Fortunately it looks like the team may have found the largest performance bottleneck on Friday. Looking forward to sharing the results. Thanks, Matt

Hi all, We had a bit of a rough week that required us to split time in Customer Service between older tickets and immediate item delivery issues on GamersFirst.com caused by moving the Citadel database. Everything has smoothed out now, but we fell behind a little overall. As of today we have: 195 unanswered/new tickets 336 total tickets We are now responding to tickets from 1/25 (almost 1/26) which puts us at 13 days to respond. Thanks, Matt

Yes. We are prioritizing those above others, but there really isn't a need to open a ticket now. We're working proactively through about 100 Citadel accounts that were affected. Some of these were accounts that missed the delivery of an item going back as far as November. We got about half of them sorted out today, and we'll hit the rest over the next 2 days.

Hi there, You should have received an email receipt for the purchase. You can just paste the details of the receipt to Support. Otherwise, you can let Support know what you purchased, and they can look up your account and find missing items, but that might take a bit longer. Thanks, Matt

Hi everyone, We had an issue with G1 purchases this morning when Citadel came back online. For a period of about 3 hours, item purchases weren't being delivered to players on that world. This was due to an issue caused by moving around the database during maintenance. The issue has been solved, and all purchase should work now. However, for anyone who is still missing weapons, you'll need to open a ticket at http://support.gamersfirst.com. Give them the transaction id from your purchase, and we'll escalate delivery as fast as possible. Thanks, Matt

You may want to experiment with different action districts. Some are West Coast, and some are East Coast. The team is nearly done putting in a region ping/latency indicator. That will show up on the Advanced tab to show which servers are better for you.

The team is nearly done with this. Hoping it will go into the next major patch.

Hi all, Can you test again? We made some adjustments today. Thanks, Matt

Hi there, This could be a case of selecting a New York server or a Los Angeles server. We have action districts in both. The team is nearly done with a new patch that will show players a ping/latency indicator to the closest servers. If you get in and out of districts, you can see your latency go up and down depending on what is closer. Thanks, Matt

Hi there, We've fixed the problem, so new purchases will work correctly. However, you'll need to open a ticket at http://support.gamersfirst.com to get the items delivered. Give them the transaction id from your purchase, and we'll escalate delivery as fast as possible. Thanks, Matt

Hi all, Servers are back online with the Valentine's Day patch. This maintenance was extra long because we were moving databases and more resources into the Europe data center. This also means we finally have enough resources to turn back on threat for both worlds. Thanks, Matt

Hi all, We shipped a couple APB 2.1 builds to SPCT this week. All of them were focused on performance. The engineers were able to fix the issue that was identified last week, and the game is running a lot better. We also fixed several of the top game play bugs including an issue with destructible props and shoot windows. The biggest blockers are the memory issue on Xbox and a frame rate drop when looking around. The team is currently memory mapping the game profiling builds to identify the fixes. Thanks, Matt

Hi there, For the record, we don't ever account wipe. More than likely your account got caught up in the password reset from January 2018 that forced everyone to either reset their password or contact support to regain access. Thanks, Matt

Hi all, We're now very close to finally being completed. The remaining Europe servers will be delivered on Monday, and it is my hope to have them up and running by the end of next week. As soon as all servers are online, then we can re-activate threat. Thanks, Matt

Hi all, Another big push this week to try and get caught up. As of today we have: 180 unanswered/new tickets 339 total tickets We are now responding to tickets from 1/21 which puts us at 10 days to respond. This is the lowest response time we have ever had. The goal is to continue bringing it down until we reach industry standards. Thanks, Matt

Hi all, About a week ago we swapped in a bunch of more powerful CPUs for the district servers. However it looks like some important configuration settings were missed. The plan is to take them down and make adjustments during maintenance. Thanks, Matt