MattScott

Hi all, This will be a shorter update this week, and I think it will be obvious why in the next couple days. At this point, we have run 3 Open Betas, and we need more information related to the player hardware along side the performance data. We have a lot of players who say they have had good experiences, but we also have many players who have given negative feedback. We took the time this week to significantly improve our stat collection technique. It will now also include any latency tests. So if you're experiencing what appear to be server lag, then please run /latencytest once before you exit the game. The new data collection will run and upload a dxdiag as well. Before the data had nothing identifiable, and while it is still anonymous, it does contain added hardware specs and other info. With that in mind, we have added a popup when you exit the game that asks whether you want to contribute data. We don't want to violate anyone's privacy, but we hope that as many players as possible submit data on this next test. Aside from APB 2.1 work, we also continued work on console. The build is edging right up to the memory limits on PS4, which is creating some issues debugging. But we're making good progress. Lastly, there will be more info soon on the next Open Beta, but this time we're going to do things in 2 parts. The first part will be a shorter more concentrated stress test, and the second part will be another normal play period. I would love to see as many players join for both tests, and as a little incentive we have added something fun to check out in Social. Thanks, Matt

Hi all, Backtracking a bit. In all the election run up, you may have missed the severe fires that hit Orange County during the week of Oct 26. Half our team had to be evacuated from their homes, and many spent 2-3 days in hotels or with relatives. It didn't make sense to try and conduct business with everyone's focus elsewhere, so we got a very slow start after the Beta weekend. Fortunately, our amazing fire fighters in Southern California kept all the buildings and homes safe, and no property was lost. This last week was much more productive. Here are a list of the major Beta issues that were reported (or picked up from stats), and what items have already been addressed: Graphical glitches in the UI made worse by turning on VSync. These primarily showed up during the login/character/district select flow. "Skinny" characters or loss of character customizations in faces and body proportions. (FIXED) Lots of audio issues. Missing audio. Directional audio not correct. This included left and right sounds or 3D positioned sounds. (FIXED) Graphic/chrome issue on player vehicles caused by spawning a second vehicle from the same spawner. Periodic lag / latency in districts. (POSSIBLE FIX THAT NEEDS TESTING) Stutters on the loading screen. Game crash on accepting a clan invite. (FIXED) Frame rate loss when moving the mouse around very fast. Input delay in the new Fullscreen Windowed mode. For the console players, we also jumped back into those builds to merge over all the recent work on PC. Both PS4 and XB1 are now compiling again with all the new content. We'll be looking at console crashes next week. Thanks, Matt

Hi all, Just a quick update before the Halloween festivities start. I want to thank everyone who jumped into the Open Beta 3 weekend. We collected lots of good data from the test. While there was some engine work this week, we are still collecting and digesting all the logs and player feedback. Next week, we will be putting together the next sprint based on what we prioritize. I hope everyone has a safe holiday. Thanks, Matt

Servers have been restarted. Let's see if players start getting opposed missions and the dev gun should be gone from everyone's inventory. Thanks, Matt

Hi everyone, Rough start on Open Beta #3. We're looking at mission problems and removing some guns that shouldn't have been available. Bear with us. Thanks, Matt

Hi everyone, I thought I would give the Community Events Team a shout today today. This is an entirely player run group that recently organized a fun APB Tournament that they just finished. These guys put in a lot of work behind the scenes, and you can join them on their Discord here: https://discord.gg/ZzefFfm Little Orbit has put up all the matches through the Semi Finals. We're still working on editing down the final matches. If you missed them, here's a link to the new CET Sept-Oct 2020 video playlist with the matches in order on our YouTube channel. I want to pass along a personal note of thanks to everyone who participated, the commentators, and to Blank for all the great graphics, Great job. Thanks, Matt

Hi everyone, Here is a quick update: If you don't want to read the details, so far we are still on track to run the next Open Beta - but work isn't complete yet. So there could still be hiccups. As I mentioned last week, the big remaining task is to migrate new or changed code and content from 1.20 to 2.1 so that new players that have their characters copied don't crash. We still have to do a couple other minor things like removing the AVX requirement from the installer, now that we have removed it from the build. Our internal goal is to get a candidate build to start testing by Monday EOD. That will need to be approved by our QA by Tuesday EOD for our schedule to hold. So far we have finished the code migrations from 1.20 into 2.1. That leaves data and content. For those two, we have a couple options: We can try and automatic merge We can accept the newly migrated content from 1.20 We can ignore the 1.20 content and keep the 2.1 content because it's already modified properly Or we can hand merge I believe everything left right now falls into that last category. There are 9 Unreal packages to fix Some audio work and rebuilding the sound banks Some minor map changes And data changes We have divided all of the remaining items across the team, and everyone is working in parallel to finish. More in a couple days. Thanks, Matt

Yep. Missed it. We'll get that pulled down too. Anyone who previously purchased it can keep them. We aren't removing them from inventory - just not selling them any more.

For the OP, Our predecessors made the Trump mask, and after some analysis I made the call to remove it a while back. This was not done for political reasons. Like many other mafia style games, APB is set in an alternate reality where San Paro represents a hybrid of cities in the US. While the world of APB does have politicians, none of them are real figures like Trump. Thanks, Matt

Hi everyone, I knew aiming for the Open Beta this weekend was aggressive, and unfortunately we have hit a minor setback yesterday. As we were testing the new build, we found that new players joining the Open Beta might crash if they have any of the new Joker Store items. This is because we haven't migrated content from 1.20 into 2.1 since Open Beta #2. That process is now started, but it means we won't be able to run the test this weekend. I'll keep you posted. Thanks, Matt

Hi everyone, Small Monday update: We haven't done extensive testing, but here are the early results comparing a build with AVX support to a build without AVX. This was done on a fairly high end PC. The first test was Financial: The second test was Waterfront: NOTE: There is a margin of variance for running these kinds of tests based on pedestrians and cars that go by which means essentially the results are the same. This is just one of the cases where APB does the opposite of what you would expect. In light of this, we are dropping the AVX requirement moving forward. Thanks, Matt

Hi everyone, I'll start with the good news: We're ready for Open Beta #3 and this time we will have all districts online for testing. PROGRESS WITH THE ENGINE UPDATE: The team is pushing hard to do the Open Beta this next weekend so we don't interfere with the Halloween event. There is only 1 blocker left, and it's a last minute item I wanted to try for the community: Removing AVX During development of APB 2.1, we decided to require support for AVX. After the initial Open Beta, we learned that this decision prevented a significant number of players from joining. So as an experiment I am having the team remove AVX from the client to make sure everyone can still play once we migrate over. It's about 3 days of work to recompile all of the supporting libraries without AVX, check them all in, and then recompile a new 2.1 build without AVX. If there is little to no difference, then we'll test without it. But if there is a big enough impact, then we'll have to move forward with it in. Assuming that work gets done by EOD Monday, then I would feel comfortable that we have enough QA time to identify and fix any last minute problems. For this test, we are also leaving a series of Advanced multithreading options visible for you guys to play with. None of these options require restarting the game or even getting out of the district. They dynamically enable and disable multithreading features. I would love to collect some feedback on which options worked best overall for your configuration (CPU, GPU, Memory, Hard drive) so we can set those on by default once we roll out. In terms of bugs, the team has mostly cleared the things we knew about: Vivox should now work. Customizations shouldn't look wonky (after making some larger changes, we couldn't reproduce the skinny character issues) Doors shouldn't get out of sync over the network I am hoping we fixed the character baking hitch that was occurring when new players joined the district. This is a very difficult thing to test in a lab, so the engineers will be capturing performance stats during the test. MOVING FORWARD WITH THE TEST: I want to finish this update by talking a little about expectations. This has been an incredible difficult project to finish. Obviously we are massively off the timeline I originally set. There are still bugs, and we are still not done optimizing the renderer, but weighing the pros and cons, I have decided we are hopefully "good enough" to launch. There are simply too many other more important tasks that are blocked by not releasing APB 2.1. I want to get into better district management, better matchmaking, and cross district play between worlds. Post the 2.1 launch, the team is committed to continuing to optimize performance. There are definitely places where the 2.1 FPS significantly under performs Live. We have some larger changes that we would like to make to the renderer, but they will require much more time to engineer. However, we have asked the internal testing group how the game "feels". And for those that have given feedback and been involved in daily testing, they say that 2.1 "feels" better than Live. So this is where I need the community's help. For this test, let's put away the FPS counter. There will be plenty of time for that down the road. Here is what I would like to see from testers for Open Beta #3: First, ahead of the test, get on Live. Try a mission or Fight Club. Try raising or lowering the graphics settings on Live to zero in some of the visual elements that make or break the game for you. Focus on the feel of Live. The hitches. The mouse input latency. etc. Next, get into the Open Beta and try to do the same things in the same districts. Play around with the graphics settings to try and see if you can get to a similar visual quality. Then try to compare the feel of Live versus the feel of 2.1 and give us some feedback. If we can pull off next weekend for Open Beta #3, then we'll get an announcement up in a couple days. WHAT IS LEFT IF THIS OPEN BETA GOES WELL: During the test you'll see that some things are not up to date with Live. So the only major task that we need to complete before launching is to migrate the most recent changes from Live into APB 2.1. (That in addition to any big blocker bugs we encounter with the test) Thanks, Matt

Hi everyone, This week's update is frustrating, because we are so close to new benchmarks. Unfortunately 2 specific areas were a bit more complicated and couldn't be completed on Friday. I may post mid next week once I get a chance to see where we are at. Thanks, Matt

Hi everyone, I'm long overdue for a more comprehensive update on the Engine Upgrade, so here is a recap of the last couple of weeks. Back on September 6th, I outlined a list of things we were working on. Here is where we stand with those items. Progress with the Engine Upgrade: There is still a lot of thread safety work left to do, but this was a big source of the errors. This work has been completed. We *believe* we have re-architected the mulithreaded code so that all the previous errors have been eliminated. There is still one issue related to Occlusion Query multithreading on RTX cards that causes a bizarre slowdown, so we have that code disabled till we can fix it. We want to allow secondary render threads to use the cache and then track down the specific case where we are setting the per material/mesh state and making sure those invalidate the cache properly so everything gets updated. This work has also been completed. I'm pretty excited about how we fixed this, because after we started testing the new code, we found another issue where the game was destroying the shader cache multiple times per frame. That has also been addressed. We are investigating an issue where specific items are forced to update every frame when they shouldn't. This last item is fixed. We started test driving the new code this week, and while there is some improvement, we're still not seeing anywhere near the kind of performance gains we were expecting. SIDE NOTE: The team has affectionately adopted the phrase "As expected, APB code does the unexpected". For the later half of this week, the team went back and refreshed some older code that integrates with 3rd party tools that allow us to better analyze multithreaded performance. In the past, we have been relying on Visual Studio and 3rd party tools that were working when we took over. These existing tools measure where the most CPU time is being spent. Overall, that technique has been invaluable in finding brute force code doing large tasks in a single loop that could benefit from being multithreaded. However, these tools do not measure things on the GPU, the time spent waiting on a lock or when a process is blocked. On Friday, we finished integrating some new tools that enabled us to stage a full test, capture results, and generate more comprehensive reports. As suspected, we found a significantly larger problem related to how the engine renders Meshes overall. To give an example, our previous efforts to multithread code lower down in the pipeline only account for a fraction of the processing time. The higher level system for actually rendering everything that was processed accounts for the majority of processing time - but it's mostly spent waiting or being blocked which is why these areas never showed up in the previous analysis. The overall rendering is effectively drawing elements on only 1 or 2 threads. It's incredibly inefficient, and we already have a good idea on how to fix this. The team will be moving onto this new issue next week. Alongside these 3 areas listed above, the rest of the team has been fixing VOIP, desync, and other more minor areas. We need those out of the way so we can launch once the renderer is finished. As always, I'm providing these updates in real time with the best information I have at hand. I am (again) hopeful we'll see good progress this coming week, and that I can finally get some better mission district benchmarks. The plan is to spin up Open Beta #3 with mission districts as fast as we can after that. NOTE: After I posted this, I thought I would provide some nerdy details. Rendering in a 3D engine mostly happens in 3 stages. Stage 1 processes the scene. This involves lots of complex routines that gather up and process all the things we need to draw. Some of things we do in this stage are: Identify only the things that are in view of the player's camera. This happens in multiple ways: bounding checks on the field of view, eliminating things too far in the distance, testing to see if an object is occluded (covered) by another object, etc. We want to eliminate everything not in view to avoid drawing far too many things. Recalculate lighting receivers on each object that we are going to render. Process objects for extra data that we need to pass into the next 2 stages. Stage 2 actually draws the big list of things to the scene, which also involves its own set of stages to properly sort objects back to front and render transparency correctly. Stage 3 handles post processing of the final frame buffer (effectively a composited image of the scene once the objects are drawn). This handles a variety of effects including shadow rendering, bloom, iris response, motion blur and more. Stage 3 requires a lot of extra information that was captured in Stages 1 and 2. TL;DR: The majority of our multithreaded work has been in Stage 1. Now we've identified big areas in Stage 2 that are affecting the overall speed of rendering a single frame. Thanks, Matt

Hey everyone, My schedule has been incredibly busy - busy enough that I completely missed last week's update and only realized it today. Sorry about that. Part of the problem is that in addition to daytime meetings here in Europe, I am also leading a massive project in the evenings with the US that keeps me up till 2-3am every night. Big deadline today, so that should free up some of my time to resume these updates tomorrow. Apologies, Matt