Tracking the Engine Upgrade

[MattScott 15245]

Hi everyone,

 

This week was a mixture of success and frustration.

 

We made a lot of progress knocking out bugs, and the only thing blocking the release of the Beta are some inconsistent crashes with RTX and higher end cards. At this point, we're starting to get more specific issues reported. These are things like "The stairs in this part of Asylum cause a jitter as I go up them" or "This object needs to allow shots through it." I'm not sure how much longer I will hold the Beta while we fix some of these smaller issues, but until the last of the major crash bugs is fixed, I have no choice but to hold this back.

 

Lighting is getting very very close, but this continues to be one of the more frustrating areas.

 

The main issue right now is color balance using Color Look Up Tables (Color LUTs). These are images that modify how specific color ranges are displayed.

 

Here is an example of what a ColorLUT image looks like. We use this one for color grading Financial at noon.

 

Here are some references from Live (if you look closely you'll notice some flawed lighting with foliage).

All of these shots accomplish giving the game a moody look and feel while also keeping the character lit enough to see where he is.

 

Midnight

 

Noon

 

5pm

 

 

Here are some shots from APB 2.1 as it stands right now.

All of these shots have much better lighting consistency (ie. foliage doesn't glow).

However the scenes don't have the same punchy color as Live, and the character is far too muddy while standing in shadow.

This gives the overall scene a very flat feel.

 

Midnight

 

Noon

 

5pm

 

This is all very correctable with ColorLUTs, because we punch up the contrast on specific colors.

Except.. that system isn't doing what it is supposed to do.

 

Here are some pretty awful examples of what we see right now when the team tries to enable some pretty basic ColorLUTs.

This system works fine right now on console, so somewhere along the way it broke.

 

Noon

 

5pm

 

Thanks,

Matt

[MattScott 15245]

Hi everyone,

 

This was another productive week.

We fixed the blown out ColorLUT issue that I posted pictures of last week.

We are knee deep in fixing an in-game browser issue that causes a crash when opening up ARMAS.

We are also working through more collision and game-play issues in Asylum.

 

And lastly, that's right, we are working on lighting issues in the districts.

 

I know this is something that has been taking forever, and several players suggested that we release the Beta with unfinished lighting.

However, just to show you some of what we are fixing, I pulled some images that went back and forth between the team this week.

 

Building lighting in Unreal is actually 2 different processes: (1) Baking lightmaps and (2) building indirect lighting

 

1. Baking lightmaps

 

This is an ancient technique for generating fast lighting that doesn't require very powerful graphics cards.

The primary downside to lightmaps is resolution. Lighting is baked to a texture and then wrapped around objects just like normal textures, but if your lightmap is too small, then you'll end up with very jagged results. Conversely, if your lightmaps are too big, then you'll eat up a lot of texture space. Also, you can only build lightmaps for objects that are static - meaning they don't move. So it's not a technique you can apply to everything in the scene. Ultimately, even though this technique is very old, lightmapping works well if you blend it with several other techniques.

 

In our case, we don't build lightmaps, we build shadow maps.

This enables us to change their color and how much they influence the scene at different times of day.

We can only build shadowmaps for static objects that don't move, and those objects require a separate UV channel to map that generated texture into.

Buildings are the best example of objects that we bake lighting for.

 

However, Unreal is very finicky about the UVs it maps into. For those that don't know, UVs are a 2D position on a texture that is assigned to each vertex in a mesh. This defines how textures map onto the mesh. Lightmaps require "unique" UVs. That means each vertex maps to a unique spot in the texture, which makes sense because the lighting generated into the lightmap would be unique for each part of the building. If any of those UVs are slightly overlapping or touching the edge of the texture, then Unreal errors and aborts lighting that object.

 

We run lighting with error color coding turned on to see which objects have issues. This looks something like:

 

The orange area indicates that part of the lighthouse entrance doesn't have correct UVs. 

This specific case likely wouldn't be a big deal for Beta.

 

However, this next case is, because every ceiling in this part of the district has no lighting and looks terrible.

 

 

As you imagine some districts have thousands of individual meshes that require lightmap UVs, so we have been going through by hand to fix all the areas that aren't building correctly.

Corrected lighting ends up looking like this:

 

 

2. Building indirect lighting

 

This is a new system that was introduced in Unreal 3.5.

 

I'm sure Reloaded felt this would be the biggest visual upgrade for players, and they aren't wrong. However, the new lighting engine in Unreal 3.5 also costs quite a bit more to render, so all of our initial performance tests were terrible. We have spent a lot of time optimizing so that players will get the benefits of better lighting, but not take a huge performance hit.

 

Indirect lighting is built into a 3D grid and it samples the various lights in the scene at that location, so that when a moving or dynamic object is in that area, it is quick to apply the lights.

 

This next image shows off some of the better lighting which is a combination of overhead dynamic lights, dynamic shadows, baked shadowmaps, ambient occlusion, and built indirect lighting on the player).

It also shows off one of the big flaws in how indirect lighting is built.

 

The indirect lighting in this shot is the orange light casting on the player.

The flaw in the shot is that there is no obvious source of that lighting. In fact, that orange light is coming from just inside that building.

This only happens if they get too close to the wall because the 3D grid is 4ft x 4ft, which doesn't have the resolution to sample lighting from both inside and outside the building.

 

To fix this, it requires us to build lighting volumes that "trap" the light inside the volume by inserting extra cells in the 3D grid for indirect lighting to emulate where walls would prevent light from leaking outside. The art team has been building lighting volumes for these cases.

 

Lastly, we had a couple players mention that we needed to turn on ambient occlusion.

It's there, it just has problems.

 

Here is what ambient occlusion looks like right now in Asylum.

 

It's very faint, and it's being calculated on a very dark frame buffer in the pipeline.

That's also being looked at.

 

Thanks,

Matt

[MattScott 15245]

Hi everyone,

 

There isn't an Engine Update this week.

 

Instead we are announcing the first Live stream of the Beta on Friday, April 17, 2020 at 11am PST.

 

You can get more details here: 

 

Thanks,
Matt

[MattScott 15245]

Hi everyone,

 

I hope you are enjoying your respective holidays this weekend and connecting with family and friends either virtually or in real life.

 

Several players reached out to me asking if/how we managed to fix the problems I discussed in the last update, and one even sent some ideas on how to fix the issues. After responding to them individually, I realized that I had nearly written a blog. So I thought I would share those answers here, since I didn't post an Engine Update yesterday.

 

First, a little bit of history.

 

When the upgrade to Unreal 3.5 was started at Reloaded Productions, they were faced with the difficult task of exporting all the geometry for the districts from Unreal 3 to Unreal 3.5. You might think this is easy, but Unreal 3 has very limited export formats. The final solution was to write a commandlet in Unreal 3 that exported meshes through the OBJ format as pieces along with some extra data. Then they wrote a commandlet in Unreal 3.5 that reconstructed all those individual files and imported everything, and when they were done, they exported a single FBX file for each mesh. This is a pretty impressive bit of code, because now we have a decent starting place for all the basic buildings (like that lighthouse entrance from last week).

 

It's important to know that APB was the result of a different project at RTW which was focused on procedural world building. As good as that tech was, it was also somewhat messy. So a lot of manual 3D art work had to be done to clean things up. Since the RTW days everything has been done in 3DS Max. So that's what we used.

 

When we hit issues like last week for a specific building, we often have to fix it in the original FBX and then re-import it back into Unreal.

 

But that leads to 5 basic problems:

 

1) There are unwelded verts all over the place that leave stray polygons (polygon soup) detached from each other. The easy fix for this is to select all the verts in 3DS Max and then weld everything within a specific tolerance back together. But you have to find the right weld tolerance. APB is modeled in centimeters. When we welded verts within 1cm of each other, we ended up welding verts that needed to remain distinct, and some texturing got ruined. The best tolerance ended up being 0.5cm. Here is an example of what unwelded verts look like when the lightmapping tools try to bake lighting.

 

 

2) Since these FBX files were programmatically exported, they have no smoothing group data. Bad smoothing groups is another place where lighting will bake extra seams and edges that you don’t want. There are thousands of building pieces in APB and many of them have LOD variants, so we needed an approach that could be run automatically across a number of files. We tried automating "autosmooth" in 3DS Max and generated disastrous results. In the end, since the majority of buildings have squared off edges, we wrote a maxscript that was able to sample faces for 30 different vectors (angles) to rebuild the smoothing group data. From there everything else had to be hand tweaked.

 

Here is an example of a more complex dock building after we applied autosmoothing. I have selected the faces from a specific smoothing group. You can see how some faces on the same side of that building are not highlighted. This means they aren't in the same smoothing group and can cause problems when we bake lighting.

 

 

Here is the same smoothing group after using our custom script. All of the faces on the side facing the camera are in the same smoothing group. This will insure they will all be lit more uniformly together.

 

 

3) After those two issues, then we get to the uvs. In many cases, the original set of lightmap UVs can simply be repacked to use the texture space better. But if you had stray polygons to begin with, then chances are those will still be messed up in the UV channels. So in about 20-30% of cases, we had to create all new unique UVs required for lightmapping to work. Again, the automated uvmapping tools failed us because they all use a single angle to determine where to unwrap the mesh. So this had to be hand tweaked by artists mostly selecting chunks of contiguous walls where we wanted uniform lighting, then planar mapping those selections as a single set of uv faces, and then repacking the whole uv channel to fit 0,0 to 1,1 with no overlapping faces.

 

For those who are unfamiliar with what uvs are, they are best described with a picture. UVs are a 2D coordinate system for mapping textures onto a 3D object. Each face (triangle) of an object is defined by 3 vertices in 3D x,y,z coordinates. In order to texture those faces, we have to map each vertex into 2D space with just x,y which are renamed to u and v so we don't confuse them with the 3D coordinates. Each object can have multiple uv channels. For lightmapping, we need a channel where none of the faces are overlapping. The building above looks like this: 

 

 

And here is an example of the lighthouse itself when we tried to use Flatten uvmapping to generate a set of unique UVs and then tried to bake lighting. Since the lighthouse tower is round, you can't apply a technique that unwraps faces based on angle.

 

 

 

4) Sometimes we simply needed to increase the lightmap resolution. When the districts were rebuilt in Unreal 3.5, part of that commandlet tried to guess the surface area and set the appropriate lightmap texture size. In some cases that estimate was way too small. Increasing the size fixed a lot of lighting artifacts. The problem with APB is that texture memory is already stretched pretty thin. So we have to be careful when increasing these.

 

Fortunately we have tools in Unreal that allow us to view the entire district with lightmapping resolution grids turned on. Larger / darker will be worse with chunky shadows or visual artifacts. Smaller / lighter is better and more defined. Here is what that looks like on the same dock building. Some of the areas that appear solid colored actually have a grid so small you can't see it in this shot.

 

 

5) Lastly, for hyper detailed meshes, there is an option in Unreal to use high precision UVs. I’m guessing by default they use traditional floating point values, which lose accuracy for small numbers with a lot of decimal places. Checking this option probably switches to double precision which preserves better accuracy. In any case, this also cleared up some lighting artifacts.

 

Hopefully some of this information will help you understand why the Engine Upgrade has taken so long - even from the point that we took over. It's also one of the reasons we decided to launch the Beta with only Asylum and Social, since those districts are farther along in being cleaned up.

 

Thanks,
Matt

[MattScott 15245]

Hi all,

 

Missed the post yesterday here. Instead we uploaded the highlights from our Beta stream on Friday.

 

Here is the thread: 

 

Thanks,

Matt

[MattScott 15245]

Hi all,

 

This week started a bit unorganized. I think the team was a bit burnt out having worked long hours to prep the streams last week, so some staff took Monday off.

The rest of us took the day to digest all the data from the streamed playtest of APB 2.1, so we could create a new dev plan to finish the Beta.

 

Problem #1: Poor frame rate

 

Initially we thought the slowdown was from all the characters in the district. In order to test that, we needed to revive some old code that allows us to run Bots in districts that can emulate players for missions. Don't get excited. These Bots are horrible. They literally move around the district firing and interacting with things randomly. SPCT has been fascinated by seeing these AI abominations running around and trying to determine where they are going.

 

After some testing, we determined the major part of the slowdown isn't tied to the number of players in the district. However, the Bots did allow us to spend some time identifying game loop areas in missions that were underperforming. We got a build out to testers by Wednesday and there was some improvement.

 

We also found that Scaleform continues to drag our framerate down, so we have decided to remove it entirely from the game play HUD. That will free us from having to constantly update that system for the couple screens that use it. We optimized the Scoreboard HUD, Actionmessage HUD, and Taskmarker HUD.

 

Problem #2: Audio issues

This is the last part of the code that we haven't touched.

 

I'm sure many players noticed how bad it was in the streams. Right before the playtest we pulled down a new version of Wwise, so this week we dove into that part of the code where we found... a mess. It looks like during the port to console (and possibly even in Live) the dev team didn't really implement sound to utilize all of the proper features. We did make progress, and we're hoping to get a build to testers soon so they can verify some of the worst sound issues are corrected. We rebuilt the PS4 and XB1 sound banks, fixed a threading issue with audio memory, and reduced some info calls to improve performance. 

 

Ongoing work:

 

We carried over a task from last week to implement a new SSAO shader to get rid of the flicker when you run. The flicker is caused by changing the Field of View which breaks the math in the shader. It's a known issue in Unreal.

 

Thanks,
Matt

[MattScott 15245]

Hi all,

 

Sorry for the late post today. I try to get these out a bit earlier, so they can be read by our European player base.

 

We had another big push this week to try and clean up remaining items for the Beta.

We completed a lot of the audio re-work late on Friday, but I had an SPCT member go in, and we saw many of the issues were fixed.
Still a couple stragglers, but with the limited testing, it looks like we got about 80-85% of these bugs.

 

Next, let's deep dive frame rate issues.

 

Last week, I walked through our work on optimizing the UI during missions in Asylum.

Today, I thought I would post some of the performance statistics from several builds.

 

These all came off the same machine with the following specs:

Intel(R) Core(TM) i7-6700K CPU @ 4.00GHz with 16GB of ram and an NVIDIA GeForce RTX 2080 SUPER

 

The test is based on the average frame rate running around for 10 minutes in Asylum while actively doing objectives and missions.

 

I should make the disclaimer that we always accept these benchmarks with a grain of salt, because with an online game it is impossible to recreate the same exact testing conditions. But we use them as a general feedback mechanism to let us know if we're on the right track. I'll apologize in advance for not having any of Skay's pretty charts.

 

We started with the performance on Live with ~18-19 players in the district:

Mean Frame time = 8.08ms

Mean Frame rate = 123.75 FPS

Average Frame time = 8.125ms

Average Frame rate = 123.07 FPS

Maximum 1% Frame time = 6.95ms

Maximum 1% Frame rate = 143.8 FPS

Minimum 1% Frame time = 11.12ms (ignoring hitches)

Minimum 1% Frame rate = 89.87 FPS

UI accounted for 1.41ms on average

 

We actually pulled two logs from two different sessions during the 4/17 Beta. I'm going to share them both, because it illustrates the variance in benchmarking.

 

Session #1 during the Beta stream with ~19-24 players in the district:

Mean Frame time = 12.02ms

Mean Frame rate = 83 FPS

Average Frame time = 12.19ms

Average Frame rate = 82 FPS

Maximum 1% Frame time = 6.62ms

Maximum 1% Frame rate = 151 FPS

Minimum 1% Frame time = 18.16ms (no hitches)

Minimum 1% Frame rate = 55 FPS

UI accounted for 4.43ms on average (more than 3ms higher than Live)

 

Session #2 during the Beta stream with ~19-24 players in the district (but only half that for the last couple minutes):

Mean Frame time = 10.98ms

Mean Frame rate = 91 FPS

Average Frame time = 11.21ms

Average Frame rate = 89 FPS

Maximum 1% Frame time = 5.55ms

Maximum 1% Frame rate = 180 FPS

Minimum 1% Frame time = 16.94ms (no hitches)

Minimum 1% Frame rate = 59 FPS

UI accounted for 3.49ms on average (only 2ms higher than Live)

 

You can see that just dropping some opponents during the stream for the last couple minutes made a significant difference in the frame rate between Session #1 and Session #2.

 

All three of these tests had real players running around firing and jumping a lot more often in concentrated areas with customizations.

The bots don't do that. There is literally no AI at all. They randomly path around the district, fire and interact with things.

They are great for us to trace code and diagnose issues, because we can start up missions in Asylum without gathering a bunch of real people.

In retrospect, I wish we had a bot run from the 4/17 Beta build.

 

We got an original benchmark today, and I had to throw it out because we didn't have enough bots running and the bots that were in the district were super far from each other.

So the stats were clearly skewed and too good.

 

We fired up more (although couldn't get 19-24 to compare), and this time they clustered up a bit near the Criminal spawn point.

But even this next benchmark wont represent an apples to apples test with the first three.

 

Here are the results from today's session in the new Beta build with 15 bots:

Mean Frame time = 6.03ms

Mean Frame rate = 165.78 FPS

Average Frame time = 6.07ms

Average Frame rate = 164.8 FPS

Maximum 1% Frame time = 4.82ms

Maximum 1% Frame rate = 207.4 FPS

Minimum 1% Frame time = 7.80ms (no hitches)

Minimum 1% Frame rate = 128.2 FPS

UI accounted for 2.56ms on average (we are still running both Unreal UI and Scaleform for now during game play in APB 2.1)

 

Based on this test, the next step will be to organize another large playtest between SPCT, the streamers, and LO staff so we can get a better benchmark.

I'll let you guys know when that will happen.

 

FOOTNOTE: Also for those that are interested, we started work on the unfinished DAM (District Allocation Management) system that was designed by Reloaded. This system dynamically collapses and expands threat by spinning down and spinning up new districts during the day. This is the first major system we will launch directly aimed at solving match making in a new way. It requires APB 2.1 to work, but I wont hold back the Beta for it. I wanted to make sure we get that feature pushed out as soon as possible.

 

Thanks,

Matt

[MattScott 15245]

Hi everyone,

 

I missed yesterday's post due to some hard core number crunching that needed to be done for this update. Sorry about that. My schedule has been extremely busy lately. That also means I'm several months behind responding to messages in my forum inbox. I plan on trying to squeeze some time in this week to catch up a little.

 

With that out of the way, today marks the 2 year anniversary of Little Orbit taking over APB Reloaded in 2018. To be honest, I had hoped that the Engine Upgrade would be shipped by now. But once it was clear that many of the issues plaguing us were not going to be easily resolved, I decided to start this thread a little over 7 months ago.

 

Here is a quick recap of where we are at this point.

 

- We found a new way to multithread using an upgrade to an existing library in the game. We had avoided doing this previously, because the library was only supported on Windows. But the upgrade seems to support console now, so we took a shot. Our first test is focused on how we update lights in the game. The new system is far more efficient and has the capacity to manage more cores at higher utilization. This is still pretty experimental, but we're pretty excited about it. If this test holds up, then we'll likely implement this across the other multithreaded areas.

- The team spent last week continuing work on UI optimizations to frame rate. As I said last week, there is still a significant difference in time spent rendering the UI between Live and the Beta.

- We also found some clean up audio work that was causing an occasional crash.
- And we are exploring a persistent crash popping up for the in-game browser. There has been light weight work going on there for the last couple weeks to get it sorted before we ship the Beta.

 

That brought us to Friday and a rather quickly thrown together stress test to see how all the changes performed.

 

This time we kept it private, because we knew were pushing the envelope. The plan was for each player to first spend some time on OTW playing the Live build of Asylum for a base line benchmark. We encouraged many of the players to load up their most complex customizations, and we tried to fill the district as much as possible. Then we did the same thing in our Beta environment. Each player submitted both logs to us, so we could process the results. At the same time, we had our devs in the Beta environment running around and capturing performance profiles to determine the functions that execute the most and require optimizations.

 

In retrospect, I'm glad I kept things private, because the Beta portion of the playtest was a bit of a disaster. It appears that several of the "rare" or "occasional" crashes scaled up under more load and became "constant". I doubt anyone was able to play for more than 3-5 minutes without crashing and having to load back in.

 

However, we did accomplish most of what we wanted. We got some usable comparisons from testers. We also captured many crash logs, and the devs got quite a number of good profiles that revealed some things we knew, some thing we suspected, and some things we didn't. 

 

Based on the logs we did get, here are the results.

 

DISCLAIMER #1: This was a lot less scientific than I prefer, so please don't read too much into the results. Due to the consistent crashes, it was really hard to pull apples-to-apples comparisons, but I did my best by grabbing specific segments of time when things were more equal. At a minimum there might be some good data here for players on APB 1.20 looking to figure out how to improve their performance.

 

DISCLAIMER #2: Our logs don't capture whether the machine is overclocked, so there may be performance discrepancies.

 

I'll start with system configurations and their FPS stats for APB 1.20:

AMD Ryzen 9 3950X 16-Core Processor  + NVIDIA GeForce GTX 1080 Ti - 112 FPS average with unknown players (grapher bug processing log - but probably 3x as many as the 2.1 log)

AMD Ryzen 7 3800X 8-Core Processor + NVIDIA GeForce GTX 1070 - 112 FPS average with ~13 players

AMD Ryzen 7 2700X 8-Core Processor + NVIDIA GeForce GTX 1060 - 89 FPS average with ~14 players

AMD Ryzen 7 1800X 8-Core Processor + NVIDIA GeForce GTX 1080 Ti - 65 FPS average with ~20 players

AMD Ryzen 5 3600 6-Core Processor + NVIDIA GeForce RTX 2060 SUPER - 134 FPS average with ~8 players

Intel(R) Core(TM) i9-9900K CPU @ 3.60GHz + NVIDIA GeForce RTX 2080 Ti - 184 FPS average with ~11 players

Intel(R) Core(TM) i9-9900K CPU @ 3.60GHz + NVIDIA GeForce GTX 1080 Ti - 144 FPS average with ~13 players

Intel(R) Core(TM) i7-9700K CPU @ 3.60GHz + NVIDIA GeForce GTX 1080 - 92 FPS average with ~9 players

Intel(R) Core(TM) i7-8700K CPU @ 3.70GHz (OC to 5.0GHz) + NVIDIA GeForce GTX 1070 - 151 FPS average with ~8 players

Intel(R) Core(TM) i7-7700K CPU @ 4.20GHz (OC to 4.60GHz) + NVIDIA GeForce GTX 1080 Ti - 117 FPS with ~13 players

Intel(R) Core(TM) i7-7700K CPU @ 4.20GHz + Radeon RX Vega - 117 FPS average with ~22 players

Intel(R) Core(TM) i7-6700K CPU @ 4.00GHz (OC to 4.4GHz) + NVIDIA GeForce RTX 2070 - 111 FPS average with ~11 players

Intel(R) Core(TM) i7-6700K CPU @ 4.00GHz + NVIDIA GeForce GTX 1080 Ti - 127 FPS average with ~13 players

Intel(R) Core(TM) i7-6700K CPU @ 4.00GHz + NVIDIA GeForce GTX 970 - 61 FPS average with ~8 players

Intel(R) Core(TM) i7-6700 CPU @ 3.40GHz + NVIDIA GeForce RTX 2060 - 96 FPS average with ~13 players

Intel(R) Core(TM) i7-4820K CPU @ 3.70GHz + NVIDIA GeForce RTX 2060 - 72 FPS average with ~14 players

Intel(R) Core(TM) i7-4770K CPU @ 3.50GHz + NVIDIA GeForce RTX 2070 - 65 FPS average with ~8 players

Intel(R) Core(TM) i7-3820 CPU @ 3.60GHz + NVIDIA GeForce GTX 660 Ti - 62 FPS average with ~8 players
Intel(R) Core(TM) i5-7500 CPU @ 3.40GHz + NVIDIA GeForce GTX 1060 - 75 FPS average with unknown players (grapher bug processing log - but probably 3x as many as the 2.1 log)

 

Next, here are their FPS stats for APB 2.1:

AMD Ryzen 9 3950X 16-Core Processor  + NVIDIA GeForce GTX 1080 Ti - 177 FPS average with unknown players (grapher bug processing log - player count likely 1/3 of the 1.20 log)

AMD Ryzen 7 3800X 8-Core Processor + NVIDIA GeForce GTX 1070 - 123 FPS average with ~11 players

AMD Ryzen 7 2700X 8-Core Processor + NVIDIA GeForce GTX 1060 - 132 FPS average with unknown players (grapher bug processing log - player count likely 1/2 of the 1.20 log)

AMD Ryzen 7 1800X 8-Core Processor + NVIDIA GeForce GTX 1080 Ti - 117 FPS average with ~9 players

AMD Ryzen 5 3600 6-Core Processor + NVIDIA GeForce RTX 2060 SUPER - 140 FPS average with ~5 players

Intel(R) Core(TM) i9-9900K CPU @ 3.60GHz + NVIDIA GeForce RTX 2080 Ti - 197 FPS average with ~11 players

Intel(R) Core(TM) i9-9900K CPU @ 3.60GHz + NVIDIA GeForce GTX 1080 - 214 FPS average with ~11 players

Intel(R) Core(TM) i7-9700K CPU @ 3.60GHz + NVIDIA GeForce GTX 1080 - 130 FPS average with ~9 players

Intel(R) Core(TM) i7-8700K CPU @ 3.70GHz (OC to 5.0GHz) + NVIDIA GeForce GTX 1070 - 194 FPS average with unknown players ( player count likely close to the 1.20 log)

Intel(R) Core(TM) i7-7700K CPU @ 4.20GHz (OC to 4.60GHz) + NVIDIA GeForce GTX 1080 Ti - 169 FPS average with ~8 players

Intel(R) Core(TM) i7-7700K CPU @ 4.20GHz + Radeon RX Vega - 147 FPS average with ~11 players

Intel(R) Core(TM) i7-6700K CPU @ 4.00GHz (OC to 4.4GHz) + NVIDIA GeForce RTX 2070 - 154 FPS average with ~7 players

Intel(R) Core(TM) i7-6700K CPU @ 4.00GHz + NVIDIA GeForce GTX 1080 Ti - 134 FPS average with ~13 players

Intel(R) Core(TM) i7-6700K CPU @ 4.00GHz + NVIDIA GeForce GTX 970 - 59 FPS average with unknown players (grapher bug processing log - player count likely close to the 1.20 log)

Intel(R) Core(TM) i7-6700 CPU @ 3.40GHz + NVIDIA GeForce RTX 2060 - 113 FPS average with unknown players (grapher bug processing log - player count likely 1/2 of the 1.20 log)

Intel(R) Core(TM) i7-4820K CPU @ 3.70GHz + NVIDIA GeForce RTX 2060 - 72 FPS average with ~13 players
Intel(R) Core(TM) i7-4770K CPU @ 3.50GHz + NVIDIA GeForce RTX 2070 - 164 FPS average with ~6 players (don't read into this.. no idea what was happening or not)

Intel(R) Core(TM) i7-3820 CPU @ 3.60GHz + NVIDIA GeForce GTX 660 Ti - 62 FPS average with unknown players (grapher bug processing log - player count likely close to the 1.20 log)

Intel(R) Core(TM) i5-7500 CPU @ 3.40GHz + NVIDIA GeForce GTX 1060 - 112 FPS average with unknown players (grapher bug processing log - player count likely 1/3 of the 1.20 log)

 

Whew! That is a relatively small amount of info, but took quite a while to compile.

We are planning another play test this week, so I should have a better update on the 16th.

 

Thanks,

Matt

[MattScott 15245]

Hi everyone,

 

This is going to be a shorter update with a follow up tomorrow or Monday.

 

We spent the week fixing the bugs and crashes that plagued last week's play test.

Yesterday did a new play test with ~30 players.

 

Everyone started by getting on APB 1.20 and testing Asylum for about 30 minutes. Several of the players last week didn't have their logging setup correctly, so we re-ran that portion to establish a benchmark. With people jumping in and out it looks like we had a pretty solid average of ~22 players over the entire test.

 

Next, they got back on APB 2.1 and tested Asylum for the same amount of time. Looks like we got ~22 players during this test too.

There were a few crashes, so we have some things to fix this week, but overall the stability was way better.

Unfortunately something was going on with the server. There was definitely a latency problem.

 

There are a lot more logs to go through this time, and I had other obligations this weekend, so I haven't finished collating everything 

This process involves matching up CPU/GPUs to stats for both OTW1 and OTW2, and instead of 19 like last week, there are 29.

 

However, here are the stats from the first player in my list.

 

APB 1.20:

AMD Ryzen 7 3800X 8-Core Processor + NVIDIA GeForce GTX 1070 - 102FPS with ~22 players

Highest 1% FPS = 173FPS

Lowest 1% FPS = 65FPS

 

APB 2.1:

AMD Ryzen 7 3800X 8-Core Processor + NVIDIA GeForce GTX 1070 - 125FPS with ~22 players

Highest 1% FPS = 248FPS

Lowest 1% FPS = 63FPS

 

Thanks,
Matt

[MattScott 15245]

Hi everyone,

 

I didn't quite make Monday, but this was a lot to organize. I probably wont be able to post this level of detail in the future.

 

These stats are admittedly a little disappointing. I know many of the testers complained about latency issues on APB 2.1.

We're looking to what happened there, but the stats also show that OTW2 didn't perform across the board like last time.

It is also important to know that we ran the most complicated customizations that we could, and we still have hitches when new players are entering the district and their customizations are being baked.

With so many players versus the 5/8 that could be affecting the stats as well.

 

Regardless, here are the numbers:

 

5 testers performed lower on APB 2.1 than APB 1.20. And 6 testers from 5/8 weren't able to make it this time. I left their entries blank.

We do have more lower spec machines this time, and everyone is running Maximum in APB 1.20 and Very High in APB 2.1.

You can definitely see how much raw processing speed affects APB 1.20.

The biggest gains for us in APB 2.1 are tied to having more cores.

I might be missing some over clocking data for a few of these, which might explain some of the oddball discrepancies.

 

Like I mentioned on Saturday, we had about 10 crashes. They all turned out to be related to the same thing, and we believe we have a fix for that now.
We'll spend the next couple days chasing down the rest of the issues, continuing to multi thread new bits of the game, and fixing the hitch when baking customizations.

 

We plan on running another test on Friday 5/22, and this time we'll try to make sure the APB 2.1 server is running better.

 

UPDATE 5/19: I put a new listing up that had more details for overlocking by several testers. I think that helps put the numbers in better perspective.

 

Thanks,
Matt

[MattScott 15245]

Hi everyone,

This update isn't going to be very long.

 

We finished a number of fixes last week, and ran another playtest.
We knew there was a potential issue with the test district server that popped up on our 5/15 test, so this time we staged a backup district server.

We were right and the same issue popped up, but switching to the backup failed because our provider didn't open up the right access, so it was useless.

Frustratingly, we still haven't gotten a decent run without crashes on APB 2.1. 
However, the devs were still able to do a few profiles and capture more areas that we can look at for optimizations.

 

Right now, the most common question I get asked is "When will the Beta launch?"

 

I can't answer that yet, but I can share the remaining tasks/bugs blocking the release:

• A crash bug that has already been fixed (we hope)
• A bug on the server related to performance (this has been blocking our last two play tests)
• A bug related to the crash reporter that is stopping it from working all the time. (this is critical for getting good data from the Beta)
• A client fix related to fixing hitches every time a new player loads into the district (this is the client taking a hit to generate the customizations for the new character)
• A backend task related to copying the Live characters to Beta the first time the player logs in

Enjoy the Memorial Day weekend, and stay safe.

 

Thanks,
Matt

[MattScott 15245]

Hi everyone,

 

Today is my APB AMA on Reddit, so I wont be updating this thread until maybe tomorrow or Monday.

But you can get Reddit and ask me questions directly here:

 

 

Thanks,

Matt

[MattScott 15245]

Hi everyone,

My schedule has been out of control for the last couple weeks. I've made it through the worst of it, but some of those commitments cut into my time interacting with the community.

All of my various inboxes are overflowing with messages from around February. I am hoping to start responding soon.

 

The last two playtests have been somewhat smaller. After reviewing the results from the 29th, there wasn't much to comment on. We did collect a number of very good crash reports, and the district server issue got fixed.

Yesterday's test was very positive, but we had less testers, so I can't do an apples to apples. I don't want to get lulled in by good numbers based on less concurrency.

 

For this update, I'm going to track progress on the few remaining items blocking the launch of the Open Beta.

I have highlighted the actual blocker bugs, so you can see how close we are.

• The original crash bug that I listed on Memorial Day. (fixed) 
  • 3 more crashes popped up on the 29th, but our test yesterday only had a single crash. So we're making progress.
• Still investigating the new crash from yesterday. (not yet started)
• One of the new crashes seems to be related to exclusive fullscreen mode (in progress)
  • This is very close. We are implementing borderless windowed fullscreen that can run at different resolutions than the desktop. Nearly all modern games today have dropped exclusive fullscreen in favor of windowed fullscreen.
• A bug related to the in-game browser (not yet complete but non-blocking)
  • We have a work around in place to force the browser to open externally for now.
• A bug on the server related to performance (fixed)
• A bug related to the crash reporter that is stopping it from working all the time. (fixed)
• A client fix related to hitches every time a new player loads into the district related to baking out customizations. (partially fixed) 
  • This ended up being quite a bit of work. The partial fix was tested yesterday, and we saw some improvement. The full fix is finished now, and we're hoping to get tested internally by Tuesday. That will allow us to get another playtest organized for Wednesday and possibly a second on Friday this week if necessary.
• A backend task related to copying the Live characters to Beta the first time the player logs in (fixed but in testing)
• Copy new code and content from APB 1.20 to APB 2.1 (not yet started)
  • This usually takes about a week to do, and we've done it plenty of times before. It's necessary so that when characters are copied over from APB 1.20 for Open Beta they don't crash the new version with clothing or guns that don't exist.
• There are also a lot of minor bugs, but the whole purpose of the Open Beta is to get lots more players involved, so we can fix the remaining issues.

I'm excited to get this behind us, so we can start working on more impactful things like matchmaking and phasing.

I'll be back on the 13th with another update.

 

Thanks,

Matt

[MattScott 15245]

Hi everyone,

 

This week we got in 2 play tests. One on 6/10 and another on 6/12.

We still had 2 crashes on 6/12. The team is looking at those.

 

These new builds focused on a couple things that our previous comparable test on 5/29 didn't.

• BattlEye performance compared to EAC
• Windowed Fullscreen mode
• Character baking hitches when new players enter the district

 

So I thought I would go back compare play test stats from 5/29, 6/10, and 6/12:

• BattlEye: I found that in general it runs faster than EAC did.
• Windowed Fullscreen mode: I'm not sure our stats capture input lag, and some testers mentioned some noticeable difference vs APB 1.20. That's something we'll have to keep working on.
• Character baking hitches: I can see a significant improvement in the 6/10 & 6/12 Rendering stats. 5/29 was much slower. Most players reported a reduction or elimination of the hitches, but since the engine is multithreaded and mostly not render bound, the change didn't affect the FPS a lot.

 

Here are some of the stats over our testing period going back to April.

 

For next week:

• We should have a new build that incorporates much more functionality that supports different refresh rates across different monitors and more.
• We should finish the latest migration of code and content from APB 1.20 to APB 2.1.
• We should get some solid time to work through more of the crashes to try and eliminate those.

 

Beyond that, there are some tester concerns related to:

• Input lag from Windowed Fullscreen
• Visual and audio bugs
• Doors getting out of sync with the server and showing open when they are closed or closed when they are open
• Issues with Field of View and mouse sensitivity in Marksman mode. The more zoom, the more problematic this becomes.
• Possible bugs in the UI that are slowing down FPS
• Issue with stutters that was cleared after rebooting
• Removal of 1024x768 resolution

 

This last one was called out very strongly by a couple testers. For APB 2.0 (console) they expanded UI screens to show more information and generally use the screen space better, so those screens no longer fit in 1024x768. We will have to do a bunch of extra work to put back in support for this low resolution.

 

My personal feeling is to launch the Beta as soon as the content migration and crashes are fixed. Then iterate on Windowed Fullscreen and other bugs.

But I've also been told that the first impression is everything, and to some extent I believe that. But I feel like that would apply more to the APB 2.1 full launch and less to the Beta.

 

Thoughts? I would be interested in seeing a thread on these topics.

 

Thanks,

Matt

[MattScott 15245]

Hi everyone,

 

I spent some time reviewing the thread that Flight made for last week's question.

It looks to me like the vast majority of players responding would just like us to release the Beta and then finish work with feedback from the community.

I'm taking that under heavy consideration.

 

This week involved a lot of activity:

• Improvements to monitor settings to help with input lag
• Migrating content from APB 1.20
• Adding more crash logging around GPU crashes

 

Our normal Friday play test was pushed to today, so I'm still reviewing logs. 

But here are the general notes:

• Lots of good feedback on the performance.
• The expanded monitor options seemed to go over well.
• Still some issues with input lag - enough that testers said the game felt far below the reported FPS.

We got 2 crashes from today, but neither triggered the GPU logging.

We've had 5 crashes in the last 2 weeks that are logged in our defect tracking software.

 

We are still looking at mouse sensitivity issues when in Marksman and the crashes.

 

Thanks,

Matt

[MattScott 15245]

Hi everyone,

I'm going to be posting early this week.

Hopefully everyone knows we're running the first Open Beta test tomorrow, Saturday, June 27th at 11am PDT to 2pm PDT.

I expect this test will be a mess. That's why we're only running it for 3 hours. That should give us enough data to work with, while also giving a lot of players the chance to log in.

The Known Issues are here: 

 

Please be patient with us, and know that some systems just wont be able to get in right now.
That includes systems that don't support AVX. We'll do a comparison build with AVX turned off to try and see if this optimization is worth it. You also wont be able to run 1024x768. This is something we inherited from Reloaded. They increased the minimum resolution for console and changed several UIs. It will be a pretty big task to undo those changes. And lastly, if you're running Win 7 or Win 8, you might experience more input lag than Win 10 players in Fullscreen Windowed mode. I highly recommend upgrading to Win 10 if possible. There are many optimizations that help gamers.

 

These are all areas we will continue to look at as development moves forward.

For this week, I thought I would share some of the Fullscreen Windowed work we did, and how we evaluated our progress.

Several testers and several developers ran benchmarks in Asylum to get apples to apples comparions.
Here is a sample of what those looked like.

1) This is what my machine looks like on Live with Fullscreen Exclusive mode. 

 

Couple key stats to point out:

• FPS average of 149.5 with some stuttering 0.23% of the time
• Input latency ranging from 18.2 to 24.9
• No dropped frames

 

Here was our first set of benchmarks for the APB 2.1 Beta right after we put in Fullscreen Windowed Mode.

 

By comparison, this is mostly a mess:

• FPS is the bright spot with 237.1 average but 1.26% stutters
• Input latency was between 34.9 and 39.2 which is nearly double FSE in Live
• And dropped frames.. lots of dropped frames (25.2%)

 

 

Finally here is is my last capture against the Beta build for Saturday:

• 241.7 FPS which is solid and 0.5% stutters, but not the big spikes like in Live
• Input latency is actually slightly better than FSE on Live at 16.1 to 20.3
• No dropped frames

The testers were great about helping us work through this.

 

NOTE: We still have some work to do here. If you run at a lower resolution than your desktop, you will introduce the kind of performance from the earlier Beta build.

If you want to run the game at a lower resolution, change your desktop resolution and then match it in game.

Crossing my fingers for tomorrow.
 

Thanks,
Matt

[MattScott 15245]

Hi everyone,

 

I missed the update yesterday with the holiday.

 

Last week was a maintenance week. That meant we backtracked over existing code and systems to diagnose the issues from the Beta.

Part of the team resurrected a series of old stress tester applications. We've almost got them working for the Login and World server.

That should let us try and narrow down the bad server problems.

We also fixed a number of the big crashes. There will be more this week, as we dig into one section of troublesome multithreaded code.

 

Hope everyone got a nice break over the weekend.

 

Thanks,
Matt

[MattScott 15245]

Hi everyone,

 

This week was a bit frustrating again. All of the issues currently blocking the Open Beta are based on bugs in code or systems that were originally created ages ago.

Worst of all, the areas we are working on are... boring.

 

That means no sexy graphs or cool pictures to highlight.

 

Instead I'm going to run down the list of issues we are working on and try my best to explain them:

 

• Login/World server issues handling a lot of players - We spent last week excavating some stress test tools from 2012. We got them working in the middle of this week, which enabled us to start testing why the vast majority of players weren't able to log in or stay connected to the world server during the Beta. We have already verified that the hardware and installations between Live and the Beta environment are exactly the same. Moving to new hardware or reinstalling made no difference to the connection issue. In APB 1.20 (Live), our Login and World servers handle lots of spikes when a bunch of players try connecting at the same time. It also appears that APB 2.0 (Live Consoles) handle load just fine too, although the concurrency on consoles is a lot lower than PC, so there is a possibility that a less severe version of this issue exists there. But there is definitely a problem with APB 2.1. Any time we try to get more than about 50 players trying to connect, we start hanging sessions. Our best guess is that we carried over some piece of code from 1.20 somewhere in the last 1.5 years that appeared to work during our small SPCT/closed beta tests, but fails at scale. The team is continuing to review code step by step to fix the issue.

 

• Problems debugging crashes submitted by Open Beta players - This appeared to work fine with our small testing group. In the past, we have noticed small inconsistencies from time to time, but we were still able to get enough info to fix issues. But crash reports submitted during the Open Beta showed lots of inaccurate call stacks and debug info. This has made tracking down bugs much more difficult than it should have been. I can't really get into details on why this system is so complex. It just is, and we're stuck with it. So we have started rewriting code from 2015 that took several developers 4 months to complete. Similar to the first issue, this task is "open ended", which means it's difficult to put a time estimate on since we have to work through each line of code to figure out what is failing. With all of that in mind, we have been making good progress, and it doesn't appear like it will take too much longer to fix. Hoping for a better update next week.

 

• Crashing fixes / Multi-threading fixes - This is also ongoing. We feel like we've fixed the majority of crashes from the Open Beta, but some pesky DirectX11 crashes remain. To address those we are going to kill two birds with one stone. The team is moving ahead with some multi-threaded work that we had put off in order to get the Open Beta out. We believe these remaining crashes are caused by a lower level issue in Unreal 3's task handler related to shared memory across threads. The base code for Unreal 3 is nearly 20 years old now and even the newer sections are around 8 years old. Our solution is to use a newer, more powerful library that has become the industry standard for this type of work. We started implementing that library about 3 months ago because it is very stable with more sophisticated ways to schedule tasks and ramp up processing on more powerful machines, while also being much more efficient on lower end machines. So in theory we can fix the remaining crashes while also improving performance. However we can't just "plug it in" for an easy win. The code works substantially different to Unreal's code. So each area where multiple threads are used has to be individually migrated to the new library. The team is moving quickly, and we've already converted some of the bigger systems like Shadows and Occlusion. As soon as we have the rest converted, we can start testing this again internally.

 

 

Thanks,

Matt

[MattScott 15245]

Hi everyone,

 

Running a day behind on this.

 

In the last update I laid out the three major areas and we continued to work on those areas this week.

 

Unfortunately our lead on the Login/World Server issue was on vacation. However the other team members continued that work and may have come up with the fix.

I'll need the lead to go back through the work and determine whether the analysis is correct.

 

Debugging work continued. The library we were working on now compiles and runs, but we're still working through errors that come up.

 

On the Multi-threading, we are still working through Occlusion. That's a bit of a rabbit hole unfortunately due to some memory that needs to be accessible across threads.

 

Thanks,
Matt

[MattScott 15245]

Hi everyone,

 

Lots of work this last week.

We'll be making an announcement soon.

Stay tuned.

 

Thanks,
Matt

 

EDIT: The 2nd Open Beta has been announced for August 1st.

https://www.gamersfirst.com/apb/news/2020/7/27/second-open-beta

[MattScott 15245]

Hi everyone,

 

I want to post a brief update this week after yesterday's test.

 

To echo what @Sakebee has already said, I want to thank everyone who came out for the Open Beta #2 yesterday.

• We had ~500 players on at the same time
• We had a total of just over 2,600 total players who logged in to test

Based on all of the hiccups in the first attempt, I view this as a our first real opportunity to get wide spread data and feedback from players. 

 

I want to acknowledge that while some players had okay performance, many players had frustrating experiences. 

Here is a quick recap of what we will be looking at from this test.

• All players saw stutters when new characters would load into the district. This is something we have been working on, and we're close to solving.
• All players saw their frame rate drop from more items in the UI. This has been an ongoing effort to optimize, but the UI is much worse with lots of players on.
• All players saw the status of doors desync with the client, so they appeared closed when actually open.
• Most players saw pretty bad server lag and some saw warping. This is a bit of a mystery right now. The servers are the same ones in the same locations we use in Live.
• Most players saw higher input lag. This is a critical item for us to look at more closely.
• Most players experienced at least 1 crash, which was expected. We have some work on this that didn't make the build.
• Some players saw issues with their imported characters that made them skinny or changed their face proportions. This is purely a bug and will be addressed.
• The font is still being swapped out. While you can still see the newer console font in some places, we have already swapped many other places back to the older APB PC font. 
• A number of art issues were reported, which is great. Please be aware we wont be working on those till we fix the big ticket performance items.
• Controller

On the bright side, I had numerous players report a smooth experience with only the stutters related to character loading even if their FPS was slower than Live. 

 

Overall, my biggest concern right now is that it looks like the build for Open Beta #2 ran about 15-20% slower than the Open Beta #1 build, and in some cases much slower. I want to make sure we didn't add some bad code at the last minute. It also looks like we had bad "hot spots" that caused large frame drops if you looked in that direction during game play. We're still collecting some more detailed information to narrow down the issue.

 

I got some negative feedback that 3 hours for the test was too short. I will take that into consideration for the next test. We don't want to exclude anyone, but we also didn't know if the Login server fix would work, so we needed a clear endpoint in case things weren't working well. 

 

Moving forward, the team will be finish off work that didn't make it into this build. I'm still going to be selective in when we turn on and off the Beta servers, because I don't want to build up testing fatigue like we have seen in the past with Weapon Balance Districts. For each test, we need as many players as possible on at the same time.

 

Lastly, I know not everyone is a fan of Fight Club, but we need to stick with Social and Asylum until those districts run better. Financial and Waterfront are *mostly* done from an art perspective. We're still adjusting lighting, but we know those districts perform worse due to their size. There is no value in putting those online yet.

 

Hopefully, we have finally put to rest the idea that the Engine Upgrade was a myth.

 

I purposefully didn't limit streaming or videos of the event, so those who didn't make it for the test could see the good, bad, and the ugly. I will continue with that policy. At the end of the day, this is a community effort. We need your continued help, so I look forward to seeing the new engine improve over time.

 

Thanks,
Matt

[MattScott 15245]

Hi everyone,

 

I completely missed an update last weekend. Sorry about that. My schedule is a bit out of control lately. I have been putting off travel all summer, but now I need to head over to Europe at the end of the month (if I am allowed), and it's been a scramble.

 

That also means I won't be available for this weekend either. So today will have to serve as both a late and an early update.

 

After Open Beta #2, we looked at a number of big ticket issues.

• We sorted out the network latency. So that should be better in the next test.
• We also looked at the crashes and found a fair amount of unfinished code relating to Settings. That whole section has been cleaned up to prevent crashes and random values from being used in some cases.
• We also took the opportunity to clean up rendering distances and connect them to gameplay instead of settings, so players should always be on equal footing regardless of graphic quality.

Due to things moving around, the 'High' graphical setting is now the equivalent to the 'Maximum' setting on Live.

 

There are a number of bugs that we are setting aside, so we can focusing on more multithreaded work that didn't make the last public build. That work is specific to getting action districts into the Beta. We have completed 3 different pieces of that project related to Occlusion, which is a way to speed up rendering and provide better performance on larger maps (think Asylum versus Waterfront).

 

NOTE: Unfortunately we lost at least a day of work after Microsoft released a bad Visual Studio 2017 update. We wrote some temporary workaround code, but our hope is that Microsoft patches again soon because the workarounds are slightly slower.

 

Initial tests looked very good, but disappointingly other machines ground to a halt with 100% CPU usage across all cores, locking up the machine for seconds at a time.

 

As of just this afternoon, we think we now understand what is happening.

 

It looks like each GPU has different speeds and different limits to how much memory you're allowed to push in some Occlusion calls per frame. If you go over this limit, the driver will basically stall until it's done processing the previous buffer changes to free up more memory.

 

We think that we're now pushing so many occlusion queries (each requiring specific buffers) to the card early in each frame, that the card runs out of buffer space. 

 

Yep. You guessed it. It's too fast. Seriously I couldn't make this up if I wanted to. Before the multi-threaded code was slow enough that we likely didn't hit this limit, although I know some players complained of bad stutters which would have been this issue.

 

We need to fix this issue and then we'll get a build up for SPCT to brea- I mean test. Then we'll do another public test.

 

Thanks,

Matt

[MattScott 15245]

Hi everyone,

 

I'm going to be moving my Engine Updates to Friday, now that I don't post CS stats.

That means I don't have to interrupt my weekend to get the the updates out.

 

We got back in on Monday and spent the first part of the week digging through nVidia driver documentation and then trying various code paths/options for getting a better sense of how much we can throw at the card before it stalls. It appears the problem is a bit more nuanced that we thought, and may take a bit longer to address. So we're splitting efforts. Part of the team will move onto the other optimizations that we had planned, while another part of the team continues to look at this issue.

 

As of this afternoon, I don't have a status on the stalls.

 

But yesterday, I got into Asylum and Financial to run my own tests, and much to my dismay I found that the game ran slightly slower than before. Classic APB, I thought.

 

After sharing logs and performance data with the team, I was asked what setting did I run the test at... and of course I had forgotten and left everything at Very High expecting the benchmark to line up with our previous Very High settings. I shared last week that we fixed a number of bad issues related to Settings, and that Very High is now not comparable to Maximum. So I went back and tried a test at High, and then at Medium. For the most part, Medium looks as good as Maximum on Live - with one exception related to how far away dynamic lights render. So in theory, the proper benchmark is between High and Medium with some custom values. With that new setup, I was relieved to find that performance had improved for both Asylum and Financial. 

 

Once this goes out, I will reiterate - don't jump to Very High and expect it to match Maximum on Live.

You guys will need to play with the settings and find what works best for your systems.

 

NOTE: I might add to this later tonight as the devs submit end of week work.

Thanks,
Matt

[MattScott 15245]

Hi everyone,

 

I'm going to sneak this update in under-the-wire for today.

I want to start by acknowledging that this part of the Engine work is frustrating. It's noodley and tedious, because we have to run the game over and over under certain conditions to find bottlenecks or crashes. 

I know sometimes these updates seem endless. I'm sure it appears that I talk about the same stuff or the same goals (like speed optimization) all the time.

 

However, this is necessary to get Action Districts up to speed.

 

I feel very good about this week's progress.

• We have started work to address the stalls. Over the last two updates, I talked about how much we are throwing at the Graphics card (GPU) and the problems that causes. This week we hand inspected logs for all the calls per frame, and found that a large percentage of them are executing the same command over and over. Each of these instructions is wasteful and costly if we don't need it. So we have started work on a State Caching system that saves values every time we pass them to the GPU across a wide number of different types of variables like Shaders, Samplers (textures), Parameters, and Buffers. Then when we go to run that command again, we check the value against the new value, and if its the same, we can skip that instruction. I am honestly shocked there wasn't a system like this in Unreal already (I am 99% that Unreal 4 has a system like this). Some early work was done on this system, but based on my math, we should see a significant speed improvement.
• We also finally figured out the source of the D3D crashes. This was super hard to reproduce, but we finally found that deep in the multi-threaded rendering code it was saving copies of how Meshes attach to Shaders. But other threads can actually overwrite the Shader, which then means drawing the Mesh is now invalid (wrong Shader). This sort of code is not "thread safe". We did a temporary fix for the issue, but it involves locking all the threads when a mesh is drawn. That's incredibly bad, because it slows things down. But we now have a benchmark build that doesn't crash. Starting on Monday, we will be re-working all those bad places where values are stored, so we can remove the lock and make drawing Meshes thread safe. Since we have a benchmark, we will see the speed improvements as we go.
• We added optimizations for 2 types of Shadow rendering. We are focusing on the Central Park area in Financial, which is one of the slowest areas in the game. There are a number of issues that make this area bad, and they are related to how we handle shadows from foliage. These two optimizations went in fairly smoothly, and they are the "pattern" for what we want to do throughout other parts of the code. Early tests indicated a 10% improved FPS. We have other areas that are very similar which we can now easily apply this new pattern to in order to gain more speed.

Thanks,
Matt

[MattScott 15245]

Hi everyone,

 

Sorry I missed Friday. I'm currently traveling in Europe, and my schedule is a bit off. 

 

This week I'm going to talk about progress first, and then I'm going to talk about the overall state of the player base and what the plan is for rebuilding the player base.

 

Progress with the Engine Upgrade:

• We had another solid week of working on complex code buried deep in the render system.
• A significant portion of rendering involves caching material shader parameters, and we were able to clean that code up so it is now thread safe. This greatly reduced the amount of locks.
• Still to do: There is still a lot of thread safety work left to do, but this was a big source of the errors.
• Still to do: We are investigating an issue where specific items are forced to update every frame when they shouldn't.
• Still to do: We want to allow secondary render threads to use the cache and then track down the specific case where we are setting the per material/mesh state and making sure those invalidate the cache properly so everything gets updated.

At that point, we'll be testing the new changes against our old benchmark to see if we can run the next phase of Open Beta with mission districts.

 

State of the player base:

 

I've seen a couple threads mentioning the decline in players, and I want to take a moment to comment on that.

 

I think there are macro issues related to games in general right now, and there are certainly micro issues related to APB specifically.

 

If you look at the industry as a whole, there are declining CCUs in most games right now. This is because in many places coronavirus restrictions have been lifted so players who are sick of being cooped up all summer are doing something different. On top of that, students are going back to school, which takes away free time for games. I'm not saying every game out there has dropped players, but in general we are seeing that trend right now.

 

For APB, we find ourselves in a weird transitional place. APB 1.20 (Live) has some systemic issues that can create a pretty bad play experience - mostly centering on matchmaking, new player experience, and various forms of griefing/dethreating. We have a lot of things in the works for 2.1 that will address these issues which have either been completed or are nearly done. But work on the engine has taken light years longer than I anticipated. While we have done 2 Open Betas now, I can understand the frustration in waiting for the rest of the districts and the actual launch. Hopefully everyone has seen our hard work on Asylum and Social, and they know we are committed to getting this finished as soon as possible.

 

I believe we have made good use of this delay by overhauling the game's entire monetization system. You'll be seeing more of this in September. And I think once APB 2.1 does launch, new players will find the game much more attractive because of our changes.

 

Rebuilding the player base:

 

With the current state of the game in mind, I have been faced with the pretty tough decision of whether to start spending lots of money now on advertising and promoting 1.20 or continuing to wait for the launch of 2.1. That money has been set aside, but timing is everything. I believe we will only get one shot to bring back players and show them the new APB experience. For that reason, I have intentionally held off promoting 1.20 because that would largely be a short term gain and a long term loss as new players enter 1.20 and then bounce for the same reasons we are upgrading everything.

 

Waiting does have its consequences.

 

Just know that advertising and promoting this game is not a question of IF.

It's a question of WHEN.

 

APB 2.1 has a lot of features under the hood that will either address or start to fix the remaining core issues in the game. It has much improved console experiences which at one point during their launch added significant amounts of players, who then left when the performance was so bad. With server cross play, we can bring the entire community together in a much better way. It also has integration with Anzu for in-game ads that will take more pressure off players to spend money. Anzu has also committed to helping us with a large Twitch campaign for awareness at launch.

 

Hang in there.

 

I want to end this week's update with a note to those of you who have continued to support us both in words and with purchases. 

Thank you. We wouldn't be able to do this without you.

 

Thanks,
Matt