Tracking the Engine Upgrade

[MattScott 15242]

Hi everyone,

 

The entire team has been crunching for September on the Engine Upgrade (including me), so I haven't had much time to blog lately.

However, I have seen a lot of requests for updates, so I thought I would try something a little different.

 

As I said on the latest Q&A, we're done merging all the content and code. We are just fixing bugs now.

 

So below I'm going to list the exact bugs that we need to fix before we do the public OTW test.

You'll notice we're not quite at the stage of fixing actual gameplay or collision issues yet.

I'll try updating this thread each Friday to clear or add bugs as we move forward.

 

All platform issues:

 

• Performance benchmarks - We've made a ton of changes recently, so we need all new benchmarks. From our earlier work, we feel the game is running well in terms of multithreaded rendering. The render threads are out performing the game thread. So all of our latest optimizations have been surrounding ways to improve core game logic.
  • STATUS: Ongoing

 

• Various disconnect crashes - This only happens in "final" builds that we release to players. There is a crash in the game whenever you get disconnected from a server for any reason. This causes the client to flag itself for a full file repair, which is very annoying for testers.
  • STATUS: In progress

 

• Level of Detail issue - This was extremely noticeable in the video that got released. Buildings popping in horribly. Trees disappearing. Etc. Turns out there was a tremendous amount of custom code in Live used to dynamically morph between various building LODs and to support individually lit windows at night. Reloaded tossed all this original code because none of it was supported under Unreal 3.5, and they just adopted the stock LOD system. Obviously this left many bugs. The team was able to pull in all the code we needed from the old engine and combine it with the new Unreal 3.5 LOD system.
  • STATUS: Mostly fixed / Window lights on LOD0 at night don't work in "final" builds for players

 

• Texture streaming bug - Even though this isn't a crash bug, it has been our #1 issue. Texture streaming is a unique system RTW wrote to prioritize textures based on various methods, so that when the player's machine or console runs out of texture memory, the game can downrez textures that aren't used very much (or aren't very noticable). This is a lot harder than it sounds, and there are many different methods used (texture size rendered to the screen, distance from the furthest object using the texture, special flags for signage and other highly detailed textures, etc). For the most part, this is working MUCH better now. The game looks amazing. There are a bunch of signs in the districts that need to get special texture streaming flags, and in some cases we see oddball textures get downrezzed that shouldn't.
  • STATUS: Mostly fixed

 

• NPC / Living city character lighting issues - This was very weird. We started noticing that sometimes many of the pedestrians were heavily under lit with a weird orange lighting that didn't match the scene. We've been chasing this for a bit, and found that the code for reflective lights was broken. The team was able to determine that reflective light values were being initialized based on whatever main direction light was active (or inactive) in the scene when they loaded into the district. This is being fixed so that we can control the reflective light values from our Time of Day data.
  • STATUS: In progress / Mostly fixed

 

• Character customization saving is broken - There's a database change necessary in our external SPCT testing environment. The next push should fix it.
  • STATUS: In progress

 

• Color Lookup Table grading is broken - This is a general post processing feature in Unreal where artists can apply color LUT files to modify the look of the scene. Right now, we're seeing some high thresholding / banding when LUTs are used. 
  • STATUS: Removed so the artifact doesn't show up / Awaiting dev resource

 

• Time of Day color values are broken - The lighting model for Unreal 3.5 is completely different than Unreal 3.0. As such, all of the values used to handle colors, depth of field, bloom, etc for midnight, dawn, noon, evening and night time need to be polished up.
  • STATUS: In progress

 

• Streetlamps in Financial - It looks like an artist at Reloaded swapped all the spotlights to point lights for any lighting that comes from streetlamps in the district. In order to create the light halo on the sidewalk, they had to make the radius quite large. This means when the streetlamp is next to a building there is a bright patch where the light is affecting the wall. The team went back through and swapped back in the spotlights, but we have a bug in the prefab that needs our Financial map rebuilt.
  • STATUS: Fixed but 2nd issue needs fixing

 

• Building feature LOD issue - We have a visual artifact because the rendering of mission features on buildings is not synced to the building LOD code. This means that for a small period of time we sometimes see holes in buildings where the features should be.
  • STATUS: Awaiting dev resource

 

 

We have submitted two console builds, and gotten kicked back for defects.

Right now we're 94% compliant with the necessary checks that both Microsoft and Sony require to approve a title.

 

Xbox One: 21 total must fix bugs (hightlights below)

• Title terminates on disconnecting the network cable anywhere in the title
• Null Characters are displayed on launching the title in any supported language other than English
• Background art and MAP fails to render on Xbox One X kit
• Title terminates on launching the tile in an Xbox One X kit 
• The title crashes on suspending the title during the splash screen
• Title crashes in keeping the console idle for more than 20 minutes on the Music player
• Game Event limitation popup is displayed during gameplay 
• SpaceBar is displayed for jump in Tutorial during gameplay
• Unnecessary Resolution option is displayed in Video Options
• Title crashes on adding friends from inside the game
• Title displays a black screen for more than 12 seconds right after the initial splash screen
• Title displays unnecessary button call out during gameplay
• Title allows the muted user to communicate through text during online gameplay
• Title allows the blocked user to send Group Invite
• User is unable to dismiss the interact menu by pressing B button after interacting with another user
• Title randomly crashes during normal gameplay
• The achievement 'Olympic Contender' fails to unlock on meeting its criteria
• User is unable to scroll thorough the description using RS in descriptions 
• Title does not provide an option to invite another user to the game

 

PlayStation 4: 12 total must fix bugs (highlights below)

• The term 'Sony Entertainment Network account' is displayed in the title screen
• Application provides 'Resolution' setting in the 'Video' tab
• Application soft locks upon selecting a user from the ‘Search’ tab of 'GROUP & FRIENDS'
• Gameplay - Incorrect naming convention is displayed in ‘Tutorial’
• Incomplete online ID is displayed in the ‘Friends’ tab in ‘GROUP & FRIENDS’
• Application fails to transition to a mode which supports play together
• User B fails to join the User A's session using 'Join' feature
• Application crashes on exploding a bunch of cars during gameplay
• Application incorrectly displays the SIE-specified text in the ToS
• Gameplay - Incorrect naming convention is displayed in 'Drive the vehicle 500 meters' tutorial
   

Thanks,
Matt

 

 

[MattScott 15242]

Hi everyone,

 

I am traveling right now, so I missed yesterday's update.

I'm going to summarize the week here, and then modify the main post.

• Performance updates - this has been a little frustrating. We've seen side by side comps where the new engine outperforms or matches Live. But right now, the build being tested by SPCT is performing terrible compared to Live. We spent part of the week rebuilding our stat collecting system to gather deeper data related to the game loop.
• Various disconnect crashes - These have all been fixed (for the moment).
• Texture streaming bug - We addressed a couple more issues, and I'm treating this as finished.
• NPC / Living city character lighting issue - This is fixed but I have a couple non-essential changes I want before go live.
• Character customization saving is broken - This is fixed and needs to be deployed with the next build.
• Streetlamps in Financial - This turned into a huge rabbit hole, but we managed to get it all fixed. We need to re-run lighting for Financial before the next build goes up. This process runs overnight in the office and uses all the available computers on the network, and even then it still takes more than 8 hours to complete.

 

We found a lot of new bugs that have been logged, but I don't have access to the list at the moment. I'll try to rework the list in the main post on my next update.

 

Thanks,
Matt

[MattScott 15242]

Hi everyone,

 

These are turning into Saturday updates, since that seems to be time when I can sit down and recap the work from the week.

 

Right now, our #1 priority is performance. Performance for the Engine Upgrade was pretty good prior to when we merged all the code and assets from Live. After that, our performance has really struggled.

 

For this post, I'm going to breakdown how performance is measured by our team, and what we're looking at.

 

First, we spent this week finishing off items from above, and re-writing/improving the statistics that we collect in-game. This took significant effort to revamp old code and measure specific parts of the rendering and game loops so we can see where most of the time is spent. We also synced this new code between APB 1.20 (Unreal 3) and APB 2.1 (Unreal 3.5), and we created a new export process to chart performance in Excel.

 

Second, we took this new stats engine and ran a series of test runs that isolate as many variables as possible. For instance, we tracked performance in Financial at midnight across APB 1.20 and APB 2.1. We use night time, because the game generally runs worse at night due to the extra lights that turn on. Each of us used different test machines, but mine was on the less powerful side with a 4 core i7-7700HQ @ 2.80GHz, an Nvidia GTX 1050, and 32GB ram. For the test we ran at the highest settings but used 1024x768 windowed mode to generate a lot of windows background interference. We also run no vsync and no smooth frame rate to get the highest rates possible.

 

We measure a game's performance based on the low, high, and average frame time in milliseconds. Here are some common frame time to frame rate benchmarks values.

• 10ms frame time = 100FPS
• 16ms frame time = 60FPS
• 25ms frame time = 40FPS
• 40ms frame time = 25FPS

 

Unreal 3 has always run the Render thread and Game Loop thread in parallel, but each process runs its tasks one at a time in sequence. The game's total frame time is the overlapping time from the Render thread and the Game Loop thread per frame. The Render Thread is mostly work done on your graphics card. The Game Loop thread is all the logic and keeping track of everything else including world state, physics, and player state. The Game Loop is dominated heavily by time spent ticking (processing) each relevant thing on the screen to make sure it's updated.

 

All of this work resulted in a series of benchmarks that all showed the same general result.

 

Live (Unreal 3.0 / APB 1.20) Financial at Midnight on my test machine. 

• Peak performance = 10ms (100FPS)
• Low performance = 19ms (50FPS) *There are some bad hitches in Live that we throw out
• Average frame time = 12ms (83FPS)
• Average Rendering frame time = 12ms
• Average Game Loop frame time = 12ms

 

Engine Upgrade (Unreal 3.5 / APB 2.1) Financial at Midnight on my test machine.

• Peak performance = 10ms (100FPS)
• Low performance = 19ms (50FPS)
• Average performance = 15ms (66FPS)
• Average Rendering frame time = 8.5ms
• Average Game Loop frame time = 15ms

 

Right away, we can see that the Peak and Low performance of the game are the same. But there is a 3ms frame time / 17FPS difference in the Average frame time. Where APB 1.20 is more balanced between Rendering and Game Loop, we see a lopsided workload in APB 2.1. The Rendering thread is quite fast, but the entire frame time is held back by how long the Game Loop takes to run.

 

For the rest of the week, we iterated on this process to improve our stat collection system and dig deeper into specific chunks of the Game Loop so that we could directly compare the performance of those smaller sections between Live and the Engine Upgrade. We were successful in identifying 2 Game Loop sections (out of 35) where APB 2.1 ran a total of 4.5ms slower than APB 1.20.

 

We are now digging further into those sections to create more detailed stats to help further breakdown performance until we can define a fix to the performance difference.

 

I'll go on record saying a couple things.

1. The performance in APB 2.1 is already significantly better than what is running on consoles right now (APB 2.0). We will be finishing console blockers as soon as possible and submitting so the console players can get a much better experience.
2. If I felt the performance was good enough right now, then I would be opening the Beta and putting this on public OTW. We don't have any other major blockers. The art issues and minor bugs can all be addressed after we let players in to test.

 

Thanks,

Matt

[MattScott 15242]

Hi everyone,

This week we made good progress on performance and character lighting.

These changes will help both the PC and console builds.

 

Performance

 

I have changed the test resolution to Fullscreen Windowed 1920x1080 instead of my previous smaller Windowed 1024x768 resolution to see how far I could stress the engine. That means you can't really compare the results from last week with this week. We also just got our test build out to the SPCT members, so we're still collecting data, but here are the initial results:

 

Live (Unreal 3.0 / APB 1.20) Financial at Midnight on my test machine @ 1920x1080 / Ultra settings. 
Peak performance = 10.33ms (97FPS)
Low performance = 19.22ms (52FPS) *There are some bad hitches in Live that we throw out
Average frame time = 16.20ms (61.70FPS)
Average Rendering frame time = 16ms
Average Game Loop frame time = 16.20ms

 

Engine Upgrade (Unreal 3.5 / APB 2.1) Financial at Midnight on my test machine @ 1920x1080 / Ultra settings.
Peak performance = 9.6ms (104FPS)
Low performance = 22.38ms (45FPS)
Average frame time = 16.05ms (62.31FPS)
Average Rendering frame time = 11.7ms
Average Game Loop frame time = 16ms

 

PERFORMANCE NOTE: There is a decent difference between Day and Night on APB 2.1.

I'm running these tests at night for "worse case" performance. During the day, we see APB 2.1 run 1-2ms faster.

 

There are a million things that could render this benchmark invalid, and I know on the surface this looks like APB 2.1 is now running slightly faster than APB 1.20, but we'll need more performance tests off SPCT machines to really assess where we are.

 

Lighting

 

Here are some before and after shots to illustrate some of the character shadow/lighting issues we have been fixing.

If you've been playing on console, then you are well aware of how dark and blobby the game can look some times.

 

LIGHTING NOTE: We're not done with scene lighting or character lighting, so even the updated "new" shots have issues. We aren't rendering ground character shadows properly, and indirect reflected lighting isn't showing up.

 

This is a cropped section of the Enforcer police scene.

 

In the first shot, the characters are all very dark and badly lit with strange hard shadows that don't make a lot of sense with the lighting in the scene.

In the second shot, you can see proper lighting on characters. This is still being tuned, but it's 100x better.

 

APB 2.1 old:

 

APB 2.1 new:

 

Here is a cropped section of the new District Select screen.

In the first shot, you can see the blobby hard shadows rendering in weird places.

In the bottom shot, you can see everyone rendering properly.

 

APB 2.1 old:

 

APB 2.1 new:

 

Thanks,
Matt

[MattScott 15242]

Hi everyone,

 

I'm working remotely, but this was another week spent diagnosing and attempting to fix rather complex issues in the Engine Upgrade. 

 

Unfortunately my initial stats from last week didn't hold up across our pool of testers. This is often the frustrating part of this project. We will see good things internally, but once we get the build to testers, the larger sample pool doesn't come back as well.

 

We did figure out that my work laptop is a bad source of data. It has an embedded Intel 3D card and a high performance nVidia 3D card for gaming. However, the nVidia card is daisy chained off the Intel card. SO when I run Fullscreen Windowed, a Desktop Windows Manager process runs at 50% using my Intel GPU while APB 2.1 takes up 80%+ on my nVidia GPU. For whatever reason, that equalizes the performance between APB 1.20 (Live) and APB 2.1 (Engine Upgrade) and over time they run comparably. 

 

The team finished the following tasks:

- Secondary night time lights are now working
- Secondary night time lights are now running nearly as fast as day time. Console players will be happy with that change once we re-submit.

- Added some fixes for texture caching, so that as you run around the district we aren't wasting so much CPU to figure out what textures should be loaded.

 

Still working on the following issues:

- Rendering shadows from secondary night time lights

- Rendering indirect lighting when characters aren't affected by the main light (sun or moon)

- Interactive building features still disappear when the player is far enough from a building that it shifts from LOD0 (the highest level of detail) to LOD1. This leaves an empty hole for a period of time where the shop front or window should be.

 

This week marked the first time we ran all the testers through a series of benchmarks in both APB 1.20 and APB 2.1 to try and get accurate performance comparison stats.

 

I know this may sound exciting, but it's not. Often testers have to patch, run the benchmark once to make sure the game finishes loading and caching all the geometry, and then they run the benchmark a second time for accuracy. If anything on their system accidentally interferes, then they have to start over and do it again. Some die hard testers did this for each district. Once they had accurate benchmarks, they sent us their logs, and then yesterday we processed all the logs through our internal tools to graph, identify CPU and GPU, and call out specific bottlenecks. The end of the day was spent posting the results to our internal channels for testers to see everyone's results.

 

Based on that round of testing, we did see an overall increase in performance by 1-2 milliseconds per frame between 10/21 and 10/26.

We're now back to being Render locked instead of Game Loop locked.

 

The team is looking at an automated way to prompt beta testers to upload their logs at the end of a session, so we can collate and process them automatically across a larger number of machines to get better insight into performance.

 

At some point, we will likely need to bite the bullet and throw this out to the community to see how it performs.

 

Thanks,
Matt

[MattScott 15242]

Hi everyone,

 

This week was a bit frustrating marked with stalls, blockers, and rabbit holes.

 

We ran another series of benchmarks and things are steadily looking better.

 

- Interactive building features / windows are almost done

This is an example of something that was on our schedule as a week long item.

Instead we have had to massively re-write the system to pull together better LOD (Level of Detail) techniques but also preserve some of the old RTW code.

The system now works properly, but we're missing the data that links interactive features to the proper LOD on the building they are attached to.

We spent days going back through old data, and finally found it.

Then we spent more days parsing it different ways to reapply it.

Then we spent more days fixing crashes that would appear after an hour of running the tool to apply the data to Financial and Waterfront.

As of yesterday, we were running the tool and it hadn't crashed. Hopefully this will be done early next week.

 

I recognize this issue is hard to describe, and even harder to visualize, so we took some screenshots to compare.

 

Here is what the game looks like now in APB 1.20.

There is a lot of cheating going on, because they can hide poor Levels of Detail behind the massive amounts of fog.

 

Next up is the what APB 2.1 looks like from roughly the same shot.

We don't have the same level of fog now, so our LODs need to look better.

For this shot, we are at the edge of rendering LOD1 (the 2nd most detailed version of buildings).

 

And this is the bug we're fixing.

For this shot, we are now rendering buildings at LOD0 (the best detail on buildings).

But the interactive features and some lower windows are too far away to render.

This leaves "holes" where the feature/windows should be. If I move closer the features and windows will reappear.

 

The solution we are implementing is to sync the distance that we turn off features to when we render LOD1.

 

Side note: The dev in charge of this task found a bug that relates to why traffic signals are broken. 

Apparently these have not worked properly for some time, and he's going to take a look on Monday to see what can be done to fix those.

 

- Shadows have been a pain

This effort has been to properly render all the shadows around a character based on interior lights, night time lights, and the sun/moon.

So far, we have rewritten elements of the core Unreal shader to get this working properly, and characters now render great.

However, we've had a steady stream of issues with shadows from props acting up and not rendering properly. 
Currently there is a very irritating flicker where some props have lights embedded in them that are now casting shadows using the prop's geometry.

The team has made excellent progress, and cleaning up this code has led to other optimizations and fixes.

But I'm hoping this will finally be done next week.

 

To get this working properly, we had to create a small test scene with a ton of lights at various locations.

 

- Indirect lighting is still not working

This got tabled till shadows are fixed.

Lastly, just like the traffic light bug, sometimes we end up finding other things that can be fixed.

This week it was Anti-aliasing. Our old technique, MSAA was removed from DirectX 11, and AA has never worked in APB 2.1.

A different dev ran across some code that looked to be causing the problem, and he was able to run some quick tests and get AA working again.

 

Here is a quick shot of what APB 2.1 looks like right now with AA.

 

Thanks,
Matt

[MattScott 15242]

Hi everyone,

 

I’m going to keep this week’s update a bit shorter.

 

Fixing lighting, shadows, and building features continues to be a bit of a rabbit hole. There is very good progress, and with respect to lighting and shadows, I am calling it ‘good enough’. There are some straggling items that we can revisit post launch.

 

Features on buildings (cProps) is the last area plaguing development. We discovered a horrible issue where props with embedded lights were recreating those lights over and over. We also have a lot of duplicate props in the district (again). This is an issue that popped up a month and a half ago that we solved. But it seems to be back.

 

EDIT: I got the actual stats today. There are currently 12,700+ duplicate objects in Financial alone. 
 

There's also a secondary issue that popped up related to props that use the physics engine. Apparently we were at the limit of how many objects can use physics in a single district. It is very possible that we are occasionally seeing crashes in Live due to this issue. So we also spent some time fixing this in APB 2.1.

 

We finished AA (anti-aliasing) and have started work on AO (ambient occlusion). It works for the most part except when the game animated the field of view which causes a flicker.

 

The hope is to get a build to SPCT by Tuesday/Wednesday that has these items all addressed.

 

EDIT: It’s taking a fair bit longer to clean up the dupes. So we wont have a new build tomorrow.

 

Thanks,

Matt

[MattScott 15242]

Hi everyone,

 

I missed yesterday’s update, and I’m currently out of town, but here is a quick rundown:

 

- Part of the team had to take a break from the Engine Upgrade to help finish off our new Christmas mini-game. They should wrap up those tasks this week and jump back in soon.

 

- The biggest focus continues to be the building features and removing duplicate items. I added an update to last week’s post once we identified more than 12K duped items. It took 3 days longer than expected to hammer out the commandlet that could properly identify dupes (lights, brushes, static meshes, etc). That process is a bit crazy. Each object type has unique code that compares it with each other object of the same type. On Friday we had cleaned up all the smaller maps and Financial. We’re working on Waterfront over the weekend. Hopefully we’ll have a new build to test soon.

 

- We made some progress with Ambient Occlusion, but that engineer got pulled into other tasks related to the network move happening this weekend.

 

- We shifted some focus to PS4 and XB1 bugs. We tend to pause console development while larger efforts are underway in the code, so that we don’t end up having to work through the same code twice. Currently neither build runs well due to the dupe issue, but we made some good progress anyway.

 

TL;DR - Christmas tasks and the network move ate up some bandwidth. But everyone should be back on APB 2.1 after Thanksgiving.

 

Thanks,

Matt

[MattScott 15242]

Hi everyone,

 

Only three working days this week, and lots of interruptions with the terrible network move.

 

I don't have a comprehensive update today, but I was told that we are finally "done" with all the major blockers.

However we need to make a build and get our test environments put back together, and then we'll see how all the fixes hold together.

I'll likely update later this week once I have more information.

 

Thanks,

Matt

[MattScott 15242]

Hi everyone,

 

Here is a quick rundown of the active blockers:

 

- Lighting

Now that we have de-duped all the lights and props, we need to re-run lighting so we can bake the static portions of each district that combine with the dynamic portions at run time.

 

Due to a bug we fixed last week, up until now all the lights were running dynamically, which carries a big performance penalty at night.

 

We started baking lighting for Financial on Wednesday. But Friday it wasn't done. This is unusually long compared to light builds we have run in the past. There are quite literally thousands of lights in Financial that all switched to baked instead of dynamic. So it's possible that the longer build time is normal. However, we also suspect that there was an issue on the build machine during the week due to a network issue. So ultimately we elected to kill the job, make sure all the workstations on the network were on and running the Swarm agent, and then re-run it starting Friday night. We'll see if it finishes by Monday.

 

- Physics

This is a great example of unplanned complexity in the Engine Upgrade project.

 

In order to support the latest Microsoft and Sony libraries for console, we did a massive series of upgrades to external libraries. One of those was the physics library. Now it looks like that developer changed a core part of how physics objects are initialized, because SPCT previously reported that interactive props and other physics objects would immediately explode and then respawn as players entered the district. Part of the team spent time this week troubleshooting (and ultimately fixing) several of these physics issues.

 

I'm guessing that this is the sort of frustrating loop that I'm sure Reloaded found themselves in. You upgrade one part of the code to support one platform, and then break other parts of the code. Then to fix it, you end up needing more upgrades elsewhere. And so on. 

 

- Memory issue

On Monday, we ran our test build and found a massive memory leak that was able to allocate 7GB of ram before even getting into a district. By Friday, we had fixed two possible areas, but we are still investigating.

 

- Console blocker

The memory leak blocked all console development, because those platforms can't allocate that much memory. Work there was paused this week.

 

- Other build blockers

Making builds is a big part of development. Devs produce local builds all the time, but those have all the optimizations turned off and don't really provide any sort of real performance. We have a number of internal systems that we use to identify specific versions of the code and generate end-user optimized builds.

 

This week we had a number of build issues starting on Tuesday that prevented us from getting a new APB 2.1 build done. Most of the issues were related to the network move and the locations of various resources. The team worked through most of these, but the process takes several hours to get a successful (or unsuccessful) build, so it is extremely tedious. At this point it looks like we have worked out all the kinks. But we still don't have our Test Worlds back online yet, so we can't share anything with SPCT. I'm told our Test Worlds will be back online this week.

 

For next week, hopefully lighting will be done for Financial so that we can get a new end-user build generated to test all the work we have done over the last 3 weeks. We will also need to run lighting for all the other districts during the week. This new build should help us identify remaining tasks and perform critical benchmarks.

 

Thanks,

Matt

[MattScott 15242]

Hi everyone,

 

I'm a little late getting this posted today, but here is the update.

 

It's easier to start with what we fixed:

 

- Baking Lighting

Not sure what happened the first time around, but Financial ran in 3 hours over last weekend once we restarted it. All major districts are done now.

- Memory issue

This is mostly fixed. We're operating within proper limits on all platforms now which unblocks console development.

There will likely be some ongoing work to reduce the memory footprint.

 

- Performance

Once proper lighting was baked, we did uncover an underlying problem with the way we re-wrote interactive features. Many of them include lights, and previously those lights were destroyed and rebuilt every time a player would go in and out of range of the feature. That had the upside of making sure the game was only processing and rendering lights within a specific distance, but the horrible downside of invalidating all the baked lighting forcing those lights to run as dynamic lights. Consoles got away with this, because they are locked at 1pm, so that version never runs at night.

 

We had already fixed the code so that baked lighting is used all the time, but then we realized that the game was processing and rendering every light attached to a feature... every frame... regardless of how far away it was. This new issue got fixed by Friday, and the subsequent performance tests were very very promising. 

 

What is left:

 

- Still missing our APB 2.1 Test World environment

We can't deploy anything for SPCT to test yet.

 

- Pedestrians popping through raised floors

We didn't have quite enough time last week to look at this.

 

- Console specific updates

I have previous listed a number of console SDK issues that need to be fixed so we can submit to Sony and Microsoft again.

 

- Lots of minor issues through out the levels

These are gameplay bugs reported by SPCT and our internal QA such as bad collisions on specific rooftops.

As I have previous stated, I wont hold up the release to OTW for these kinds of issues. 

 

- New localization

Lots and lots of text in the game is new or changed, and it needs to be translated to French, Italian, German, Spanish and Russian.

Again, I wont hold up the release to OTW waiting on localization. I may not even hold up the full public release.

We'll fill this in as soon as we can, but I want the game to settle down first, so we can make one big pass.
 

Thanks,

Matt

[MattScott 15242]

Hi all,

 

I missed the update yesterday, and I have been out all week with a nasty cold, so I don’t have a lot of new info.

 

We did release a build to SPCT last week (our first in a while due to the network migration). That build had some problems and ran horribly bad. Looks like it was missing a huge chunk of the work that was completed. The team is organizing a new build for testing, and I’ll post results as soon as I have them.

 

Hopefully all the APB players are gearing up for a nice holiday break.

 

Thanks,

Matt

[MattScott 15242]

Hi all,

 

I was hoping for more of an update this week, but the holiday got in the way.

 

As of Friday, we believe we have fixed the massive lighting issue that was plaguing us during December. I’m waiting for a build to evaluate performance.

 

For console players, we noticed that both Sony and Microsoft have required upgrades to their latest SDKs before we can resubmit again. The work on the Sony SDK is complete, and we are 50% done with Microsoft.

 

Looking forward to seeing progress next week now that vacations are done.

 

Thanks,

Matt

[MattScott 15242]

Hi all,

 

We had a very busy week on APB 2.1.

We fixed the NPCs popping through raised floors.

We fixed the remaining missing props.

And we made significant progress on the massive lighting issue. Before we were seeing steady framerates when the player wasn't moving, but then you would drop 30-50FPS once you moved.

Now we're seeing steady framerates.

 

Unfortunately its clear we're game thread capped (as opposed to the render thread). This means the CPU is doing more work than the GPU, and CPU computations are preventing the FPS from running higher.

Overall performance is well below Live at the moment. We need to spend a little more time this week looking at where the game thread is spending all its time.

Since the game is now playable, we are starting to also focus on smaller bugs such as some gates not being destructible, issues shooting through windows, minimal settings not improving the FPS much, some character lighting is too dark, and marksmanship sensitivity issues.

 

I haven't shown any screenshots in a while, and one of our SPCT testers was kind enough to gather up some, so here is what the game looks like right now.

All screenshots using Ultra graphics settings at 1920x1080.

GPU: GeForce GTX 1080

CPU: Intel Core i9-9900KS

 

 

 

 

 

 

Edited 12/01/2020 - 06:51 am PT

Note from SPCT @rooq:

Hey guys! I made a huge mistake when capturing the original screenshots. Turns out that I had anisotropic filtering completely disabled, giving all textures a low resolution appearance. I tried to recreate the same screenshots with proper settings, sorry for the mishap!

 

Thanks,
Matt

[MattScott 15242]

Hi all,

 

This is just a small early update for PS4 players.

 

I made a tweet yesterday that I had seen a very good looking PS4 build, and a couple players private messaged me asking for video.

So today I tweeted that video. You can see it here on my personal twitter. The main @apb_reloaded on Twitter retweets anything I post about the game.

 

 

Thanks,
Matt

 

[MattScott 15242]

Hi all,

 

I missed Saturday's post. Just got busy with my weekend and was out of town.

I did put up an early update for PS4 players on my Twitter feed and then linked it here on Thursday.

 

We're working on a short list of major tasks, and then sorting through the bug list.

 

The big items are:

• Performance continues to be sub par.

This is very frustrating because some scenes are identical to what we had mid last year such as the 3D login scene. Last year that scene was running way above Live's framerate. Now it's running terrible, and nothing in the scene has changed. So during the many many code and asset merges between Live and 3.5, I think we introduced a performance bug that is slowing everything down. The team is profiling this scene now to figure it out.

• Raycasting issues shooting through windows and other see thru objects. This was recently reported by SPCT.
• Distance culling is an issue for props. Should be an easy fix. Culling values are just missing.
• Code optimization. We found something that pops up in the code quite a bit which is an expensive way of doing things. This is an alternate very fast method, so we are switching those calls.
• XB1 performance is atrocious (sub 15FPS). We are using the Login scene as a test. This scene is identical to what XB1 is running in live. The XB1 render system is identical to what is running on XB1 in live. Yet our current version performs worse. Solving this will likely shed light on the performance issues for PC as well. Interestingly, PS4 uses an entirely different render system, and it seems to be performing the same as PS4 Live.

 

NOTE: We solved the XB1 issue and it currently runs at 28FPS which only 2FPS slower than PS4. We're having some trouble getting districts to connect to that environment, but I'll try to post a video of us running around as soon as I can.

 

QA has made 1 full regression pass on APB 2.1, and they are in the middle of a second one.

All bugs have been prioritized and passed to the dev team, and we have a couple guys working down that list, while the rest of the team focuses on the big stuff.

Currently we have 94 open bugs. The estimate is that 20 or so can be closed of this second round of internal testing.

 

Thanks,
Matt 

[MattScott 15242]

Hi all,

 

This week we made significant (exciting) progress, but it's going to be a bit complicated to discuss, so bear with me.

 

In July, we had a build that was missing correct lighting, had a bunch of visual issues, and most importantly was missing all the content and latest code changes from Live. However, the frame rate from this build when tested by SPCT was within 1ms frame time (5-10FPS) of Live. We felt based on testing that this was an acceptable place to start testing on OTW while we finished optimizing the performance.

 

In September, we got our first look at a candidate build with all the new code and content migrated over, all action districts cleaned up from duplicate objects in the scene, and correct lighting implemented. Unfortunately the frame time had plummeted. We were 5ms (~40FPS) slower than Live. I'm sure you can imagine the disappointment experienced by the team and our SPCT testers.

 

Since then, the team has been hammering away at optimizations and critical visual issues such as building LODs and features. It has been a game of inches. Sometimes lowering 0.2 ms at a time. Several systems in APB 1.20 had to be completely abandoned and rebuilt. A week ago, based on new systems and many optimizations that we have added, we were 3ms (~20FPS) slower than Live.

 

As of last night, we finally identified a massive 2ms bottleneck that got introduced back in August. The issue happens as part of our build process, which skews our ability to debug and log it making it virtually impossible to track down. To be clear, we know what the issue is, but the engineers haven't yet proposed a solution. This will be the #1 area we focus on next week to try and finally get the game close to Live's FPS.

 

During the last 2 weeks, our internal QA did a full regression on our current APB 2.1 build. We have 59 active bugs in the database including console submission bugs. All platforms (PC, XB1, and PS4) are using the same exact code. Performance on both consoles is better than the console builds we have in Live, however our APB 2.1 build for XB1 runs out of memory after running around action districts for a while. The issue was introduced during the rewrite of 2 other major systems, so we need to tread very carefully. I think this will be the biggest remaining issue after we solve the primary performance bottleneck.

 

Moving forward, I'll countdown the bugs in the database. Just know that as QA finds more bugs and the number will shift around a little.

 

Thanks,

Matt

[MattScott 15242]

Hi all,

 

We shipped a couple APB 2.1 builds to SPCT this week. All of them were focused on performance.

The engineers were able to fix the issue that was identified last week, and the game is running a lot better.

We also fixed several of the top game play bugs including an issue with destructible props and shoot windows.

 

The biggest blockers are the memory issue on Xbox and a frame rate drop when looking around.

The team is currently memory mapping the game profiling builds to identify the fixes.

 

Thanks,
Matt

[MattScott 15242]

Hi all,

 

Not much to report this week.

 

Work was finished a number of bugs, but the focus has been on performance.

Fortunately it looks like the team may have found the largest performance bottleneck on Friday.

Looking forward to sharing the results.

 

Thanks,
Matt

[MattScott 15242]

HI all,

 

I missed yesterday's update, but the TL;DR is that this week was disappointing.

 

In an effort to explain, I'm going to talk a little bit about the innerworkings of the APB engine.

 

I'll start by sharing performance from scenes that have no actual gameplay in them. These are small contained scenes with not a lot of objects in them.

For the Login scene on my test machine, APB 1.20 (Live) runs at an average of 194FPS and APB 2.1 (Upgrade) runs at an average of 221FPS.

We tested a number of other non-gameplay scenes, and clearly our new multithreaded renderer is faster - even with the more complex lighting and more objects overall.

 

However, that is not true in Mission districts. These are much more expansive scenes where it's not feasible to render everything. As an example, in Financial there are 30,000+ objects. There are also a lot of special cases where task objects can be enabled/disabled, buildings can show various levels of detail, NPCs animate and move, and lighting is combination of Time of Day + lightmaps. All of that requires special case code.

 

As mentioned previously, the Engine itself has two main threads - one for game play and one for rendering.  The rendering thread performs all the CPU functions necessary to throw geometry to the graphics card (GPU), which then splits those draw calls into more processes. The trick to getting the game to run fast is to balance the tasks in each of these three areas so that from the beginning of a frame to the end, they take the same time to run.

 

To help optimize and balance performance, we have a number of tools that do performance profiling and tell us general areas, shaders, or functions that are taking a lot of time. This part can get very tricky because we have various versions of builds we can make such as Debug, Release, and Final builds. Debug is the slowest, but has the most diagnostics information we can see. Final is the build we ship, but it provides the least diagnostic information. Also running the game at a slower speed can wreak havoc with the balance of threads, which massively skews profiling results.

 

With all of that in mind, a couple weeks ago we zeroed in on a performance bottleneck in the render thread where we do Culling (and its associated functions). This area of the code got reworked when we implemented multithreaded rendering, and clearly it is now under performing.

 

Culling is the process of eliminating as many items as possible from being sent to the graphics card for rendering. Even though GPUs are very fast, it still takes time to copy data between the CPU and GPU, and we take a hit every time we switch materials/shaders between rendering each object. Since APB is so old, our version of Unreal has no "batching" or "instancing", which is what allows modern engines to push so many objects. In Financial, each frame we end up culling those 30,000+ objects down to 800-1000 objects that get processed for rendering to the GPU.

 

So Culling is very important, and we use a bunch of different techniques to cull including render distance, object state, and clipping against the view frustum. All of that starts with accessing the scene objects in an Octree, a 3 dimensional grid that sub divides each cell until only a few objects are present. Each frame we start by looping down the Octree to eliminate objects that are behind or to the side of the player and out of view. The Octree is both our best way to cull, but also it's the hardest thing to multithread.

 

This last week, we took a stab at multithreading that. This was a massive effort, and it ultimately didn't work. We got the code working, but on Thursday night when we started testing, we noticed that some objects in the scene didn't render. By Friday night, we figured out that the original devs on APB store scene data for objects in two places, the Octree and a main list of objects in the scene. For various reasons, those objects don't match.
 

So we need to rework the approach and try again this week.

 

Thanks,

Matt

[MattScott 15242]

Hi all,

 

We've made some good strides with Culling. There is no silver bullet, but it runs better than it did.

 

There's a big slowdown related to audio that we're hunting down.

After one of the content merges we left 6000+ audio emitters in Financial.

They are all legitimate places where audio can play, but we need to do a better job managing them so they don't cause other parts of the code that search the scene to hit those objects too.

 

During the week, we fixed several lighting bugs.

 

On Friday we found a series of bugs that need to be addressed related to memory crashes.

We also found a bug in the new Building Feature code that was preventing that system from working correctly.

 

I'm going to try and post some benchmarks next week to illustrate where we are related to past benchmarks.
 

Thanks,
Matt

[MattScott 15242]

Hi all,

 

I wish I had more visuals to show for this week. Those are always more fun to post than the bland technical write-ups.

 

We fixed several memory crashes, but the Xbox One still has an issue where we run out of memory entirely. There are three big blocks that we're looking into (shaders, textures, and audio). We'll need to investigate ways to cut those down. The crash doesn't show up right away. You have to run around a bit in Financial. We believe it's only barely over the limit based on new content that was added since the last update for that platform.

 

Audio work is ongoing. That code is a mess. We've addressed several areas, but there are a bunch more where audio arbitrarily loops through every emitter in the scene -- every frame. The team was in the middle of designing a caching mechanism so that the audio system doesn't have to do that unless anything moves, and even then most of the cache would still be valid.

 

We got secondary shadows and the sky lighting working correctly.. but on Friday QA found areas in Financial where the odd self shadowing issues reappear that we fixed months ago. This is the kind of thing that drives me crazy with the APB code base. 2 steps forward and 1 step back. We'll be looking into that this week.

 

I opted out of benchmarks this week, because there was too much work in progress.

 

Thanks,
Matt 

 

[MattScott 15242]

Hi everyone,

 

The last two updates have been pretty sparse, so this week I'll take a little more time to explain where we are at.

 

First, a portion of the team has now moved back to bugs, which is good. As much as we want the performance to get better, we also need to fix actual gameplay issues too.

Right now we are tracking 20 bugs identified by SPCT or QA. 2 of which were determined to be duplicates. 4 of which were checked in as fixed by Friday, which leaves 14 unaddressed bugs.

 

Second, there are two overall issues we are still working on - Lighting and Performance.

For lighting, we've narrowed down the lighting issue to a system in Unreal 3.5 that handles "Indirect" lights. This is a process that gets pre-processed and packaged when we ship the game. It looks like because Reloaded never turned on Day/Night, they also never tuned indirect lights that are largely only visible at night. Part of the team has now started diving into these tools to work through all our maps to fix this.

 

For performance, I've started to realize there is no silver bullets - something that we'll find and fix that will make the game suddenly run great. However, the good news is that we've tested Asylum across a number of systems, and APB 2.1 runs that district better or at the same frame rate as APB 1.20 (Live). 

 

On one of my higher end systems with an i7 with an RTX 2080, here is the comparison with both running 1920x1080:

 

APB 1.20 Asylum on Maximum (ignoring micro stutters)

1% Min = 6.5ms 153FPS

Average = 4.85ms 206FPS

1% Max = 3.15ms 316FPS

 

APB 2.1 Asylum on Very High (which is equivalent to Maximum)

1% Min = 4.68ms 213FPS

Average = 3.71 269FPS

1% Max = 3.02ms 330FPS

 

I ran a test on a lower end laptop with an i7 and a GTX 1050. Amazingly APB ran at an average of 100FPS on APB 2.1 which was about 10FPS faster that APB 1.20.

 

We have spent months getting the engine to take advantage of more cores.

Here is what APB 1.20 Asylum looks like on that same i7 with 12 logical processors.

 

And here is what APB 2.1 Asylum looks like on the same i7.

 

Other districts are more hit or miss right now, but using Asylum as a benchmark, we started looking at differences between the other districts to help us narrow down the remaining performance issues. We started to recognize very specific areas where frame rates went down significantly, and that didn't happen in Asylum. Unfortunately there are so many areas in this engine that simply weren't finished sometimes I feel like we're looking for a needle in a haystack.

 

Or as it turns out this week: a blade of grass in a park

 

For those playing at home, I'll give you a little sample of what we deal with on a weekly basis.

Here are two screenshots. If you look specifically at the foliage, can you spot a potential issue?

 

APB 1.20:

 

APB 2.1:

 

The foliage in the top shot (APB 1.20) is set to only render very close in. You can barely see it from where the screenshot is taken, and that is on Maximum settings.

The foliage in the bottom shot (APB 2.1) can be seen all the way across the district. In fact it has no draw distance limit set. That means 2.1 is currently rendering thousands of tiny polygons that each cast shadows on each other while the scene elements cast shadows onto them.

 

The team started working on this Friday, and will be cleaning it up this week.

 

Thanks,
Matt

 

EDITED: Added frame rate stats and cleaned up grammar.

[MattScott 15242]

Hi everyone,

 

First, for those who haven't seen my announcement from last night, Little Orbit is now requiring all our staff to work remotely to insure their safety and prevent the spread of coronavirus.

 

Next, we spent a portion of the week organizing for this move, but we were able to solve the bad character shadows and lighting issues.

This has been a major blocker, and it allowed us to rebuild the lighting for each district.

The results are already 100% better - even without any actual lighting changes from the art team.

 

However, on Friday my team was finally able to get a 100% reproducible scenario to fix a memory crash that has been plaguing us. The old code was 32-bit, and the new code is supposed to be 64-bit across the board, but there are many tiny exceptions that need fixing to provide overall stability. Unfortunately that means the game isn't in a runnable state right now.

 

Once we get this issue fixed and a new Beta build is released internally, I'll post some shots at various times of day and in some of the other districts so you can see the difference.

 

Thanks,

Matt

[MattScott 15242]

Hi everyone,

 

We had a great week with the Engine Upgrade with one notable problem that I'm going to talk about first.

 

Due to juggling things around, we inadvertently exposed the new Beta client, and one adventurous player got a copy and was able to get in. I'm not going to block the screenshots on the forum, and he later contacted me to reveal how he did it, so we have fixed the issue.

Many players sometimes assume we aren't working on this or that the Engine Upgrade will come out in 2025. And while I want to discourage other players from digging through our systems to find stuff like this, I have chosen to leave this post up simply to show that what we're doing is real.

 

Now that the Beta client has leaked, there are a couple things that I should cover:

• We are only working on lighting and 8 remaining bugs at this point unless anything new comes up (which has tended to happen on this game a lot)
• The Beta is going to be limited to only Social and Asylum districts. APB is a massive game, and rather than continuing to grind through every possible problem in the hopes of some perfect launch, I have chosen to focus the team on a couple districts so we could polish those and let players in to test sooner.
• I'm very happy with the system performance in Asylum. During this week's tests, I heard a lot of positive feedback on how that district felt to play.

The plan had been to release the Beta client 1 week ahead of when servers will be opened to give the players a chance to download it. Then we were going to do an initial test for a single weekend, where we open up the servers for a limited time. There has been very little testing done with a lot of players on the same district, so we want to see how the APB 2.1 servers handle everything.

 

However, with our recent influx of players, we don't have the server capacity to host a lot of beta testers. I'm already working on a solution. As soon as we can host more than 500 players on this environment, then I'll get the gears turning.

 

Thanks,

Matt