Jump to content

Tracking the Engine Upgrade

Recommended Posts

Hi everyone,


The entire team has been crunching for September on the Engine Upgrade (including me), so I haven't had much time to blog lately.

However, I have seen a lot of requests for updates, so I thought I would try something a little different.


As I said on the latest Q&A, we're done merging all the content and code. We are just fixing bugs now.


So below I'm going to list the exact bugs that we need to fix before we do the public OTW test.

You'll notice we're not quite at the stage of fixing actual gameplay or collision issues yet.

I'll try updating this thread each Friday to clear or add bugs as we move forward.


All platform issues:


  • Performance benchmarks - We've made a ton of changes recently, so we need all new benchmarks. From our earlier work, we feel the game is running well in terms of multithreaded rendering. The render threads are out performing the game thread. So all of our latest optimizations have been surrounding ways to improve core game logic.
    • STATUS: Ongoing


  • Various disconnect crashes - This only happens in "final" builds that we release to players. There is a crash in the game whenever you get disconnected from a server for any reason. This causes the client to flag itself for a full file repair, which is very annoying for testers.
    • STATUS: In progress


  • Level of Detail issue - This was extremely noticeable in the video that got released. Buildings popping in horribly. Trees disappearing. Etc. Turns out there was a tremendous amount of custom code in Live used to dynamically morph between various building LODs and to support individually lit windows at night. Reloaded tossed all this original code because none of it was supported under Unreal 3.5, and they just adopted the stock LOD system. Obviously this left many bugs. The team was able to pull in all the code we needed from the old engine and combine it with the new Unreal 3.5 LOD system.
    • STATUS: Mostly fixed / Window lights on LOD0 at night don't work in "final" builds for players


  • Texture streaming bug - Even though this isn't a crash bug, it has been our #1 issue. Texture streaming is a unique system RTW wrote to prioritize textures based on various methods, so that when the player's machine or console runs out of texture memory, the game can downrez textures that aren't used very much (or aren't very noticable). This is a lot harder than it sounds, and there are many different methods used (texture size rendered to the screen, distance from the furthest object using the texture, special flags for signage and other highly detailed textures, etc). For the most part, this is working MUCH better now. The game looks amazing. There are a bunch of signs in the districts that need to get special texture streaming flags, and in some cases we see oddball textures get downrezzed that shouldn't.
    • STATUS: Mostly fixed


  • NPC / Living city character lighting issues - This was very weird. We started noticing that sometimes many of the pedestrians were heavily under lit with a weird orange lighting that didn't match the scene. We've been chasing this for a bit, and found that the code for reflective lights was broken. The team was able to determine that reflective light values were being initialized based on whatever main direction light was active (or inactive) in the scene when they loaded into the district. This is being fixed so that we can control the reflective light values from our Time of Day data.
    • STATUS: In progress / Mostly fixed


  • Character customization saving is broken - There's a database change necessary in our external SPCT testing environment. The next push should fix it.
    • STATUS: In progress


  • Color Lookup Table grading is broken - This is a general post processing feature in Unreal where artists can apply color LUT files to modify the look of the scene. Right now, we're seeing some high thresholding / banding when LUTs are used. 
    • STATUS: Removed so the artifact doesn't show up / Awaiting dev resource


  • Time of Day color values are broken - The lighting model for Unreal 3.5 is completely different than Unreal 3.0. As such, all of the values used to handle colors, depth of field, bloom, etc for midnight, dawn, noon, evening and night time need to be polished up.
    • STATUS: In progress


  • Streetlamps in Financial - It looks like an artist at Reloaded swapped all the spotlights to point lights for any lighting that comes from streetlamps in the district. In order to create the light halo on the sidewalk, they had to make the radius quite large. This means when the streetlamp is next to a building there is a bright patch where the light is affecting the wall. The team went back through and swapped back in the spotlights, but we have a bug in the prefab that needs our Financial map rebuilt.
    • STATUS: Fixed but 2nd issue needs fixing


  • Building feature LOD issue - We have a visual artifact because the rendering of mission features on buildings is not synced to the building LOD code. This means that for a small period of time we sometimes see holes in buildings where the features should be.
    • STATUS: Awaiting dev resource



We have submitted two console builds, and gotten kicked back for defects.

Right now we're 94% compliant with the necessary checks that both Microsoft and Sony require to approve a title.


Xbox One: 21 total must fix bugs (hightlights below)

  • Title terminates on disconnecting the network cable anywhere in the title
  • Null Characters are displayed on launching the title in any supported language other than English
  • Background art and MAP fails to render on Xbox One X kit
  • Title terminates on launching the tile in an Xbox One X kit 
  • The title crashes on suspending the title during the splash screen
  • Title crashes in keeping the console idle for more than 20 minutes on the Music player
  • Game Event limitation popup is displayed during gameplay 
  • SpaceBar is displayed for jump in Tutorial during gameplay
  • Unnecessary Resolution option is displayed in Video Options
  • Title crashes on adding friends from inside the game
  • Title displays a black screen for more than 12 seconds right after the initial splash screen
  • Title displays unnecessary button call out during gameplay
  • Title allows the muted user to communicate through text during online gameplay
  • Title allows the blocked user to send Group Invite
  • User is unable to dismiss the interact menu by pressing B button after interacting with another user
  • Title randomly crashes during normal gameplay
  • The achievement 'Olympic Contender' fails to unlock on meeting its criteria
  • User is unable to scroll thorough the description using RS in descriptions 
  • Title does not provide an option to invite another user to the game


PlayStation 4: 12 total must fix bugs (highlights below)

  • The term 'Sony Entertainment Network account' is displayed in the title screen
  • Application provides 'Resolution' setting in the 'Video' tab
  • Application soft locks upon selecting a user from the ‘Search’ tab of 'GROUP & FRIENDS'
  • Gameplay - Incorrect naming convention is displayed in ‘Tutorial’
  • Incomplete online ID is displayed in the ‘Friends’ tab in ‘GROUP & FRIENDS’
  • Application fails to transition to a mode which supports play together
  • User B fails to join the User A's session using 'Join' feature
  • Application crashes on exploding a bunch of cars during gameplay
  • Application incorrectly displays the SIE-specified text in the ToS
  • Gameplay - Incorrect naming convention is displayed in 'Drive the vehicle 500 meters' tutorial




  • Like 45
  • Thanks 39
  • Haha 2
  • Sad 3
  • Dislike 2

Share this post

Link to post
Share on other sites

Hi everyone,


I am traveling right now, so I missed yesterday's update.

I'm going to summarize the week here, and then modify the main post.

  • Performance updates - this has been a little frustrating. We've seen side by side comps where the new engine outperforms or matches Live. But right now, the build being tested by SPCT is performing terrible compared to Live. We spent part of the week rebuilding our stat collecting system to gather deeper data related to the game loop.
  • Various disconnect crashes - These have all been fixed (for the moment).
  • Texture streaming bug - We addressed a couple more issues, and I'm treating this as finished.
  • NPC / Living city character lighting issue - This is fixed but I have a couple non-essential changes I want before go live.
  • Character customization saving is broken - This is fixed and needs to be deployed with the next build.
  • Streetlamps in Financial - This turned into a huge rabbit hole, but we managed to get it all fixed. We need to re-run lighting for Financial before the next build goes up. This process runs overnight in the office and uses all the available computers on the network, and even then it still takes more than 8 hours to complete.


We found a lot of new bugs that have been logged, but I don't have access to the list at the moment. I'll try to rework the list in the main post on my next update.



  • Like 43
  • Thanks 10
  • Haha 1
  • Sad 2
  • Dislike 1

Share this post

Link to post
Share on other sites

Hi everyone,


These are turning into Saturday updates, since that seems to be time when I can sit down and recap the work from the week.


Right now, our #1 priority is performance. Performance for the Engine Upgrade was pretty good prior to when we merged all the code and assets from Live. After that, our performance has really struggled.


For this post, I'm going to breakdown how performance is measured by our team, and what we're looking at.


First, we spent this week finishing off items from above, and re-writing/improving the statistics that we collect in-game. This took significant effort to revamp old code and measure specific parts of the rendering and game loops so we can see where most of the time is spent. We also synced this new code between APB 1.20 (Unreal 3) and APB 2.1 (Unreal 3.5), and we created a new export process to chart performance in Excel.


Second, we took this new stats engine and ran a series of test runs that isolate as many variables as possible. For instance, we tracked performance in Financial at midnight across APB 1.20 and APB 2.1. We use night time, because the game generally runs worse at night due to the extra lights that turn on. Each of us used different test machines, but mine was on the less powerful side with a 4 core i7-7700HQ @ 2.80GHz, an Nvidia GTX 1050, and 32GB ram. For the test we ran at the highest settings but used 1024x768 windowed mode to generate a lot of windows background interference. We also run no vsync and no smooth frame rate to get the highest rates possible.


We measure a game's performance based on the low, high, and average frame time in milliseconds. Here are some common frame time to frame rate benchmarks values.

  • 10ms frame time = 100FPS
  • 16ms frame time = 60FPS
  • 25ms frame time = 40FPS
  • 40ms frame time = 25FPS


Unreal 3 has always run the Render thread and Game Loop thread in parallel, but each process runs its tasks one at a time in sequence. The game's total frame time is the overlapping time from the Render thread and the Game Loop thread per frame. The Render Thread is mostly work done on your graphics card. The Game Loop thread is all the logic and keeping track of everything else including world state, physics, and player state. The Game Loop is dominated heavily by time spent ticking (processing) each relevant thing on the screen to make sure it's updated.


All of this work resulted in a series of benchmarks that all showed the same general result.


Live (Unreal 3.0 / APB 1.20) Financial at Midnight on my test machine. 

  • Peak performance = 10ms (100FPS)
  • Low performance = 19ms (50FPS) *There are some bad hitches in Live that we throw out
  • Average frame time = 12ms (83FPS)
  • Average Rendering frame time = 12ms
  • Average Game Loop frame time = 12ms


Engine Upgrade (Unreal 3.5 / APB 2.1) Financial at Midnight on my test machine.

  • Peak performance = 10ms (100FPS)
  • Low performance = 19ms (50FPS)
  • Average performance = 15ms (66FPS)
  • Average Rendering frame time = 8.5ms
  • Average Game Loop frame time = 15ms


Right away, we can see that the Peak and Low performance of the game are the same. But there is a 3ms frame time / 17FPS difference in the Average frame time. Where APB 1.20 is more balanced between Rendering and Game Loop, we see a lopsided workload in APB 2.1. The Rendering thread is quite fast, but the entire frame time is held back by how long the Game Loop takes to run.


For the rest of the week, we iterated on this process to improve our stat collection system and dig deeper into specific chunks of the Game Loop so that we could directly compare the performance of those smaller sections between Live and the Engine Upgrade. We were successful in identifying 2 Game Loop sections (out of 35) where APB 2.1 ran a total of 4.5ms slower than APB 1.20.


We are now digging further into those sections to create more detailed stats to help further breakdown performance until we can define a fix to the performance difference.


I'll go on record saying a couple things.

  1. The performance in APB 2.1 is already significantly better than what is running on consoles right now (APB 2.0). We will be finishing console blockers as soon as possible and submitting so the console players can get a much better experience.
  2. If I felt the performance was good enough right now, then I would be opening the Beta and putting this on public OTW. We don't have any other major blockers. The art issues and minor bugs can all be addressed after we let players in to test.




  • Like 34
  • Thanks 13
  • Dislike 1

Share this post

Link to post
Share on other sites

Hi everyone,

This week we made good progress on performance and character lighting.

These changes will help both the PC and console builds.




I have changed the test resolution to Fullscreen Windowed 1920x1080 instead of my previous smaller Windowed 1024x768 resolution to see how far I could stress the engine. That means you can't really compare the results from last week with this week. We also just got our test build out to the SPCT members, so we're still collecting data, but here are the initial results:


Live (Unreal 3.0 / APB 1.20) Financial at Midnight on my test machine @ 1920x1080 / Ultra settings. 
Peak performance = 10.33ms (97FPS)
Low performance = 19.22ms (52FPS) *There are some bad hitches in Live that we throw out
Average frame time = 16.20ms (61.70FPS)
Average Rendering frame time = 16ms
Average Game Loop frame time = 16.20ms


Engine Upgrade (Unreal 3.5 / APB 2.1) Financial at Midnight on my test machine @ 1920x1080 / Ultra settings.
Peak performance = 9.6ms (104FPS)
Low performance = 22.38ms (45FPS)
Average frame time = 16.05ms (62.31FPS)
Average Rendering frame time = 11.7ms
Average Game Loop frame time = 16ms


PERFORMANCE NOTE: There is a decent difference between Day and Night on APB 2.1.

I'm running these tests at night for "worse case" performance. During the day, we see APB 2.1 run 1-2ms faster.


There are a million things that could render this benchmark invalid, and I know on the surface this looks like APB 2.1 is now running slightly faster than APB 1.20, but we'll need more performance tests off SPCT machines to really assess where we are.




Here are some before and after shots to illustrate some of the character shadow/lighting issues we have been fixing.

If you've been playing on console, then you are well aware of how dark and blobby the game can look some times.


LIGHTING NOTE: We're not done with scene lighting or character lighting, so even the updated "new" shots have issues. We aren't rendering ground character shadows properly, and indirect reflected lighting isn't showing up.


This is a cropped section of the Enforcer police scene.


In the first shot, the characters are all very dark and badly lit with strange hard shadows that don't make a lot of sense with the lighting in the scene.

In the second shot, you can see proper lighting on characters. This is still being tuned, but it's 100x better.


APB 2.1 old:



APB 2.1 new:



Here is a cropped section of the new District Select screen.

In the first shot, you can see the blobby hard shadows rendering in weird places.

In the bottom shot, you can see everyone rendering properly.


APB 2.1 old:



APB 2.1 new:




  • Like 37
  • Thanks 10
  • Dislike 4

Share this post

Link to post
Share on other sites

Hi everyone,


I'm working remotely, but this was another week spent diagnosing and attempting to fix rather complex issues in the Engine Upgrade. 


Unfortunately my initial stats from last week didn't hold up across our pool of testers. This is often the frustrating part of this project. We will see good things internally, but once we get the build to testers, the larger sample pool doesn't come back as well.


We did figure out that my work laptop is a bad source of data. It has an embedded Intel 3D card and a high performance nVidia 3D card for gaming. However, the nVidia card is daisy chained off the Intel card. SO when I run Fullscreen Windowed, a Desktop Windows Manager process runs at 50% using my Intel GPU while APB 2.1 takes up 80%+ on my nVidia GPU. For whatever reason, that equalizes the performance between APB 1.20 (Live) and APB 2.1 (Engine Upgrade) and over time they run comparably. 


The team finished the following tasks:

- Secondary night time lights are now working
- Secondary night time lights are now running nearly as fast as day time. Console players will be happy with that change once we re-submit.

- Added some fixes for texture caching, so that as you run around the district we aren't wasting so much CPU to figure out what textures should be loaded.


Still working on the following issues:

- Rendering shadows from secondary night time lights

- Rendering indirect lighting when characters aren't affected by the main light (sun or moon)

- Interactive building features still disappear when the player is far enough from a building that it shifts from LOD0 (the highest level of detail) to LOD1. This leaves an empty hole for a period of time where the shop front or window should be.


This week marked the first time we ran all the testers through a series of benchmarks in both APB 1.20 and APB 2.1 to try and get accurate performance comparison stats.


I know this may sound exciting, but it's not. Often testers have to patch, run the benchmark once to make sure the game finishes loading and caching all the geometry, and then they run the benchmark a second time for accuracy. If anything on their system accidentally interferes, then they have to start over and do it again. Some die hard testers did this for each district. Once they had accurate benchmarks, they sent us their logs, and then yesterday we processed all the logs through our internal tools to graph, identify CPU and GPU, and call out specific bottlenecks. The end of the day was spent posting the results to our internal channels for testers to see everyone's results.


Based on that round of testing, we did see an overall increase in performance by 1-2 milliseconds per frame between 10/21 and 10/26.

We're now back to being Render locked instead of Game Loop locked.


The team is looking at an automated way to prompt beta testers to upload their logs at the end of a session, so we can collate and process them automatically across a larger number of machines to get better insight into performance.


At some point, we will likely need to bite the bullet and throw this out to the community to see how it performs.



  • Like 19
  • Thanks 10
  • Haha 1
  • Sad 1
  • Dislike 1

Share this post

Link to post
Share on other sites

Hi everyone,


This week was a bit frustrating marked with stalls, blockers, and rabbit holes.


We ran another series of benchmarks and things are steadily looking better.


- Interactive building features / windows are almost done

This is an example of something that was on our schedule as a week long item.

Instead we have had to massively re-write the system to pull together better LOD (Level of Detail) techniques but also preserve some of the old RTW code.

The system now works properly, but we're missing the data that links interactive features to the proper LOD on the building they are attached to.

We spent days going back through old data, and finally found it.

Then we spent more days parsing it different ways to reapply it.

Then we spent more days fixing crashes that would appear after an hour of running the tool to apply the data to Financial and Waterfront.

As of yesterday, we were running the tool and it hadn't crashed. Hopefully this will be done early next week.


I recognize this issue is hard to describe, and even harder to visualize, so we took some screenshots to compare.


Here is what the game looks like now in APB 1.20.

There is a lot of cheating going on, because they can hide poor Levels of Detail behind the massive amounts of fog.



Next up is the what APB 2.1 looks like from roughly the same shot.

We don't have the same level of fog now, so our LODs need to look better.

For this shot, we are at the edge of rendering LOD1 (the 2nd most detailed version of buildings).



And this is the bug we're fixing.

For this shot, we are now rendering buildings at LOD0 (the best detail on buildings).

But the interactive features and some lower windows are too far away to render.

This leaves "holes" where the feature/windows should be. If I move closer the features and windows will reappear.


The solution we are implementing is to sync the distance that we turn off features to when we render LOD1.



Side note: The dev in charge of this task found a bug that relates to why traffic signals are broken. 

Apparently these have not worked properly for some time, and he's going to take a look on Monday to see what can be done to fix those.


- Shadows have been a pain

This effort has been to properly render all the shadows around a character based on interior lights, night time lights, and the sun/moon.

So far, we have rewritten elements of the core Unreal shader to get this working properly, and characters now render great.

However, we've had a steady stream of issues with shadows from props acting up and not rendering properly. 
Currently there is a very irritating flicker where some props have lights embedded in them that are now casting shadows using the prop's geometry.

The team has made excellent progress, and cleaning up this code has led to other optimizations and fixes.

But I'm hoping this will finally be done next week.


To get this working properly, we had to create a small test scene with a ton of lights at various locations.

image (18).png


- Indirect lighting is still not working

This got tabled till shadows are fixed.

Lastly, just like the traffic light bug, sometimes we end up finding other things that can be fixed.

This week it was Anti-aliasing. Our old technique, MSAA was removed from DirectX 11, and AA has never worked in APB 2.1.

A different dev ran across some code that looked to be causing the problem, and he was able to run some quick tests and get AA working again.


Here is a quick shot of what APB 2.1 looks like right now with AA.




  • Like 43
  • Thanks 12
  • Dislike 2

Share this post

Link to post
Share on other sites

Hi everyone,


I’m going to keep this week’s update a bit shorter.


Fixing lighting, shadows, and building features continues to be a bit of a rabbit hole. There is very good progress, and with respect to lighting and shadows, I am calling it ‘good enough’. There are some straggling items that we can revisit post launch.


Features on buildings (cProps) is the last area plaguing development. We discovered a horrible issue where props with embedded lights were recreating those lights over and over. We also have a lot of duplicate props in the district (again). This is an issue that popped up a month and a half ago that we solved. But it seems to be back.


EDIT: I got the actual stats today. There are currently 12,700+ duplicate objects in Financial alone. 

There's also a secondary issue that popped up related to props that use the physics engine. Apparently we were at the limit of how many objects can use physics in a single district. It is very possible that we are occasionally seeing crashes in Live due to this issue. So we also spent some time fixing this in APB 2.1.


We finished AA (anti-aliasing) and have started work on AO (ambient occlusion). It works for the most part except when the game animated the field of view which causes a flicker.


The hope is to get a build to SPCT by Tuesday/Wednesday that has these items all addressed.


EDIT: It’s taking a fair bit longer to clean up the dupes. So we wont have a new build tomorrow.




  • Like 21
  • Thanks 15
  • Dislike 2

Share this post

Link to post
Share on other sites

Hi everyone,


I missed yesterday’s update, and I’m currently out of town, but here is a quick rundown:


- Part of the team had to take a break from the Engine Upgrade to help finish off our new Christmas mini-game. They should wrap up those tasks this week and jump back in soon.


- The biggest focus continues to be the building features and removing duplicate items. I added an update to last week’s post once we identified more than 12K duped items. It took 3 days longer than expected to hammer out the commandlet that could properly identify dupes (lights, brushes, static meshes, etc). That process is a bit crazy. Each object type has unique code that compares it with each other object of the same type. On Friday we had cleaned up all the smaller maps and Financial. We’re working on Waterfront over the weekend. Hopefully we’ll have a new build to test soon.


- We made some progress with Ambient Occlusion, but that engineer got pulled into other tasks related to the network move happening this weekend.


- We shifted some focus to PS4 and XB1 bugs. We tend to pause console development while larger efforts are underway in the code, so that we don’t end up having to work through the same code twice. Currently neither build runs well due to the dupe issue, but we made some good progress anyway.


TL;DR - Christmas tasks and the network move ate up some bandwidth. But everyone should be back on APB 2.1 after Thanksgiving.




  • Like 20
  • Thanks 7
  • Haha 1
  • Confused 1
  • Dislike 3

Share this post

Link to post
Share on other sites

Hi everyone,


Only three working days this week, and lots of interruptions with the terrible network move.


I don't have a comprehensive update today, but I was told that we are finally "done" with all the major blockers.

However we need to make a build and get our test environments put back together, and then we'll see how all the fixes hold together.

I'll likely update later this week once I have more information.




  • Like 27
  • Thanks 8
  • Confused 1
  • Dislike 4

Share this post

Link to post
Share on other sites

Hi everyone,


Here is a quick rundown of the active blockers:


- Lighting

Now that we have de-duped all the lights and props, we need to re-run lighting so we can bake the static portions of each district that combine with the dynamic portions at run time.


Due to a bug we fixed last week, up until now all the lights were running dynamically, which carries a big performance penalty at night.


We started baking lighting for Financial on Wednesday. But Friday it wasn't done. This is unusually long compared to light builds we have run in the past. There are quite literally thousands of lights in Financial that all switched to baked instead of dynamic. So it's possible that the longer build time is normal. However, we also suspect that there was an issue on the build machine during the week due to a network issue. So ultimately we elected to kill the job, make sure all the workstations on the network were on and running the Swarm agent, and then re-run it starting Friday night. We'll see if it finishes by Monday.


- Physics

This is a great example of unplanned complexity in the Engine Upgrade project.


In order to support the latest Microsoft and Sony libraries for console, we did a massive series of upgrades to external libraries. One of those was the physics library. Now it looks like that developer changed a core part of how physics objects are initialized, because SPCT previously reported that interactive props and other physics objects would immediately explode and then respawn as players entered the district. Part of the team spent time this week troubleshooting (and ultimately fixing) several of these physics issues.


I'm guessing that this is the sort of frustrating loop that I'm sure Reloaded found themselves in. You upgrade one part of the code to support one platform, and then break other parts of the code. Then to fix it, you end up needing more upgrades elsewhere. And so on. 


- Memory issue

On Monday, we ran our test build and found a massive memory leak that was able to allocate 7GB of ram before even getting into a district. By Friday, we had fixed two possible areas, but we are still investigating.


- Console blocker

The memory leak blocked all console development, because those platforms can't allocate that much memory. Work there was paused this week.


- Other build blockers

Making builds is a big part of development. Devs produce local builds all the time, but those have all the optimizations turned off and don't really provide any sort of real performance. We have a number of internal systems that we use to identify specific versions of the code and generate end-user optimized builds.


This week we had a number of build issues starting on Tuesday that prevented us from getting a new APB 2.1 build done. Most of the issues were related to the network move and the locations of various resources. The team worked through most of these, but the process takes several hours to get a successful (or unsuccessful) build, so it is extremely tedious. At this point it looks like we have worked out all the kinks. But we still don't have our Test Worlds back online yet, so we can't share anything with SPCT. I'm told our Test Worlds will be back online this week.


For next week, hopefully lighting will be done for Financial so that we can get a new end-user build generated to test all the work we have done over the last 3 weeks. We will also need to run lighting for all the other districts during the week. This new build should help us identify remaining tasks and perform critical benchmarks.




  • Like 20
  • Thanks 8
  • Sad 1
  • Dislike 3

Share this post

Link to post
Share on other sites
This topic is now closed to further replies.

  • Recently Browsing   1 member

  • Create New...