Jump to content
MattScott

Tracking the Engine Upgrade

Recommended Posts

Hi everyone,

 

The entire team has been crunching for September on the Engine Upgrade (including me), so I haven't had much time to blog lately.

However, I have seen a lot of requests for updates, so I thought I would try something a little different.

 

As I said on the latest Q&A, we're done merging all the content and code. We are just fixing bugs now.

 

So below I'm going to list the exact bugs that we need to fix before we do the public OTW test.

You'll notice we're not quite at the stage of fixing actual gameplay or collision issues yet.

I'll try updating this thread each Friday to clear or add bugs as we move forward.

 

All platform issues:

 

  • Performance benchmarks - We've made a ton of changes recently, so we need all new benchmarks. From our earlier work, we feel the game is running well in terms of multithreaded rendering. The render threads are out performing the game thread. So all of our latest optimizations have been surrounding ways to improve core game logic.
    • STATUS: Ongoing

 

  • Various disconnect crashes - This only happens in "final" builds that we release to players. There is a crash in the game whenever you get disconnected from a server for any reason. This causes the client to flag itself for a full file repair, which is very annoying for testers.
    • STATUS: In progress

 

  • Level of Detail issue - This was extremely noticeable in the video that got released. Buildings popping in horribly. Trees disappearing. Etc. Turns out there was a tremendous amount of custom code in Live used to dynamically morph between various building LODs and to support individually lit windows at night. Reloaded tossed all this original code because none of it was supported under Unreal 3.5, and they just adopted the stock LOD system. Obviously this left many bugs. The team was able to pull in all the code we needed from the old engine and combine it with the new Unreal 3.5 LOD system.
    • STATUS: Mostly fixed / Window lights on LOD0 at night don't work in "final" builds for players

 

  • Texture streaming bug - Even though this isn't a crash bug, it has been our #1 issue. Texture streaming is a unique system RTW wrote to prioritize textures based on various methods, so that when the player's machine or console runs out of texture memory, the game can downrez textures that aren't used very much (or aren't very noticable). This is a lot harder than it sounds, and there are many different methods used (texture size rendered to the screen, distance from the furthest object using the texture, special flags for signage and other highly detailed textures, etc). For the most part, this is working MUCH better now. The game looks amazing. There are a bunch of signs in the districts that need to get special texture streaming flags, and in some cases we see oddball textures get downrezzed that shouldn't.
    • STATUS: Mostly fixed

 

  • NPC / Living city character lighting issues - This was very weird. We started noticing that sometimes many of the pedestrians were heavily under lit with a weird orange lighting that didn't match the scene. We've been chasing this for a bit, and found that the code for reflective lights was broken. The team was able to determine that reflective light values were being initialized based on whatever main direction light was active (or inactive) in the scene when they loaded into the district. This is being fixed so that we can control the reflective light values from our Time of Day data.
    • STATUS: In progress / Mostly fixed

 

  • Character customization saving is broken - There's a database change necessary in our external SPCT testing environment. The next push should fix it.
    • STATUS: In progress

 

  • Color Lookup Table grading is broken - This is a general post processing feature in Unreal where artists can apply color LUT files to modify the look of the scene. Right now, we're seeing some high thresholding / banding when LUTs are used. 
    • STATUS: Removed so the artifact doesn't show up / Awaiting dev resource

 

  • Time of Day color values are broken - The lighting model for Unreal 3.5 is completely different than Unreal 3.0. As such, all of the values used to handle colors, depth of field, bloom, etc for midnight, dawn, noon, evening and night time need to be polished up.
    • STATUS: In progress

 

  • Streetlamps in Financial - It looks like an artist at Reloaded swapped all the spotlights to point lights for any lighting that comes from streetlamps in the district. In order to create the light halo on the sidewalk, they had to make the radius quite large. This means when the streetlamp is next to a building there is a bright patch where the light is affecting the wall. The team went back through and swapped back in the spotlights, but we have a bug in the prefab that needs our Financial map rebuilt.
    • STATUS: Fixed but 2nd issue needs fixing

 

  • Building feature LOD issue - We have a visual artifact because the rendering of mission features on buildings is not synced to the building LOD code. This means that for a small period of time we sometimes see holes in buildings where the features should be.
    • STATUS: Awaiting dev resource

 

 

We have submitted two console builds, and gotten kicked back for defects.

Right now we're 94% compliant with the necessary checks that both Microsoft and Sony require to approve a title.

 

Xbox One: 21 total must fix bugs (hightlights below)

  • Title terminates on disconnecting the network cable anywhere in the title
  • Null Characters are displayed on launching the title in any supported language other than English
  • Background art and MAP fails to render on Xbox One X kit
  • Title terminates on launching the tile in an Xbox One X kit 
  • The title crashes on suspending the title during the splash screen
  • Title crashes in keeping the console idle for more than 20 minutes on the Music player
  • Game Event limitation popup is displayed during gameplay 
  • SpaceBar is displayed for jump in Tutorial during gameplay
  • Unnecessary Resolution option is displayed in Video Options
  • Title crashes on adding friends from inside the game
  • Title displays a black screen for more than 12 seconds right after the initial splash screen
  • Title displays unnecessary button call out during gameplay
  • Title allows the muted user to communicate through text during online gameplay
  • Title allows the blocked user to send Group Invite
  • User is unable to dismiss the interact menu by pressing B button after interacting with another user
  • Title randomly crashes during normal gameplay
  • The achievement 'Olympic Contender' fails to unlock on meeting its criteria
  • User is unable to scroll thorough the description using RS in descriptions 
  • Title does not provide an option to invite another user to the game

 

PlayStation 4: 12 total must fix bugs (highlights below)

  • The term 'Sony Entertainment Network account' is displayed in the title screen
  • Application provides 'Resolution' setting in the 'Video' tab
  • Application soft locks upon selecting a user from the ‘Search’ tab of 'GROUP & FRIENDS'
  • Gameplay - Incorrect naming convention is displayed in ‘Tutorial’
  • Incomplete online ID is displayed in the ‘Friends’ tab in ‘GROUP & FRIENDS’
  • Application fails to transition to a mode which supports play together
  • User B fails to join the User A's session using 'Join' feature
  • Application crashes on exploding a bunch of cars during gameplay
  • Application incorrectly displays the SIE-specified text in the ToS
  • Gameplay - Incorrect naming convention is displayed in 'Drive the vehicle 500 meters' tutorial
     

Thanks,
Matt

 

 

  • Like 45
  • Thanks 37
  • Haha 2
  • Sad 3
  • Dislike 1

Share this post


Link to post
Share on other sites

Hi everyone,

 

I am traveling right now, so I missed yesterday's update.

I'm going to summarize the week here, and then modify the main post.

  • Performance updates - this has been a little frustrating. We've seen side by side comps where the new engine outperforms or matches Live. But right now, the build being tested by SPCT is performing terrible compared to Live. We spent part of the week rebuilding our stat collecting system to gather deeper data related to the game loop.
  • Various disconnect crashes - These have all been fixed (for the moment).
  • Texture streaming bug - We addressed a couple more issues, and I'm treating this as finished.
  • NPC / Living city character lighting issue - This is fixed but I have a couple non-essential changes I want before go live.
  • Character customization saving is broken - This is fixed and needs to be deployed with the next build.
  • Streetlamps in Financial - This turned into a huge rabbit hole, but we managed to get it all fixed. We need to re-run lighting for Financial before the next build goes up. This process runs overnight in the office and uses all the available computers on the network, and even then it still takes more than 8 hours to complete.

 

We found a lot of new bugs that have been logged, but I don't have access to the list at the moment. I'll try to rework the list in the main post on my next update.

 

Thanks,
Matt

  • Like 43
  • Thanks 10
  • Haha 1
  • Sad 2

Share this post


Link to post
Share on other sites

Hi everyone,

 

These are turning into Saturday updates, since that seems to be time when I can sit down and recap the work from the week.

 

Right now, our #1 priority is performance. Performance for the Engine Upgrade was pretty good prior to when we merged all the code and assets from Live. After that, our performance has really struggled.

 

For this post, I'm going to breakdown how performance is measured by our team, and what we're looking at.

 

First, we spent this week finishing off items from above, and re-writing/improving the statistics that we collect in-game. This took significant effort to revamp old code and measure specific parts of the rendering and game loops so we can see where most of the time is spent. We also synced this new code between APB 1.20 (Unreal 3) and APB 2.1 (Unreal 3.5), and we created a new export process to chart performance in Excel.

 

Second, we took this new stats engine and ran a series of test runs that isolate as many variables as possible. For instance, we tracked performance in Financial at midnight across APB 1.20 and APB 2.1. We use night time, because the game generally runs worse at night due to the extra lights that turn on. Each of us used different test machines, but mine was on the less powerful side with a 4 core i7-7700HQ @ 2.80GHz, an Nvidia GTX 1050, and 32GB ram. For the test we ran at the highest settings but used 1024x768 windowed mode to generate a lot of windows background interference. We also run no vsync and no smooth frame rate to get the highest rates possible.

 

We measure a game's performance based on the low, high, and average frame time in milliseconds. Here are some common frame time to frame rate benchmarks values.

  • 10ms frame time = 100FPS
  • 16ms frame time = 60FPS
  • 25ms frame time = 40FPS
  • 40ms frame time = 25FPS

 

Unreal 3 has always run the Render thread and Game Loop thread in parallel, but each process runs its tasks one at a time in sequence. The game's total frame time is the overlapping time from the Render thread and the Game Loop thread per frame. The Render Thread is mostly work done on your graphics card. The Game Loop thread is all the logic and keeping track of everything else including world state, physics, and player state. The Game Loop is dominated heavily by time spent ticking (processing) each relevant thing on the screen to make sure it's updated.

 

All of this work resulted in a series of benchmarks that all showed the same general result.

 

Live (Unreal 3.0 / APB 1.20) Financial at Midnight on my test machine. 

  • Peak performance = 10ms (100FPS)
  • Low performance = 19ms (50FPS) *There are some bad hitches in Live that we throw out
  • Average frame time = 12ms (83FPS)
  • Average Rendering frame time = 12ms
  • Average Game Loop frame time = 12ms

 

Engine Upgrade (Unreal 3.5 / APB 2.1) Financial at Midnight on my test machine.

  • Peak performance = 10ms (100FPS)
  • Low performance = 19ms (50FPS)
  • Average performance = 15ms (66FPS)
  • Average Rendering frame time = 8.5ms
  • Average Game Loop frame time = 15ms

 

Right away, we can see that the Peak and Low performance of the game are the same. But there is a 3ms frame time / 17FPS difference in the Average frame time. Where APB 1.20 is more balanced between Rendering and Game Loop, we see a lopsided workload in APB 2.1. The Rendering thread is quite fast, but the entire frame time is held back by how long the Game Loop takes to run.

 

For the rest of the week, we iterated on this process to improve our stat collection system and dig deeper into specific chunks of the Game Loop so that we could directly compare the performance of those smaller sections between Live and the Engine Upgrade. We were successful in identifying 2 Game Loop sections (out of 35) where APB 2.1 ran a total of 4.5ms slower than APB 1.20.

 

We are now digging further into those sections to create more detailed stats to help further breakdown performance until we can define a fix to the performance difference.

 

I'll go on record saying a couple things.

  1. The performance in APB 2.1 is already significantly better than what is running on consoles right now (APB 2.0). We will be finishing console blockers as soon as possible and submitting so the console players can get a much better experience.
  2. If I felt the performance was good enough right now, then I would be opening the Beta and putting this on public OTW. We don't have any other major blockers. The art issues and minor bugs can all be addressed after we let players in to test.

 

Thanks,

Matt

  • Like 34
  • Thanks 13

Share this post


Link to post
Share on other sites

Hi everyone,


This week we made good progress on performance and character lighting.

These changes will help both the PC and console builds.

 

Performance

 

I have changed the test resolution to Fullscreen Windowed 1920x1080 instead of my previous smaller Windowed 1024x768 resolution to see how far I could stress the engine. That means you can't really compare the results from last week with this week. We also just got our test build out to the SPCT members, so we're still collecting data, but here are the initial results:

 

Live (Unreal 3.0 / APB 1.20) Financial at Midnight on my test machine @ 1920x1080 / Ultra settings. 
Peak performance = 10.33ms (97FPS)
Low performance = 19.22ms (52FPS) *There are some bad hitches in Live that we throw out
Average frame time = 16.20ms (61.70FPS)
Average Rendering frame time = 16ms
Average Game Loop frame time = 16.20ms

 

Engine Upgrade (Unreal 3.5 / APB 2.1) Financial at Midnight on my test machine @ 1920x1080 / Ultra settings.
Peak performance = 9.6ms (104FPS)
Low performance = 22.38ms (45FPS)
Average frame time = 16.05ms (62.31FPS)
Average Rendering frame time = 11.7ms
Average Game Loop frame time = 16ms

 

PERFORMANCE NOTE: There is a decent difference between Day and Night on APB 2.1.

I'm running these tests at night for "worse case" performance. During the day, we see APB 2.1 run 1-2ms faster.

 

There are a million things that could render this benchmark invalid, and I know on the surface this looks like APB 2.1 is now running slightly faster than APB 1.20, but we'll need more performance tests off SPCT machines to really assess where we are.

 

Lighting

 

Here are some before and after shots to illustrate some of the character shadow/lighting issues we have been fixing.

If you've been playing on console, then you are well aware of how dark and blobby the game can look some times.

 

LIGHTING NOTE: We're not done with scene lighting or character lighting, so even the updated "new" shots have issues. We aren't rendering ground character shadows properly, and indirect reflected lighting isn't showing up.

 

This is a cropped section of the Enforcer police scene.

 

In the first shot, the characters are all very dark and badly lit with strange hard shadows that don't make a lot of sense with the lighting in the scene.

In the second shot, you can see proper lighting on characters. This is still being tuned, but it's 100x better.

 

APB 2.1 old:

EnforcerCharacter35old.PNG

 

APB 2.1 new:

EnforcerCharacter35new.PNG

 

Here is a cropped section of the new District Select screen.

In the first shot, you can see the blobby hard shadows rendering in weird places.

In the bottom shot, you can see everyone rendering properly.

 

APB 2.1 old:

DistrictSelect35old.PNG

 

APB 2.1 new:

DistrictSelect35new.PNG

 

Thanks,
Matt

  • Like 36
  • Thanks 9
  • Dislike 3

Share this post


Link to post
Share on other sites

Hi everyone,

 

I'm working remotely, but this was another week spent diagnosing and attempting to fix rather complex issues in the Engine Upgrade. 

 

Unfortunately my initial stats from last week didn't hold up across our pool of testers. This is often the frustrating part of this project. We will see good things internally, but once we get the build to testers, the larger sample pool doesn't come back as well.

 

We did figure out that my work laptop is a bad source of data. It has an embedded Intel 3D card and a high performance nVidia 3D card for gaming. However, the nVidia card is daisy chained off the Intel card. SO when I run Fullscreen Windowed, a Desktop Windows Manager process runs at 50% using my Intel GPU while APB 2.1 takes up 80%+ on my nVidia GPU. For whatever reason, that equalizes the performance between APB 1.20 (Live) and APB 2.1 (Engine Upgrade) and over time they run comparably. 

 

The team finished the following tasks:

- Secondary night time lights are now working
- Secondary night time lights are now running nearly as fast as day time. Console players will be happy with that change once we re-submit.

- Added some fixes for texture caching, so that as you run around the district we aren't wasting so much CPU to figure out what textures should be loaded.

 

Still working on the following issues:

- Rendering shadows from secondary night time lights

- Rendering indirect lighting when characters aren't affected by the main light (sun or moon)

- Interactive building features still disappear when the player is far enough from a building that it shifts from LOD0 (the highest level of detail) to LOD1. This leaves an empty hole for a period of time where the shop front or window should be.

 

This week marked the first time we ran all the testers through a series of benchmarks in both APB 1.20 and APB 2.1 to try and get accurate performance comparison stats.

 

I know this may sound exciting, but it's not. Often testers have to patch, run the benchmark once to make sure the game finishes loading and caching all the geometry, and then they run the benchmark a second time for accuracy. If anything on their system accidentally interferes, then they have to start over and do it again. Some die hard testers did this for each district. Once they had accurate benchmarks, they sent us their logs, and then yesterday we processed all the logs through our internal tools to graph, identify CPU and GPU, and call out specific bottlenecks. The end of the day was spent posting the results to our internal channels for testers to see everyone's results.

 

Based on that round of testing, we did see an overall increase in performance by 1-2 milliseconds per frame between 10/21 and 10/26.

We're now back to being Render locked instead of Game Loop locked.

 

The team is looking at an automated way to prompt beta testers to upload their logs at the end of a session, so we can collate and process them automatically across a larger number of machines to get better insight into performance.

 

At some point, we will likely need to bite the bullet and throw this out to the community to see how it performs.

 

Thanks,
Matt

  • Like 19
  • Thanks 10
  • Haha 1
  • Sad 1

Share this post


Link to post
Share on other sites

Hi everyone,

 

This week was a bit frustrating marked with stalls, blockers, and rabbit holes.

 

We ran another series of benchmarks and things are steadily looking better.

 

- Interactive building features / windows are almost done

This is an example of something that was on our schedule as a week long item.

Instead we have had to massively re-write the system to pull together better LOD (Level of Detail) techniques but also preserve some of the old RTW code.

The system now works properly, but we're missing the data that links interactive features to the proper LOD on the building they are attached to.

We spent days going back through old data, and finally found it.

Then we spent more days parsing it different ways to reapply it.

Then we spent more days fixing crashes that would appear after an hour of running the tool to apply the data to Financial and Waterfront.

As of yesterday, we were running the tool and it hadn't crashed. Hopefully this will be done early next week.

 

I recognize this issue is hard to describe, and even harder to visualize, so we took some screenshots to compare.

 

Here is what the game looks like now in APB 1.20.

There is a lot of cheating going on, because they can hide poor Levels of Detail behind the massive amounts of fog.

Live_LOD01.png

 

Next up is the what APB 2.1 looks like from roughly the same shot.

We don't have the same level of fog now, so our LODs need to look better.

For this shot, we are at the edge of rendering LOD1 (the 2nd most detailed version of buildings).

LOD01.png

 

And this is the bug we're fixing.

For this shot, we are now rendering buildings at LOD0 (the best detail on buildings).

But the interactive features and some lower windows are too far away to render.

This leaves "holes" where the feature/windows should be. If I move closer the features and windows will reappear.

 

The solution we are implementing is to sync the distance that we turn off features to when we render LOD1.

LOD00.png

 

Side note: The dev in charge of this task found a bug that relates to why traffic signals are broken. 

Apparently these have not worked properly for some time, and he's going to take a look on Monday to see what can be done to fix those.

 

- Shadows have been a pain

This effort has been to properly render all the shadows around a character based on interior lights, night time lights, and the sun/moon.

So far, we have rewritten elements of the core Unreal shader to get this working properly, and characters now render great.

However, we've had a steady stream of issues with shadows from props acting up and not rendering properly. 
Currently there is a very irritating flicker where some props have lights embedded in them that are now casting shadows using the prop's geometry.

The team has made excellent progress, and cleaning up this code has led to other optimizations and fixes.

But I'm hoping this will finally be done next week.

 

To get this working properly, we had to create a small test scene with a ton of lights at various locations.

image (18).png

 

- Indirect lighting is still not working

This got tabled till shadows are fixed.


Lastly, just like the traffic light bug, sometimes we end up finding other things that can be fixed.

This week it was Anti-aliasing. Our old technique, MSAA was removed from DirectX 11, and AA has never worked in APB 2.1.

A different dev ran across some code that looked to be causing the problem, and he was able to run some quick tests and get AA working again.

 

Here is a quick shot of what APB 2.1 looks like right now with AA.

ScreenShot00003.png

 

Thanks,
Matt

  • Like 41
  • Thanks 12

Share this post


Link to post
Share on other sites

Hi everyone,

 

I’m going to keep this week’s update a bit shorter.

 

Fixing lighting, shadows, and building features continues to be a bit of a rabbit hole. There is very good progress, and with respect to lighting and shadows, I am calling it ‘good enough’. There are some straggling items that we can revisit post launch.

 

Features on buildings (cProps) is the last area plaguing development. We discovered a horrible issue where props with embedded lights were recreating those lights over and over. We also have a lot of duplicate props in the district (again). This is an issue that popped up a month and a half ago that we solved. But it seems to be back. There's also a secondary issue that popped up related to props that use the physics engine. Apparently we were at the limit of how many objects can use physics in a single district. It is very possible that we are occasionally seeing crashes in Live due to this issue. So we also spent some time fixing this in APB 2.1.

 

We finished AA (anti-aliasing) and have started work on AO (ambient occlusion). It works for the most part except when the game animated the field of view which causes a flicker.

 

The hope is to get a build to SPCT by Tuesday/Wednesday that has these items all addressed.

 

Thanks,

Matt

  • Like 18
  • Thanks 14

Share this post


Link to post
Share on other sites
Guest
This topic is now closed to further replies.

  • Recently Browsing   0 members

    No registered users viewing this page.

×
×
  • Create New...