Tracking the Engine Upgrade

[MattScott 15242]

Hi all,

 

Time zones are messing with my updates. I got a brief update on where the Engine Upgrade was at Friday morning Pacific time.

Progress continues to be good. It's a rat's nest of old code, but we've nearly unraveled it all and put most of it back together better.

 

Internally the big test will be this upcoming week, when we can take everything for a test drive.

At this point I am hopeful we will have some benchmarks by next week's update.

 

Thanks,

Matt

[MattScott 15242]

Hey everyone,

 

My schedule has been incredibly busy - busy enough that I completely missed last week's update and only realized it today. Sorry about that.

 

Part of the problem is that in addition to daytime meetings here in Europe, I am also leading a massive project in the evenings with the US that keeps me up till 2-3am every night.

 

Big deadline today, so that should free up some of my time to resume these updates tomorrow.

 

Apologies,

Matt

[MattScott 15242]

Hi everyone,

 

I'm long overdue for a more comprehensive update on the Engine Upgrade, so here is a recap of the last couple of weeks.

 

Back on September 6th, I outlined a list of things we were working on.

Here is where we stand with those items.

 

 

Progress with the Engine Upgrade:

• There is still a lot of thread safety work left to do, but this was a big source of the errors.

This work has been completed. We *believe* we have re-architected the mulithreaded code so that all the previous errors have been eliminated.

There is still one issue related to Occlusion Query multithreading on RTX cards that causes a bizarre slowdown, so we have that code disabled till we can fix it.

• We want to allow secondary render threads to use the cache and then track down the specific case where we are setting the per material/mesh state and making sure those invalidate the cache properly so everything gets updated.

This work has also been completed. I'm pretty excited about how we fixed this, because after we started testing the new code, we found another issue where the game was destroying the shader cache multiple times per frame. That has also been addressed.

• We are investigating an issue where specific items are forced to update every frame when they shouldn't.

This last item is fixed.

 

We started test driving the new code this week, and while there is some improvement, we're still not seeing anywhere near the kind of performance gains we were expecting.

SIDE NOTE: The team has affectionately adopted the phrase "As expected, APB code does the unexpected".

 

For the later half of this week, the team went back and refreshed some older code that integrates with 3rd party tools that allow us to better analyze multithreaded performance. In the past, we have been relying on Visual Studio and 3rd party tools that were working when we took over. These existing tools measure where the most CPU time is being spent. Overall, that technique has been invaluable in finding brute force code doing large tasks in a single loop that could benefit from being multithreaded. However, these tools do not measure things on the GPU, the time spent waiting on a lock or when a process is blocked.

 

On Friday, we finished integrating some new tools that enabled us to stage a full test, capture results, and generate more comprehensive reports. 

 

As suspected, we found a significantly larger problem related to how the engine renders Meshes overall. To give an example, our previous efforts to multithread code lower down in the pipeline only account for a fraction of the processing time. The higher level system for actually rendering everything that was processed accounts for the majority of processing time - but it's mostly spent waiting or being blocked which is why these areas never showed up in the previous analysis. The overall rendering is effectively drawing elements on only 1 or 2 threads. It's incredibly inefficient, and we already have a good idea on how to fix this. The team will be moving onto this new issue next week.

 

Alongside these 3 areas listed above, the rest of the team has been fixing VOIP, desync, and other more minor areas. We need those out of the way so we can launch once the renderer is finished.

 

As always, I'm providing these updates in real time with the best information I have at hand. I am (again) hopeful we'll see good progress this coming week, and that I can finally get some better mission district benchmarks. The plan is to spin up Open Beta #3 with mission districts as fast as we can after that.

 

NOTE: After I posted this, I thought I would provide some nerdy details.

 

Rendering in a 3D engine mostly happens in 3 stages.

 

Stage 1 processes the scene. This involves lots of complex routines that gather up and process all the things we need to draw. Some of things we do in this stage are:

• Identify only the things that are in view of the player's camera. This happens in multiple ways: bounding checks on the field of view, eliminating things too far in the distance, testing to see if an object is occluded (covered) by another object, etc. We want to eliminate everything  not in view to avoid drawing far too many things.
• Recalculate lighting receivers on each object that we are going to render.
• Process objects for extra data that we need to pass into the next 2 stages. 

 

Stage 2 actually draws the big list of things to the scene, which also involves its own set of stages to properly sort objects back to front and render transparency correctly.

 

Stage 3 handles post processing of the final frame buffer (effectively a composited image of the scene once the objects are drawn). This handles a variety of effects including shadow rendering, bloom, iris response, motion blur and more. Stage 3 requires a lot of extra information that was captured in Stages 1 and 2.

 

TL;DR: The majority of our multithreaded work has been in Stage 1. Now we've identified big areas in Stage 2 that are affecting the overall speed of rendering a single frame.

 

Thanks,

Matt

[MattScott 15242]

Hi everyone,

 

This week's update is frustrating, because we are so close to new benchmarks.

 

Unfortunately 2 specific areas were a bit more complicated and couldn't be completed on Friday.

I may post mid next week once I get a chance to see where we are at.

 

Thanks,
Matt

[MattScott 15242]

Hi everyone,

 

I'll start with the good news:

We're ready for Open Beta #3 and this time we will have all districts online for testing.

PROGRESS WITH THE ENGINE UPDATE:
The team is pushing hard to do the Open Beta this next weekend so we don't interfere with the Halloween event.

 

There is only 1 blocker left, and it's a last minute item I wanted to try for the community:

Removing AVX

 

During development of APB 2.1, we decided to require support for AVX. After the initial Open Beta, we learned that this decision prevented a significant number of players from joining.

So as an experiment I am having the team remove AVX from the client to make sure everyone can still play once we migrate over.

It's about 3 days of work to recompile all of the supporting libraries without AVX, check them all in, and then recompile a new 2.1 build without AVX.

If there is little to no difference, then we'll test without it. But if there is a big enough impact, then we'll have to move forward with it in.

 

Assuming that work gets done by EOD Monday, then I would feel comfortable that we have enough QA time to identify and fix any last minute problems.

 

For this test, we are also leaving a series of Advanced multithreading options visible for you guys to play with.

None of these options require restarting the game or even getting out of the district. They dynamically enable and disable multithreading features.

I would love to collect some feedback on which options worked best overall for your configuration (CPU, GPU, Memory, Hard drive) so we can set those on by default once we roll out.

 

In terms of bugs, the team has mostly cleared the things we knew about:

• Vivox should now work.
• Customizations shouldn't look wonky (after making some larger changes, we couldn't reproduce the skinny character issues) 
• Doors shouldn't get out of sync over the network

I am hoping we fixed the character baking hitch that was occurring when new players joined the district.
This is a very difficult thing to test in a lab, so the engineers will be capturing performance stats during the test.

 

MOVING FORWARD WITH THE TEST:

I want to finish this update by talking a little about expectations.

 

This has been an incredible difficult project to finish.  

Obviously we are massively off the timeline I originally set.

There are still bugs, and we are still not done optimizing the renderer, but weighing the pros and cons, I have decided we are hopefully "good enough" to launch.

There are simply too many other more important tasks that are blocked by not releasing APB 2.1.

I want to get into better district management, better matchmaking, and cross district play between worlds.

 

Post the 2.1 launch, the team is committed to continuing to optimize performance. 
There are definitely places where the 2.1 FPS significantly under performs Live.

We have some larger changes that we would like to make to the renderer, but they will require much more time to engineer.

 

However, we have asked the internal testing group how the game "feels".

And for those that have given feedback and been involved in daily testing, they say that 2.1 "feels" better than Live.

 

So this is where I need the community's help.

 

For this test, let's put away the FPS counter. There will be plenty of time for that down the road.

Here is what I would like to see from testers for Open Beta #3:

• First, ahead of the test, get on Live. Try a mission or Fight Club.
• Try raising or lowering the graphics settings on Live to zero in some of the visual elements that make or break the game for you.
• Focus on the feel of Live. The hitches. The mouse input latency. etc.
• Next, get into the Open Beta and try to do the same things in the same districts.
• Play around with the graphics settings to try and see if you can get to a similar visual quality.
• Then try to compare the feel of Live versus the feel of 2.1 and give us some feedback.

If we can pull off next weekend for Open Beta #3, then we'll get an announcement up in a couple days.

 

WHAT IS LEFT IF THIS OPEN BETA GOES WELL:

During the test you'll see that some things are not up to date with Live.

So the only major task that we need to complete before launching is to migrate the most recent changes from Live into APB 2.1.

(That in addition to any big blocker bugs we encounter with the test)

 

Thanks,
Matt

[MattScott 15242]

Hi everyone,

 

Small Monday update:

We haven't done extensive testing, but here are the early results comparing a build with AVX support to a build without AVX.

This was done on a fairly high end PC.

 

The first test was Financial:

 

The second test was Waterfront:

 

NOTE: There is a margin of variance for running these kinds of tests based on pedestrians and cars that go by which means essentially the results are the same.

This is just one of the cases where APB does the opposite of what you would expect.

In light of this, we are dropping the AVX requirement moving forward.

 

Thanks,

Matt

[MattScott 15242]

Hi everyone,

 

I knew aiming for the Open Beta this weekend was aggressive, and unfortunately we have hit a minor setback yesterday.

 

As we were testing the new build, we found that new players joining the Open Beta might crash if they have any of the new Joker Store items. This is because we haven't migrated content from 1.20 into 2.1 since Open Beta #2.

 

That process is now started, but it means we won't be able to run the test this weekend.

 

I'll keep you posted.

 

Thanks,

Matt

[MattScott 15242]

Hi everyone,

 

Here is a quick update:

If you don't want to read the details, so far we are still on track to run the next Open Beta - but work isn't complete yet. So there could still be hiccups.

As I mentioned last week, the big remaining task is to migrate new or changed code and content from 1.20 to 2.1 so that new players that have their characters copied don't crash.

We still have to do a couple other minor things like removing the AVX requirement from the installer, now that we have removed it from the build.

 

Our internal goal is to get a candidate build to start testing by Monday EOD.

That will need to be approved by our QA by Tuesday EOD for our schedule to hold.

 

So far we have finished the code migrations from 1.20 into 2.1.

 

That leaves data and content. For those two, we have a couple options:

• We can try and automatic merge
• We can accept the newly migrated content from 1.20
• We can ignore the 1.20 content and keep the 2.1 content because it's already modified properly
• Or we can hand merge

I believe everything left right now falls into that last category.

• There are 9 Unreal packages to fix
• Some audio work and rebuilding the sound banks
• Some minor map changes
• And data changes

 

We have divided all of the remaining items across the team, and everyone is working in parallel to finish.

 

More in a couple days.

 

Thanks,
Matt

[MattScott 15242]

Hi all,

 

Just a quick update before the Halloween festivities start.

 

I want to thank everyone who jumped into the Open Beta 3 weekend. We collected lots of good data from the test.

While there was some engine work this week, we are still collecting and digesting all the logs and player feedback.

Next week, we will be putting together the next sprint based on what we prioritize.

 

I hope everyone has a safe holiday.

 

Thanks,

Matt

[MattScott 15242]

Hi all,

 

Backtracking a bit. 

 

In all the election run up, you may have missed the severe fires that hit Orange County during the week of Oct 26. Half our team had to be evacuated from their homes, and many spent 2-3 days in hotels or with relatives. It didn't make sense to try and conduct business with everyone's focus elsewhere, so we got a very slow start after the Beta weekend. Fortunately, our amazing fire fighters in Southern California kept all the buildings and homes safe, and no property was lost.

 

This last week was much more productive.

 

Here are a list of the major Beta issues that were reported (or picked up from stats), and what items have already been addressed:

• Graphical glitches in the UI made worse by turning on VSync. These primarily showed up during the login/character/district select flow.
• "Skinny" characters or loss of character customizations in faces and body proportions. (FIXED)
• Lots of audio issues. Missing audio. Directional audio not correct. This included left and right sounds or 3D positioned sounds. (FIXED)
• Graphic/chrome issue on player vehicles caused by spawning a second vehicle from the same spawner. 
• Periodic lag / latency in districts. (POSSIBLE FIX THAT NEEDS TESTING)
• Stutters on the loading screen.
• Game crash on accepting a clan invite. (FIXED)
• Frame rate loss when moving the mouse around very fast.
• Input delay in the new Fullscreen Windowed mode.

For the console players, we also jumped back into those builds to merge over all the recent work on PC. Both PS4 and XB1 are now compiling again with all the new content.
We'll be looking at console crashes next week.

 

Thanks,
Matt

[MattScott 15242]

Hi all,

 

This will be a shorter update this week, and I think it will be obvious why in the next couple days.

 

At this point, we have run 3 Open Betas, and we need more information related to the player hardware along side the performance data. We have a lot of players who say they have had good experiences, but we also have many players who have given negative feedback. We took the time this week to significantly improve our stat collection technique. It will now also include any latency tests. So if you're experiencing what appear to be server lag, then please run /latencytest once before you exit the game. The new data collection will run and upload a dxdiag as well.

 

Before the data had nothing identifiable, and while it is still anonymous, it does contain added hardware specs and other info. With that in mind, we have added a popup when you exit the game that asks whether you want to contribute data. We don't want to violate anyone's privacy, but we hope that as many players as possible submit data on this next test. 

 

Aside from APB 2.1 work, we also continued work on console. The build is edging right up to the memory limits on PS4, which is creating some issues debugging. But we're making good progress.

 

Lastly, there will be more info soon on the next Open Beta, but this time we're going to do things in 2 parts.

The first part will be a shorter more concentrated stress test, and the second part will be another normal play period.

I would love to see as many players join for both tests, and as a little incentive we have added something fun to check out in Social.

 

Thanks,
Matt

[MattScott 15242]

Hi all, 

It's been a little while since my last update. We ran our 4th Open Beta test on the 21st/22nd, and last weekend was a holiday for me, so I'm just getting caught up.

 

I will admit that our 4th Open Beta was very frustrating. The team worked hard to clear up the issues from the 3rd Open Beta (or so we thought), but yet once we started testing many of those same issues popped up again. Fortunately this time we had a lot of extended data logging and devs online capturing specific moments to narrow down items. More than 250 players submitted feedback and logs.

 

We've still got a lead developer out on holiday, but here is where we stand:

• SSAO caused client slowdowns (this was supposed to be disabled in this build)
• Shadows and time of day lighting not finalized
• Server spikes in both the world server and hosting code (world server has been fixed)
• Character baking issue (we believe this is an alternate UI flow to the previous issue that we already fixed)
• Scaleform/UI tearing (mostly on the opening screens like login)
• Audio problems (very clear what the issues are with the Vegas)
• Chrome showing up on cars (we think this is a settings issue where anyone running below a certain graphic setting will see the chrome)
• Keybinds not being able to be rebound (reported but not confirmed)

To be clear, there were many other issues reported, but these were the largest that we felt we needed to fix before diving deeper to start addressing collisions or other gameplay issues.

I'll keep you posted at the end of this week on our progress.

 

Thanks,
Matt

[MattScott 15242]

Hi all,

 

It's been far too long since I posted an update. I'm actually working on the Engine Upgrade project as a dev, so my free time has become pretty heavily impacted.

 

At this point we are focused on two systems that got removed during the initial console project.

 

One involves a custom lighting solution that eliminates the overhead of dynamically lighting and shadowing thousands of objects in the scene. We found just enough hints of it to have the guys find the rest of it in APB 1.20. It's a lot of code to port in, and it touches the editor, tools, and the runtime game. Work is well underway, but I won't have a sense of how long it will take for another couple weeks. It's not as simple as copying code in and hitting compile. Unfortunately there are extensive differences between the rendering pipelines. I'm sure this is why it wasn't brought over in the original console work.

 

Alongside that, there is another custom solution that Real Time Worlds created to batch render and efficiently handle building features and interiors. Again, there is a lot of code that was removed, but this part will be slightly more difficult. We are also missing all the data that drives the proxy system. That will have to be gathered from APB 1.20 and then imported into 2.1 once we are done.

 

Work continues.

 

Thanks,

Matt

[MattScott 15242]

Hi all,

 

This week I'm going to try and make up for all the missed updates.

Bear with me, this is a bit lengthy.

 

I've seen the "Where is Matt" thread, and I appreciate the concern. I'm fine. Just busy.

 

I figured I would start this update with a bit of context for what is happing in the studio before I jump into APB.

 

At the beginning of January, we started a very large project alongside a longtime partner of ours. These are important projects for the health of the studio because they augment our cashflow and ability to continue investing in projects like APB.

 

This first period of any game is a ton of work. The team is organizing milestones, delivery dates, the checklist for how to evaluate those milestones, and then the detailed overall schedule that has resource allocations for every staff member on the project. We are also working through wireframes for the user experience (UX) flow, doing mockups for the User Interface (UI), polishing up the Game Design Document (GDD), and writing the Technical Design Document (TDD). This project was based on some older tech that was written back in 2013/2014, so there has also been a technology upgrade effort to bring everything up to current standards. Per usual, I've been knee deep in all of that reviewing, asking for changes here and there, and generally building the momentum we need to finish pre-production and get into production. We're submitting the deliverables for Month #2 today, so now I can step back and let my production team take over the reins to manage the staff and continue progress. Now before I start seeing posts and panic, the team working on APB are separate from this new project.

 

Alongside all of that, we have a second new project that we have finished organizing and reached terms on. Just waiting for paperwork and then we'll start that one too. Yep. You guessed it, I've been handling the more complex parts of business development and scope on that title as well. Some of the work will be done in-house, but most of development will be with an external group we have used before. Little Orbit's main focus is the publishing and launch for the game later this year. I've done the majority of my part already, so this is getting handed off to a producer for the next steps.

 

We also have a big round of changes being completed for Unsung Story to get back out to our Early Access backers. 

 

And lastly, I saw mention of Descent on the forums. It's been a long hard road. I appreciate the patience of all the Kickstarter backers and Early Access fans who supported that title. I want to get the game in their hands as much as they want to play it. I promise we will share news as soon as humanly possible when we are legally able to. We are nearly there.

 

TL;DR - 2021 has been insane for me so far, but I'm getting caught up.

 

So that brings me to APB. 

 

Even with those various other projects happening in the background, my primary role right now has been managing the major overhaul to APB 2.1. As I mentioned in my last update, there are two main systems that were dropped during the initial console work. These systems were replaced with standard graphics techniques and removed, so it wasn't obvious to us they were ever missing. The team managed to uncover these missing systems during a line by line comparison with APB 1.20.

 

The systems are:

#1: Night lights / AO shadowing

#2: Building Feature rendering

 

We're focused on System #1 right now. As of today, we believe all of the missing code has been merged into 2.1, and it finally compiles again. Imagine spending a month moving code into a project and only now getting it all to compile. That's how much was missing. I seriously doubt that all of the logic is perfect, so there are going to be bugs littered throughout. Regardless, this is a big first milestone. Now we're recompiling the game's Unreal scripts, which isn't quite working due to some custom classes that can't be autogenerated. Once that is resolved, then here is the rest of the steps to get this part of the project completed:

 

1. Get the Unreal Editor working with no crashes.

 

Let's cross our fingers. In theory, this part shouldn't take a lot of effort. There is some Day/Night Weather data that needs to be reimported from APB 1.20 through the Editor, and then this step is done.

 

2. Test in a very small Unreal test scene for both APB 2.1 and APB 1.20 that only has part of a street, a main directional light and a "night light" such as the street lamps.

 

We need to generate lighting for this small scene and then compare the results side by side. Not just the visual results - but stepping through both sets of code during the process and comparing the actual data. Night lights generate individual custom shadow maps that can be turned on and off depending on the time of day. Those maps light all the static geometry they effect, but it's virtually free to render at runtime. Right now, every night light in APB 2.1 was altered during the console conversion to be a dynamic light.

 

There's a double whammy here.

 

First, dynamic lights themselves are super expensive. But second, the shadows for night lights are being calculated all the time. They are just set to 100% transparent during the day!

 

3. Do the same test to work on the missing "ambient occlusion" map data.

 

In APB 2.1, they decided to go with Screen Space Ambient Occlusion (SSAO) which is done in GPU shaders and rendered at runtime. But even after selecting that system and getting it working, the console devs disabled it when APB was launched on console. We found out why when we turned it back on.

 

It doesn't work.

 

During the Open Beta tests, you guys all saw the flicker that happens when running or entering a car. This is because the camera's field of view changes which in turn causes the flicker as an artifact. We spent quite a bit of time looking for how to fix this, and nearly everything we found said it couldn't be solved. The math just doesn't work.

 

Back in APB 1.20 they cheated and baked out AO to a different style map that could be overlaid on the scene. Not only is this much faster to render, but it doesn't flicker. It also solves another problem - small shadows. In Step 2, I talked about all those dynamic night lights. Each of those casts shadows on everything includes foliage/grass. You can imagine how expensive it gets during portions of the map that have lots of foliage. The AO map takes care of those small shadows without all the overhead we have today.

 

4. Merge Unreal 3 and Unreal 3.5 render techniques.

 

With all the data in place, then we can actually bake lights for a simple test district and run around. This is the trickiest part. We want to preserve the dynamic lighting from Lightmass, the new lighting engine in Unreal 3.5. Lightmass has a very slick pre-calculated grid of light samples that are used to cheaply light moving or animating objects in the scene without having to resort to expensive dynamic lights. But for static objects, we want to use the older shadow map / AO data combined with our newer multithreaded techniques. That's a bit of a jigsaw puzzle to work out, but there's already been some work being done to try and identify all the shaders that need to operate together.

 

As soon as step 4 is done, we'll benchmark everything again. It's possible we may be able to launch APB 2.1 based on this single system, and then we can work on System #2 in the background for more improved performance. Additionally, for all those players who felt the districts were too dark or the lighting was odd, my hope is that we'll capture the color corrections from APB 1.20 but with slightly better looking techniques.

 

I know this sounds like a lot, and it is. Unfortunately there aren't a lot of small milestones that I can talk about in between each step, which makes these updates more difficult to write.

I'll do my best moving forward. I'm hoping once we get to Step 2, I can start posting screenshots. But right now I can't commit to the same weekly updates like before.

 

That's it for now.

 

Again, sorry about the lack of updates or responses to my inbox or discord.

My schedule is nonstop right now. We want to get this completed for you guys as soon as possible.

 

Thanks,
Matt

[MattScott 15242]

Hi all,

 

I'm going to keep the weekly updates going as best I can.

 

Last week, I walked through our four part plan to get APB 2.1 optimized properly.

 

This week we finished troubleshooting the scripts and got those compiling.

 

For those that aren't familiar with Unreal, script files are written in Unrealscript. It's sort of a janky language that Epic has since done away with in Unreal 4.

Devs in Unreal 3 used it as glue between the engine and C++ files to expose variables and functions for non-programmers to build off of in the Editor.

 

Now we're working on #1: Getting the Unreal Editor working.

I'll admit upfront, I shouldn't have tempted fate when I said "In theory, this part shouldn't take a lot of effort." 

 

Turns out all of the shaders for the game have to be compiled and working before we can open the Editor. 

All modern day engines use vertex shaders and pixel shaders to render geometry and perform calculations on the Graphics Processing Unit (GPU).

Vertex shaders handle data that relevant to geometry you're rendering. Triangles. Normals. Vertices.

Pixel shaders handle data at the pixel level which can output textures or entire screens.

I'm going to make some sweeping generalizations for illustration sake, but during each frame, the game grabs batches of geometry it needs to render or process.

It runs that geometry through a vertex shader and then it pumps those results into a pixel shader which results in.. wait for it: a color for a single pixel.
Unreal has a modular design that allows the behavior of both vertex shaders and pixel shaders to modified and then compiled over and over with different features.

In fact, vertex shader and pixel shaders are reusable. They can be mix and matched to produce different effects.

 

APB has hundreds of shader files that each have inputs and outputs with a bunch of logic in between.

That logic is written in yet another language. High-level Shader Language (HLSL). Original, right?

Fortunately it reads very similar to C.

 

Right now we're going line by line through all of those shaders to diagnose areas where code was merged from 1.20 into 2.1.

There are two categories of issue here:

The first is that the shader is simply invalid and doesn't compile.

The second is that the shader compiles but the results are broken based on data that is missing or not processed properly.

 

For Getting the Unreal Editor working, we are only focused on the first category of shader issues.

Once we get into the editor, then we'll start sifting through maps and characters to find shaders that fall into the second category.

 

This week we made steady progress, but until every shader compiles 100%, I wont be able to show any screenshots.

 

Thanks,
Matt

[MattScott 15242]

Hi all,

 

I'm still on Central Europe time for another couple weeks, so I'm posting this on my Saturday morning.

 

We made our way through all of the blockers for compiling scripts. Looks like there is definitely a lighting calculation issue that we will have to come back and adjust after everything is working.

But for now we have set that aside in order to move on.

 

This week is a bit of a boring update. The team was able to start fixing actual coding bugs this week that occur during the Editor start up process.

Lots of them were just missed bits of code from 1.20. We seem to be making good progress so far.
 

I'll keep you posted.

 

Thanks,
Matt

[MattScott 15242]

Hi all,

Just a small note to say that we got the Editor up and running. Now comes the fun of fixing bugs so we can open a map.

 

Thanks,
Matt

[MattScott 15242]

Hi all,

 

I'm going to try and keep up the schedule of posting at least one update a week (even when they are small).

 

We are now on Step 2 from roadmap I posted here.

 

This week was spent testing, fixing, and testing again the work that has been to done. So far we have merged a ton of code, recompiled UE scripts, recompiled shaders, and patched up that code for the two systems we are merging over to get the editor fully compiling and loading without crashes.

 

As I mentioned in the roadmap, now the goal is to get load a simple scene. For this first test, we created a small scene with a sphere primitive in APB 1.20 that has the default material on it. After fixing a number of crashes, we finally got that map loading successfully in APB 2.2 (the internal version for this new expanded version of APB 2.1). Unfortunately the sphere itself didn't render in any of the editor panels. So there was still work to be done, but we can see that all the data is there properly.

 

Just to experiment, we spent some time attempting to load a more complex scene. For that test we used the simplest APB map that can actually be loaded by the game as a district to play in. We use this empty district to test various gameplay elements while they are being worked on. Again, this is a map that was created in APB 1.20 that we tried loading in APB 2.2. The reason we're loading APB 1.20 maps is because that engine fully supports the two systems that are being merged over. So those maps have data that we can use to properly test. And after a bit of hunting down crashes and bugs, we got that scene to load. Right now there are about 10 objects in that scene that are dismissed by the editor due to bad data. But in all cases, they are objects from the two systems we're merging over. Aside from a couple brushes, nothing rendered properly in that scene either.

 

From there, we set about diving into the scripts, shaders, and code to find the needle in the haystack preventing objects from rendering. For the coders following along at home, we use a Visual Studio tool called the Graphic Analyzer for this. That tool allows us to capture a renderer frame and then step through every instruction sent to the Graphics Processing Unit to see what the data was that it sent. To make this exercise a bit easier, we took a rendered frame from APB 2.1 using the same scene that renders properly and compared them side by side. The tool itself is invaluable, but it has one big drawback - it crashes very very frequently. Fortunately the frames are saved, so it's fairly easy to reload. But that involves finding where you were and continuing another half dozen instructions before it decides to crash again. 

 

Along the way we discovered lots of places where code hadn't quite been merged over properly and where some smaller systems were overlooked entirely.

Friday was pretty frustrating, because we knew were close but couldn't quite get anything rendering before the close of business.

 

So today, we put in a little extra time, and finally got the test sphere rendering properly.

 

Tada! Impressive, right? Behold a completely unlit sphere. Look upon its greatness and marvel. 

 

I know this does not seem impressive - at all - but given how much of the code in this game we tore up, it's good progress.

 

There are still bugs with rendering. Some panels in the Editor go completely blank at odd times and you have to reset their view to get them start rendering again, but this demonstrates a clean path through the most basic rendering pipeline. It also means once we clean up the leftover issues, we can get into the heart of Step 2 - adding the rest of the code one chunk and a time to support the new rendering pipeline for the 2 missing systems and baking out the missing data in APB 2.2.

 

As an final experiment this week, we also loaded our simple gameplay scene and that rendered as well, but I can't post a screenshot because it contains some proprietary info onscreen I can't share.

 

Till next week.

 

Thanks,
Matt

[MattScott 15242]

Hi all,

 

I took off last weekend to celebrate Easter. I hope everyone got a nice break.

 

There isn't much to talk about this week.

 

We can now load very complex scenes into the Editor and they render fine.
There are bugs with Decals and Skeletal Meshes that are being hunted down.

 

We tried running the game to see how the login scene rendered, and it wasn't pretty.

There were a number of really bad shader issues that made that scene look like a nightmare.

We'll be working on clearing the Decal and Skeletal Mesh issues this week so that all levels load properly in the Editor.
Then we'll start troubleshooting in-game rendering which goes through a much more complicated pipeline.

 

Thanks,
Matt

[MattScott 15242]

Hi all,

 

This was a bit of a frustrating week. 

 

The main issue right now is a combination of data that was discarded in the original upgrade and the fact that between Unreal 3 and Unreal 3.5, Epic started compressing lighting textures. Epic did this to reduce texture memory by 1/3rd for Directional Lightmaps from 3 to 2. So all of the maps for APB 2.1 ran code during the conversion to get them into the proper format. However during this conversion, the original dev team also dropped other data that they felt they wouldn't be needing. In order to finish, we need that missing data.

 

On top of that, the newer districts have been modified so they are no longer 1:1 with the APB 1.20 districts.

 

This puts us in a weird spot. We can try keeping the newer APB 2.1 districts and then work out how to recreate the deleted data. Or we can modify the conversion process to properly handle the data they deleted and then go back and re-convert the APB 1.20 districts to APB 2.1. Doing that will mean losing all the work that Reloaded Games did on the newer districts to spruce them up. 

 

We'll make the decision on that this week. Either way, we are adding new resources to the upgrade project next week that will be able to focus on specific problematic areas, so I'm hoping that progress will pick back up again.

 

Thanks,

Matt

[MattScott 15242]

Hi all,

 

Another slightly frustrating week, but only because we had to backtrack a bit.

 

We have decided to go back to the APB 1.20 assets and reimport them. It was a tough call, but I feel it will eliminate any further hidden land mines.

 

As we prepped to reimport, we found issues in the previous import related to how collisions were handled and a couple other items. Last week was spent going over that code.

 

Reimporting will likely also involve a little bit of debugging and possible bug fixing to make sure the missing data can be loaded properly from the old 1.20 files and saved into the new 2.2 files. Then we can go back to debugging and fixing the issues that were blocking us in 2.2.

 

Welcome to Development 301, how the sausage is made.

 

Thanks,

Matt

[MattScott 15242]

Hi all,

 

I was pushing right up to tonight to see if we could get something to show for the progress this week. But I want to stick with at least an update every week, and I'm out of time. Oh well.

 

We're quagmired somewhere between re-compiling the last of adjusted shader code, sorting out an optimization that relied on 3 lighting coefficients so APB can store light tangent vectors in the alpha of each one, and serialization issues nested deep down in LOD data for static meshes. 

The work continues.

 

Thanks,
Matt

[MattScott 15242]

Hi all,

 

I hope you had a great Mother's Day break. I missed my update yesterday.

 

We made good progress last week. Serialization issues are fixed. Objects load again, and we're troubleshooting lighting now.

We need to finish up one more piece, and then we will reimport all of the 1.20 packages.

 

I had thought we would have screenshots this week.

But nothing was as impressive as the Sphere shot from before. 

 

Thanks,
Matt

[MattScott 15242]

Hi all,

 

Back to being frustrated. Spent the week deep into shaders and working out some kinks that led to mismatched inputs and outputs between Vertex and Pixel shaders. From the outside, you would think this would be trivial to solve. But APB generates shaders for every possible unique combination of lighting, rendering, fog, and other variables. It's a lot. So the effort this week was like finding a needle in a haystack. Run a map, catch the crash, figure out which object it is and what material was attached, and then examine why those shaders aren't in sync.

More next week.

 

Thanks,
Matt

[MattScott 15242]

Hi all,

 

It's been three weeks since my last update. 

 

I'll try to make this more comprehensive to make up for that.

 

I'm going to start by acknowledging that I'm now leading and handling the majority of the programming for the upgrade on APB. That may already be obvious based on the detailed previous posts, but I figured it was worth publicly stating so you guys can understand some of the slowness. While APB is a priority, I only have so many hours in the day after running the company. There are a number of reasons why, but it really comes down to the fact that everything left to do is very very specific to APB. Simply knowing Unreal 3 or GPU shaders isn't enough at this point. No one else has enough knowledge across all the game's systems (art, shaders, etc). I still stand by this decision. The progress is farther than we've gotten in the last 2 years, but it has come with a cost to my own personal life and free time. The burnout is real. However, this is something I committed to, and I want to see it through so we can get back into full production.

 

Also, I want to address the last couple weeks specifically. In general when Friday/Saturday comes around, I make an extra push to try and get to some screenshots for something more visual rather than these long, word heavy updates. Unfortunately, the last couple weeks have been spent in the weeds cleaning up bugs which prevented that. In every case, I felt tantalizingly close. So I held off on the update thinking that I could get the kind of progress I wanted before the weekend was up. But even with the extra push that never happened, so I ended up waiting another week, hoping for better results.

 

With that out of the way, let's dig into progress.

 

Before I jump in, I should note that Programming 101 says to try and introduce new code in discrete batches so you can test each piece and know that it works before moving on. This is especially important when bugs pop up, because you know exactly the code that was touched, and it's much easier to track things down. When you have to move code over in bulk, you end up not knowing exactly where to start looking when stuff breaks. I did my best to try and move bits over in chunks, but everything is so intertwined that it wasn't really feasible. 

 

Here are a couple of the "needle in a haystack" tasks that occurred over the last couple weeks.

 

1) Tracked down an issue that caused everything to render black (both in the Editor and in-game).

This ultimately ended up as a bug in the primary BassPassPixelShader. But even when I fixed that issue, objects remained largely dark due to the missing baked lighting data. I have temporarily disabled the lighting calculations in the main pixel shader. Once I have finished a couple other tasks, I'll bake lighting and back out this change. For clarity, the game is doing all the lighting calculations so benchmarks will be accurate, I just short circuit the final output to make sure everything actually renders for now.

 

2) Tracked down an issue that caused all skeletal meshes not to render (both in the Editor and in-game).

Skeletal meshes are any object that has bones we can animate. This includes not just player characters and NPCs, but many props and cars. The issue was related to some new transparency optimizations that had been badly ported over from 1.20. Effectively the game was treating all skeletal meshes as transparent, but then not rendering them when we render all the transparent objects. That is now fixed.

 

3) Tracked down an issue that was creating polygon soup in-game.

I probably should have taken a screenshot of the login scene to show what this looked like, but I worried it would be too disheartening to you guys. Essentially "polygon soup" is when you have a serious problem with vertices and points for a mesh. It creates a bizarre scene of stretched polygons everywhere that look nothing like what you're trying to render. At the time, everything in the Editor rendered fine, so this was some in-game specific problem. It ended up being some bad code in an APB 1.20 system that batches mesh data to the GPU. It's a smaller system that I moved over to get the speed benefits, but there ended up being some DirectX 9 -> DirectX 11 bugs. 

 

Here is where that leaves us:

• All major systems ported over from APB 1.20 are working.
• All individual objects render in the Editor (albeit without proper lighting)
• The login scene renders fine (again with no lighting)
• I'm working on getting district maps loading properly in the Editor. This was working, but after fixing some code that supports the features attached to buildings, the maps now error because they are missing data. I need to go back and finish re-importing all the APB 1.20 content so I have the feature data that was lost.
• Once the district maps load, then I can spin up a district and do the benchmarks.

This week, instead of showing off the almighty sphere, I figured I would include some behind-the-scenes shots that reveal a bit more about how the sausage is made.

 

For a lot of the initial work  a couple months ago, I was using a relatively small test level to slowly add objects and fix the bugs. This is setup so that I can actually spawn it and run around. This map only exists in APB 2.2, so it's not super useful for testing side by side with APB 1.20.

 

After that I moved to an existing map in APB 1.20 that we use to test new gameplay. If you look closely, you can see some RIOT bombs leftover. It's not pretty, but it gets the job done.

 

Thanks,
Matt