Tracking the Engine Upgrade

[MattScott 15242]

Hi all,

 

Doing a quick Tuesday morning update.

I can now get into all of the action and fight club districts.

 

There are a fair number of new bugs to work on. Collisions are the first thing I need to fix.

 

Thanks,
Matt

[MattScott 15242]

Hi all,

 

Progress has been slow as I got into the weeds on a number of items.

However, we are bringing on a full time specialist to help, so things should pick up pace.

 

Thanks,
Matt

[MattScott 15242]

Hi all,

I'm way overdue for an update on the engine.

We brought on a new resource to add some momentum and pick up several tasks from me. The new resource had experience with the engine, specifically this customized version used by Reloaded, but it's been quite a few years, and he's still getting caught up on APB specific changes. We will be filling out the team more as we go, but right now we're spending a bit of time to onboard.

The primary issue during my last post were the collisions in Social, which we are using as a test level.

After a number of red herrings, we determined some collisions are loading, but not inside buildings. This is because the old Building Feature code was migrated and is already working. On the one hand, this is great news, because the system appears to be working and we need it to remove 1,000s of objects from the scene that we were processing in 2.x. But right now this is bad because the Building Feature data is empty. It was created with a Real Time Worlds era tool and baked into the APB 1.x maps. If you recall that map data was wiped out during the migration from 1.x to 2.x years ago. Essentially, without the proper data, the game has no idea what building you are in or near, so it rejects loading any of the assets inside - including the collisions.

We are current focusing exclusively on re-migrating the data from 1.20 to 2.2. That involves picking through and understanding that RTW-era system, and then building the tools to modify the data in that system, which is separate from any existing Unreal 3 systems. These are techniques we can utilize later to re-optimize the data, but the important part for now is that the data stays the same, and that the data loads correctly. You guys might remember the 'contrust-like' Enforcer trainee abomination from years ago. We're basically aligning the data to prevent 'creating' more of those, and to actually have the data appear as it was before.

Once the data loads correctly, we should have proper collisions. We'll do some quick fixes to the shaders and then run benchmarks.

Thanks,
Matt 

[MattScott 15242]

Hi all,

This week had some ups and downs, but overall I would say there's a net positive amount of progress going on.

 

We had an issue where loading Static Mesh data was crashing in some cases. That seems to be fixed now, which means the new format that has the additional data from 1.20 is working. We will have to see if the data itself works in the next couple steps.

 

From there, we have been in the quagmire of resaving all the game's content to import 1.20 3D objects and maps back into 2.2. We do that through a commandlet. These are small Unreal programs that don't bring up the editor. Right away we hit a crash once maps got fully loaded.

 

The crash is part of the new code that was pulled over related to lights. The engine needs to do a single 'update' to the entire scene to activate the correct lights. That update is currently crashing when the level is fully loaded. APB's handling of lights is entire custom to RealTime Worlds and deviates significantly from Epic's techniques. For the initial upgrade, we tried backing out the RTW changes going with standard Epic Games code. Now we are going back to the 1.20 technique so we can compare apples to apples in 2.2. Once we've fixed the crash, we can continue on with the package fixup process.

 

From there, we'll attempt to load maps and debug as-needed.

Thanks,
Matt 

[MattScott 15242]

Hi all,

Apologies about the lack of an update last week. There wasn't much exciting to post about. But my team sees you guys in Discord, and I know you're waiting for more news, so I'm going to try and provide that now.

We are still stuck migrating data. So far this week, we have focused on cleaning and stabilizing the codebase so we can take the next step of merging in data from 1.20 to 2.2. We fixed the issue related to Static Meshes loading properly, and then resolved an issue involving Octree data. In order to do this, we had to take several steps backwards. We ultimately decided to focus on getting the pipeline of feature data loading but with placeholder data, so we could make sure the resulting data works without crashing in-game. We were able to complete that work. It doesn't render yet because there is no feature data to render. But at least we know the underlying structure works.

 

Now we are working on porting over 1.20's feature data to fill in the blanks.

Here is where I made a pretty big assumption about the project overall. I assumed that over the last 10 years of engine changes, the previous devs always maintained backwards compatibility. Little Orbit constantly merged 1.20 packages up to 2.1 for the last couple years. But it appears that data between 1.20 and 2.2 aligns perfectly in some places and not in others. These are all in areas that never popped up before.

Our contractor is currently working on a solution to identify the version of the data we're loading so we can use the proper structure in memory. Basically, we know how the data is supposed to be aligned, and we know 1.20 data is saved all with the same version number. That means the data should convert one-to-one to the new format, but only if we add back in the original structures from 1.20 across the board - even they aren't used anywhere in the actual game. The process involves going through, line by line, about 200 separate blocks of conditionals and adding in the proper support. We have to pay close attention while we upgrade each condition because if the wrong comparison operator is applied, it will break the entire conversion process.

Once that process is completed, then we can finally extract the old data we need from the 1.20 packages and resave it in the 2.2 packages.

A lot of this work is purely code based. We aren't loading maps in the editor except to run commandlets or to test how something functions in the scene graph. So no screenshots yet. Our progress right now is measured by the lack of crash dumps and warning text on converting the Social district map. I'll grab more shots once we've made a bit more progress.

Thanks,
Matt

[MattScott 15242]

Hi everyone,

We made some good progress on the loading of older 1.20 packages this week.

 

We are still not able to load the Social district. However, I feel like we are very very close now. As a reminder, Feature Data was baked into the 1.20 maps for displaying static mesh actors and culling/rendering them under appropriate scenarios. That data was removed when the maps were upgraded to 2.0 (console) and 2.1 (Engine upgrade PC). While we have upgraded various packages regularly since the original work in 2014, the maps themselves were upgraded once way back then, and then permanently modified with new art. 

 

No attempt has been made to upgrade them from 1.20 to 2.x since then.

 

Here is some of the work we finished on the Feature Data this week:

• We're now parsing and loading the feature data correctly in most scenarios.
• We added support for alignment of lightmap data under 1.20. Byte alignment is a big problem we recently uncovered. The original 1.20 maps are mostly 4 byte aligned, but most of the new engine is 16 byte aligned. Of course, we cannot actually use the lightmap data stored in the maps. But generating new lightmaps should be trivial, provided we re-save the package and discard of the old lightmap data.
• The same goes for shadow maps. We've made progress on aligning the shadowmap data, and will treat it the same as light map data. We will remove that data and re-save when we generate lighting.

Another issue we have been sorting out are Prefabs. Prefabs are references in a map to instances of a static mesh. This was used heavily in 1.20 and removed for 2.x.  Unfortunately, in some cases now the saved Prefab references in older maps point to invalid engine classes. This coming week we are going to work on cleaning up all the bad references.

 

We're also utilizing a new test map that contains feature data in 1.20 to get the data ported over: The APB Login Level.

 

That single map, which can be tested by simply loading the game, contains examples for most of the data we're porting over. Most of it is off-screen and can't be seen by player, but the map actually contains parts of the Financial District and is small enough so that we have sample data to test without running into red herrings. Once we can successfully load this map into 2.2, we will try porting over the Social district from 1.20 again. Once that district loads properly with collisions, we will attempt to load the Financial district from 1.20. That will let us do another internal benchmark, and continue the work from there.

 

Currently, we don't see too many blockers in getting this task done, but it is worth acknowledging that none of this has been attempted before. There's bound to be ways it will break. Once again, there are no fun pictures to show the community yet. We're still aligning data and almost all of the work thus far is code work.

 

I'm feeling good about this week.

Ready to show more as soon as possible.

 

Thanks,
Matt

[MattScott 15242]

Hi everyone,

 

It's been nearly a month since my last post, and for that I apologize.

 

As I have mentioned previously, these updates are sometimes tricky to write when we are deep in the weeds fixing issues.

It always feels like we are right at the cusp of a big break through, so it's tempting to wait "just one more week" till we have something more to show.

 

I don't have something big to show - yet. But I do have significant progress to talk about.

 

I'll admit, the last 4 weeks have been a slog. We've hit very nuanced issues surrounding exactly how to load the new 1.20 data we were trying to preserve. These issues crash our conversion process. Each time, we spend nearly a day converting packages to 2.2, hit a crash, go back to the 1.20 code, step thought side by side, find the problem, work out how to fix it, and then re-run the lengthy conversion till it crashes again. Some of that nuance comes from the fact we simply can't take all of the 1.20 data packages. There are specific APB 2.x screens or console-based packages that we need to preserve and merge.

 

It seemed by the end of last week that we had finally addressed all the bugs, and that we might get a full map to load into the editor. Then we hit actual bad data in the 1.20 files. Empty meshes and broken references to missing 3D objects. The works. It is a mystery to me how APB 1.20 runs as well as it does. But each of these new problems would break as APB 2.2 tried to load those files. So this week we have been playing whack-a-mole with bad data and adding code to handle each case.

 

Today, the entire APB 1.20 Social map loaded successfully in the editor. No palm trees. No odd foliage. Just the live version of Social including all of the feature data we need to move forward running in the APB 2.2 editor. 

 

It was a pretty awesome day.

 

The team pushed hard to get Social up and running as an actual district in APB 2.2, but we hit an entirely new problem related to how levels are loaded at runtime. We will be addressing that first thing next week. Our next steps are:

• Get Social running at runtime so we can snap off a couple screenshots and run a benchmark against APB 1.20
• Run the conversion process on Financial and run a benchmark against APB 1.20
• Convert the rest of the maps.
• Re-bake all of the lighting with UE 3.5's Lightmass.
• Fix the main rendering shader so that lighting is applied properly for the special case light maps that Real Time Worlds created.
• Assuming the performance is where we want it, then we will get a small group of community testers onto our test servers. 

I know this has been a long process, but no one has attempted this new direction before. It's uncharted territory. As I have stated before, the original team at Reloaded felt it was impossible to port the game this way. But today we saw our first big sign of progress.

 

Hang in there with us. I have lots of plans, but they all rest on making the game sustainable. Then we can dig into the fun stuff.

 

Thanks,
Matt

[MattScott 15242]

Hi everyone,

 

Sorry for the delayed update. I know there's been a few weeks without an update. In general, I prefer to only share significant milestones with the community.

 

As I previously mentioned, our focus has been on making sure we can load the converted 1.20 Social District in our 2.2 build.

 

With a bit of software engineering work, we were finally able to get the Social district to load in the editor, though without proper lighting, shadowmaps - and importantly, collision did not work on the legacy feature data meshes.

 

APB is an extremely large and complex game. It has taken extreme discipline in addressing specific parts of the game that do not work.

 

When we go to fix a system, such as collisions, it's very easy to stray off-course and try and fix two or three other systems, only to have two more break in their place.
By hyper-focusing on feature collision, we were able to track down where the system lost its data, and also add in collisions automatically on level conversion.

 

As as a result of that effort, we now have collisions working in the map.

 

The team would like to share with you a few screenshots of the 1.20 Social District in our 2.2 client build. Veterans of APB will notice the lack of palm trees.

 

 

This is substantial progress.

 

We have a minor crash to address, and then the team will start work on the other maps and missing functionality.

 

Thanks,

Matt

[MattScott 15242]

Hi everyone,

 

Apologies to the community. It's now been 45 days since my last update. I have been in Europe for the last 5 weeks, and my schedule was been incredibly busy during the trip.

 

If I had to choose a single word to recap progress, it would be: "frustrating"

 

Back on October 2nd, we had a big breakthrough, and I was feeling really good about next steps.

You can see from the screenshots in that post that we had finally gotten collisions working on the sidewalk.

That update was posted minutes after we got a build that worked.

 

What we didn't realize at the time, was that if you stepped onto the grass, you immediately fell through the world again.

Additionally, several different types of placed props and static meshes in the map also had collision problems.

 

During the weeks that followed, we fell down a lot of rabbit holes attempting to find the issues.

 

While we could have individually made those objects conform to the new collision system, doing so would have been wildly unsustainable.

Once we start work on Financial, there are thousands of objects and checking everyone of the them would be a nightmare.

Instead, we chose to take additional time to step through line by line in the collision code on the current live version of the game versus the engine upgrade.

We need the conversion process to properly rebuild each map and all of the collisions in one pass.

 

After a bit of digging, we were able to identify the core issue for terrain, so that is working properly now.

 

As with all things APB, of course there are still issues. For some reason we have some oddball props and items that still aren't colliding properly. 

So the team is still working on restoring old 1.20 collision code to handle every specific case. 

 

Below is a video running around Social with nearly all of the collisions working. As I stated in the last update, ignore the lighting, we haven't turned that code back on yet, so everything is either fully lit or not lit at all. You can see once we get to the placed crates, those don't work. And the statues don't have collisions. As soon as we have those sorted out, we will reprocess all of Financial to see how that district does.

 

 

Thanks,
Matt

[MattScott 15242]

Hi everyone,

 

I hope you enjoyed the Thanksgiving holiday. We had some days off, but still made decent progress.

At this point everything in Social is working except for collisions on the skinned meshes like statues.

 

We started work on Financial this week as well. There are some issues with streamed assets that are getting loaded twice due to older and newer code running at the same time.
We'll dive into that next week.

 

For now, here's another video with progress.

Working collisions are:

• Joker Ammo vending machines
• Mailboxes (specifically, the spot in Social where people gather most often)
• Marketplace Kiosks
• Static Meshes
• Streamed Terrain 

 

Thanks,

Matt