APB 1.30 64-bit upgrade

[MattScott 15242]

Hi everyone,

I've locked this thread similar to our original Engine Upgrade thread, so that I can post on a regular basis for people to follow.

This week I'm traveling to Europe in prep for Gamescom next week. Then I'm staying over another week to wrap up business there before I return in early September.
With that in mind, I'm setting a goal of posting every 2 weeks.

 

Right now, we’re progressing nicely through with APB 1.30 (also known as the 64-bit port of APB).

We’ve been monitoring feedback after my blogpost came out, and I get it. There is a lot to be frustrated with ‘yet another engine upgrade’ and the lack of tangible progress for players. 

Here is my update this week on exactly what actual work has been completed:

1)     We have finished porting most of the core engine functionality, excluding APB-specific gameplay code, over to 64-bit. This involved a lot of different changes, most of which are backports from the 2.x code, however, some are entirely new changes to the codebase so that 1.20’s functionality is fundamentally preserved. The engine code had thousands of errors and warnings that we had to work through. At this point, I'm assuming that any warnings are really errors that need to be resolved before we ship 1.30.

Here are specifics for some of what we had to do:

• Set up projects to have a consistent struct member alignment across the entire project (backport from 2.x)
• Ensure compatibility with the VS2017 toolchain.
• Identify and set up versions of third-party libraries, such as Scaleform and PhysX, that will suit a 64-bit client build of the 1.20 codebase
• Changing hard-coded, casted pointer types from implied 4-byte pointers to implied 8-byte pointers.
• Clean up declarations of local variables hiding class variables.
• Clean up declarations of local variables hiding other declared local variables.
• Modify the codebase to support newer versions of specific middleware libraries and ensure functionality is not compromised

2)      We’re in the process of cleaning up the ‘core’ components of APB, such as the middleware made specific for APB. This is mostly complete but may need further adjustments.
 

3)      We are beginning to do the same conversion process for the Unreal Editor components.

 

Our next steps will be to finish converting the Unreal Editor components, and the APB-specific components. Once that process is done, we will attempt to launch the Unreal Engine Editor and attempt to get a map loaded in-editor. This was the same strategy we successfully implemented for the APB 2.x upgrade before we started beta testing and realized that the underlying implementation in 2.x was never going to perform very well.
 

This time, we have taken the right steps to ensure (crosses fingers) that the Editor process will go smoothly. My hope is that there will be minimal time from getting a map loading in-editor to having it running in our actual client/server setup. That will allow us to do some performance and stability testing on both the APB client and its district servers. I'll be looking for opportunities to share ‘client havoc’ as it comes up, as that seems to be a community favorite.

It's been a little over a week and a half since my last update.

I'm hoping to do my next update in 2 more weeks once we are in the editor resolving issues with APB’s Social District’s map.

 

Thanks,

Matt

[MattScott 15242]

Hi everyone,

It’s been exactly two weeks since our last update.

Our progress continues with APB 1.30, and I promised to give more frequent updates on a bi-monthly schedule. My hope is that keeping that schedule will provide some insight into how the team is coming along.

Firstly, we have cleaned up all of APB’s dependency projects to compile under 64-bit. For the most part, we tracked down the appropriate libraries that would work with 1.20’s codebase out of the box in 64-bit, and in some cases, we modified existing libraries ourselves to fit the older version of Unreal. Between middleware integrations (such as Scaleform and Wwise), a rewrite of some 32-bit specific optimizations and texture utility code written in assembly, APB’s custom Music Studio and Web Browser libraries, Windows API support, and APB’s User Interface and Editor modifications, there has been plenty of work completed.

A total of about 3,000 warnings and errors have been resolved to date.

Next, there is only one more project to finish converting to 64-bit – the gameplay code for APB itself.

After the entire codebase compiles without warning or error, we will have to resolve issues that may arise from the process of linking the projects together. And then we can fire up the Unreal Editor and load a district. From there, we expect to find problems with serialized package structure alignment similar to what we encountered with the 2.x upgrade. Since most of these issues were already resolved in 2.2, I'm hoping we can reuse that work to do an ‘in-place’ upgrade of APB 1.20’s serialized content packages to support 64-bit architecture.

Even though we missed our previous mark of wanting to be in editor attempting to load packages by the next two-week update, we still completed a significant amount of work in a small amount of time, and we have not encountered any unknowns that we haven’t overcome before.

I’m hoping that we will be in-editor resolving issues with APB’s Social District map for the next two week update.

Thanks,
Matt

[MattScott 15242]

Hi everyone,

 

It’s been another two weeks since my last update.
Here’s another update on our schedule.

We did pretty well.

 

At this stage, every project compiles and links under 64-bit without warnings or errors!
This is a significant milestone, as it means most of our time will no longer be spent merging in code, but instead debugging and getting the editor, game, and servers running.

We are now at the ‘get the editor running’ stage of the build process.
This involves resolving startup / initialization issues with dynamic link libraries, initialization, and instantiation of core engine components.

Then, we will ensure that we can rebuild and regenerate APB: Reloaded’s Unreal scripts.

After that, we will deal with existing disk-serialized package misalignments that would prevent maps from loading.

So right now, the focus is on instantiation of core engine components.

When we have all the items above sorted, then we will be able to attempt to load the Social District map.

Thanks,
Matt

[MattScott 15242]

Hi everyone,

It's that time again - we have reached another two weeks since my last update. 

We are still at the ‘get the editor running’ stage of the build process.

The script generation commandlet needed some unexpected additional work.
We identified several new issues while building the scripts mostly related to serialization of classes. 

SIDEBAR "What is Serialization?"

I don't like using jargon that isn't easily understood by everyone, so since this update is pretty short, I'll explain what "Serialization" is. At a high level serialization is the concept of translating structured memory into a format that allow us to store it - more specifically APB data is stored in files on disk. Structured memory is another way of saying data structures or in-game objects. These are things like 3D mesh data, Gun stats, or data that controls how parts of a district show up in a mission. The critical (and exhausting) part of serialization is making sure that each member of a data structure is translated at the proper size into a format that can be reversed back into memory giving us the same original value. There are a lot of different reasons why sizes can be problematic or truncate data such as 32-bit -> 64-bit conversions that might have smaller or larger definitions of a memory block. This is made worse by the fact that many parts of APB had hard coded block sizes set at the top of the data structure. So we have to go in and find every one of them to make sure they are all handled the same.

END SIDEBAR

And of course... there are lots and lots of data structures for APB (and Unreal).

To address some of the issues we found, we spent some additional time pulling across relevant code sections from the 2.x upgrade back to 1.30.

Our plan has not changed, but this process of pulling across code is still going so that serialization works properly.
Then we can rebuild and regenerate APB: Reloaded’s Unreal scripts.. and then the editor will load.

Our current benchmark is still to get APB’s Social District map loading.

Thanks,
Matt

[MattScott 15242]

Hi everyone,

Tuesday is here, and it's time for another update.

 

We’ve completed the work to get the editor’s script generation commandlet working.
 

We’re in the process of making the UnrealScript ‘source’ files match the headers we’ve manually modified in various places of the engine so that 64-bit support will work. This process is honestly more complicated than it needs to be, but that's just the way it is working with this version of Unreal. Essentially, it's a chicken or egg problem. Unreal compiles scripts to generate mores source code. But the old scripts wont compile with the new version of our 64-bit source because they are incompatible. So we have to create all source code that would normally be generated to get the source to compile. Then we can process all the scripts which will replace our manually generated code with the correct version.

It's a complicated process filled with potential ‘rabbit holes’, but at least it's mostly straightforward and mostly grunt engineering work.
 

We’ve also completed the work to regenerate the “Core” project’s scripts.

As of Friday, we ran into an issue parsing the source version of the script themselves that we’re investigating that currently results in a crash. Unfortunately, due to the age of the engine and the way older Unreal outputs text to logfiles, this has been a particularly painful issue as logging doesn't finish dumping once a crash occurs. So there's no help there. We’re currently in the process of identifying that crash, using a combination of three elements – working copies of the codebase that function as intended, debugger output messages, and our gut intuition on what the problem may be (insert all the community jokes here about how bad our gut intuition is).

Once this issue is resolved, we will likely have to fix the rest of the project’s UnrealScript package compilation. As I mentioned above, these UnrealScript files will be processed to generate native c++ source code that needs to match the files we’ve manually merged into the 1.30 codebase. It's a crucial step so we can continue forward and launch the editor.

We’re hoping to be done with the UnrealScript portion of 1.30 by the end of the month.

After that, we can begin work on loading existing content packages.

Our current benchmark is still to get APB’s Social District map loading.

Thanks,
Matt

[MattScott 15242]

Hi everyone,

 

It’s a very spooky Tuesday in San Paro, and with the event comes another update on the progress on 1.30!

We’re still working on crashes since the last update.

We’re in the process of narrowing those crashes down and have a few leads on the root cause.

Unfortunately, this crash is about exciting to share with you guys as it is to fix, and it's also a blocking crash. The rest of the 64-bit upgrade is dependent on resolving this crash.
 

For the moment, I have our most senior engineer working through the issue. The nature of the crash itself means that we need to complete the fix before the rest of the project works. Allocating multiple team members to an engineering task like this does not speed up the work in question. Other APB team members are working on creating new rewards and enabling the Halloween event, which is scheduled to go live later this week, while the team allocated to working on 1.30 continues their work.
 

We’re still progressing.

Once we get over this irritating hurdle, we will complete the rest of the tasks necessary to get the Editor loading.

 

Still hoping that once we have the Editor loading, the rest of the components should ‘just work’, due to the nature in which we have been careful while upgrading the engine.

NOTE: I've received a community letter from a group of the players that I'm working on addressing in a much larger post. We're working through the strategy of possibly splitting each section into separate forum threads that the community can weigh in on, and I'll do a live AMA as part of my response. Stay tuned.

Thanks,
Matt

[MattScott 15242]

Hi everyone,

 

The last couple weeks had unfortunate timing as several of our staff had vacation time scheduled, and we were blocked on a fairly tough crash.

 

However, everyone is back, and the team has resolved the blocking crash in 1.30 as of Monday of this week.
 

We’re continuing progress on the remainder of 1.30’s UnrealScript changes.

We hope that we can quickly move past these changes to loading the Editor and working to ensure existing packages load correctly.

 

Other members of the team aren't back till tomorrow, and there was a problem with removing Halloween, so we will be extending that event one more week. 
 

As mentioned in the previous update, I'm still working through a series of forum threads and scheduling a Live AMA with the community to walk through the community letter from several concerned APB community members. We’ll be sure to post information regarding that AMA so you can tune in and participate.

Thanks,
Matt

[MattScott 15242]

Hi everyone,

 

It’s been another 2 weeks, and it’s time for an update.

 

We’re progressing through UnrealScript crashes. We’ve managed to complete script serialization work for all non-APB packages, and now have moved to APB-specific UnrealScript. This is great progress, and we managed to get as far as trying to load an Audio package as part of serialization this week.

 

The team ran into a 32-bit/64-bit alignment issue with the Audio package. We believe we have a grasp on the issue now, and it should be addressed shortly so we can keep moving forward. UnrealScript compiling and serialization is required to get into the Editor. The game primarily runs off "packages" which contain all the assets and scripts. But since the UnrealScript serialization process includes all of the package serialization, it’s my hope that the team will not need to spend as much time getting other types of packages loading.

 

Also, I'm running a bit behind on the community letter. I promise I'm not avoiding it. I just ran out of time before I left for vacation. (It was important to me to keep our cadence of updates, so I took time away from my break to post this.) I am hoping to get the community letter threads up as soon as I'm back in early December. Then we can do the AMA after everyone has had a chance to engage on the threads.

Thanks,
Matt

[MattScott 15242]

Hi everyone,

Time for another update, and this time it's a bit more exciting.

Going into last week, we had hit a number of irritating issues with byte alignment blocking our progress, so our engineers brainstormed through a task to revert a slice of our changes and focus on keeping only the memory management from the 2.x. This meant going back and revising code to try and stay inline a bit closer to 1.20 - except in situations where 64-bit or VS2017 compatibility was required.

We worked on that task last week, and it proved to be very successful. The code is much more reliable than what we were trying to do before. We were able to complete the serialization and UnrealScript building part of the 1.30 upgrade on Monday. Now that we’re finally able to build the game’s UnrealScript packages, we can finally begin the process of attempting to load the Editor and the game’s content.

I'm happy to report we are finally booting the Editor and working through ensuring shaders build properly!
I'll report more progress next time.

It's worth noting that separately I have created a series of forum threads for the Open Letter that was sent to my team. The plan is to work on responses to each issue and then schedule a Live AMA with the community soon. 

Thanks,
Matt

[MattScott 15242]

Hi everyone,

 

I couldn't think of a better way to illustrate what stage we just hit with 1.30, so I'll just leave this here.

 

 

Thanks,

Matt

[MattScott 15242]

Hi everyone,

It’s been another 2 weeks since my late update (1 week since the sphere), and what a two weeks it has been!

First, I hoped everyone enjoyed the AMA today. We had over 1,000 unique viewers who tuned in to listen to Frosi and I, ask questions, and get a sneak peek at 1.30.
I want to personally thank everyone who showed up. If you’ve missed the AMA, you can watch the VOD on Twitch here:
https://youtu.be/K2MGdEOhbKY
 

As I mentioned earlier on the livestream, we’ve made it past the Editor stage and into the game.
 

Next, during the last 2 weeks, our team has completed the following (in order of completion):

• Fixed a critical bug relating to DirectX 9 and 64-bit compatibility while running under Visual Studio 2017
• Ability to launch and run the Editor using 1.30’s codebase
• Ability to use Editor tools, such as the Skeletal Mesh Viewer, and the Static Mesh Viewer
• Built a sphere
• Load Editor assets properly
• Save Editor assets properly
• Load District Maps in-editor
• Fixed a few merge-related issues related to backporting 2.x memory work into 1.30
• Re-baked Lighting for 1.30 within Social and Baylan Shipping Storage
• Re-built PhysX meshes for all assets
• Completed PhysX compatibility work
• Ability to run a test level offline
• Ability to enter the test level networked
• Created a workaround for compressed textures that resulted in player character meshes appearing transparent
• Ability to enter the Social district networked
• Ability to enter Baylan Shipping Storage networked
• Implemented a fix for a crash issue in DirectX related to a depth pass shader.
• Ability to enter the Financial district (!!)
• Implemented a fix for traffic lights which was causing a crash when recalculating whether a car could pass through a light.
• Implemented a fix for pedestrians which was causing a crash.
• Implemented a fix for various allocation issues within the audio subsystem that prevented the game from playing specific sounds due to being out of memory.

Needless to say, I am extremely happy. There was no way I could have planned for us to have video ready for today's AMA. Just saw more progress than I expected in the last two weeks.

We are now in Financial running tests. You can see the footage here:
https://youtu.be/MqQUjXlITJI
 

Our remaining issues are as follows, and will be tackled in the following order:

• Get the video player working
• Re-bake lighting for all maps (Only Social, Financial, and the Baylan Shipping Storage districts have been fully baked)
• Test missions and ensure they work
• Implement the crash dump reporting system from 2.x
• Upgrade the production configuration for the game’s code and assets and make a "Shipping" binary that has all the optimizations that are in the Live build.
• Get the testing environment back online

Our target is to get 1.30 in the hands of our beta testing group no later than February. As mentioned in the AMA, we may also opt to remove the NDA for 1.30 testing so more people can see results for themselves on Twitch in a couple matches between testers.

As mentioned elsewhere on the forums, San Paro will have its annual Holiday event starting tomorrow!
You can get all the details about that event here:  https://forums.gamersfirst.com/topic/20504-holiday-event-2022/

 

We hope you enjoy the Holiday festivities into the new year, and hope you are as excited about 1.30’s progress as we are!

Thanks,
Matt

[MattScott 15242]

Hi everyone,

 

I am traveling this week in Europe, so my schedule is a bit off, but I think I can just barely squeak this update in on time.

 

We’re chugging along with 1.30. Every map now has lighting built, and we have been able to get into all districts in our production (shipping) build.

 

If you missed my Twitter update, we did a very quick comparison of APB 1.20 (LIVE) and 1.30 (the x64 Upgrade) and there are some solid performance improvements, notably with regards to stutters. So good news there.

 

The following tasks still need to be completed before we can ship 1.30:

• Integrate 1.30 into existing Continuous Integration systems for automated builds.
• Find and fix an issue relating to specific localized strings in production builds.
• Resolve an issue relating to a third-party library responsible for loading intro movies.
• Test EasyAntiCheat integration (yes, EAC is back. I mentioned this in my AMA)
• Ensure the crash dump reporting system from 2.x is implemented and works as intended.
• Create the 1.30 environment that community beta testers will use.
• Perform internal QA on the 1.30 build.
• Deploy & test the build with community testers.
• Process regressions encountered during the community test.

 

The build for our community beta testers is still a couple weeks away. Depending on the severity of the issues encountered, we may opt to keep the NDA in place for some of the early tests. But we are excited to get 1.30 in your hands as quickly as possible, so the team is pushing as hard as they can.

 

Thanks,

Matt

[MattScott 15242]

Hi everyone,

 

Time for another 1.30 update!

 

We’ve made progress this week on various tasks from the last two weeks.

We created a workaround for the localization problem we encountered and have continued progress on all other tasks.

 

Out of those tasks, the following need to be completed before we can ship 1.30:

 

• Fix an issue where a crash occurs on exit during the build process. (This is an irritating bug, because our continuous integration build system thinks the build didn't work - but it did. Its some other crash at the end of the process throwing things off)
• Resolve an issue relating to a third-party library responsible for loading intro movies.
• Test EasyAntiCheat integration.
• Deploy the builds from the build server to a 1.30 environment that Beta Testers can use.
• Perform internal QA on the 1.30 build.
• Deploy & test the build with Beta Testers.
• Process regressions encountered during the Beta Test.

 

We still intend to get a build to our Beta Testers on schedule. As previously mentioned, we may opt to keep the NDA for the test.

Thanks,
Matt

[MattScott 15242]

Hi everyone,
 

We have a very exciting announcement this week.
 

First, a majority of 1.30 work has been completed. We have run tests internally, and things seem to be holding together.
Second, a test server is now online for Beta Testing the 64bit upgrade. We had a group of community players in test districts yesterday.

I am going to be making a separate post on how to join this effort, but here is a quick rundown.

 

Notes for the test:

• Effective this evening, we have opened access to all players. There is no whitelist or NDA for the test. Anyone may download the client and provide feedback on the forums.

• You can hop on to OTW by downloading the launcher here: https://apb.patch.ot.gamersfirst.com/APB_OTW_Installer.exe

• Streaming and open discussion of the test is allowed as of this post.
• EasyAntiCheat is currently present – but we are using Battleye for the time being. As previously mentioned, our plan is to integrate EasyAntiCheat as quickly as possible.

Known Bugs:

• If you are encountering issues running the game, run the launcher’s repair tool.
  • There is an issue with BattlEye and multiple APB directories that running “Repair” on the OTW launcher will fix.
• There are known issues involving doors respawning in specific parts of maps.
• There is an issue with Joker Ammo machines repeating audio.
• Baked shaders and lighting may be incorrect in certain areas of the maps.
• A highly specific crash related to the music studio is known to exist.
• A highly specific crash related to the clothing editor is known to exist.

Provided there are no major blockers during the test, we plan on cutting over to LIVE with 1.30 as soon as possible.

We hope to see you in-game testing!

Thanks,

Matt

[MattScott 15242]

Hi all,

I'm going to unpin this thread since we're moving onto the next phase of development.

I appreciate everyone who followed along on this part of the journey.

 

Thanks,
Matt