MattScott

Hi everyone, Sorry for the delayed update. I know there's been a few weeks without an update. In general, I prefer to only share significant milestones with the community. As I previously mentioned, our focus has been on making sure we can load the converted 1.20 Social District in our 2.2 build. With a bit of software engineering work, we were finally able to get the Social district to load in the editor, though without proper lighting, shadowmaps - and importantly, collision did not work on the legacy feature data meshes. APB is an extremely large and complex game. It has taken extreme discipline in addressing specific parts of the game that do not work. When we go to fix a system, such as collisions, it's very easy to stray off-course and try and fix two or three other systems, only to have two more break in their place. By hyper-focusing on feature collision, we were able to track down where the system lost its data, and also add in collisions automatically on level conversion. As as a result of that effort, we now have collisions working in the map. The team would like to share with you a few screenshots of the 1.20 Social District in our 2.2 client build. Veterans of APB will notice the lack of palm trees. This is substantial progress. We have a minor crash to address, and then the team will start work on the other maps and missing functionality. Thanks, Matt

Hi everyone, It's been nearly a month since my last post, and for that I apologize. As I have mentioned previously, these updates are sometimes tricky to write when we are deep in the weeds fixing issues. It always feels like we are right at the cusp of a big break through, so it's tempting to wait "just one more week" till we have something more to show. I don't have something big to show - yet. But I do have significant progress to talk about. I'll admit, the last 4 weeks have been a slog. We've hit very nuanced issues surrounding exactly how to load the new 1.20 data we were trying to preserve. These issues crash our conversion process. Each time, we spend nearly a day converting packages to 2.2, hit a crash, go back to the 1.20 code, step thought side by side, find the problem, work out how to fix it, and then re-run the lengthy conversion till it crashes again. Some of that nuance comes from the fact we simply can't take all of the 1.20 data packages. There are specific APB 2.x screens or console-based packages that we need to preserve and merge. It seemed by the end of last week that we had finally addressed all the bugs, and that we might get a full map to load into the editor. Then we hit actual bad data in the 1.20 files. Empty meshes and broken references to missing 3D objects. The works. It is a mystery to me how APB 1.20 runs as well as it does. But each of these new problems would break as APB 2.2 tried to load those files. So this week we have been playing whack-a-mole with bad data and adding code to handle each case. Today, the entire APB 1.20 Social map loaded successfully in the editor. No palm trees. No odd foliage. Just the live version of Social including all of the feature data we need to move forward running in the APB 2.2 editor. It was a pretty awesome day. The team pushed hard to get Social up and running as an actual district in APB 2.2, but we hit an entirely new problem related to how levels are loaded at runtime. We will be addressing that first thing next week. Our next steps are: Get Social running at runtime so we can snap off a couple screenshots and run a benchmark against APB 1.20 Run the conversion process on Financial and run a benchmark against APB 1.20 Convert the rest of the maps. Re-bake all of the lighting with UE 3.5's Lightmass. Fix the main rendering shader so that lighting is applied properly for the special case light maps that Real Time Worlds created. Assuming the performance is where we want it, then we will get a small group of community testers onto our test servers. I know this has been a long process, but no one has attempted this new direction before. It's uncharted territory. As I have stated before, the original team at Reloaded felt it was impossible to port the game this way. But today we saw our first big sign of progress. Hang in there with us. I have lots of plans, but they all rest on making the game sustainable. Then we can dig into the fun stuff. Thanks, Matt

Hi everyone, We made some good progress on the loading of older 1.20 packages this week. We are still not able to load the Social district. However, I feel like we are very very close now. As a reminder, Feature Data was baked into the 1.20 maps for displaying static mesh actors and culling/rendering them under appropriate scenarios. That data was removed when the maps were upgraded to 2.0 (console) and 2.1 (Engine upgrade PC). While we have upgraded various packages regularly since the original work in 2014, the maps themselves were upgraded once way back then, and then permanently modified with new art. No attempt has been made to upgrade them from 1.20 to 2.x since then. Here is some of the work we finished on the Feature Data this week: We're now parsing and loading the feature data correctly in most scenarios. We added support for alignment of lightmap data under 1.20. Byte alignment is a big problem we recently uncovered. The original 1.20 maps are mostly 4 byte aligned, but most of the new engine is 16 byte aligned. Of course, we cannot actually use the lightmap data stored in the maps. But generating new lightmaps should be trivial, provided we re-save the package and discard of the old lightmap data. The same goes for shadow maps. We've made progress on aligning the shadowmap data, and will treat it the same as light map data. We will remove that data and re-save when we generate lighting. Another issue we have been sorting out are Prefabs. Prefabs are references in a map to instances of a static mesh. This was used heavily in 1.20 and removed for 2.x. Unfortunately, in some cases now the saved Prefab references in older maps point to invalid engine classes. This coming week we are going to work on cleaning up all the bad references. We're also utilizing a new test map that contains feature data in 1.20 to get the data ported over: The APB Login Level. That single map, which can be tested by simply loading the game, contains examples for most of the data we're porting over. Most of it is off-screen and can't be seen by player, but the map actually contains parts of the Financial District and is small enough so that we have sample data to test without running into red herrings. Once we can successfully load this map into 2.2, we will try porting over the Social district from 1.20 again. Once that district loads properly with collisions, we will attempt to load the Financial district from 1.20. That will let us do another internal benchmark, and continue the work from there. Currently, we don't see too many blockers in getting this task done, but it is worth acknowledging that none of this has been attempted before. There's bound to be ways it will break. Once again, there are no fun pictures to show the community yet. We're still aligning data and almost all of the work thus far is code work. I'm feeling good about this week. Ready to show more as soon as possible. Thanks, Matt

Hi all, Apologies about the lack of an update last week. There wasn't much exciting to post about. But my team sees you guys in Discord, and I know you're waiting for more news, so I'm going to try and provide that now. We are still stuck migrating data. So far this week, we have focused on cleaning and stabilizing the codebase so we can take the next step of merging in data from 1.20 to 2.2. We fixed the issue related to Static Meshes loading properly, and then resolved an issue involving Octree data. In order to do this, we had to take several steps backwards. We ultimately decided to focus on getting the pipeline of feature data loading but with placeholder data, so we could make sure the resulting data works without crashing in-game. We were able to complete that work. It doesn't render yet because there is no feature data to render. But at least we know the underlying structure works. Now we are working on porting over 1.20's feature data to fill in the blanks. Here is where I made a pretty big assumption about the project overall. I assumed that over the last 10 years of engine changes, the previous devs always maintained backwards compatibility. Little Orbit constantly merged 1.20 packages up to 2.1 for the last couple years. But it appears that data between 1.20 and 2.2 aligns perfectly in some places and not in others. These are all in areas that never popped up before. Our contractor is currently working on a solution to identify the version of the data we're loading so we can use the proper structure in memory. Basically, we know how the data is supposed to be aligned, and we know 1.20 data is saved all with the same version number. That means the data should convert one-to-one to the new format, but only if we add back in the original structures from 1.20 across the board - even they aren't used anywhere in the actual game. The process involves going through, line by line, about 200 separate blocks of conditionals and adding in the proper support. We have to pay close attention while we upgrade each condition because if the wrong comparison operator is applied, it will break the entire conversion process. Once that process is completed, then we can finally extract the old data we need from the 1.20 packages and resave it in the 2.2 packages. A lot of this work is purely code based. We aren't loading maps in the editor except to run commandlets or to test how something functions in the scene graph. So no screenshots yet. Our progress right now is measured by the lack of crash dumps and warning text on converting the Social district map. I'll grab more shots once we've made a bit more progress. Thanks, Matt

Hi all, This week had some ups and downs, but overall I would say there's a net positive amount of progress going on. We had an issue where loading Static Mesh data was crashing in some cases. That seems to be fixed now, which means the new format that has the additional data from 1.20 is working. We will have to see if the data itself works in the next couple steps. From there, we have been in the quagmire of resaving all the game's content to import 1.20 3D objects and maps back into 2.2. We do that through a commandlet. These are small Unreal programs that don't bring up the editor. Right away we hit a crash once maps got fully loaded. The crash is part of the new code that was pulled over related to lights. The engine needs to do a single 'update' to the entire scene to activate the correct lights. That update is currently crashing when the level is fully loaded. APB's handling of lights is entire custom to RealTime Worlds and deviates significantly from Epic's techniques. For the initial upgrade, we tried backing out the RTW changes going with standard Epic Games code. Now we are going back to the 1.20 technique so we can compare apples to apples in 2.2. Once we've fixed the crash, we can continue on with the package fixup process. From there, we'll attempt to load maps and debug as-needed. Thanks, Matt

Hi all, I'm way overdue for an update on the engine. We brought on a new resource to add some momentum and pick up several tasks from me. The new resource had experience with the engine, specifically this customized version used by Reloaded, but it's been quite a few years, and he's still getting caught up on APB specific changes. We will be filling out the team more as we go, but right now we're spending a bit of time to onboard. The primary issue during my last post were the collisions in Social, which we are using as a test level. After a number of red herrings, we determined some collisions are loading, but not inside buildings. This is because the old Building Feature code was migrated and is already working. On the one hand, this is great news, because the system appears to be working and we need it to remove 1,000s of objects from the scene that we were processing in 2.x. But right now this is bad because the Building Feature data is empty. It was created with a Real Time Worlds era tool and baked into the APB 1.x maps. If you recall that map data was wiped out during the migration from 1.x to 2.x years ago. Essentially, without the proper data, the game has no idea what building you are in or near, so it rejects loading any of the assets inside - including the collisions. We are current focusing exclusively on re-migrating the data from 1.20 to 2.2. That involves picking through and understanding that RTW-era system, and then building the tools to modify the data in that system, which is separate from any existing Unreal 3 systems. These are techniques we can utilize later to re-optimize the data, but the important part for now is that the data stays the same, and that the data loads correctly. You guys might remember the 'contrust-like' Enforcer trainee abomination from years ago. We're basically aligning the data to prevent 'creating' more of those, and to actually have the data appear as it was before. Once the data loads correctly, we should have proper collisions. We'll do some quick fixes to the shaders and then run benchmarks. Thanks, Matt

Hi all, Progress has been slow as I got into the weeds on a number of items. However, we are bringing on a full time specialist to help, so things should pick up pace. Thanks, Matt

Hi everyone, Rest assured the entire team is still here. I'll be more candid here. We followed the same strategy for over 2 years with mixed results. It makes no sense to keep following that same course and expect a different outcome. Instead, I came up with a new strategy that the existing team had avoided since the beginning of the port to 3.5. In fact, there is a segment of my team that feels my approach 'cant be done'. So I felt the need to take over this portion of the development and then hand the game back off. The work has involved tearing apart the base of the engine to re-implement several core systems from 1.20. That has meant having to go backward and get the data ported over again. This is why you're seeing all the rework. Thanks, Matt

Hi all, Doing a quick Tuesday morning update. I can now get into all of the action and fight club districts. There are a fair number of new bugs to work on. Collisions are the first thing I need to fix. Thanks, Matt

Hi all, It's been three weeks since my last update. I'll try to make this more comprehensive to make up for that. I'm going to start by acknowledging that I'm now leading and handling the majority of the programming for the upgrade on APB. That may already be obvious based on the detailed previous posts, but I figured it was worth publicly stating so you guys can understand some of the slowness. While APB is a priority, I only have so many hours in the day after running the company. There are a number of reasons why, but it really comes down to the fact that everything left to do is very very specific to APB. Simply knowing Unreal 3 or GPU shaders isn't enough at this point. No one else has enough knowledge across all the game's systems (art, shaders, etc). I still stand by this decision. The progress is farther than we've gotten in the last 2 years, but it has come with a cost to my own personal life and free time. The burnout is real. However, this is something I committed to, and I want to see it through so we can get back into full production. Also, I want to address the last couple weeks specifically. In general when Friday/Saturday comes around, I make an extra push to try and get to some screenshots for something more visual rather than these long, word heavy updates. Unfortunately, the last couple weeks have been spent in the weeds cleaning up bugs which prevented that. In every case, I felt tantalizingly close. So I held off on the update thinking that I could get the kind of progress I wanted before the weekend was up. But even with the extra push that never happened, so I ended up waiting another week, hoping for better results. With that out of the way, let's dig into progress. Before I jump in, I should note that Programming 101 says to try and introduce new code in discrete batches so you can test each piece and know that it works before moving on. This is especially important when bugs pop up, because you know exactly the code that was touched, and it's much easier to track things down. When you have to move code over in bulk, you end up not knowing exactly where to start looking when stuff breaks. I did my best to try and move bits over in chunks, but everything is so intertwined that it wasn't really feasible. Here are a couple of the "needle in a haystack" tasks that occurred over the last couple weeks. 1) Tracked down an issue that caused everything to render black (both in the Editor and in-game). This ultimately ended up as a bug in the primary BassPassPixelShader. But even when I fixed that issue, objects remained largely dark due to the missing baked lighting data. I have temporarily disabled the lighting calculations in the main pixel shader. Once I have finished a couple other tasks, I'll bake lighting and back out this change. For clarity, the game is doing all the lighting calculations so benchmarks will be accurate, I just short circuit the final output to make sure everything actually renders for now. 2) Tracked down an issue that caused all skeletal meshes not to render (both in the Editor and in-game). Skeletal meshes are any object that has bones we can animate. This includes not just player characters and NPCs, but many props and cars. The issue was related to some new transparency optimizations that had been badly ported over from 1.20. Effectively the game was treating all skeletal meshes as transparent, but then not rendering them when we render all the transparent objects. That is now fixed. 3) Tracked down an issue that was creating polygon soup in-game. I probably should have taken a screenshot of the login scene to show what this looked like, but I worried it would be too disheartening to you guys. Essentially "polygon soup" is when you have a serious problem with vertices and points for a mesh. It creates a bizarre scene of stretched polygons everywhere that look nothing like what you're trying to render. At the time, everything in the Editor rendered fine, so this was some in-game specific problem. It ended up being some bad code in an APB 1.20 system that batches mesh data to the GPU. It's a smaller system that I moved over to get the speed benefits, but there ended up being some DirectX 9 -> DirectX 11 bugs. Here is where that leaves us: All major systems ported over from APB 1.20 are working. All individual objects render in the Editor (albeit without proper lighting) The login scene renders fine (again with no lighting) I'm working on getting district maps loading properly in the Editor. This was working, but after fixing some code that supports the features attached to buildings, the maps now error because they are missing data. I need to go back and finish re-importing all the APB 1.20 content so I have the feature data that was lost. Once the district maps load, then I can spin up a district and do the benchmarks. This week, instead of showing off the almighty sphere, I figured I would include some behind-the-scenes shots that reveal a bit more about how the sausage is made. For a lot of the initial work a couple months ago, I was using a relatively small test level to slowly add objects and fix the bugs. This is setup so that I can actually spawn it and run around. This map only exists in APB 2.2, so it's not super useful for testing side by side with APB 1.20. After that I moved to an existing map in APB 1.20 that we use to test new gameplay. If you look closely, you can see some RIOT bombs leftover. It's not pretty, but it gets the job done. Thanks, Matt

Hey everyone, You've been more than patient, and you deserve a better update. As a company, we need to clear two projects off our plate to be able to dedicate proper resources to my plans for Fallen Earth. Having said that, I have started looking at intermediate ways I can put things online for you guys to experiment with. Please don't over hype this. I just realize this is taking longer than I planned, so I'm looking for near term solutions. Thanks, Matt

Hi all, Back to being frustrated. Spent the week deep into shaders and working out some kinks that led to mismatched inputs and outputs between Vertex and Pixel shaders. From the outside, you would think this would be trivial to solve. But APB generates shaders for every possible unique combination of lighting, rendering, fog, and other variables. It's a lot. So the effort this week was like finding a needle in a haystack. Run a map, catch the crash, figure out which object it is and what material was attached, and then examine why those shaders aren't in sync. More next week. Thanks, Matt

Hi all, I hope you had a great Mother's Day break. I missed my update yesterday. We made good progress last week. Serialization issues are fixed. Objects load again, and we're troubleshooting lighting now. We need to finish up one more piece, and then we will reimport all of the 1.20 packages. I had thought we would have screenshots this week. But nothing was as impressive as the Sphere shot from before. Thanks, Matt

Hi all, I was pushing right up to tonight to see if we could get something to show for the progress this week. But I want to stick with at least an update every week, and I'm out of time. Oh well. We're quagmired somewhere between re-compiling the last of adjusted shader code, sorting out an optimization that relied on 3 lighting coefficients so APB can store light tangent vectors in the alpha of each one, and serialization issues nested deep down in LOD data for static meshes. The work continues. Thanks, Matt

Sorry for the delayed response. 2.0 has the old version. We upgraded it once we started 2.1.