MattScott

Hi everyone, Lots of work this last week. We'll be making an announcement soon. Stay tuned. Thanks, Matt EDIT: The 2nd Open Beta has been announced for August 1st. https://www.gamersfirst.com/apb/news/2020/7/27/second-open-beta

Hi all, I am very pleased with the CS team. Our 3 new agents alongside everyone else have dug in and finally gotten our tickets under control. We still have a ways to go, but I'm feeling a lot better. As of today we have: 85 new tickets (a massive drop from last week) 245 total tickets (another big drop from last week) We are responding to tickets submitted on 7/14 which brings the wait time to down to 10 days. Thanks, Matt

Hi everyone, It's been a while since my last update, and several members of the Fallen Earth community have asked for more frequent posts. The team at Little Orbit is very busy with several high priority tasks right now, and my schedule is the worst out of everyone. Unfortunately that means I can't commit to monthly updates right now, but I will do my best to post more often. I do need to reiterate that out of the 3 projects going on in the studio, Fallen Earth is currently the lowest priority. I expect that to change as we get closer to the end of September or early October. I know that's not a fun thing to hear, but I want to set expectations correctly. Work on the project is continuing on a number of different fronts. In May, I laid out the short term road map of areas we are working on which includes finalizing terrain, getting characters up and running, and implementing basic movement. There is a good amount of work in progress on those items, but much of it is behind the scenes which means I don't have good visuals for them. One of the biggest setbacks, is that we found a number of places in our conversion code that were mishandled because we didn't fully understand the proper logic needed. The conversion code is critical to all of the immediate areas we are working on, because it is responsible for migrating assets from the old Icarus proprietary format into modern day formats that can be used by Unity. Simply put, the more we looked at objects that were moved across, the more we found they looked very odd. Some where see through in places they shouldn't be. Some where shiny in places they shouldn't be. Here is a quick primer before I continue. If you're not interested in the technical bits, feel free to jump down to the Texture map examples below. Most games use a model that looks like this Shader->Material->3D Model. Shaders define inputs and how lighting will get applied. Materials define texture maps, colors, and other properties that are passed into the Shader they are linked to. Texture maps can be Grayscale with 1 channel (black), RGB with 3 channels (Red, Green, Blue) of information or RGBA with 4 channels (Red, Green, Blue, Alpha). And finally, all in-game 3D Models (Objects) have a Material assigned to them. When Fallen Earth was created, they only had Fixed Function pipelines to work with. That means their Shaders were hard coded in C++. Nowadays Shaders are much more diverse. They can operate at the geometry level (Vertex Shaders) or they can operate at the pixel level (Pixel Shaders). And they are mostly written in High Level shader languages like HLSL. The original FE Shaders are split up and named for the lighting model they render such as: Normal Tangent (or DotBump) = materials that will have a Normal map Gloss = materials that will have reflections Alpha Test = materials that will have sections that are see through Fallen Earth also used an older set of Texture maps for all of its Materials. This system attempted to "cheat" the look of real lighting across an object. Diffuse = all color for an object including lighting and shadowing Normal = detailed pixel by pixel curvature data that gives low polygon surfaces much more detail Specular = this is only a grayscale image that defines the specular reflectivity of the object and a couple more Today we use custom Shader Graphs or the more standardized Physics Based Rendering approach (https://en.wikipedia.org/wiki/Physically_based_rendering). PBR attempts to model how light really works as opposed to cheating the look and feel. Unity offers two approaches to PBR: Specular or Metallic Since Fallen Earth already had Specular texture maps, we chose the Specular implementation. This uses different texture maps such as: Albedo = similar to Diffuse with all color for an object but without any shadowing or lighting Normal = detailed pixel by pixel curvature data that gives low polygon surfaces much more detail Occlusion = grayscale texture showing ambient shadows that would occur without specific lighting Specular = RGB texture showing both the color that is reflected and the degree to which parts of the object absorb light or reflect Smoothness = grayscale texture showing what parts of the object are rough versus glossy Now back to our work and some examples. We assumed the Fallen Earth Diffuse texture maps would have coloring in the RGB channels and alpha in the Alpha channel, because this is the default setup in PBR. However this is not the case in a majority of the Fallen Earth materials. In retrospect, this should have been more obvious to us as very few Materials require alpha or transparency. Most objects in the world of Fallen Earth aren't see through. So having empty Alpha channels everywhere would have been wasteful. Instead, they used a "flags" field in each Material to let the system know what the Alpha channel data in the Diffuse texture map would be used for. We found that in nearly 80% of cases, the Specular grayscale data was embedded in the Alpha of the Diffuse texture map. So more recently we had to go back through all the imported texture maps and split out the Alpha channel into a separate image. This allows us to merge that channel into other texture maps to migrate everything properly depending on what that alpha channel actually contains. I'm going to walk through samples of texture maps for the Heavy Assault Battle Suit (this is what it's called in the data - but necessarily what it is called in-game). It contains Phoenix Plate, pants, and boots. For this part, I think it's handy to see the final product of what we imported into Unity to give you some reference on what you're looking at. Look at the red heart painted on the left side of the chest. It will help you figure out what you're looking at later on. In our previous attempts, all of the shiny areas on this model were transparent. I'm sure you can imagine how weird that looked. Next, here is a shot to give you an example of what this suit looks like without any texture maps. Notice the lack of detail and how flat everything is. Here are the texture maps. Diffuse (with the Alpha removed): Notice the red painted heart on the right side. That part of the texture appears to be mapped backwards to the chest so the heart shows up on the left side and not the right. Diffuse Alpha which turned out to be the Specular texture map: White areas have the most shine while black areas are the most dull. Normal texture map: This looks a bit bizarre to most people, and it should. The Normal map isn't painted like the other maps. It's generated from high polygon versions of the same model. Someone very smart decided lower polygon objects could look much better if we mapped the curvature in X,Y,Z values from higher polygon objects on a per pixel level to the R,G,B channels of the Normal map. All the detail you see in that top image comes from this texture map. As part of our conversion process, we alter the Diffuse to remove shadows and lighting (as much as possible) to create the Albedo texture map used by PBR. We got lucky with the Diffuse texture map in this case, because it didn't have a lot of lighting or shadows painted in. And finally here is an example of what the new Occlusion map looks like. As we continued work on the Character System, we came across another recent problem. Fallen Earth has a relatively complex clothing system, and we found that sometimes our new clothing or armor pieces would interpenetrate to show the body parts poking through. This was because we never imported a set of data on the body called "Selection Sets" (we call these Mat Ids today). This data splits up the body into 17 different pieces that can be shown or hidden depending what you're wearing. To fix this, we had to go back, correct the code to import the Selection Set data, then split the meshes up properly, and finally re-import everything back in. Here are some examples of how the Selection Set data is used to show or hide bits of the body depending on what you are wearing. The full Male body looks like this. NOTE: no custom skin color has been applied yet, so it's pure white right now. And here is an outfit assembled from random pieces of clothing we imported (don't judge me for my fashion sense ) Here is what the body looks like using the Selection Set data if I hide the outfit. You can see how areas covered by clothing are hidden so they work properly. Lastly we also went back and captured some data that allows multiple pieces of equipment to be created from a single 3D model. This part of the system uses the definition of equipment items and allows them to show or hide parts of the model to make it more unique. As an example, here is the same Heavy Assault Battle Suit, but the pants, boots and gloves have been hidden to create a Jacket instead. That's it for now. We're going to keep working through all the equipment, heads, hair, weapons and accessories so that we can reconstitute a character properly, and then we'll start work on the basic character movement. Thanks, Matt

Hi everyone, Running a day behind on this. In the last update I laid out the three major areas and we continued to work on those areas this week. Unfortunately our lead on the Login/World Server issue was on vacation. However the other team members continued that work and may have come up with the fix. I'll need the lead to go back through the work and determine whether the analysis is correct. Debugging work continued. The library we were working on now compiles and runs, but we're still working through errors that come up. On the Multi-threading, we are still working through Occlusion. That's a bit of a rabbit hole unfortunately due to some memory that needs to be accessible across threads. Thanks, Matt

Hi all, Missed posting this last night. We had another strong week. I feel like our new agents are working out well. As of today we have: 207 new tickets (continuing to drop from last week) 421 total tickets (also continuing to drop) We are 1 ticket away from responding to tickets submitted on 6/29. This means as of yesterday we are responding in 18 days. Thanks, Matt

Hi everyone, This week was a bit frustrating again. All of the issues currently blocking the Open Beta are based on bugs in code or systems that were originally created ages ago. Worst of all, the areas we are working on are... boring. That means no sexy graphs or cool pictures to highlight. Instead I'm going to run down the list of issues we are working on and try my best to explain them: Login/World server issues handling a lot of players - We spent last week excavating some stress test tools from 2012. We got them working in the middle of this week, which enabled us to start testing why the vast majority of players weren't able to log in or stay connected to the world server during the Beta. We have already verified that the hardware and installations between Live and the Beta environment are exactly the same. Moving to new hardware or reinstalling made no difference to the connection issue. In APB 1.20 (Live), our Login and World servers handle lots of spikes when a bunch of players try connecting at the same time. It also appears that APB 2.0 (Live Consoles) handle load just fine too, although the concurrency on consoles is a lot lower than PC, so there is a possibility that a less severe version of this issue exists there. But there is definitely a problem with APB 2.1. Any time we try to get more than about 50 players trying to connect, we start hanging sessions. Our best guess is that we carried over some piece of code from 1.20 somewhere in the last 1.5 years that appeared to work during our small SPCT/closed beta tests, but fails at scale. The team is continuing to review code step by step to fix the issue. Problems debugging crashes submitted by Open Beta players - This appeared to work fine with our small testing group. In the past, we have noticed small inconsistencies from time to time, but we were still able to get enough info to fix issues. But crash reports submitted during the Open Beta showed lots of inaccurate call stacks and debug info. This has made tracking down bugs much more difficult than it should have been. I can't really get into details on why this system is so complex. It just is, and we're stuck with it. So we have started rewriting code from 2015 that took several developers 4 months to complete. Similar to the first issue, this task is "open ended", which means it's difficult to put a time estimate on since we have to work through each line of code to figure out what is failing. With all of that in mind, we have been making good progress, and it doesn't appear like it will take too much longer to fix. Hoping for a better update next week. Crashing fixes / Multi-threading fixes - This is also ongoing. We feel like we've fixed the majority of crashes from the Open Beta, but some pesky DirectX11 crashes remain. To address those we are going to kill two birds with one stone. The team is moving ahead with some multi-threaded work that we had put off in order to get the Open Beta out. We believe these remaining crashes are caused by a lower level issue in Unreal 3's task handler related to shared memory across threads. The base code for Unreal 3 is nearly 20 years old now and even the newer sections are around 8 years old. Our solution is to use a newer, more powerful library that has become the industry standard for this type of work. We started implementing that library about 3 months ago because it is very stable with more sophisticated ways to schedule tasks and ramp up processing on more powerful machines, while also being much more efficient on lower end machines. So in theory we can fix the remaining crashes while also improving performance. However we can't just "plug it in" for an easy win. The code works substantially different to Unreal's code. So each area where multiple threads are used has to be individually migrated to the new library. The team is moving quickly, and we've already converted some of the bigger systems like Shadows and Occlusion. As soon as we have the rest converted, we can start testing this again internally. Thanks, Matt

Hi all, This week was much better. We are still training a couple new customer service agents, but they are all picking up speed. As of today we have: 285 new tickets (a lot lower than last week) 498 total tickets (a lot lower than last week) We are responding to tickets submitted on 6/17. Thanks, Matt

Hi everyone, I missed the update yesterday with the holiday. Last week was a maintenance week. That meant we backtracked over existing code and systems to diagnose the issues from the Beta. Part of the team resurrected a series of old stress tester applications. We've almost got them working for the Login and World server. That should let us try and narrow down the bad server problems. We also fixed a number of the big crashes. There will be more this week, as we dig into one section of troublesome multithreaded code. Hope everyone got a nice break over the weekend. Thanks, Matt

Hi all, We finally sorted out some internal networking issues on Tuesday that were preventing all CS from working. This was a short week for us with the holiday, so the numbers aren't great. As of today we have: 401 new tickets 573 total tickets We are responding to tickets submitted on 6/10. Thanks, Matt

Servers will be back online by noon PDT. We are fixing a technical issue with the Login servers caused by copying the characters of many players to the Beta.

The login servers / world server are getting hammered with players joining. Those servers are doing a one time copy of all characters on join. We are working on it.

We are open now.

We are open now.

There is a technical issue preventing access. We are working on it now.

Hi all, Bear with us. In all the networking chaos yesterday, the new Beta servers we flagged in a way that prevents regular users from getting on. This is still going to happen today. We are working to clear this now. We'll extend the Beta time frame to make up for it. Thanks, Matt NOTE: This was resolved We are extending the Beta by one more hour to 3pm PDT.

Hi everyone, Today I had to make a tough call. Both yesterday and today, our internal network had massive problems causing long outages at a time which prevent staff from making a lot of progress on their work. This includes Customer Support, Quality Assurance, and Development. We had a number of last minute bugs we wanted to address that might have made tomorrow a bit smoother, but only one of them got addressed. The Open Beta final build has been released. Everyone should try to patch now. Against my better judgment, I'm sticking withtomorrow's test as planned, because I said we'd do it. So we're doing it. Details here: http://gamersfirst.com/apb/news/2020/6/22/legendary-beta I don't recommend anyone stream this build. There are bound to be lots of problems. It just isn't the build I had hoped for. Please also check the Known Issues here: Lastly, we will be doing more of these as regularly as we can. Sorry, Matt Edited to clear up questions.

Hi all, This week started great, but ended horribly. I'm going to post on this separately, but our internal network decided to have massive problems on Thursday and today which prevented CS from answering a lot of support tickets. As of today we have: 290 new tickets 460 total tickets We are 2 tickets away from responding to tickets submitted on 6/8. Thanks, Matt

Hi everyone, I'm going to be posting early this week. Hopefully everyone knows we're running the first Open Beta test tomorrow, Saturday, June 27th at 11am PDT to 2pm PDT. I expect this test will be a mess. That's why we're only running it for 3 hours. That should give us enough data to work with, while also giving a lot of players the chance to log in. The Known Issues are here: Please be patient with us, and know that some systems just wont be able to get in right now. That includes systems that don't support AVX. We'll do a comparison build with AVX turned off to try and see if this optimization is worth it. You also wont be able to run 1024x768. This is something we inherited from Reloaded. They increased the minimum resolution for console and changed several UIs. It will be a pretty big task to undo those changes. And lastly, if you're running Win 7 or Win 8, you might experience more input lag than Win 10 players in Fullscreen Windowed mode. I highly recommend upgrading to Win 10 if possible. There are many optimizations that help gamers. These are all areas we will continue to look at as development moves forward. For this week, I thought I would share some of the Fullscreen Windowed work we did, and how we evaluated our progress. Several testers and several developers ran benchmarks in Asylum to get apples to apples comparions. Here is a sample of what those looked like. 1) This is what my machine looks like on Live with Fullscreen Exclusive mode. Couple key stats to point out: FPS average of 149.5 with some stuttering 0.23% of the time Input latency ranging from 18.2 to 24.9 No dropped frames Here was our first set of benchmarks for the APB 2.1 Beta right after we put in Fullscreen Windowed Mode. By comparison, this is mostly a mess: FPS is the bright spot with 237.1 average but 1.26% stutters Input latency was between 34.9 and 39.2 which is nearly double FSE in Live And dropped frames.. lots of dropped frames (25.2%) Finally here is is my last capture against the Beta build for Saturday: 241.7 FPS which is solid and 0.5% stutters, but not the big spikes like in Live Input latency is actually slightly better than FSE on Live at 16.1 to 20.3 No dropped frames The testers were great about helping us work through this. NOTE: We still have some work to do here. If you run at a lower resolution than your desktop, you will introduce the kind of performance from the earlier Beta build. If you want to run the game at a lower resolution, change your desktop resolution and then match it in game. Crossing my fingers for tomorrow. Thanks, Matt

Hi everyone, I want to personally apologize for an incident that occurred during yesterday's stream. I was running late to setup the stream and it didn't occur to me that I would need the pictures of the winning designs. On the fly, I flipped to the forums to show the winners, and accidentally exposed the IP addresses of several players because I was logged in as an Admin. I should have known better. I am sorry. I can do better. Sincerely, Matt

Hi everyone, I also want to acknowledge the hard work put in by SPCT and our closed testing group of players. This was very tedious work, and a lot of it wasn't fun. It wouldn't have been possible with them getting on week after week and reporting issues. I also want to level set expectations. I'm releasing the Beta because we need a lot more concurrency and mass testing now. Expect crashes and issues. This is going to be bumpy, but at this point we can't do this without the community. We needed to cross this threshold so we can start focusing on mission districts and better matchmaking. Thanks, Matt

To the OP: Okay. https://www.gamersfirst.com/apb/news/2020/6/22/legendary-beta

Hi everyone, I spent some time reviewing the thread that Flight made for last week's question. It looks to me like the vast majority of players responding would just like us to release the Beta and then finish work with feedback from the community. I'm taking that under heavy consideration. This week involved a lot of activity: Improvements to monitor settings to help with input lag Migrating content from APB 1.20 Adding more crash logging around GPU crashes Our normal Friday play test was pushed to today, so I'm still reviewing logs. But here are the general notes: Lots of good feedback on the performance. The expanded monitor options seemed to go over well. Still some issues with input lag - enough that testers said the game felt far below the reported FPS. We got 2 crashes from today, but neither triggered the GPU logging. We've had 5 crashes in the last 2 weeks that are logged in our defect tracking software. We are still looking at mouse sensitivity issues when in Marksman and the crashes. Thanks, Matt

Hi all, Lots of progress on tickets. Those numbers keep coming down. The straggler tickets from the 26th and 27th are complicated and taking days to resolve. So the date isn't great, but I'm waiting for date to move quicker as we close out these tickets. As of today we have: 221 new tickets (more than 60 down from last week) 354 total tickets (more than 90 down from last week) We are responding to tickets submitted on 5/26, which is at 24 days. Thanks, Matt

Yes. To start just those two. But the plan is to expand as quickly as possible. One reason for launching sooner, is to free up testers to start hitting the mission districts.

For those that have asked, the Open Beta will run in a completely new environment. So it will be completely optional, and not interrupt APB 1.20 (Live).