MattScott

Hi everyone, There isn't an Engine Update this week. Instead we are announcing the first Live stream of the Beta on Friday, April 17, 2020 at 11am PST. You can get more details here: Thanks, Matt

Hi everyone, 6 months ago I decided that the community had been kept in the dark too long about the Engine Upgrade. In development all kinds of things can and do happen, and even the best teams run into issues and blockers. So to help pull back the curtain, I started posting a dev diary of sorts focusing on our weekly progress. If you haven't been following it, you can check it out here: This week there is no update - at least not on the engine itself. Instead we are going to try something different with the community. On Friday, April 17, 2020 at 11am PST we have invited a number of APB streamers to show off the new Beta live. Over the coming week, we'll be posting a list of Twitch channels that will be participating so you can support your favorite. Each streamer will start with some quick benchmarks on their system from APB 1.20 (Live). Then they will load up the Beta and do another quick series of benchmarks before jumping into a couple Fight Club matches on Asylum with our dev team. Everyone will have codes and goodies to give away, and they will be gathering your questions and comments to give me for later. After about an hour all of the feeds will hand everyone off to the Little Orbit Twitch channel (https://www.twitch.tv/littleorbit), where I'll do my best to answer your questions and talk a bit more in-depth about the upgrade and the upcoming timeline. EDIT: Below are the names of the streamers that will be showing off the Beta on Friday. This was a bit difficult to manage, because many of these are controversial figures in the community, but I didn't want to get into the politics of approving or denying streamers. The idea behind this is to allow the community to take a good look at the game with your favorite streamer. You can choose to watch any of these, so let's not flame this thread with criticism. This is a big milestone for the game. Let's enjoy it! Here is the list: WitchQueen https://twitch.tv/witchqueen Frosi https://twitch.tv/frosi Lie https://twitch.tv/liee Rich https://twitch.tv/cuve RuschGaming http://twitch.tv/ruschgaming Kempington https://twitch.tv/kempington Flaws https://twitch.tv/Flvws Gordo https://twitch.tv/gordoismyname CombatMedic02 https://twitch.tv/0utbreakgaming Exo https://twitch.tv/exoticzlol Zeal https://twitch.tv/zeal PoundOfFlesh https://twitch.tv/thepoundofflesh Shini https://twitch.tv/shini Jam https://twitch.tv/amsterjam Ntec https://twitch.tv/ntec Sadira https://www.twitch.tv/sadira SKay will be joining Matt Scott on twitch.tv/littleorbit for a Q&A at 12pm PDT / 7pm UTC. Thanks, Matt

Hi all, Still trying to keep the response time low, but there has been a massive surge of new tickets. The team is working hard to keep pace. As of today we have: 318 new tickets 490 total tickets We are responding to tickets submitted on 4/2, so we're were at 8 days to respond. EDIT: Sometimes a picture is worth a thousand words... Here is a report showing how many new tickets we have been getting since the beginning of 2020 through last week. We went from ~200 new tickets per day in late February up to nearly 400 tickets on a single day last week. In order to stay current with tickets right now, we have to close ~1500 tickets per week. Here is a report showing our average response time since January 2019. At the beginning of 2019 it was taking between 30-36 days to response on average. For the most recent 12 weeks we have been responding between 6.8 days and 4.9 days. Thanks, Matt

Servers were back online at the time of my original post.

Hi everyone, This is just an informational post to let you guys know that the Citadel servers got hit extremely hard this morning. Our normal DDoS protections weren’t enough, and those districts were affected intermittently for about an hour. If you had connectivity or latency issues, that wasn’t your provider. It was on our end. It’s been quite a while since we have had an attack get through. The network team has made corrections to block this form attack moving forward. Apologies, Matt

Hi everyone, This was another productive week. We fixed the blown out ColorLUT issue that I posted pictures of last week. We are knee deep in fixing an in-game browser issue that causes a crash when opening up ARMAS. We are also working through more collision and game-play issues in Asylum. And lastly, that's right, we are working on lighting issues in the districts. I know this is something that has been taking forever, and several players suggested that we release the Beta with unfinished lighting. However, just to show you some of what we are fixing, I pulled some images that went back and forth between the team this week. Building lighting in Unreal is actually 2 different processes: (1) Baking lightmaps and (2) building indirect lighting 1. Baking lightmaps This is an ancient technique for generating fast lighting that doesn't require very powerful graphics cards. The primary downside to lightmaps is resolution. Lighting is baked to a texture and then wrapped around objects just like normal textures, but if your lightmap is too small, then you'll end up with very jagged results. Conversely, if your lightmaps are too big, then you'll eat up a lot of texture space. Also, you can only build lightmaps for objects that are static - meaning they don't move. So it's not a technique you can apply to everything in the scene. Ultimately, even though this technique is very old, lightmapping works well if you blend it with several other techniques. In our case, we don't build lightmaps, we build shadow maps. This enables us to change their color and how much they influence the scene at different times of day. We can only build shadowmaps for static objects that don't move, and those objects require a separate UV channel to map that generated texture into. Buildings are the best example of objects that we bake lighting for. However, Unreal is very finicky about the UVs it maps into. For those that don't know, UVs are a 2D position on a texture that is assigned to each vertex in a mesh. This defines how textures map onto the mesh. Lightmaps require "unique" UVs. That means each vertex maps to a unique spot in the texture, which makes sense because the lighting generated into the lightmap would be unique for each part of the building. If any of those UVs are slightly overlapping or touching the edge of the texture, then Unreal errors and aborts lighting that object. We run lighting with error color coding turned on to see which objects have issues. This looks something like: The orange area indicates that part of the lighthouse entrance doesn't have correct UVs. This specific case likely wouldn't be a big deal for Beta. However, this next case is, because every ceiling in this part of the district has no lighting and looks terrible. As you imagine some districts have thousands of individual meshes that require lightmap UVs, so we have been going through by hand to fix all the areas that aren't building correctly. Corrected lighting ends up looking like this: 2. Building indirect lighting This is a new system that was introduced in Unreal 3.5. I'm sure Reloaded felt this would be the biggest visual upgrade for players, and they aren't wrong. However, the new lighting engine in Unreal 3.5 also costs quite a bit more to render, so all of our initial performance tests were terrible. We have spent a lot of time optimizing so that players will get the benefits of better lighting, but not take a huge performance hit. Indirect lighting is built into a 3D grid and it samples the various lights in the scene at that location, so that when a moving or dynamic object is in that area, it is quick to apply the lights. This next image shows off some of the better lighting which is a combination of overhead dynamic lights, dynamic shadows, baked shadowmaps, ambient occlusion, and built indirect lighting on the player). It also shows off one of the big flaws in how indirect lighting is built. The indirect lighting in this shot is the orange light casting on the player. The flaw in the shot is that there is no obvious source of that lighting. In fact, that orange light is coming from just inside that building. This only happens if they get too close to the wall because the 3D grid is 4ft x 4ft, which doesn't have the resolution to sample lighting from both inside and outside the building. To fix this, it requires us to build lighting volumes that "trap" the light inside the volume by inserting extra cells in the 3D grid for indirect lighting to emulate where walls would prevent light from leaking outside. The art team has been building lighting volumes for these cases. Lastly, we had a couple players mention that we needed to turn on ambient occlusion. It's there, it just has problems. Here is what ambient occlusion looks like right now in Asylum. It's very faint, and it's being calculated on a very dark frame buffer in the pipeline. That's also being looked at. Thanks, Matt

Hi all, I missed posting this yesterday. With the significant rise in population, we are continuing to see lots of new tickets. As of today we have: 229 new tickets 364 total tickets We are responding to tickets submitted on 3/27, so we're were at 7 days to respond yesterday. Thanks, Matt

Hi all, Let me step in here to provide some clarity. First, I want to say that I respect the OP’s opinion. I’m going to disagree in my answer, but I can appreciate the dialog this has raised on this thread. Next, I think it is important to remember that the game itself is rated M and has pre-existing references to prostitution and the term itself. I’m not going to start diluting RTW’s vision for a gritty world with mature themes. As has already been pointed out, the term refers to a male or female, and more importantly the title was meant to be more of a joke than anything since it refers to candy on Easter rather than prostitution. This is a case where I do think the OP is taking this a bit too seriously. Having said that, if you don’t approve of the term, you have agency on whether to apply the title to yourself or not. Lastly, the forums are PG-13, and have always had different rules than the game. Previous management felt the need to blacklist the term here, which I’ll respect for the time being. Thanks, Matt

For anyone experiencing high latency, please go in-game and run /latencytest Then open a ticket at http://support.gamersfirst.com and attach the .CSV file from the APBGame/Logs directory. Right now we are seeing 1 of 3 scenarios: 1) We had an issue in the last patch that was spamming the network which got rolled back. We fixed the issue, so that code is active again since the Easter patch (players are helping diagnose with latency tests) 2) Some home network providers are overloaded due to all the people working from home 3) Our servers have been hit relentlessly with DDoS - for the most part this has had zero effect on players, but occasionally a large spike results in momentary lag Thanks, Matt

Hi everyone, This week was a mixture of success and frustration. We made a lot of progress knocking out bugs, and the only thing blocking the release of the Beta are some inconsistent crashes with RTX and higher end cards. At this point, we're starting to get more specific issues reported. These are things like "The stairs in this part of Asylum cause a jitter as I go up them" or "This object needs to allow shots through it." I'm not sure how much longer I will hold the Beta while we fix some of these smaller issues, but until the last of the major crash bugs is fixed, I have no choice but to hold this back. Lighting is getting very very close, but this continues to be one of the more frustrating areas. The main issue right now is color balance using Color Look Up Tables (Color LUTs). These are images that modify how specific color ranges are displayed. Here is an example of what a ColorLUT image looks like. We use this one for color grading Financial at noon. Here are some references from Live (if you look closely you'll notice some flawed lighting with foliage). All of these shots accomplish giving the game a moody look and feel while also keeping the character lit enough to see where he is. Midnight Noon 5pm Here are some shots from APB 2.1 as it stands right now. All of these shots have much better lighting consistency (ie. foliage doesn't glow). However the scenes don't have the same punchy color as Live, and the character is far too muddy while standing in shadow. This gives the overall scene a very flat feel. Midnight Noon 5pm This is all very correctable with ColorLUTs, because we punch up the contrast on specific colors. Except.. that system isn't doing what it is supposed to do. Here are some pretty awful examples of what we see right now when the team tries to enable some pretty basic ColorLUTs. This system works fine right now on console, so somewhere along the way it broke. Noon 5pm Thanks, Matt

Hi Ketog, You raise some good points, and I think the answers are a lot more simple than you might expect. When we took over APB, we only had Reloaded as an example of how to run things. So largely we started by following their previous policies with some smaller changes. I also took a very conservative approach, because aspects of the community were very scary (ie. calling bomb threats to Tigg's home). For the safety of my staff, I chose to expose my identity up front, but that allowed everyone else at LO to remain somewhat anonymous till we got a better feel for things. Beyond that, we stepped into a very difficult spot. Towards the last couple years, Reloaded promised a lot of things and never delivered. Initially, we tried being more transparent with dates and things coming up, but most players responded either neutrally or negatively because those dates are often challenging to hit. This is really the only comment that confused me. I post every week about the Engine Upgrade - good, bad or ugly. I also post Customer Support stats every week. Those are areas where we are already giving 100% transparency. For the most part, I see those efforts as successful, so I am looking for more opportunities to continue doing that. Lixil lasted more than a year, and while she wasn't perfect, I think it's tough for a CM to "compete" with me. I try to get on the forums, twitter, Facebook, and discord as much as possible to stay aware of what is happening. I also try to respond when I can. We made some changes with SakeBee, so that he would have a clearer space to manage without me constantly jumping in. He's still new, but we are working on ways he can hang out with the community and get to know you guys better. I'm not going to defend RIOT. It had its benefits for my team, but we did it mostly because it was "low hanging fruit" that Reloaded had mostly finished. There were some hard lessons learned there. There are going to be a lot of opportunities for this coming up. But right now, we have to focus on the Engine Upgrade. It's blocking everything else, and a lot of the tasks for the Engine Upgrade are more "mechanical" to just getting it working like Live. However, the upcoming Beta will very much be work in progress. It will be a chance for the community to jump in and give us feedback, so we can make corrections. Thanks, Matt

Yes. Vehicles will work in Asylum.

Hi all, We had a lot of new tickets this week from more players returning.. up about 25% per day. The team did a stellar job trying to answer as quickly as possible. As of today we have: 198 new tickets 322 total tickets We are responding to the last 6 tickets submitted on 3/20, so we're at 7 days to respond. Thanks, Matt

I'm looking into South America and Asia servers as well. We have a pretty specific architecture that needs to be supported, so it takes time to identify data centers we can operate out of. It's coming. Some of the crossplay work was paused to get the Engine Upgrade out the door. But getting in support for regions was a big first step.

Hi everyone, This week was a total surprise for me. On Monday, the entire US watched as our government slowly tightened up social rules to help suppress the spread of COVID-19. At Little Orbit, we took the plunge and moved everyone to working remote, and I had to hope things would all work out. And thanks to all of you, we've seen a spectacular rise in population. Keep reading below the graphs to get an update on a new data center location for APB, my Engine Upgrade has big news, and a nod to our folks in Customer Support who have worked hard to get our response times down. First, the population increase. This is our PC traffic from two weeks ago on Saturday the 7th. It shows a total peak concurrency of ~900 PC players. While this is a good measure of Citadel (which is in yellow at 700 peak players). Jericho's peak comes later in the day beyond this graph at around 8pm. This is our PC traffic from today on Saturday the 21st. It shows a total peak concurrency of ~1500 players. That's a net gain of 600 players or 66% growth. I'm extremely grateful that players have chosen to spend their free time on APB during this world wide situation. We had a couple issues with district disconnections, and a client-side issue with this week's patch that causes a hitch every so often which we will correct on Wednesday. But overall I feel like the new server architecture and anti-DDoS got a chance to shine. If this continues, we will be out of server space in the next 2 weeks. With that in mind, we have started the process to open more servers in Warsaw, Poland so our Eastern Europe and Russian players can get better latency. Next, if you haven't been following my weekly updates on the Engine Upgrade, you should check out this week. We are making our preparations for Beta. One player in our community was able to get a copy, and I am now sharing details about the rollout plan. And finally, for those who don't look at my Customer Support updates I also made an exciting post there this week. Despite moving our entire team off site, we are now at only a 4 day initial response time for tickets. We will continue to work hard to bring that down as new and old players join APB, but that is our lowest ever. Thanks, Matt

Hi everyone, We had a great week with the Engine Upgrade with one notable problem that I'm going to talk about first. Due to juggling things around, we inadvertently exposed the new Beta client, and one adventurous player got a copy and was able to get in. I'm not going to block the screenshots on the forum, and he later contacted me to reveal how he did it, so we have fixed the issue. Many players sometimes assume we aren't working on this or that the Engine Upgrade will come out in 2025. And while I want to discourage other players from digging through our systems to find stuff like this, I have chosen to leave this post up simply to show that what we're doing is real. Now that the Beta client has leaked, there are a couple things that I should cover: We are only working on lighting and 8 remaining bugs at this point unless anything new comes up (which has tended to happen on this game a lot) The Beta is going to be limited to only Social and Asylum districts. APB is a massive game, and rather than continuing to grind through every possible problem in the hopes of some perfect launch, I have chosen to focus the team on a couple districts so we could polish those and let players in to test sooner. I'm very happy with the system performance in Asylum. During this week's tests, I heard a lot of positive feedback on how that district felt to play. The plan had been to release the Beta client 1 week ahead of when servers will be opened to give the players a chance to download it. Then we were going to do an initial test for a single weekend, where we open up the servers for a limited time. There has been very little testing done with a lot of players on the same district, so we want to see how the APB 2.1 servers handle everything. However, with our recent influx of players, we don't have the server capacity to host a lot of beta testers. I'm already working on a solution. As soon as we can host more than 500 players on this environment, then I'll get the gears turning. Thanks, Matt

Hi all, It appears that being told to stay home has inspired many players to check out or come back to APB. We saw a massive amount of new and old players join the game this week which has led to much higher player populations on the servers. We also saw a fairly large influx of new tickets asking for access to get back into accounts and other things with a spike on Monday and Tuesday. Along side that, my team transitioned to working remotely due to coronavirus which meant several days of getting setup and moving computers around. However, I am very proud of our CS team. They hunkered down and powered through tickets - despite all the setbacks. As of today we have: 148 new tickets 298 total tickets And most impressively, we are now (for real this time) responding to tickets from 3/16, which means we are getting back to players in 4 days. This is the lowest we have ever been. Please stay safe and enjoy the extra mini-game events with our GMs this weekend. Thanks, Matt

Hi all, This is a totally valid criticism from the OP. I wasn’t aware Reloaded had done this in the past, and we didn’t do it for the Thunder or the New Glory. But it makes total sense on why players would want to see it so they can decide whether to buy the box or now. We have had a second redesign of ARMAS on the books for quite a while to fix all the small images and lack of detail. I like the production values for the video posted in this thread. Nicely done! Thanks, Matt

Hi everyone, First, for those who haven't seen my announcement from last night, Little Orbit is now requiring all our staff to work remotely to insure their safety and prevent the spread of coronavirus. Next, we spent a portion of the week organizing for this move, but we were able to solve the bad character shadows and lighting issues. This has been a major blocker, and it allowed us to rebuild the lighting for each district. The results are already 100% better - even without any actual lighting changes from the art team. However, on Friday my team was finally able to get a 100% reproducible scenario to fix a memory crash that has been plaguing us. The old code was 32-bit, and the new code is supposed to be 64-bit across the board, but there are many tiny exceptions that need fixing to provide overall stability. Unfortunately that means the game isn't in a runnable state right now. Once we get this issue fixed and a new Beta build is released internally, I'll post some shots at various times of day and in some of the other districts so you can see the difference. Thanks, Matt

Hi everyone, I'm sure by now that everyone has seen the impact of coronavirus (COVID-19) on their city, work, schools, and entertainment. Starting Monday the 16th until the foreseeable future, all Little Orbit employees will be working remotely until we get different guidance from health officials. We prepped all week to make sure teams had hardware and instructions on how to check in during the day with their leads for tasks and updates. Come hell or high water, the team is dedicated to shipping the Engine Upgrade, and we didn't get this close to be sidelined by a little virus. I know this may be a little scary right now. And APB is just a game. But since many of you will be temporarily home due to work or supporting family members, we want to insure that the game is online and available for all our players to distract themselves with some missions, fight club, or a little virtual socialization in Breakwater. I'm also going to be turning on Free Premium for all players across all platforms. NOTE: We will be snapshotting the active Premium days for everyone, and we'll extend those days once everything goes back to normal. Hopefully that will make Social a little more fun for the time being. I'm also going to ask for everyone's patience with us. We are going to do our best to make this as smooth as possible. And most of all, wash your hands. Don't touch your face. Avoid large gatherings. Stay safe. Thanks, Matt

Hi all, I need to make a correction to last week's numbers. Our CS manager was out, so Selali and I had to run the stats, and we messed up. It turns out that we hadn't yet reached 2/28, so as we moved through tickets this week several players were confused. There was also a general surge in tickets and players returning to check out the game. With that course correction in mind, here are this week's numbers. 171 new tickets 281 total tickets We are now responding to tickets from 2/29, which puts us at 13 days to respond. Thanks, Matt

Hi all, I’d like to weigh in here and try to eliminate the confusion. In the spirit of transparency, I post undoctored shots to the Engine Upgrade thread. So that is how APB 2.1 looks right now. But none of the districts have final lighting or color balance. I want to see solid performance before we spend any more time on look and feel. Having said that, there are lots of knobs and dials we hope to use soon in order to get the proper look that compliments the gameplay and still feels like APB. Thanks, Matt

Hi everyone, The last two updates have been pretty sparse, so this week I'll take a little more time to explain where we are at. First, a portion of the team has now moved back to bugs, which is good. As much as we want the performance to get better, we also need to fix actual gameplay issues too. Right now we are tracking 20 bugs identified by SPCT or QA. 2 of which were determined to be duplicates. 4 of which were checked in as fixed by Friday, which leaves 14 unaddressed bugs. Second, there are two overall issues we are still working on - Lighting and Performance. For lighting, we've narrowed down the lighting issue to a system in Unreal 3.5 that handles "Indirect" lights. This is a process that gets pre-processed and packaged when we ship the game. It looks like because Reloaded never turned on Day/Night, they also never tuned indirect lights that are largely only visible at night. Part of the team has now started diving into these tools to work through all our maps to fix this. For performance, I've started to realize there is no silver bullets - something that we'll find and fix that will make the game suddenly run great. However, the good news is that we've tested Asylum across a number of systems, and APB 2.1 runs that district better or at the same frame rate as APB 1.20 (Live). On one of my higher end systems with an i7 with an RTX 2080, here is the comparison with both running 1920x1080: APB 1.20 Asylum on Maximum (ignoring micro stutters) 1% Min = 6.5ms 153FPS Average = 4.85ms 206FPS 1% Max = 3.15ms 316FPS APB 2.1 Asylum on Very High (which is equivalent to Maximum) 1% Min = 4.68ms 213FPS Average = 3.71 269FPS 1% Max = 3.02ms 330FPS I ran a test on a lower end laptop with an i7 and a GTX 1050. Amazingly APB ran at an average of 100FPS on APB 2.1 which was about 10FPS faster that APB 1.20. We have spent months getting the engine to take advantage of more cores. Here is what APB 1.20 Asylum looks like on that same i7 with 12 logical processors. And here is what APB 2.1 Asylum looks like on the same i7. Other districts are more hit or miss right now, but using Asylum as a benchmark, we started looking at differences between the other districts to help us narrow down the remaining performance issues. We started to recognize very specific areas where frame rates went down significantly, and that didn't happen in Asylum. Unfortunately there are so many areas in this engine that simply weren't finished sometimes I feel like we're looking for a needle in a haystack. Or as it turns out this week: a blade of grass in a park For those playing at home, I'll give you a little sample of what we deal with on a weekly basis. Here are two screenshots. If you look specifically at the foliage, can you spot a potential issue? APB 1.20: APB 2.1: The foliage in the top shot (APB 1.20) is set to only render very close in. You can barely see it from where the screenshot is taken, and that is on Maximum settings. The foliage in the bottom shot (APB 2.1) can be seen all the way across the district. In fact it has no draw distance limit set. That means 2.1 is currently rendering thousands of tiny polygons that each cast shadows on each other while the scene elements cast shadows onto them. The team started working on this Friday, and will be cleaning it up this week. Thanks, Matt EDITED: Added frame rate stats and cleaned up grammar.

Hi all, As of today we have: 154 new tickets 252 total tickets While those numbers are higher, we are now responding to tickets from 2/28, which puts us at 1 week to respond. Thanks, Matt

Hi all, I wish I had more visuals to show for this week. Those are always more fun to post than the bland technical write-ups. We fixed several memory crashes, but the Xbox One still has an issue where we run out of memory entirely. There are three big blocks that we're looking into (shaders, textures, and audio). We'll need to investigate ways to cut those down. The crash doesn't show up right away. You have to run around a bit in Financial. We believe it's only barely over the limit based on new content that was added since the last update for that platform. Audio work is ongoing. That code is a mess. We've addressed several areas, but there are a bunch more where audio arbitrarily loops through every emitter in the scene -- every frame. The team was in the middle of designing a caching mechanism so that the audio system doesn't have to do that unless anything moves, and even then most of the cache would still be valid. We got secondary shadows and the sky lighting working correctly.. but on Friday QA found areas in Financial where the odd self shadowing issues reappear that we fixed months ago. This is the kind of thing that drives me crazy with the APB code base. 2 steps forward and 1 step back. We'll be looking into that this week. I opted out of benchmarks this week, because there was too much work in progress. Thanks, Matt