MattScott

Hi everyone, This week was a bit frustrating again. All of the issues currently blocking the Open Beta are based on bugs in code or systems that were originally created ages ago. Worst of all, the areas we are working on are... boring. That means no sexy graphs or cool pictures to highlight. Instead I'm going to run down the list of issues we are working on and try my best to explain them: Login/World server issues handling a lot of players - We spent last week excavating some stress test tools from 2012. We got them working in the middle of this week, which enabled us to start testing why the vast majority of players weren't able to log in or stay connected to the world server during the Beta. We have already verified that the hardware and installations between Live and the Beta environment are exactly the same. Moving to new hardware or reinstalling made no difference to the connection issue. In APB 1.20 (Live), our Login and World servers handle lots of spikes when a bunch of players try connecting at the same time. It also appears that APB 2.0 (Live Consoles) handle load just fine too, although the concurrency on consoles is a lot lower than PC, so there is a possibility that a less severe version of this issue exists there. But there is definitely a problem with APB 2.1. Any time we try to get more than about 50 players trying to connect, we start hanging sessions. Our best guess is that we carried over some piece of code from 1.20 somewhere in the last 1.5 years that appeared to work during our small SPCT/closed beta tests, but fails at scale. The team is continuing to review code step by step to fix the issue. Problems debugging crashes submitted by Open Beta players - This appeared to work fine with our small testing group. In the past, we have noticed small inconsistencies from time to time, but we were still able to get enough info to fix issues. But crash reports submitted during the Open Beta showed lots of inaccurate call stacks and debug info. This has made tracking down bugs much more difficult than it should have been. I can't really get into details on why this system is so complex. It just is, and we're stuck with it. So we have started rewriting code from 2015 that took several developers 4 months to complete. Similar to the first issue, this task is "open ended", which means it's difficult to put a time estimate on since we have to work through each line of code to figure out what is failing. With all of that in mind, we have been making good progress, and it doesn't appear like it will take too much longer to fix. Hoping for a better update next week. Crashing fixes / Multi-threading fixes - This is also ongoing. We feel like we've fixed the majority of crashes from the Open Beta, but some pesky DirectX11 crashes remain. To address those we are going to kill two birds with one stone. The team is moving ahead with some multi-threaded work that we had put off in order to get the Open Beta out. We believe these remaining crashes are caused by a lower level issue in Unreal 3's task handler related to shared memory across threads. The base code for Unreal 3 is nearly 20 years old now and even the newer sections are around 8 years old. Our solution is to use a newer, more powerful library that has become the industry standard for this type of work. We started implementing that library about 3 months ago because it is very stable with more sophisticated ways to schedule tasks and ramp up processing on more powerful machines, while also being much more efficient on lower end machines. So in theory we can fix the remaining crashes while also improving performance. However we can't just "plug it in" for an easy win. The code works substantially different to Unreal's code. So each area where multiple threads are used has to be individually migrated to the new library. The team is moving quickly, and we've already converted some of the bigger systems like Shadows and Occlusion. As soon as we have the rest converted, we can start testing this again internally. Thanks, Matt

Hi all, This week was much better. We are still training a couple new customer service agents, but they are all picking up speed. As of today we have: 285 new tickets (a lot lower than last week) 498 total tickets (a lot lower than last week) We are responding to tickets submitted on 6/17. Thanks, Matt

Hi everyone, I missed the update yesterday with the holiday. Last week was a maintenance week. That meant we backtracked over existing code and systems to diagnose the issues from the Beta. Part of the team resurrected a series of old stress tester applications. We've almost got them working for the Login and World server. That should let us try and narrow down the bad server problems. We also fixed a number of the big crashes. There will be more this week, as we dig into one section of troublesome multithreaded code. Hope everyone got a nice break over the weekend. Thanks, Matt

Hi all, We finally sorted out some internal networking issues on Tuesday that were preventing all CS from working. This was a short week for us with the holiday, so the numbers aren't great. As of today we have: 401 new tickets 573 total tickets We are responding to tickets submitted on 6/10. Thanks, Matt

Servers will be back online by noon PDT. We are fixing a technical issue with the Login servers caused by copying the characters of many players to the Beta.

The login servers / world server are getting hammered with players joining. Those servers are doing a one time copy of all characters on join. We are working on it.

We are open now.

We are open now.

There is a technical issue preventing access. We are working on it now.

Hi all, Bear with us. In all the networking chaos yesterday, the new Beta servers we flagged in a way that prevents regular users from getting on. This is still going to happen today. We are working to clear this now. We'll extend the Beta time frame to make up for it. Thanks, Matt NOTE: This was resolved We are extending the Beta by one more hour to 3pm PDT.

Hi everyone, Today I had to make a tough call. Both yesterday and today, our internal network had massive problems causing long outages at a time which prevent staff from making a lot of progress on their work. This includes Customer Support, Quality Assurance, and Development. We had a number of last minute bugs we wanted to address that might have made tomorrow a bit smoother, but only one of them got addressed. The Open Beta final build has been released. Everyone should try to patch now. Against my better judgment, I'm sticking withtomorrow's test as planned, because I said we'd do it. So we're doing it. Details here: http://gamersfirst.com/apb/news/2020/6/22/legendary-beta I don't recommend anyone stream this build. There are bound to be lots of problems. It just isn't the build I had hoped for. Please also check the Known Issues here: Lastly, we will be doing more of these as regularly as we can. Sorry, Matt Edited to clear up questions.

Hi all, This week started great, but ended horribly. I'm going to post on this separately, but our internal network decided to have massive problems on Thursday and today which prevented CS from answering a lot of support tickets. As of today we have: 290 new tickets 460 total tickets We are 2 tickets away from responding to tickets submitted on 6/8. Thanks, Matt

Hi everyone, I'm going to be posting early this week. Hopefully everyone knows we're running the first Open Beta test tomorrow, Saturday, June 27th at 11am PDT to 2pm PDT. I expect this test will be a mess. That's why we're only running it for 3 hours. That should give us enough data to work with, while also giving a lot of players the chance to log in. The Known Issues are here: Please be patient with us, and know that some systems just wont be able to get in right now. That includes systems that don't support AVX. We'll do a comparison build with AVX turned off to try and see if this optimization is worth it. You also wont be able to run 1024x768. This is something we inherited from Reloaded. They increased the minimum resolution for console and changed several UIs. It will be a pretty big task to undo those changes. And lastly, if you're running Win 7 or Win 8, you might experience more input lag than Win 10 players in Fullscreen Windowed mode. I highly recommend upgrading to Win 10 if possible. There are many optimizations that help gamers. These are all areas we will continue to look at as development moves forward. For this week, I thought I would share some of the Fullscreen Windowed work we did, and how we evaluated our progress. Several testers and several developers ran benchmarks in Asylum to get apples to apples comparions. Here is a sample of what those looked like. 1) This is what my machine looks like on Live with Fullscreen Exclusive mode. Couple key stats to point out: FPS average of 149.5 with some stuttering 0.23% of the time Input latency ranging from 18.2 to 24.9 No dropped frames Here was our first set of benchmarks for the APB 2.1 Beta right after we put in Fullscreen Windowed Mode. By comparison, this is mostly a mess: FPS is the bright spot with 237.1 average but 1.26% stutters Input latency was between 34.9 and 39.2 which is nearly double FSE in Live And dropped frames.. lots of dropped frames (25.2%) Finally here is is my last capture against the Beta build for Saturday: 241.7 FPS which is solid and 0.5% stutters, but not the big spikes like in Live Input latency is actually slightly better than FSE on Live at 16.1 to 20.3 No dropped frames The testers were great about helping us work through this. NOTE: We still have some work to do here. If you run at a lower resolution than your desktop, you will introduce the kind of performance from the earlier Beta build. If you want to run the game at a lower resolution, change your desktop resolution and then match it in game. Crossing my fingers for tomorrow. Thanks, Matt

Hi everyone, I want to personally apologize for an incident that occurred during yesterday's stream. I was running late to setup the stream and it didn't occur to me that I would need the pictures of the winning designs. On the fly, I flipped to the forums to show the winners, and accidentally exposed the IP addresses of several players because I was logged in as an Admin. I should have known better. I am sorry. I can do better. Sincerely, Matt

Hi everyone, I also want to acknowledge the hard work put in by SPCT and our closed testing group of players. This was very tedious work, and a lot of it wasn't fun. It wouldn't have been possible with them getting on week after week and reporting issues. I also want to level set expectations. I'm releasing the Beta because we need a lot more concurrency and mass testing now. Expect crashes and issues. This is going to be bumpy, but at this point we can't do this without the community. We needed to cross this threshold so we can start focusing on mission districts and better matchmaking. Thanks, Matt

To the OP: Okay. https://www.gamersfirst.com/apb/news/2020/6/22/legendary-beta

Hi everyone, I spent some time reviewing the thread that Flight made for last week's question. It looks to me like the vast majority of players responding would just like us to release the Beta and then finish work with feedback from the community. I'm taking that under heavy consideration. This week involved a lot of activity: Improvements to monitor settings to help with input lag Migrating content from APB 1.20 Adding more crash logging around GPU crashes Our normal Friday play test was pushed to today, so I'm still reviewing logs. But here are the general notes: Lots of good feedback on the performance. The expanded monitor options seemed to go over well. Still some issues with input lag - enough that testers said the game felt far below the reported FPS. We got 2 crashes from today, but neither triggered the GPU logging. We've had 5 crashes in the last 2 weeks that are logged in our defect tracking software. We are still looking at mouse sensitivity issues when in Marksman and the crashes. Thanks, Matt

Hi all, Lots of progress on tickets. Those numbers keep coming down. The straggler tickets from the 26th and 27th are complicated and taking days to resolve. So the date isn't great, but I'm waiting for date to move quicker as we close out these tickets. As of today we have: 221 new tickets (more than 60 down from last week) 354 total tickets (more than 90 down from last week) We are responding to tickets submitted on 5/26, which is at 24 days. Thanks, Matt

Yes. To start just those two. But the plan is to expand as quickly as possible. One reason for launching sooner, is to free up testers to start hitting the mission districts.

For those that have asked, the Open Beta will run in a completely new environment. So it will be completely optional, and not interrupt APB 1.20 (Live).

Hi everyone, This week we got in 2 play tests. One on 6/10 and another on 6/12. We still had 2 crashes on 6/12. The team is looking at those. These new builds focused on a couple things that our previous comparable test on 5/29 didn't. BattlEye performance compared to EAC Windowed Fullscreen mode Character baking hitches when new players enter the district So I thought I would go back compare play test stats from 5/29, 6/10, and 6/12: BattlEye: I found that in general it runs faster than EAC did. Windowed Fullscreen mode: I'm not sure our stats capture input lag, and some testers mentioned some noticeable difference vs APB 1.20. That's something we'll have to keep working on. Character baking hitches: I can see a significant improvement in the 6/10 & 6/12 Rendering stats. 5/29 was much slower. Most players reported a reduction or elimination of the hitches, but since the engine is multithreaded and mostly not render bound, the change didn't affect the FPS a lot. Here are some of the stats over our testing period going back to April. For next week: We should have a new build that incorporates much more functionality that supports different refresh rates across different monitors and more. We should finish the latest migration of code and content from APB 1.20 to APB 2.1. We should get some solid time to work through more of the crashes to try and eliminate those. Beyond that, there are some tester concerns related to: Input lag from Windowed Fullscreen Visual and audio bugs Doors getting out of sync with the server and showing open when they are closed or closed when they are open Issues with Field of View and mouse sensitivity in Marksman mode. The more zoom, the more problematic this becomes. Possible bugs in the UI that are slowing down FPS Issue with stutters that was cleared after rebooting Removal of 1024x768 resolution This last one was called out very strongly by a couple testers. For APB 2.0 (console) they expanded UI screens to show more information and generally use the screen space better, so those screens no longer fit in 1024x768. We will have to do a bunch of extra work to put back in support for this low resolution. My personal feeling is to launch the Beta as soon as the content migration and crashes are fixed. Then iterate on Windowed Fullscreen and other bugs. But I've also been told that the first impression is everything, and to some extent I believe that. But I feel like that would apply more to the APB 2.1 full launch and less to the Beta. Thoughts? I would be interested in seeing a thread on these topics. Thanks, Matt

Hi all, We're trying hang in there. The good news is that we started training with the new CS, and both CS candidates from last week are being hired. As of today we have: 287 new tickets 448 total tickets We are responding to tickets submitted on 5/21, which is slightly over 3 weeks. Thanks, Matt

This first phase does not include the ability to rebuy ARMAS weapons, but the system was designed so that we can do that as soon as possible. I am excited to see this go in across the board for all purchases.

We are still running DX11, and Windows 10 has not removed FSE. We are dropping FSE because it crashes the game. But if we find a way to salvage it, then I have no problem putting it back - for the time being. We are looking at going to DX12 at some point. They improved a lot of the deferred rendering systems that we are using for multithreaded rendering. DX11 has its own options for supporting Windowed mode. The team is making sure we do this right.

Hi all, There seems to be some confusion about Full screen mode, so I'll try to clear up the information we are working from, and then we discuss the issue further. There are two forms of Fullscreen - Exclusive and Borderless Windowed. As of DX12, Microsoft is no longer supporting Fullscreen Exclusive. Essentially this is going away whether any of us like it or not. It is worth pointing out that every game that runs on DX12 has already removed Fullscreen Exclusive and replaced it with a Borderless Windowed version. Fullscreen Exclusive literally doesn't work in DX12. They just weren't as transparent about it. Microsoft has written a pretty concise explanation of their strategy here: https://devblogs.microsoft.com/directx/demystifying-full-screen-optimizations/ They are basically saying that Fullscreen Optimizations under Windows 10 equalizes the performance between Windowed and Exclusive, which is why they are phasing out Exclusive. For those that don't know, Fullscreen Borderless Windowed mode looks just like Fullscreen Exclusive. It just interacts with the OS differently. Based on that information, we looked at the distribution of Windows versions across players. Here is what that currently looks like on Steam: Out of the 95% represented by all Windows versions on Steam, Windows 10 accounts for 85% (so 90% of Windows gamers overall). Even with all of that in mind, I recognize that still leaves 10% running Windows 7 or Windows 8 at a disadvantage because of increased input lag. I know that up until January 1 2020, Microsoft was still giving away copies of Windows 10, and believe there are still some ways to get a legitimate copy for free - even with a fresh install. But lastly, we are making this change for a specific reason: During playtests, we kept encountering a common crash related to a vague DirectX error where the DX device was "lost". After spending a lot of time researching this, we narrowed the error down to a combination of problems with Fullscreen Exclusive mode. In our last playtest, we had all the players run in some form of windowed mode. That test only had 1-2 crashes down from 25 for the previous test. I don't want to hold back the Open Beta any more than I need to, so for now based on the information at hand, we are going to remove Fullscreen Exclusive. If the team finds a way to make DX11 Fullscreen Exclusive work without crashing during the Beta, then we'll put that back in. Thanks, Matt