MattScott

Thanks for the reminder. I'll ask the team to see what we can do. Thank you very much. I'll definitely pass it along.

Hi everyone, Over the last year and half, the Little Orbit team has produced quite a bit of original art which has been incorporated into skins, clothing, and events. These pieces have all been original works created by our talented in-house artists, and a couple months ago, I had a player reach out for permission to use the high rez art I posted from the Kabuki Weapon Skin Collection for a t-shirt. With that in mind, my art team have created a Redbubble store and put most of our original art up there. For those who aren't familiar with Redbubble, they make print-on-demand items showcasing original art. I was pretty skeptical at first, but their capabilities are impressive including custom made prints, mugs, clothing and even phone cases. Having spent some time looking at all the items, I admit there's a part of me that loves the idea of taking virtual customizations back into real life content. To be clear, none of these designs are APB specific art, characters, or logos. So if this isn't your thing, no worries. If you're interested in seeing more, you can see the new Little Orbit channel here: https://www.redbubble.com/people/littleorbit/shop?asc=u As a small Easter Egg, we've also included a couple new designs that haven't been released for APB clothing that you can see first on Redbubble. And we'll be updating the store over the coming months as we release new art. Thanks, Matt

Hi all, This week we began staging networking infrastructure with an entirely new provider. This includes new district server hardware based on the west coast, east coast, and Europe. Please bear with us. We are trying to keep the downtime at a minimum while this is being staged, but we have an aggressive timeline to get entirely cutover by Nov 25th. I'd rather plow through this and get it over with, versus having some rambling effort that takes months to complete. For the next week, we will have periodic windows where systems will be unavailable. @Lixil will do her best to make announcements on the forums. EDIT: We are still working on the world consolidation. We hit some extremely frustrating blockers once we built our proof of concept. The strategy has shifted to worlds that can seamlessly loan their players/characters to other worlds, giving us the same end result of NA - EU cross play. EDIT: District servers for the West and East coast will be consolidated under the Jericho world. For now, players will be able to choose them off the advanced list. Thanks, Matt

Hi there, We had to do a unscheduled reboot. The Login Server should be back online now. Thanks, Matt

Hi all, I'm checking with the network team to see if the fix was applied to all servers yet. Thanks, Matt

Hi everyone, I’m going to keep this week’s update a bit shorter. Fixing lighting, shadows, and building features continues to be a bit of a rabbit hole. There is very good progress, and with respect to lighting and shadows, I am calling it ‘good enough’. There are some straggling items that we can revisit post launch. Features on buildings (cProps) is the last area plaguing development. We discovered a horrible issue where props with embedded lights were recreating those lights over and over. We also have a lot of duplicate props in the district (again). This is an issue that popped up a month and a half ago that we solved. But it seems to be back. EDIT: I got the actual stats today. There are currently 12,700+ duplicate objects in Financial alone. There's also a secondary issue that popped up related to props that use the physics engine. Apparently we were at the limit of how many objects can use physics in a single district. It is very possible that we are occasionally seeing crashes in Live due to this issue. So we also spent some time fixing this in APB 2.1. We finished AA (anti-aliasing) and have started work on AO (ambient occlusion). It works for the most part except when the game animated the field of view which causes a flicker. The hope is to get a build to SPCT by Tuesday/Wednesday that has these items all addressed. EDIT: It’s taking a fair bit longer to clean up the dupes. So we wont have a new build tomorrow. Thanks, Matt

Hi all, I want to take some time to walk through this, because we have been chasing it for a couple weeks trying to diagnose and fix the issue. I also want to thank all the players who sent us latency reports and helped report the issue. Based on initial reports, it seemed like this was an obvious issue with our provider. So that's where we started, but after many days of extensive research, we couldn't find anything affecting our service. It turns out the problem was in our orchestration layer that controls spinning up and spinning down districts on our many district servers. This system is also responsible for detected hung/non-responsive districts by looking at ping times and CCU counts. If the district has been sitting empty for a specific time period, then we throw it away and spin up a new one. At some point in the last month, we had an issue where the orchestration layer was killing districts with players in them. This issue was difficult to diagnose, because it only happens on Citadel, so it's not a system wide issue. While we were investigating, my network team disabled the ability for orchestration to kill districts, so that we wouldn't accidently keep kicking players. Then we got caught up in Halloween and a number of other pressing tasks, and this issue was left on the back burner. During the last two weeks, it appears that orchestration was still detecting non-responsive districts, attempting to kill them (which was disabled), and then spinning up a new district on the same server. Currently due to memory and CPU constraints, we can only run 3 districts per physical server. Over time, orchestration had fired up 7-8 districts, which caused massive amounts of processing lag and eventually started crashing districts. As of today, we believe we have figured out why orchestration was killing populated districts. It involved a small configuration error from when Nekrova physical servers were merged with Citadel servers (which also explains why only Citadel servers were affected). We have started putting a fix in place, and the lag issue should get better. Thanks, Matt

Hi all, Lots of movement this week. 537 unanswered tickets 740 total tickets Both of these are considerably below last week. We are answering tickets from 10/31 with a couple earlier stragglers. So we are now running at a ~16 day response time. If your ticket is older than October 31st, and we have not reached out, then I recommend updating your ticket and asking for status. Thanks, Matt

I've alerted the devs.

Hi everyone, This week was a bit frustrating marked with stalls, blockers, and rabbit holes. We ran another series of benchmarks and things are steadily looking better. - Interactive building features / windows are almost done This is an example of something that was on our schedule as a week long item. Instead we have had to massively re-write the system to pull together better LOD (Level of Detail) techniques but also preserve some of the old RTW code. The system now works properly, but we're missing the data that links interactive features to the proper LOD on the building they are attached to. We spent days going back through old data, and finally found it. Then we spent more days parsing it different ways to reapply it. Then we spent more days fixing crashes that would appear after an hour of running the tool to apply the data to Financial and Waterfront. As of yesterday, we were running the tool and it hadn't crashed. Hopefully this will be done early next week. I recognize this issue is hard to describe, and even harder to visualize, so we took some screenshots to compare. Here is what the game looks like now in APB 1.20. There is a lot of cheating going on, because they can hide poor Levels of Detail behind the massive amounts of fog. Next up is the what APB 2.1 looks like from roughly the same shot. We don't have the same level of fog now, so our LODs need to look better. For this shot, we are at the edge of rendering LOD1 (the 2nd most detailed version of buildings). And this is the bug we're fixing. For this shot, we are now rendering buildings at LOD0 (the best detail on buildings). But the interactive features and some lower windows are too far away to render. This leaves "holes" where the feature/windows should be. If I move closer the features and windows will reappear. The solution we are implementing is to sync the distance that we turn off features to when we render LOD1. Side note: The dev in charge of this task found a bug that relates to why traffic signals are broken. Apparently these have not worked properly for some time, and he's going to take a look on Monday to see what can be done to fix those. - Shadows have been a pain This effort has been to properly render all the shadows around a character based on interior lights, night time lights, and the sun/moon. So far, we have rewritten elements of the core Unreal shader to get this working properly, and characters now render great. However, we've had a steady stream of issues with shadows from props acting up and not rendering properly. Currently there is a very irritating flicker where some props have lights embedded in them that are now casting shadows using the prop's geometry. The team has made excellent progress, and cleaning up this code has led to other optimizations and fixes. But I'm hoping this will finally be done next week. To get this working properly, we had to create a small test scene with a ton of lights at various locations. - Indirect lighting is still not working This got tabled till shadows are fixed. Lastly, just like the traffic light bug, sometimes we end up finding other things that can be fixed. This week it was Anti-aliasing. Our old technique, MSAA was removed from DirectX 11, and AA has never worked in APB 2.1. A different dev ran across some code that looked to be causing the problem, and he was able to run some quick tests and get AA working again. Here is a quick shot of what APB 2.1 looks like right now with AA. Thanks, Matt

Hi all, We started to hand out a couple codes for Tier 2, but we immediately got reports that the skin had a bizarre issue that caused a lot of guns to show up with their default look. We hadn’t seen this behavior before, and just haven’t had a lot of time to troubleshoot. Now that Halloween is over, I’ll have the team dive back in and check it out. Thanks, Matt

Hi all, I am very proud of the CS team. They handled a lot of extra load this week due to Halloween Pumpkin issues and repeat network outages. I know these numbers came up, but we cleared twice as many tickets as normal. 680 unanswered tickets 808 total tickets We are answering tickets from 10/20 with a couple earlier stragglers. So we are still running at ~19 days. If your ticket is older than October 20th, and we have not reached out, then I recommend updating your ticket and asking for status. Thanks, Matt

Hi there, I announced a week ago that we would be making some network changes to support more local district servers starting with New York. And even though our intention is to make this transition seamless, we have encountered a couple problems. We hope to complete this work soon. Thanks, Matt

Hi all, Login servers are back up. Sorry, Matt

Hi everyone, Our login servers are having some issues. We are working on it right now. I'll post here when it is back online. Thanks, Matt

Hi all, We're currently having a significant network outage. Unfortunately I don't have enough information to predict how long this is going to last. I'll keep you posted. Thanks, Matt

Hi all, We will be fixing the tutorial to remove the RIOT elements soon. Thanks, Matt

Hi there, I 100% sympathize with the console players. Both PS4 and XB1 are terrible at the moment. I try to include console specific details in my weekly Engine Upgrade threads here: Thanks, Matt

Hi there, We are working on this issue. Our provider in Europe is having problems. Thanks, Matt

Hi there, We are working with our provider in the EU on this. The situation has been.. frustrating. Thanks, Matt

Hi there, I just posted the Engine Upgrade update. My time zones are a bit off. Apologies. For the next round of Weapon balance changes, I believe we are waiting to get clear of the Halloween event (ending at patch time early on 11/6 PST) Thanks, Matt

Hi everyone, I'm working remotely, but this was another week spent diagnosing and attempting to fix rather complex issues in the Engine Upgrade. Unfortunately my initial stats from last week didn't hold up across our pool of testers. This is often the frustrating part of this project. We will see good things internally, but once we get the build to testers, the larger sample pool doesn't come back as well. We did figure out that my work laptop is a bad source of data. It has an embedded Intel 3D card and a high performance nVidia 3D card for gaming. However, the nVidia card is daisy chained off the Intel card. SO when I run Fullscreen Windowed, a Desktop Windows Manager process runs at 50% using my Intel GPU while APB 2.1 takes up 80%+ on my nVidia GPU. For whatever reason, that equalizes the performance between APB 1.20 (Live) and APB 2.1 (Engine Upgrade) and over time they run comparably. The team finished the following tasks: - Secondary night time lights are now working - Secondary night time lights are now running nearly as fast as day time. Console players will be happy with that change once we re-submit. - Added some fixes for texture caching, so that as you run around the district we aren't wasting so much CPU to figure out what textures should be loaded. Still working on the following issues: - Rendering shadows from secondary night time lights - Rendering indirect lighting when characters aren't affected by the main light (sun or moon) - Interactive building features still disappear when the player is far enough from a building that it shifts from LOD0 (the highest level of detail) to LOD1. This leaves an empty hole for a period of time where the shop front or window should be. This week marked the first time we ran all the testers through a series of benchmarks in both APB 1.20 and APB 2.1 to try and get accurate performance comparison stats. I know this may sound exciting, but it's not. Often testers have to patch, run the benchmark once to make sure the game finishes loading and caching all the geometry, and then they run the benchmark a second time for accuracy. If anything on their system accidentally interferes, then they have to start over and do it again. Some die hard testers did this for each district. Once they had accurate benchmarks, they sent us their logs, and then yesterday we processed all the logs through our internal tools to graph, identify CPU and GPU, and call out specific bottlenecks. The end of the day was spent posting the results to our internal channels for testers to see everyone's results. Based on that round of testing, we did see an overall increase in performance by 1-2 milliseconds per frame between 10/21 and 10/26. We're now back to being Render locked instead of Game Loop locked. The team is looking at an automated way to prompt beta testers to upload their logs at the end of a session, so we can collate and process them automatically across a larger number of machines to get better insight into performance. At some point, we will likely need to bite the bullet and throw this out to the community to see how it performs. Thanks, Matt

Hi all, I’m traveling right now and missed this yesterday. However, I am told that we have sorted the Red Pumpkin issue for newer tickets to make sure everyone had a chance to complete the event over the weekend. We mostly held steady this week. 561 unanswered tickets 716 total tickets We are answering tickets from 10/13 with a couple earlier stragglers. So we are running at ~19 days. If your ticket is older than October 19th, and we have not reached out, then I recommend updating your ticket and asking for status. Thanks, Matt

Hi everyone, This week we made good progress on performance and character lighting. These changes will help both the PC and console builds. Performance I have changed the test resolution to Fullscreen Windowed 1920x1080 instead of my previous smaller Windowed 1024x768 resolution to see how far I could stress the engine. That means you can't really compare the results from last week with this week. We also just got our test build out to the SPCT members, so we're still collecting data, but here are the initial results: Live (Unreal 3.0 / APB 1.20) Financial at Midnight on my test machine @ 1920x1080 / Ultra settings. Peak performance = 10.33ms (97FPS) Low performance = 19.22ms (52FPS) *There are some bad hitches in Live that we throw out Average frame time = 16.20ms (61.70FPS) Average Rendering frame time = 16ms Average Game Loop frame time = 16.20ms Engine Upgrade (Unreal 3.5 / APB 2.1) Financial at Midnight on my test machine @ 1920x1080 / Ultra settings. Peak performance = 9.6ms (104FPS) Low performance = 22.38ms (45FPS) Average frame time = 16.05ms (62.31FPS) Average Rendering frame time = 11.7ms Average Game Loop frame time = 16ms PERFORMANCE NOTE: There is a decent difference between Day and Night on APB 2.1. I'm running these tests at night for "worse case" performance. During the day, we see APB 2.1 run 1-2ms faster. There are a million things that could render this benchmark invalid, and I know on the surface this looks like APB 2.1 is now running slightly faster than APB 1.20, but we'll need more performance tests off SPCT machines to really assess where we are. Lighting Here are some before and after shots to illustrate some of the character shadow/lighting issues we have been fixing. If you've been playing on console, then you are well aware of how dark and blobby the game can look some times. LIGHTING NOTE: We're not done with scene lighting or character lighting, so even the updated "new" shots have issues. We aren't rendering ground character shadows properly, and indirect reflected lighting isn't showing up. This is a cropped section of the Enforcer police scene. In the first shot, the characters are all very dark and badly lit with strange hard shadows that don't make a lot of sense with the lighting in the scene. In the second shot, you can see proper lighting on characters. This is still being tuned, but it's 100x better. APB 2.1 old: APB 2.1 new: Here is a cropped section of the new District Select screen. In the first shot, you can see the blobby hard shadows rendering in weird places. In the bottom shot, you can see everyone rendering properly. APB 2.1 old: APB 2.1 new: Thanks, Matt

Hi all, We're working both ends of the queue right now to fix players with issues from the Pumpkin events. We hope to have that issue resolved for those players by Tuesday. 539 unanswered tickets 714 total tickets Responding to tickets created on 10/8, so we are running at around 17 days. Hopefully next week we can get down to 2 weeks response times. If your ticket is older than October 8th, and we have not reached out, then I recommend updating your ticket and asking for status. Thanks, Matt