MattScott

Hi all, I am very proud of the CS team. They handled a lot of extra load this week due to Halloween Pumpkin issues and repeat network outages. I know these numbers came up, but we cleared twice as many tickets as normal. 680 unanswered tickets 808 total tickets We are answering tickets from 10/20 with a couple earlier stragglers. So we are still running at ~19 days. If your ticket is older than October 20th, and we have not reached out, then I recommend updating your ticket and asking for status. Thanks, Matt

Hi there, I announced a week ago that we would be making some network changes to support more local district servers starting with New York. And even though our intention is to make this transition seamless, we have encountered a couple problems. We hope to complete this work soon. Thanks, Matt

Hi all, Login servers are back up. Sorry, Matt

Hi everyone, Our login servers are having some issues. We are working on it right now. I'll post here when it is back online. Thanks, Matt

Hi all, We're currently having a significant network outage. Unfortunately I don't have enough information to predict how long this is going to last. I'll keep you posted. Thanks, Matt

Hi all, We will be fixing the tutorial to remove the RIOT elements soon. Thanks, Matt

Hi there, I 100% sympathize with the console players. Both PS4 and XB1 are terrible at the moment. I try to include console specific details in my weekly Engine Upgrade threads here: Thanks, Matt

Hi there, We are working on this issue. Our provider in Europe is having problems. Thanks, Matt

Hi there, We are working with our provider in the EU on this. The situation has been.. frustrating. Thanks, Matt

Hi there, I just posted the Engine Upgrade update. My time zones are a bit off. Apologies. For the next round of Weapon balance changes, I believe we are waiting to get clear of the Halloween event (ending at patch time early on 11/6 PST) Thanks, Matt

Hi everyone, I'm working remotely, but this was another week spent diagnosing and attempting to fix rather complex issues in the Engine Upgrade. Unfortunately my initial stats from last week didn't hold up across our pool of testers. This is often the frustrating part of this project. We will see good things internally, but once we get the build to testers, the larger sample pool doesn't come back as well. We did figure out that my work laptop is a bad source of data. It has an embedded Intel 3D card and a high performance nVidia 3D card for gaming. However, the nVidia card is daisy chained off the Intel card. SO when I run Fullscreen Windowed, a Desktop Windows Manager process runs at 50% using my Intel GPU while APB 2.1 takes up 80%+ on my nVidia GPU. For whatever reason, that equalizes the performance between APB 1.20 (Live) and APB 2.1 (Engine Upgrade) and over time they run comparably. The team finished the following tasks: - Secondary night time lights are now working - Secondary night time lights are now running nearly as fast as day time. Console players will be happy with that change once we re-submit. - Added some fixes for texture caching, so that as you run around the district we aren't wasting so much CPU to figure out what textures should be loaded. Still working on the following issues: - Rendering shadows from secondary night time lights - Rendering indirect lighting when characters aren't affected by the main light (sun or moon) - Interactive building features still disappear when the player is far enough from a building that it shifts from LOD0 (the highest level of detail) to LOD1. This leaves an empty hole for a period of time where the shop front or window should be. This week marked the first time we ran all the testers through a series of benchmarks in both APB 1.20 and APB 2.1 to try and get accurate performance comparison stats. I know this may sound exciting, but it's not. Often testers have to patch, run the benchmark once to make sure the game finishes loading and caching all the geometry, and then they run the benchmark a second time for accuracy. If anything on their system accidentally interferes, then they have to start over and do it again. Some die hard testers did this for each district. Once they had accurate benchmarks, they sent us their logs, and then yesterday we processed all the logs through our internal tools to graph, identify CPU and GPU, and call out specific bottlenecks. The end of the day was spent posting the results to our internal channels for testers to see everyone's results. Based on that round of testing, we did see an overall increase in performance by 1-2 milliseconds per frame between 10/21 and 10/26. We're now back to being Render locked instead of Game Loop locked. The team is looking at an automated way to prompt beta testers to upload their logs at the end of a session, so we can collate and process them automatically across a larger number of machines to get better insight into performance. At some point, we will likely need to bite the bullet and throw this out to the community to see how it performs. Thanks, Matt

Hi all, I’m traveling right now and missed this yesterday. However, I am told that we have sorted the Red Pumpkin issue for newer tickets to make sure everyone had a chance to complete the event over the weekend. We mostly held steady this week. 561 unanswered tickets 716 total tickets We are answering tickets from 10/13 with a couple earlier stragglers. So we are running at ~19 days. If your ticket is older than October 19th, and we have not reached out, then I recommend updating your ticket and asking for status. Thanks, Matt

Hi everyone, This week we made good progress on performance and character lighting. These changes will help both the PC and console builds. Performance I have changed the test resolution to Fullscreen Windowed 1920x1080 instead of my previous smaller Windowed 1024x768 resolution to see how far I could stress the engine. That means you can't really compare the results from last week with this week. We also just got our test build out to the SPCT members, so we're still collecting data, but here are the initial results: Live (Unreal 3.0 / APB 1.20) Financial at Midnight on my test machine @ 1920x1080 / Ultra settings. Peak performance = 10.33ms (97FPS) Low performance = 19.22ms (52FPS) *There are some bad hitches in Live that we throw out Average frame time = 16.20ms (61.70FPS) Average Rendering frame time = 16ms Average Game Loop frame time = 16.20ms Engine Upgrade (Unreal 3.5 / APB 2.1) Financial at Midnight on my test machine @ 1920x1080 / Ultra settings. Peak performance = 9.6ms (104FPS) Low performance = 22.38ms (45FPS) Average frame time = 16.05ms (62.31FPS) Average Rendering frame time = 11.7ms Average Game Loop frame time = 16ms PERFORMANCE NOTE: There is a decent difference between Day and Night on APB 2.1. I'm running these tests at night for "worse case" performance. During the day, we see APB 2.1 run 1-2ms faster. There are a million things that could render this benchmark invalid, and I know on the surface this looks like APB 2.1 is now running slightly faster than APB 1.20, but we'll need more performance tests off SPCT machines to really assess where we are. Lighting Here are some before and after shots to illustrate some of the character shadow/lighting issues we have been fixing. If you've been playing on console, then you are well aware of how dark and blobby the game can look some times. LIGHTING NOTE: We're not done with scene lighting or character lighting, so even the updated "new" shots have issues. We aren't rendering ground character shadows properly, and indirect reflected lighting isn't showing up. This is a cropped section of the Enforcer police scene. In the first shot, the characters are all very dark and badly lit with strange hard shadows that don't make a lot of sense with the lighting in the scene. In the second shot, you can see proper lighting on characters. This is still being tuned, but it's 100x better. APB 2.1 old: APB 2.1 new: Here is a cropped section of the new District Select screen. In the first shot, you can see the blobby hard shadows rendering in weird places. In the bottom shot, you can see everyone rendering properly. APB 2.1 old: APB 2.1 new: Thanks, Matt

Hi all, We're working both ends of the queue right now to fix players with issues from the Pumpkin events. We hope to have that issue resolved for those players by Tuesday. 539 unanswered tickets 714 total tickets Responding to tickets created on 10/8, so we are running at around 17 days. Hopefully next week we can get down to 2 weeks response times. If your ticket is older than October 8th, and we have not reached out, then I recommend updating your ticket and asking for status. Thanks, Matt

Hi there, We have been struggling with issues at the provider in Europe. We are rebooting servers (again) to try and clear the issue. Thanks, Matt

Hi all, We are restarting districts to fix the lag. Thanks, Matt

Hi all, This was a bug that we patched early on Thursday. For anyone stuck on pumpkins, please open a ticket with Support. I am having the team check these immediately to allow players to continue. Thanks, Matt

Hi all, We've been reading this feedback, and there were some good lessons learned on Halloween. In the future, we are looking at producing more general designs that have more re-usability. Thanks, Matt

Hi everyone, These are turning into Saturday updates, since that seems to be time when I can sit down and recap the work from the week. Right now, our #1 priority is performance. Performance for the Engine Upgrade was pretty good prior to when we merged all the code and assets from Live. After that, our performance has really struggled. For this post, I'm going to breakdown how performance is measured by our team, and what we're looking at. First, we spent this week finishing off items from above, and re-writing/improving the statistics that we collect in-game. This took significant effort to revamp old code and measure specific parts of the rendering and game loops so we can see where most of the time is spent. We also synced this new code between APB 1.20 (Unreal 3) and APB 2.1 (Unreal 3.5), and we created a new export process to chart performance in Excel. Second, we took this new stats engine and ran a series of test runs that isolate as many variables as possible. For instance, we tracked performance in Financial at midnight across APB 1.20 and APB 2.1. We use night time, because the game generally runs worse at night due to the extra lights that turn on. Each of us used different test machines, but mine was on the less powerful side with a 4 core i7-7700HQ @ 2.80GHz, an Nvidia GTX 1050, and 32GB ram. For the test we ran at the highest settings but used 1024x768 windowed mode to generate a lot of windows background interference. We also run no vsync and no smooth frame rate to get the highest rates possible. We measure a game's performance based on the low, high, and average frame time in milliseconds. Here are some common frame time to frame rate benchmarks values. 10ms frame time = 100FPS 16ms frame time = 60FPS 25ms frame time = 40FPS 40ms frame time = 25FPS Unreal 3 has always run the Render thread and Game Loop thread in parallel, but each process runs its tasks one at a time in sequence. The game's total frame time is the overlapping time from the Render thread and the Game Loop thread per frame. The Render Thread is mostly work done on your graphics card. The Game Loop thread is all the logic and keeping track of everything else including world state, physics, and player state. The Game Loop is dominated heavily by time spent ticking (processing) each relevant thing on the screen to make sure it's updated. All of this work resulted in a series of benchmarks that all showed the same general result. Live (Unreal 3.0 / APB 1.20) Financial at Midnight on my test machine. Peak performance = 10ms (100FPS) Low performance = 19ms (50FPS) *There are some bad hitches in Live that we throw out Average frame time = 12ms (83FPS) Average Rendering frame time = 12ms Average Game Loop frame time = 12ms Engine Upgrade (Unreal 3.5 / APB 2.1) Financial at Midnight on my test machine. Peak performance = 10ms (100FPS) Low performance = 19ms (50FPS) Average performance = 15ms (66FPS) Average Rendering frame time = 8.5ms Average Game Loop frame time = 15ms Right away, we can see that the Peak and Low performance of the game are the same. But there is a 3ms frame time / 17FPS difference in the Average frame time. Where APB 1.20 is more balanced between Rendering and Game Loop, we see a lopsided workload in APB 2.1. The Rendering thread is quite fast, but the entire frame time is held back by how long the Game Loop takes to run. For the rest of the week, we iterated on this process to improve our stat collection system and dig deeper into specific chunks of the Game Loop so that we could directly compare the performance of those smaller sections between Live and the Engine Upgrade. We were successful in identifying 2 Game Loop sections (out of 35) where APB 2.1 ran a total of 4.5ms slower than APB 1.20. We are now digging further into those sections to create more detailed stats to help further breakdown performance until we can define a fix to the performance difference. I'll go on record saying a couple things. The performance in APB 2.1 is already significantly better than what is running on consoles right now (APB 2.0). We will be finishing console blockers as soon as possible and submitting so the console players can get a much better experience. If I felt the performance was good enough right now, then I would be opening the Beta and putting this on public OTW. We don't have any other major blockers. The art issues and minor bugs can all be addressed after we let players in to test. Thanks, Matt

Hi all, Definitely getting caught up. 464 unanswered tickets 676 total tickets Responding to tickets created on 9/28, so we are running at around 3 weeks. If your ticket is older than September 28th, and we have not reached out, then I recommend updating your ticket and asking for status. Thanks, Matt

Hi all, We have been suffering from small, but consistent outages with our network provider in Europe for the last two to three weeks. Today we hit a significant outage in North America. These are not attacks against us. They just seem to be issues with the network. The team is working with our provider to get this fixed as soon as possible. Thanks, Matt

Hi all, Sorry for the miscommunication. I was traveling all weekend, and didn’t get a chance to connect with my network team. It looks like we got our time zones crossed. At 12:00 UTC, the Fallen Earth servers went offline. We spent all of our effort on the Last Stand event, so there was no time to make a cool countdown or other last moment. Again, I want to thank all the players who came back to say goodbye. We will be posting updates on the reboot here. Sincerely, Matt

Hi everyone, I am traveling right now, so I missed yesterday's update. I'm going to summarize the week here, and then modify the main post. Performance updates - this has been a little frustrating. We've seen side by side comps where the new engine outperforms or matches Live. But right now, the build being tested by SPCT is performing terrible compared to Live. We spent part of the week rebuilding our stat collecting system to gather deeper data related to the game loop. Various disconnect crashes - These have all been fixed (for the moment). Texture streaming bug - We addressed a couple more issues, and I'm treating this as finished. NPC / Living city character lighting issue - This is fixed but I have a couple non-essential changes I want before go live. Character customization saving is broken - This is fixed and needs to be deployed with the next build. Streetlamps in Financial - This turned into a huge rabbit hole, but we managed to get it all fixed. We need to re-run lighting for Financial before the next build goes up. This process runs overnight in the office and uses all the available computers on the network, and even then it still takes more than 8 hours to complete. We found a lot of new bugs that have been logged, but I don't have access to the list at the moment. I'll try to rework the list in the main post on my next update. Thanks, Matt

Hi all, I missed this yesterday. So the stats as of 10/11 were: 603 unanswered tickets 807 total tickets Responding to tickets created on 9/14, so we are running at around 27 days. If your ticket is older than September 14th, and we have not reached out, then I recommend updating your ticket and asking for status. Thanks, Matt

Hi all, We are about to start transitioning our architecture to support local districts and the world consolidation. We are going to do our best to eliminate as much down time as possible, but this is another big server transition. Our estimate is that we can accomplish this over the next couple weeks. When we're done, we should have districts in LA and NY for North America, which will be our first pilot of this new structure. Thanks, Matt