Jump to content
Forum rules

MattScott

CEO
 View Profile  See their activity

  • Content Count

    1281

  • Joined

  • Last visited

Posts posted by MattScott

  6. Posted

    Hi everyone,

     

    I'm working remotely, but this was another week spent diagnosing and attempting to fix rather complex issues in the Engine Upgrade. 

     

    Unfortunately my initial stats from last week didn't hold up across our pool of testers. This is often the frustrating part of this project. We will see good things internally, but once we get the build to testers, the larger sample pool doesn't come back as well.

     

    We did figure out that my work laptop is a bad source of data. It has an embedded Intel 3D card and a high performance nVidia 3D card for gaming. However, the nVidia card is daisy chained off the Intel card. SO when I run Fullscreen Windowed, a Desktop Windows Manager process runs at 50% using my Intel GPU while APB 2.1 takes up 80%+ on my nVidia GPU. For whatever reason, that equalizes the performance between APB 1.20 (Live) and APB 2.1 (Engine Upgrade) and over time they run comparably. 

     

    The team finished the following tasks:

    - Secondary night time lights are now working
    - Secondary night time lights are now running nearly as fast as day time. Console players will be happy with that change once we re-submit.

    - Added some fixes for texture caching, so that as you run around the district we aren't wasting so much CPU to figure out what textures should be loaded.

     

    Still working on the following issues:

    - Rendering shadows from secondary night time lights

    - Rendering indirect lighting when characters aren't affected by the main light (sun or moon)

    - Interactive building features still disappear when the player is far enough from a building that it shifts from LOD0 (the highest level of detail) to LOD1. This leaves an empty hole for a period of time where the shop front or window should be.

     

    This week marked the first time we ran all the testers through a series of benchmarks in both APB 1.20 and APB 2.1 to try and get accurate performance comparison stats.

     

    I know this may sound exciting, but it's not. Often testers have to patch, run the benchmark once to make sure the game finishes loading and caching all the geometry, and then they run the benchmark a second time for accuracy. If anything on their system accidentally interferes, then they have to start over and do it again. Some die hard testers did this for each district. Once they had accurate benchmarks, they sent us their logs, and then yesterday we processed all the logs through our internal tools to graph, identify CPU and GPU, and call out specific bottlenecks. The end of the day was spent posting the results to our internal channels for testers to see everyone's results.

     

    Based on that round of testing, we did see an overall increase in performance by 1-2 milliseconds per frame between 10/21 and 10/26.

    We're now back to being Render locked instead of Game Loop locked.

     

    The team is looking at an automated way to prompt beta testers to upload their logs at the end of a session, so we can collate and process them automatically across a larger number of machines to get better insight into performance.

     

    At some point, we will likely need to bite the bullet and throw this out to the community to see how it performs.

     

    Thanks,
    Matt

    • Like 23
    • Thanks 10

  7. Posted

    Hi all,

     

    I’m traveling right now and missed this yesterday. However, I am told that we have sorted the Red Pumpkin issue for newer tickets to make sure everyone had a chance to complete the event over the weekend.

     

    We mostly held steady this week.

     

    561 unanswered tickets

    716 total tickets

     

    We are answering tickets from 10/13 with a couple earlier stragglers. So we are running at ~19 days.

    If your ticket is older than October 19th, and we have not reached out, then I recommend updating your ticket and asking for status.

     

    Thanks,

    Matt

    • Like 2

  8. Posted

    Hi everyone,


    This week we made good progress on performance and character lighting.

    These changes will help both the PC and console builds.

     

    Performance

     

    I have changed the test resolution to Fullscreen Windowed 1920x1080 instead of my previous smaller Windowed 1024x768 resolution to see how far I could stress the engine. That means you can't really compare the results from last week with this week. We also just got our test build out to the SPCT members, so we're still collecting data, but here are the initial results:

     

    Live (Unreal 3.0 / APB 1.20) Financial at Midnight on my test machine @ 1920x1080 / Ultra settings. 
    Peak performance = 10.33ms (97FPS)
    Low performance = 19.22ms (52FPS) *There are some bad hitches in Live that we throw out
    Average frame time = 16.20ms (61.70FPS)
    Average Rendering frame time = 16ms
    Average Game Loop frame time = 16.20ms

     

    Engine Upgrade (Unreal 3.5 / APB 2.1) Financial at Midnight on my test machine @ 1920x1080 / Ultra settings.
    Peak performance = 9.6ms (104FPS)
    Low performance = 22.38ms (45FPS)
    Average frame time = 16.05ms (62.31FPS)
    Average Rendering frame time = 11.7ms
    Average Game Loop frame time = 16ms

     

    PERFORMANCE NOTE: There is a decent difference between Day and Night on APB 2.1.

    I'm running these tests at night for "worse case" performance. During the day, we see APB 2.1 run 1-2ms faster.

     

    There are a million things that could render this benchmark invalid, and I know on the surface this looks like APB 2.1 is now running slightly faster than APB 1.20, but we'll need more performance tests off SPCT machines to really assess where we are.

     

    Lighting

     

    Here are some before and after shots to illustrate some of the character shadow/lighting issues we have been fixing.

    If you've been playing on console, then you are well aware of how dark and blobby the game can look some times.

     

    LIGHTING NOTE: We're not done with scene lighting or character lighting, so even the updated "new" shots have issues. We aren't rendering ground character shadows properly, and indirect reflected lighting isn't showing up.

     

    This is a cropped section of the Enforcer police scene.

     

    In the first shot, the characters are all very dark and badly lit with strange hard shadows that don't make a lot of sense with the lighting in the scene.

    In the second shot, you can see proper lighting on characters. This is still being tuned, but it's 100x better.

     

    APB 2.1 old:

    EnforcerCharacter35old.PNG

     

    APB 2.1 new:

    EnforcerCharacter35new.PNG

     

    Here is a cropped section of the new District Select screen.

    In the first shot, you can see the blobby hard shadows rendering in weird places.

    In the bottom shot, you can see everyone rendering properly.

     

    APB 2.1 old:

    DistrictSelect35old.PNG

     

    APB 2.1 new:

    DistrictSelect35new.PNG

     

    Thanks,
    Matt

    • Like 41
    • Thanks 10

  9. Posted

    Hi all,

     

    We're working both ends of the queue right now to fix players with issues from the Pumpkin events. We hope to have that issue resolved for those players by Tuesday. 

     

    539 unanswered tickets

    714 total tickets

     

    Responding to tickets created on 10/8, so we are running at around 17 days.

    Hopefully next week we can get down to 2 weeks response times.

    If your ticket is older than October 8th, and we have not reached out, then I recommend updating your ticket and asking for status.

     

    Thanks,

    Matt

    • Like 5

  14. Posted

    Hi everyone,

     

    These are turning into Saturday updates, since that seems to be time when I can sit down and recap the work from the week.

     

    Right now, our #1 priority is performance. Performance for the Engine Upgrade was pretty good prior to when we merged all the code and assets from Live. After that, our performance has really struggled.

     

    For this post, I'm going to breakdown how performance is measured by our team, and what we're looking at.

     

    First, we spent this week finishing off items from above, and re-writing/improving the statistics that we collect in-game. This took significant effort to revamp old code and measure specific parts of the rendering and game loops so we can see where most of the time is spent. We also synced this new code between APB 1.20 (Unreal 3) and APB 2.1 (Unreal 3.5), and we created a new export process to chart performance in Excel.

     

    Second, we took this new stats engine and ran a series of test runs that isolate as many variables as possible. For instance, we tracked performance in Financial at midnight across APB 1.20 and APB 2.1. We use night time, because the game generally runs worse at night due to the extra lights that turn on. Each of us used different test machines, but mine was on the less powerful side with a 4 core i7-7700HQ @ 2.80GHz, an Nvidia GTX 1050, and 32GB ram. For the test we ran at the highest settings but used 1024x768 windowed mode to generate a lot of windows background interference. We also run no vsync and no smooth frame rate to get the highest rates possible.

     

    We measure a game's performance based on the low, high, and average frame time in milliseconds. Here are some common frame time to frame rate benchmarks values.

    • 10ms frame time = 100FPS
    • 16ms frame time = 60FPS
    • 25ms frame time = 40FPS
    • 40ms frame time = 25FPS

     

    Unreal 3 has always run the Render thread and Game Loop thread in parallel, but each process runs its tasks one at a time in sequence. The game's total frame time is the overlapping time from the Render thread and the Game Loop thread per frame. The Render Thread is mostly work done on your graphics card. The Game Loop thread is all the logic and keeping track of everything else including world state, physics, and player state. The Game Loop is dominated heavily by time spent ticking (processing) each relevant thing on the screen to make sure it's updated.

     

    All of this work resulted in a series of benchmarks that all showed the same general result.

     

    Live (Unreal 3.0 / APB 1.20) Financial at Midnight on my test machine. 

    • Peak performance = 10ms (100FPS)
    • Low performance = 19ms (50FPS) *There are some bad hitches in Live that we throw out
    • Average frame time = 12ms (83FPS)
    • Average Rendering frame time = 12ms
    • Average Game Loop frame time = 12ms

     

    Engine Upgrade (Unreal 3.5 / APB 2.1) Financial at Midnight on my test machine.

    • Peak performance = 10ms (100FPS)
    • Low performance = 19ms (50FPS)
    • Average performance = 15ms (66FPS)
    • Average Rendering frame time = 8.5ms
    • Average Game Loop frame time = 15ms

     

    Right away, we can see that the Peak and Low performance of the game are the same. But there is a 3ms frame time / 17FPS difference in the Average frame time. Where APB 1.20 is more balanced between Rendering and Game Loop, we see a lopsided workload in APB 2.1. The Rendering thread is quite fast, but the entire frame time is held back by how long the Game Loop takes to run.

     

    For the rest of the week, we iterated on this process to improve our stat collection system and dig deeper into specific chunks of the Game Loop so that we could directly compare the performance of those smaller sections between Live and the Engine Upgrade. We were successful in identifying 2 Game Loop sections (out of 35) where APB 2.1 ran a total of 4.5ms slower than APB 1.20.

     

    We are now digging further into those sections to create more detailed stats to help further breakdown performance until we can define a fix to the performance difference.

     

    I'll go on record saying a couple things.

    1. The performance in APB 2.1 is already significantly better than what is running on consoles right now (APB 2.0). We will be finishing console blockers as soon as possible and submitting so the console players can get a much better experience.
    2. If I felt the performance was good enough right now, then I would be opening the Beta and putting this on public OTW. We don't have any other major blockers. The art issues and minor bugs can all be addressed after we let players in to test.

     

    Thanks,

    Matt

    • Like 39
    • Thanks 13

  15. Posted

    Hi all,

     

    Definitely getting caught up.

    464 unanswered tickets

    676 total tickets

     

    Responding to tickets created on 9/28, so we are running at around 3 weeks.

    If your ticket is older than September 28th, and we have not reached out, then I recommend updating your ticket and asking for status.

     

    Thanks,

    Matt

    • Like 6

  16. Posted

    Hi all,


    We have been suffering from small, but consistent outages with our network provider in Europe for the last two to three weeks.

    Today we hit a significant outage in North America. These are not attacks against us. They just seem to be issues with the network.

     

    The team is working with our provider to get this fixed as soon as possible.

     

    Thanks,
    Matt

    • Like 8
    • Thanks 2

  17. Posted

    Hi all,

     

    Sorry for the miscommunication. I was traveling all weekend, and didn’t get a chance to connect with my network team. It looks like we got our time zones crossed.

     

    At 12:00 UTC, the Fallen Earth servers went offline. We spent all of our effort on the Last Stand event, so there was no time to make a cool countdown or other last moment. 

     

    Again, I want to thank all the players who came back to say goodbye. We will be posting updates on the reboot here.

     

    Sincerely,

    Matt

    • Like 2

  18. Posted

    Hi everyone,

     

    I am traveling right now, so I missed yesterday's update.

    I'm going to summarize the week here, and then modify the main post.

    • Performance updates - this has been a little frustrating. We've seen side by side comps where the new engine outperforms or matches Live. But right now, the build being tested by SPCT is performing terrible compared to Live. We spent part of the week rebuilding our stat collecting system to gather deeper data related to the game loop.
    • Various disconnect crashes - These have all been fixed (for the moment).
    • Texture streaming bug - We addressed a couple more issues, and I'm treating this as finished.
    • NPC / Living city character lighting issue - This is fixed but I have a couple non-essential changes I want before go live.
    • Character customization saving is broken - This is fixed and needs to be deployed with the next build.
    • Streetlamps in Financial - This turned into a huge rabbit hole, but we managed to get it all fixed. We need to re-run lighting for Financial before the next build goes up. This process runs overnight in the office and uses all the available computers on the network, and even then it still takes more than 8 hours to complete.

     

    We found a lot of new bugs that have been logged, but I don't have access to the list at the moment. I'll try to rework the list in the main post on my next update.

     

    Thanks,
    Matt

    • Like 50
    • Thanks 9

  19. Posted

    Hi all,

     

    I missed this yesterday. So the stats as of 10/11 were:

     

    603 unanswered tickets

    807 total tickets

     

    Responding to tickets created on 9/14, so we are running at around 27 days.

    If your ticket is older than September 14th, and we have not reached out, then I recommend updating your ticket and asking for status.

     

    Thanks,

    Matt

    • Like 3
    • Thanks 1

  20. Posted

    Hi all,

     

    We are about to start transitioning our architecture to support local districts and the world consolidation.

    We are going to do our best to eliminate as much down time as possible, but this is another big server transition.

    Our estimate is that we can accomplish this over the next couple weeks.

     

    When we're done, we should have districts in LA and NY for North America, which will be our first pilot of this new structure.

     

    Thanks,
    Matt

    • Like 31
    • Thanks 12

  21. Posted

    7 hours ago, Kitty Yekaterina said:

    Same happened for giving out double-b harness, @MattScott  you said you gonna fix the double b shotguns into an X angle as double b contact got, but now for a week and more nobody is mentioning it anymore. It wouldn't have happened if someone simply took 30 seconds of paying attention to details,  to login financial and literally copy pasted double b harness into armas.  I am having a feeling that somebody regarded for clothes, is often making mistakes due to hastiness.

    Same for the useless knife fix - (aka removal) on left shoulder while having katana...really no point of having pocket knife with 2 meters long katana...

    These changes have been made for the male harnesses. Just waiting on the female harnesses. We were very conservative about the angle of the shotguns for other reasons. The new angled version requires a different rig to make sure we avoid as much clipping as possible.

    • Like 4

  22. Posted

    Hi everyone,

     

    The entire team has been crunching for September on the Engine Upgrade (including me), so I haven't had much time to blog lately.

    However, I have seen a lot of requests for updates, so I thought I would try something a little different.

     

    As I said on the latest Q&A, we're done merging all the content and code. We are just fixing bugs now.

     

    So below I'm going to list the exact bugs that we need to fix before we do the public OTW test.

    You'll notice we're not quite at the stage of fixing actual gameplay or collision issues yet.

    I'll try updating this thread each Friday to clear or add bugs as we move forward.

     

    All platform issues:

     

    • Performance benchmarks - We've made a ton of changes recently, so we need all new benchmarks. From our earlier work, we feel the game is running well in terms of multithreaded rendering. The render threads are out performing the game thread. So all of our latest optimizations have been surrounding ways to improve core game logic.
      • STATUS: Ongoing

     

    • Various disconnect crashes - This only happens in "final" builds that we release to players. There is a crash in the game whenever you get disconnected from a server for any reason. This causes the client to flag itself for a full file repair, which is very annoying for testers.
      • STATUS: In progress

     

    • Level of Detail issue - This was extremely noticeable in the video that got released. Buildings popping in horribly. Trees disappearing. Etc. Turns out there was a tremendous amount of custom code in Live used to dynamically morph between various building LODs and to support individually lit windows at night. Reloaded tossed all this original code because none of it was supported under Unreal 3.5, and they just adopted the stock LOD system. Obviously this left many bugs. The team was able to pull in all the code we needed from the old engine and combine it with the new Unreal 3.5 LOD system.
      • STATUS: Mostly fixed / Window lights on LOD0 at night don't work in "final" builds for players

     

    • Texture streaming bug - Even though this isn't a crash bug, it has been our #1 issue. Texture streaming is a unique system RTW wrote to prioritize textures based on various methods, so that when the player's machine or console runs out of texture memory, the game can downrez textures that aren't used very much (or aren't very noticable). This is a lot harder than it sounds, and there are many different methods used (texture size rendered to the screen, distance from the furthest object using the texture, special flags for signage and other highly detailed textures, etc). For the most part, this is working MUCH better now. The game looks amazing. There are a bunch of signs in the districts that need to get special texture streaming flags, and in some cases we see oddball textures get downrezzed that shouldn't.
      • STATUS: Mostly fixed

     

    • NPC / Living city character lighting issues - This was very weird. We started noticing that sometimes many of the pedestrians were heavily under lit with a weird orange lighting that didn't match the scene. We've been chasing this for a bit, and found that the code for reflective lights was broken. The team was able to determine that reflective light values were being initialized based on whatever main direction light was active (or inactive) in the scene when they loaded into the district. This is being fixed so that we can control the reflective light values from our Time of Day data.
      • STATUS: In progress / Mostly fixed

     

    • Character customization saving is broken - There's a database change necessary in our external SPCT testing environment. The next push should fix it.
      • STATUS: In progress

     

    • Color Lookup Table grading is broken - This is a general post processing feature in Unreal where artists can apply color LUT files to modify the look of the scene. Right now, we're seeing some high thresholding / banding when LUTs are used. 
      • STATUS: Removed so the artifact doesn't show up / Awaiting dev resource

     

    • Time of Day color values are broken - The lighting model for Unreal 3.5 is completely different than Unreal 3.0. As such, all of the values used to handle colors, depth of field, bloom, etc for midnight, dawn, noon, evening and night time need to be polished up.
      • STATUS: In progress

     

    • Streetlamps in Financial - It looks like an artist at Reloaded swapped all the spotlights to point lights for any lighting that comes from streetlamps in the district. In order to create the light halo on the sidewalk, they had to make the radius quite large. This means when the streetlamp is next to a building there is a bright patch where the light is affecting the wall. The team went back through and swapped back in the spotlights, but we have a bug in the prefab that needs our Financial map rebuilt.
      • STATUS: Fixed but 2nd issue needs fixing

     

    • Building feature LOD issue - We have a visual artifact because the rendering of mission features on buildings is not synced to the building LOD code. This means that for a small period of time we sometimes see holes in buildings where the features should be.
      • STATUS: Awaiting dev resource

     

     

    We have submitted two console builds, and gotten kicked back for defects.

    Right now we're 94% compliant with the necessary checks that both Microsoft and Sony require to approve a title.

     

    Xbox One: 21 total must fix bugs (hightlights below)

    • Title terminates on disconnecting the network cable anywhere in the title
    • Null Characters are displayed on launching the title in any supported language other than English
    • Background art and MAP fails to render on Xbox One X kit
    • Title terminates on launching the tile in an Xbox One X kit 
    • The title crashes on suspending the title during the splash screen
    • Title crashes in keeping the console idle for more than 20 minutes on the Music player
    • Game Event limitation popup is displayed during gameplay 
    • SpaceBar is displayed for jump in Tutorial during gameplay
    • Unnecessary Resolution option is displayed in Video Options
    • Title crashes on adding friends from inside the game
    • Title displays a black screen for more than 12 seconds right after the initial splash screen
    • Title displays unnecessary button call out during gameplay
    • Title allows the muted user to communicate through text during online gameplay
    • Title allows the blocked user to send Group Invite
    • User is unable to dismiss the interact menu by pressing B button after interacting with another user
    • Title randomly crashes during normal gameplay
    • The achievement 'Olympic Contender' fails to unlock on meeting its criteria
    • User is unable to scroll thorough the description using RS in descriptions 
    • Title does not provide an option to invite another user to the game

     

    PlayStation 4: 12 total must fix bugs (highlights below)

    • The term 'Sony Entertainment Network account' is displayed in the title screen
    • Application provides 'Resolution' setting in the 'Video' tab
    • Application soft locks upon selecting a user from the ‘Search’ tab of 'GROUP & FRIENDS'
    • Gameplay - Incorrect naming convention is displayed in ‘Tutorial’
    • Incomplete online ID is displayed in the ‘Friends’ tab in ‘GROUP & FRIENDS’
    • Application fails to transition to a mode which supports play together
    • User B fails to join the User A's session using 'Join' feature
    • Application crashes on exploding a bunch of cars during gameplay
    • Application incorrectly displays the SIE-specified text in the ToS
    • Gameplay - Incorrect naming convention is displayed in 'Drive the vehicle 500 meters' tutorial
       

    Thanks,
    Matt

     

     

    • Like 61
    • Thanks 43

  23. Posted

    Hi everyone,

     

    I'm posting early today, due to meetings later.

     

    Big push this week. I think we're through a large chunk of the Nekrova players who had migration issues.

    There are now 905 total tickets and 674 tickets that have not yet had a response. That's a big drop from last week.

     

    Today we are answering tickets from September 9th, which means that we are at 25 day response time, which is nearly a week lower than last week.

    If your ticket is older than September 9th, and we have not reached out, then I recommend updating your ticket and asking for status.

     

    Thanks,
    Matt

    • Like 3
    • Thanks 1

  24. Posted

    Hi everyone,

     

    We're going to pause this effort for a number of different reasons:

     

    1) Most importantly, our email newsletter system is being rebuilt, and we have not been able to properly message individual accounts that are in danger of losing their names. It's critical we give everyone enough lead time to respond and login.

     

    2) Also I was made aware of a large number of players who are planning on griefing/squatting on released names tomorrow. I feel like this is simply going to create the same problem, and since players can create endless emails to register endless new accounts - we can't restrict how many names they can claim.

     

    3) I had made the core assumption that we needed to release these names in order to facilitate a smoother consolidation of Jericho and Citadel, but that strategy has also been changed. The end result will be the same, but we have come up with a new way of identifying players that will eliminate conflicts.

     

    In summary, the name release WILL HAPPEN. For those that bought name changes, we're not that far out.

    I just need to think on how to do it in a way that wont get abused.

     

    Thanks,
    Matt

    • Like 23
    • Thanks 5
×
×
  • Create New...