Jump to content
Forum rules

MattScott

CEO
 View Profile  See their activity

  • Content Count

    1281

  • Joined

  • Last visited

Posts posted by MattScott

  1. Posted

    Hi all,

     

    It appears that being told to stay home has inspired many players to check out or come back to APB.

    We saw a massive amount of new and old players join the game this week which has led to much higher player populations on the servers.

    We also saw a fairly large influx of new tickets asking for access to get back into accounts and other things with a spike on Monday and Tuesday. 

     

    Along side that, my team transitioned to working remotely due to coronavirus which meant several days of getting setup and moving computers around.

     

    However, I am very proud of our CS team. They hunkered down and powered through tickets - despite all the setbacks.

     

    As of today we have:

    148 new tickets

    298 total tickets

     

    And most impressively, we are now (for real this time) responding to tickets from 3/16, which means we are getting back to players in 4 days.

    This is the lowest we have ever been.

     

    Please stay safe and enjoy the extra mini-game events with our GMs this weekend.

     

    Thanks,
    Matt

    • Like 12
    • Thanks 8

  2. Posted

    Hi all,

     

    This is a totally valid criticism from the OP. I wasn’t aware Reloaded had done this in the past, and we didn’t do it for the Thunder or the New Glory. But it makes total sense on why players would want to see it so they can decide whether to buy the box or now.

     

    We have had a second redesign of ARMAS on the books for quite a while to fix all the small images and lack of detail.

     

    I like the production values for the video posted in this thread. Nicely done!
     

    Thanks,

    Matt

    • Like 1
    • Thanks 2

  3. Posted

    Hi everyone,

     

    First, for those who haven't seen my announcement from last night, Little Orbit is now requiring all our staff to work remotely to insure their safety and prevent the spread of coronavirus.

     

    Next, we spent a portion of the week organizing for this move, but we were able to solve the bad character shadows and lighting issues.

    This has been a major blocker, and it allowed us to rebuild the lighting for each district.

    The results are already 100% better - even without any actual lighting changes from the art team.

     

    However, on Friday my team was finally able to get a 100% reproducible scenario to fix a memory crash that has been plaguing us. The old code was 32-bit, and the new code is supposed to be 64-bit across the board, but there are many tiny exceptions that need fixing to provide overall stability. Unfortunately that means the game isn't in a runnable state right now.

     

    Once we get this issue fixed and a new Beta build is released internally, I'll post some shots at various times of day and in some of the other districts so you can see the difference.

     

    Thanks,

    Matt

    • Like 35
    • Thanks 12

  4. Posted

    Hi everyone,

     

    I'm sure by now that everyone has seen the impact of coronavirus (COVID-19) on their city, work, schools, and entertainment.

     

    Starting Monday the 16th until the foreseeable future, all Little Orbit employees will be working remotely until we get different guidance from health officials. We prepped all week to make sure teams had hardware and instructions on how to check in during the day with their leads for tasks and updates. 

     

    Come hell or high water, the team is dedicated to shipping the Engine Upgrade, and we didn't get this close to be sidelined by a little virus.

     

    I know this may be a little scary right now. And APB is just a game. But since many of you will be temporarily home due to work or supporting family members, we want to insure that the game is online and available for all our players to distract themselves with some missions, fight club, or a little virtual socialization in Breakwater.

     

    I'm also going to be turning on Free Premium for all players across all platforms.

    NOTE: We will be snapshotting the active Premium days for everyone, and we'll extend those days once everything goes back to normal.

    Hopefully that will make Social a little more fun for the time being.

     

    I'm also going to ask for everyone's patience with us.

    We are going to do our best to make this as smooth as possible.


    And most of all, wash your hands. Don't touch your face. Avoid large gatherings.

    Stay safe.

     

    Thanks,
    Matt

    • Like 40
    • Thanks 28

  5. Posted

    Hi all,

     

    I need to make a correction to last week's numbers. Our CS manager was out, so Selali and I had to run the stats, and we messed up.

    It turns out that we hadn't yet reached 2/28, so as we moved through tickets this week several players were confused.

    There was also a general surge in tickets and players returning to check out the game.

     

    With that course correction in mind, here are this week's numbers.

    171 new tickets

    281 total tickets

     

    We are now responding to tickets from 2/29, which puts us at 13 days to respond.

     

    Thanks,
    Matt

    • Thanks 2

  6. Posted

    Hi all,

     

    I’d like to weigh in here and try to eliminate the confusion. In the spirit of transparency, I post undoctored shots to the Engine Upgrade thread. So that is how APB 2.1 looks right now.

     

    But none of the districts have final lighting or color balance. I want to see solid performance before we spend any more time on look and feel.


    Having said that, there are lots of knobs and dials we hope to use soon in order to get the proper look that compliments the gameplay and still feels like APB.

     

    Thanks,

    Matt

    • Like 12
    • Thanks 6

  7. Posted

    Hi everyone,

     

    The last two updates have been pretty sparse, so this week I'll take a little more time to explain where we are at.

     

    First, a portion of the team has now moved back to bugs, which is good. As much as we want the performance to get better, we also need to fix actual gameplay issues too.

    Right now we are tracking 20 bugs identified by SPCT or QA. 2 of which were determined to be duplicates. 4 of which were checked in as fixed by Friday, which leaves 14 unaddressed bugs.

     

    Second, there are two overall issues we are still working on - Lighting and Performance.


    For lighting, we've narrowed down the lighting issue to a system in Unreal 3.5 that handles "Indirect" lights. This is a process that gets pre-processed and packaged when we ship the game. It looks like because Reloaded never turned on Day/Night, they also never tuned indirect lights that are largely only visible at night. Part of the team has now started diving into these tools to work through all our maps to fix this.

     

    For performance, I've started to realize there is no silver bullets - something that we'll find and fix that will make the game suddenly run great. However, the good news is that we've tested Asylum across a number of systems, and APB 2.1 runs that district better or at the same frame rate as APB 1.20 (Live). 

     

    On one of my higher end systems with an i7 with an RTX 2080, here is the comparison with both running 1920x1080:

     

    APB 1.20 Asylum on Maximum (ignoring micro stutters)

    1% Min = 6.5ms 153FPS

    Average = 4.85ms 206FPS

    1% Max = 3.15ms 316FPS

     

    APB 2.1 Asylum on Very High (which is equivalent to Maximum)

    1% Min = 4.68ms 213FPS

    Average = 3.71 269FPS

    1% Max = 3.02ms 330FPS

     

    I ran a test on a lower end laptop with an i7 and a GTX 1050. Amazingly APB ran at an average of 100FPS on APB 2.1 which was about 10FPS faster that APB 1.20.

     

    We have spent months getting the engine to take advantage of more cores.

    Here is what APB 1.20 Asylum looks like on that same i7 with 12 logical processors.

    APB1.20-performance.PNG

     

    And here is what APB 2.1 Asylum looks like on the same i7.

    APB2.1-performance.PNG

     

    Other districts are more hit or miss right now, but using Asylum as a benchmark, we started looking at differences between the other districts to help us narrow down the remaining performance issues. We started to recognize very specific areas where frame rates went down significantly, and that didn't happen in Asylum. Unfortunately there are so many areas in this engine that simply weren't finished sometimes I feel like we're looking for a needle in a haystack.

     

    Or as it turns out this week: a blade of grass in a park

     

    For those playing at home, I'll give you a little sample of what we deal with on a weekly basis.

    Here are two screenshots. If you look specifically at the foliage, can you spot a potential issue?

     

    APB 1.20:

    APB1.20-foliage.png

     

    APB 2.1:

    APB2.1-foliage.png

     

    The foliage in the top shot (APB 1.20) is set to only render very close in. You can barely see it from where the screenshot is taken, and that is on Maximum settings.

    The foliage in the bottom shot (APB 2.1) can be seen all the way across the district. In fact it has no draw distance limit set. That means 2.1 is currently rendering thousands of tiny polygons that each cast shadows on each other while the scene elements cast shadows onto them.

     

    The team started working on this Friday, and will be cleaning it up this week.

     

    Thanks,
    Matt

     

    EDITED: Added frame rate stats and cleaned up grammar.

    • Like 49
    • Thanks 17

  9. Posted

    Hi all,

     

    I wish I had more visuals to show for this week. Those are always more fun to post than the bland technical write-ups.

     

    We fixed several memory crashes, but the Xbox One still has an issue where we run out of memory entirely. There are three big blocks that we're looking into (shaders, textures, and audio). We'll need to investigate ways to cut those down. The crash doesn't show up right away. You have to run around a bit in Financial. We believe it's only barely over the limit based on new content that was added since the last update for that platform.

     

    Audio work is ongoing. That code is a mess. We've addressed several areas, but there are a bunch more where audio arbitrarily loops through every emitter in the scene -- every frame. The team was in the middle of designing a caching mechanism so that the audio system doesn't have to do that unless anything moves, and even then most of the cache would still be valid.

     

    We got secondary shadows and the sky lighting working correctly.. but on Friday QA found areas in Financial where the odd self shadowing issues reappear that we fixed months ago. This is the kind of thing that drives me crazy with the APB code base. 2 steps forward and 1 step back. We'll be looking into that this week.

     

    I opted out of benchmarks this week, because there was too much work in progress.

     

    Thanks,
    Matt 

     

    • Like 14
    • Thanks 13

  11. Posted

    Hi all,

     

    We've made some good strides with Culling. There is no silver bullet, but it runs better than it did.

     

    There's a big slowdown related to audio that we're hunting down.

    After one of the content merges we left 6000+ audio emitters in Financial.

    They are all legitimate places where audio can play, but we need to do a better job managing them so they don't cause other parts of the code that search the scene to hit those objects too.

     

    During the week, we fixed several lighting bugs.

     

    On Friday we found a series of bugs that need to be addressed related to memory crashes.

    We also found a bug in the new Building Feature code that was preventing that system from working correctly.

     

    I'm going to try and post some benchmarks next week to illustrate where we are related to past benchmarks.
     

    Thanks,
    Matt

    • Like 34
    • Thanks 7

  13. Posted

    Hi there,

     

    I can appreciate this comment, and I agree that we need to get the Engine Upgrade out. However, that’s not as simple as you think.

     

    I make the effort to post on a weekly basis to talk about our progress. You can read it here:

     


    Thanks,

    Matt

     

    • Like 10
    • Thanks 3

  14. Posted

    HI all,

     

    I missed yesterday's update, but the TL;DR is that this week was disappointing.

     

    In an effort to explain, I'm going to talk a little bit about the innerworkings of the APB engine.

     

    I'll start by sharing performance from scenes that have no actual gameplay in them. These are small contained scenes with not a lot of objects in them.

    For the Login scene on my test machine, APB 1.20 (Live) runs at an average of 194FPS and APB 2.1 (Upgrade) runs at an average of 221FPS.

    We tested a number of other non-gameplay scenes, and clearly our new multithreaded renderer is faster - even with the more complex lighting and more objects overall.

     

    However, that is not true in Mission districts. These are much more expansive scenes where it's not feasible to render everything. As an example, in Financial there are 30,000+ objects. There are also a lot of special cases where task objects can be enabled/disabled, buildings can show various levels of detail, NPCs animate and move, and lighting is combination of Time of Day + lightmaps. All of that requires special case code.

     

    As mentioned previously, the Engine itself has two main threads - one for game play and one for rendering.  The rendering thread performs all the CPU functions necessary to throw geometry to the graphics card (GPU), which then splits those draw calls into more processes. The trick to getting the game to run fast is to balance the tasks in each of these three areas so that from the beginning of a frame to the end, they take the same time to run.

     

    To help optimize and balance performance, we have a number of tools that do performance profiling and tell us general areas, shaders, or functions that are taking a lot of time. This part can get very tricky because we have various versions of builds we can make such as Debug, Release, and Final builds. Debug is the slowest, but has the most diagnostics information we can see. Final is the build we ship, but it provides the least diagnostic information. Also running the game at a slower speed can wreak havoc with the balance of threads, which massively skews profiling results.

     

    With all of that in mind, a couple weeks ago we zeroed in on a performance bottleneck in the render thread where we do Culling (and its associated functions). This area of the code got reworked when we implemented multithreaded rendering, and clearly it is now under performing.

     

    Culling is the process of eliminating as many items as possible from being sent to the graphics card for rendering. Even though GPUs are very fast, it still takes time to copy data between the CPU and GPU, and we take a hit every time we switch materials/shaders between rendering each object. Since APB is so old, our version of Unreal has no "batching" or "instancing", which is what allows modern engines to push so many objects. In Financial, each frame we end up culling those 30,000+ objects down to 800-1000 objects that get processed for rendering to the GPU.

     

    So Culling is very important, and we use a bunch of different techniques to cull including render distance, object state, and clipping against the view frustum. All of that starts with accessing the scene objects in an Octree, a 3 dimensional grid that sub divides each cell until only a few objects are present. Each frame we start by looping down the Octree to eliminate objects that are behind or to the side of the player and out of view. The Octree is both our best way to cull, but also it's the hardest thing to multithread.

     

    This last week, we took a stab at multithreading that. This was a massive effort, and it ultimately didn't work. We got the code working, but on Thursday night when we started testing, we noticed that some objects in the scene didn't render. By Friday night, we figured out that the original devs on APB store scene data for objects in two places, the Octree and a main list of objects in the scene. For various reasons, those objects don't match.
     

    So we need to rework the approach and try again this week.

     

    Thanks,

    Matt

    • Like 32
    • Thanks 7

  15. Posted

    Hi all,

     

    Lots of support requests this week.

    While there are a couple straggler Keys to the City tickets asking for back grants, we stayed on top of things pretty well.

     

    As of today we have:

    188 new tickets (which is lower than last week)

    328 total tickets (which is lower than last week)

     

    We are now responding to tickets from 1/31 (except for a couple KTTC tickets) which puts us at 15 days to respond.

     

    Thanks,
    Matt

    • Like 3
    • Thanks 2

  17. Posted

    Hi all,


    We had a bit of a rough week that required us to split time in Customer Service between older tickets and immediate item delivery issues on GamersFirst.com caused by moving the Citadel database. Everything has smoothed out now, but we fell behind a little overall.

     

    As of today we have:

    195 unanswered/new tickets

    336 total tickets

     

    We are now responding to tickets from 1/25 (almost 1/26) which puts us at 13 days to respond.

     

    Thanks,

    Matt

     

     

    • Like 3

  18. Posted

    3 hours ago, FeliciaHardy said:

    do those tickets have priority above others?

     

    Yes. We are prioritizing those above others, but there really isn't a need to open a ticket now.

     

    We're working proactively through about 100 Citadel accounts that were affected. 

    Some of these were accounts that missed the delivery of an item going back as far as November.

    We got about half of them sorted out today, and we'll hit the rest over the next 2 days.

  19. Posted

    Hi there,

     

    You should have received an email receipt for the purchase. You can just paste the details of the receipt to Support.

    Otherwise, you can let Support know what you purchased, and they can look up your account and find missing items, but that might take a bit longer.

     

    Thanks,
    Matt

    • Like 1

  20. Posted

    Hi everyone,

     

    We had an issue with G1 purchases this morning when Citadel came back online.

    For a period of about 3 hours, item purchases weren't being delivered to players on that world.
    This was due to an issue caused by moving around the database during maintenance.


    The issue has been solved, and all purchase should work now.

     

    However, for anyone who is still missing weapons, you'll need to open a ticket at http://support.gamersfirst.com.

    Give them the transaction id from your purchase, and we'll escalate delivery as fast as possible.

     

    Thanks,

    Matt

    • Thanks 1

  21. Posted

    1 hour ago, Salvick said:

    That's good news, congrats for all the progress made.

     

    What about phasing? Is it coming soon?

     

    I think asking here could work and I'm concerned because latency to West Coast is unplayable to me, I'd like to play some APB again.

     

    Thanks!

    You may want to experiment with different action districts. Some are West Coast, and some are East Coast.

    The team is nearly done putting in a region ping/latency indicator. That will show up on the Advanced tab to show which servers are better for you.

    • Thanks 1

  24. Posted

    Hi there,

     

    This could be a case of selecting a New York server or a Los Angeles server. We have action districts in both.

    The team is nearly done with a new patch that will show players a ping/latency indicator to the closest servers.

    If you get in and out of districts, you can see your latency go up and down depending on what is closer.

     

    Thanks,
    Matt

  25. Posted

    Hi there,

     

    We've fixed the problem, so new purchases will work correctly.
    However, you'll need to open a ticket at http://support.gamersfirst.com to get the items delivered.

    Give them the transaction id from your purchase, and we'll escalate delivery as fast as possible.

     

    Thanks,
    Matt

    • Like 1
×
×
  • Create New...