Jump to content
Forum rules

MattScott

CEO
 View Profile  See their activity

  • Content Count

    1281

  • Joined

  • Last visited

Posts posted by MattScott

  3. Posted

    Hi all,

     

    Unfortunately we are still working through issues with the EU servers. All of the boxes arrived as expected, however we hit a very frustrating snag - it appears the configuration we laid out was not explicitly followed, and the game server software wouldn’t run on the CPUs due to missing instruction sets that we require.

     

    The team has expedited a new batch of servers arriving tomorrow. I’ll keep you posted.

     

    Thanks,

    Matt

    • Like 4
    • Thanks 1

  4. Posted

    Hi all,

     

    I missed the update yesterday, and I have been out all week with a nasty cold, so I don’t have a lot of new info.

     

    We did release a build to SPCT last week (our first in a while due to the network migration). That build had some problems and ran horribly bad. Looks like it was missing a huge chunk of the work that was completed. The team is organizing a new build for testing, and I’ll post results as soon as I have them.

     

    Hopefully all the APB players are gearing up for a nice holiday break.

     

    Thanks,

    Matt

    • Like 9
    • Thanks 4

  6. Posted

    Hi everyone,

     

    We've just put up details for the Holiday Event 2019 including lots of new rewards.

     

    You can read them here:

    https://www.gamersfirst.com/apb/news/2019/12/18/holiday-event-2019

     

    news-26.jpg

     

    For PC, we're doing a new event called "Holiday Cheer" in Beacon.

    For Console, we still can't patch, so we'll be running the Elves and 12 Deaths.

     

    Holiday Cheer is coming to Beacon!

    A new drug has hit the streets of San Paro, will you be able to resist the scourge that is… “Holiday Cheer?” Those brave enough to try and resist the hallucinogenic properties of this powerful drug and fight to keep the kids safe from all that is merry can head to Beacon straight away! Be on the lookout for the Prince of Pushers… The Nutcracker Prince, with his “snow ball” launcher full of “Holiday Cheer.” Will you become dosed and join his army of tweaking Nutcrackers, or fight valiantly to keep this menace off the streets? Either way you’ll be earning new rewards!


    NOTE: We hit a snag yesterday with some incompatible CPUs that were accidentally ordered by our vendor and shipped to the new EU datacenter. Replacements have been organized and shipped locally. I'm working with our team over there to expedite things.

     

    At this point we're delaying the Holiday Event till Monday the 23rd in an effort to try and get EU servers online before we start.

     

    Thanks,
    Matt

    • Like 17
    • Thanks 6

  7. Posted

    Hi there,

     

    We are definitely running late on this event, but I'm the one holding it back till we can get EU servers online. In my opinion, it makes no sense to waste all the hard work and effort that we put in this year, when a significant amount of the population wont play due to terrible latency.

     

    I'm still waiting on an update from our EU hosting vendor to make the final plan.

     

    Thanks,
    Matt

    • Like 7
    • Thanks 6

  8. Posted

    Hi all,

     

    Our network provider has informed us that the EU hardware has been delivered to the new datacenter, racked, and they are working on installing OSes. I believe the plan is to give us access tonight, so we can install the game server build.

     

    I’m sharing what I’m being told, but honestly everything involving the new hardware has been 3-4 days later than they tell us.

     

    For those that have been asking, the hardware was ordered and shipped by our network provider that we lease the boxes from. Unfortunately, we had no input or window into any of the customs paperwork or issues, so I can’t shed any light on what happened there.

     

    Thanks,

    Matt

    • Like 16
    • Thanks 8

  9. Posted

    Hi everyone,

     

    I'm a little late getting this posted today, but here is the update.

     

    It's easier to start with what we fixed:

     

    - Baking Lighting

    Not sure what happened the first time around, but Financial ran in 3 hours over last weekend once we restarted it. All major districts are done now.

    - Memory issue

    This is mostly fixed. We're operating within proper limits on all platforms now which unblocks console development.

    There will likely be some ongoing work to reduce the memory footprint.

     

    - Performance

    Once proper lighting was baked, we did uncover an underlying problem with the way we re-wrote interactive features. Many of them include lights, and previously those lights were destroyed and rebuilt every time a player would go in and out of range of the feature. That had the upside of making sure the game was only processing and rendering lights within a specific distance, but the horrible downside of invalidating all the baked lighting forcing those lights to run as dynamic lights. Consoles got away with this, because they are locked at 1pm, so that version never runs at night.

     

    We had already fixed the code so that baked lighting is used all the time, but then we realized that the game was processing and rendering every light attached to a feature... every frame... regardless of how far away it was. This new issue got fixed by Friday, and the subsequent performance tests were very very promising. 

     

    What is left:

     

    - Still missing our APB 2.1 Test World environment

    We can't deploy anything for SPCT to test yet.

     

    - Pedestrians popping through raised floors

    We didn't have quite enough time last week to look at this.

     

    - Console specific updates

    I have previous listed a number of console SDK issues that need to be fixed so we can submit to Sony and Microsoft again.

     

    - Lots of minor issues through out the levels

    These are gameplay bugs reported by SPCT and our internal QA such as bad collisions on specific rooftops.

    As I have previous stated, I wont hold up the release to OTW for these kinds of issues. 

     

    - New localization

    Lots and lots of text in the game is new or changed, and it needs to be translated to French, Italian, German, Spanish and Russian.

    Again, I wont hold up the release to OTW waiting on localization. I may not even hold up the full public release.

    We'll fill this in as soon as we can, but I want the game to settle down first, so we can make one big pass.
     

    Thanks,

    Matt

    • Like 26
    • Thanks 24

  11. Posted

    Hi all,

     

    Lots of good progress on tickets, but we're hopping around working both ends of the queue depending on the issue.

    That means we closed a lot of tickets, but we have some older straggler tickets.

     

    As of today we have:

    400 unanswered tickets (-115 from last week)

    605 total tickets (-112 from last week)

     

    A few tickets from 11/20 but mostly we are answering tickets from 11/21.

     

    Thanks,

    Matt

    • Like 2

  14. Posted

    Hi all,

     

    We've had two outages today - both deserve some explanation.

     

    The first outage was at ~10pm to midnight Pacific time.

    This was caused by unannounced maintenance with the circuit provider into our East Coast provider.

    By 12/19, we will have our backup circuit provisioned, so this shouldn't happen any more.

     

    The second outage was around 10am to 11am Pacific time.

    This was caused by a technician with our provider adjusting some DDoS rules that ended up blocking all traffic for a period of about 45 minutes while we tried to figure out what happened.

    We have made it clear that no changes to our environment can occur outside of our planned maintenance windows.

     

    Apologies,

    Matt

    • Like 7
    • Thanks 4

  17. Posted

    Hi all,

     

    Many players have complained about latency or status on the Network Migration effort.

    So I'm going to use this thread to post updates.

     

    For context, here is what we did:

     

    • In May we shutdown an LA datacenter that was only being used for a couple critical services. That long standing contract that was expiring, and it was an opportunity to consolidate network services elsewhere. This datacenter had been in use for years, and as we discovered, there were a lot of buried landmines and hard coded references.
    • In November we migrated services to a new provider because that second long standing contract was expiring. The new provider offered us better hosting locations, more bandwidth, new servers, and more comprehensive DDoS protection. This effort was required to prep the upcoming World Consolidation and local districts. However, again this migration did not go smoothly. A number of new servers had bad SSDs and broken network transceivers. Additionally, the servers meant for our new EU datacenter got held up in customs due to bad paperwork filled out by our vendors.

     

    Since our departure was based on an expiring contract, we were forced to migrate and launch on a reduced set of servers.

     

    There are 6 "worlds" in our environment: NA PC, PS4, and XB1. EU PC, PS4, and XB1. Before I get a bunch of players asking why we chose to keep PS4 and XB1 worlds up, we are contractually obligated to our console publisher to keep those worlds online. There was no option other than to support them regardless of concurrency. Needless to say, all PS4 and XB1 worlds are running minimal environments.

     

    • Currently Jericho is operating on mostly Los Angeles-based servers with a couple New York servers. This is why west coast players occasionally see increased latency. We didn't have enough servers to support threat segregation, so all Jericho mission districts are open threat right now.
    • Currently Citadel is operating on New York-based servers. This is why EU players are seeing horrible latency. We also didn't have enough servers to support threat segregation, so all Citadel mission districts are open threat right now.

     

    We have been told our EU servers cleared customs today and are enroute to the new datacenter. Little Orbit has no control over this process.

    When the servers arrive, they will be racked, hooked up, and installed with our basic OS image before being turned over to our techs.

    Once we get access, then it will take a day or two for our techs to configure everything, load the game servers, and test them.

     

    Citadel will have local EU district servers, and the latency will drop down significantly.

    Then all the LA / NY servers will be re-assigned to Jericho.
    The dev team is working on exposing locations in the Advanced tab, and adding logic for how players are assigned to LA or NY servers.

     

    Thanks,

    Matt

    • Like 23
    • Thanks 21

  18. Posted

    Hi everyone,

     

    Here is a quick rundown of the active blockers:

     

    - Lighting

    Now that we have de-duped all the lights and props, we need to re-run lighting so we can bake the static portions of each district that combine with the dynamic portions at run time.

     

    Due to a bug we fixed last week, up until now all the lights were running dynamically, which carries a big performance penalty at night.

     

    We started baking lighting for Financial on Wednesday. But Friday it wasn't done. This is unusually long compared to light builds we have run in the past. There are quite literally thousands of lights in Financial that all switched to baked instead of dynamic. So it's possible that the longer build time is normal. However, we also suspect that there was an issue on the build machine during the week due to a network issue. So ultimately we elected to kill the job, make sure all the workstations on the network were on and running the Swarm agent, and then re-run it starting Friday night. We'll see if it finishes by Monday.

     

    - Physics

    This is a great example of unplanned complexity in the Engine Upgrade project.

     

    In order to support the latest Microsoft and Sony libraries for console, we did a massive series of upgrades to external libraries. One of those was the physics library. Now it looks like that developer changed a core part of how physics objects are initialized, because SPCT previously reported that interactive props and other physics objects would immediately explode and then respawn as players entered the district. Part of the team spent time this week troubleshooting (and ultimately fixing) several of these physics issues.

     

    I'm guessing that this is the sort of frustrating loop that I'm sure Reloaded found themselves in. You upgrade one part of the code to support one platform, and then break other parts of the code. Then to fix it, you end up needing more upgrades elsewhere. And so on. 

     

    - Memory issue

    On Monday, we ran our test build and found a massive memory leak that was able to allocate 7GB of ram before even getting into a district. By Friday, we had fixed two possible areas, but we are still investigating.

     

    - Console blocker

    The memory leak blocked all console development, because those platforms can't allocate that much memory. Work there was paused this week.

     

    - Other build blockers

    Making builds is a big part of development. Devs produce local builds all the time, but those have all the optimizations turned off and don't really provide any sort of real performance. We have a number of internal systems that we use to identify specific versions of the code and generate end-user optimized builds.

     

    This week we had a number of build issues starting on Tuesday that prevented us from getting a new APB 2.1 build done. Most of the issues were related to the network move and the locations of various resources. The team worked through most of these, but the process takes several hours to get a successful (or unsuccessful) build, so it is extremely tedious. At this point it looks like we have worked out all the kinks. But we still don't have our Test Worlds back online yet, so we can't share anything with SPCT. I'm told our Test Worlds will be back online this week.

     

    For next week, hopefully lighting will be done for Financial so that we can get a new end-user build generated to test all the work we have done over the last 3 weeks. We will also need to run lighting for all the other districts during the week. This new build should help us identify remaining tasks and perform critical benchmarks.

     

    Thanks,

    Matt

    • Like 25
    • Thanks 8

  19. Posted

    Hi all,

     

    Still catching up from the holiday weekend, and we have lots of network and sale related tickets.

    Many of these can be cleared because we have fixed the various issues, so I expect these numbers to drop a lot next week.

     

    As of today we have:

    515 unanswered tickets

    717 total tickets

     

    We are answering tickets from 11/14.

    If your ticket is older than November 14th, and we have not reached out, then I recommend updating your ticket and asking for status.

     

    Thanks,

    Matt

    • Like 3

  22. Posted

    Hi all,

     

    I have just confirmed that all platforms (PC, XB1, and PS4) in all regions (NA & EU) are up with all districts online again.

    We have been attempting to correct an ongoing data issue caused during our network migration. At this point, all data is current with nothing lost. 

    The remaining work will be performed during regular maintenance.

     

    Thanks,
    Matt

  23. Posted

    Hi there,

     

    This was totally a mistake on our part. The change was staged for test but the Showstopper stats got copied to Live by accident.

    We've had a lot of downtime already today, and this requires a new build to fix, so we will have to wait till our regular maintenance window next week to fix it.


    Apologies,
    Matt

  24. Posted

    Hi all,

     

    We just became aware that the experimental test changes to the Showstopper accidentally got pushed to Live.

    This was unintentional. I'll post on some of the other threads.

     

    We plan to fix this and move the change to test as soon as possible.

    However we've already had a lot of downtime today due to another issue, so I'm not going to take the game back down again this weekend.

     

    Thanks,
    Matt

    • Thanks 1

  25. Posted

    Hi all,

     

    We are having some issues with our monitoring systems, and we have been getting hit nonstop with DDoS attacks since we moved datacenters.

    However I just confirmed that XB1 login and all districts should be up now.


    Thanks,
    Matt

    • Like 1
×
×
  • Create New...