MattScott

Hi everyone, Here is a quick rundown of the active blockers: - Lighting Now that we have de-duped all the lights and props, we need to re-run lighting so we can bake the static portions of each district that combine with the dynamic portions at run time. Due to a bug we fixed last week, up until now all the lights were running dynamically, which carries a big performance penalty at night. We started baking lighting for Financial on Wednesday. But Friday it wasn't done. This is unusually long compared to light builds we have run in the past. There are quite literally thousands of lights in Financial that all switched to baked instead of dynamic. So it's possible that the longer build time is normal. However, we also suspect that there was an issue on the build machine during the week due to a network issue. So ultimately we elected to kill the job, make sure all the workstations on the network were on and running the Swarm agent, and then re-run it starting Friday night. We'll see if it finishes by Monday. - Physics This is a great example of unplanned complexity in the Engine Upgrade project. In order to support the latest Microsoft and Sony libraries for console, we did a massive series of upgrades to external libraries. One of those was the physics library. Now it looks like that developer changed a core part of how physics objects are initialized, because SPCT previously reported that interactive props and other physics objects would immediately explode and then respawn as players entered the district. Part of the team spent time this week troubleshooting (and ultimately fixing) several of these physics issues. I'm guessing that this is the sort of frustrating loop that I'm sure Reloaded found themselves in. You upgrade one part of the code to support one platform, and then break other parts of the code. Then to fix it, you end up needing more upgrades elsewhere. And so on. - Memory issue On Monday, we ran our test build and found a massive memory leak that was able to allocate 7GB of ram before even getting into a district. By Friday, we had fixed two possible areas, but we are still investigating. - Console blocker The memory leak blocked all console development, because those platforms can't allocate that much memory. Work there was paused this week. - Other build blockers Making builds is a big part of development. Devs produce local builds all the time, but those have all the optimizations turned off and don't really provide any sort of real performance. We have a number of internal systems that we use to identify specific versions of the code and generate end-user optimized builds. This week we had a number of build issues starting on Tuesday that prevented us from getting a new APB 2.1 build done. Most of the issues were related to the network move and the locations of various resources. The team worked through most of these, but the process takes several hours to get a successful (or unsuccessful) build, so it is extremely tedious. At this point it looks like we have worked out all the kinks. But we still don't have our Test Worlds back online yet, so we can't share anything with SPCT. I'm told our Test Worlds will be back online this week. For next week, hopefully lighting will be done for Financial so that we can get a new end-user build generated to test all the work we have done over the last 3 weeks. We will also need to run lighting for all the other districts during the week. This new build should help us identify remaining tasks and perform critical benchmarks. Thanks, Matt

Hi all, Still catching up from the holiday weekend, and we have lots of network and sale related tickets. Many of these can be cleared because we have fixed the various issues, so I expect these numbers to drop a lot next week. As of today we have: 515 unanswered tickets 717 total tickets We are answering tickets from 11/14. If your ticket is older than November 14th, and we have not reached out, then I recommend updating your ticket and asking for status. Thanks, Matt

Hi all, Unfortunately it looks like a data entry error accidentally pushed the prototype changes for the Showstopper into Live districts. We've already had enough downtime this week, and this change requires a new build. We'll patch this back to prototype districts during our regular maintenance next week. Thanks, Matt

Hi all, I have just confirmed that all platforms (PC, XB1, and PS4) in all regions (NA & EU) are up with all districts online again. Thanks, Matt

Hi all, I have just confirmed that all platforms (PC, XB1, and PS4) in all regions (NA & EU) are up with all districts online again. We have been attempting to correct an ongoing data issue caused during our network migration. At this point, all data is current with nothing lost. The remaining work will be performed during regular maintenance. Thanks, Matt

Hi there, This was totally a mistake on our part. The change was staged for test but the Showstopper stats got copied to Live by accident. We've had a lot of downtime already today, and this requires a new build to fix, so we will have to wait till our regular maintenance window next week to fix it. Apologies, Matt

Hi all, We just became aware that the experimental test changes to the Showstopper accidentally got pushed to Live. This was unintentional. I'll post on some of the other threads. We plan to fix this and move the change to test as soon as possible. However we've already had a lot of downtime today due to another issue, so I'm not going to take the game back down again this weekend. Thanks, Matt

Hi all, We are having some issues with our monitoring systems, and we have been getting hit nonstop with DDoS attacks since we moved datacenters. However I just confirmed that XB1 login and all districts should be up now. Thanks, Matt

Hi all, PS4 login and districts are back online. This is because Nekrova servers are offline now, and we need to update that system to remove those checks. Thanks, Matt

Hi everyone, We just made some changes, and we are seeing players on the PS4 NA districts. Can you try again? If you're getting a login error, then you may need to open a ticket with support, so we can collect more details. EDIT: This just came back up about 30 minutes ago. Thanks, Matt

Hi there, We've made some changes, and we do see players on the servers. Can you retest? Thanks, Matt

To be clear, you should have had Premium since 11/28. There is a display bug on the site that shows the wrong end date. If you're not getting the benefits of Premium, then open a trouble ticket. We are escalating those tickets to figure out if there is something else going on.

Right now Premium is turned on globally, so it should effect everyone. We tried individually assigning Premium months ago for the last compensation period, and it turned into a nightmare of waiting for days to let the database job finish. We are still investigating why some players aren't getting it based on tickets that were sent in. EDIT: Looks like its a display bug that shows the wrong Premium end date for people who already had Premium. We are in the process of fixing this on the site. EDIT: The new ARMAS sale will stack with the Premium for a total of 40% off on select items.

Hi all, We originally activated 7 Days of Premium for all players and all platforms. We have extended this till 12/22 for a total of 25 days while we continue making adjustments and get the EU servers online. Thanks, Matt

Hi all, The team did some work on this today. We will continue to look at it today. Thanks, Matt

Hi there, We are waiting for EU server hardware to clear customs. My expectation is that will be later this week. Thanks, Matt

Hi there, We have been experiencing nonstop DDoS attacks, which disrupt our authentication systems. We are in the process of bringing online new protection systems (from our new provider). So hopefully this will be resolved over the coming days. Please post here if you continue to see this issue. Thanks, Matt

Hi there, We have been getting hit with constant DDoS attacks since EU came online. Due to the rush to get back online, we hadn't yet fully implemented all the protection systems or finished training the new hardware. Hopefully this will smooth out over the coming days. Please post here if it continues to be an issue. Thanks, Matt

Hi there, Can you verify this is still happening? We have been getting a strong series of DDoS attacks all day, and we had one earlier during this period when you were having trouble. Thanks, Matt

Excellent. I'm going to lock this thread and mark it as fixed.

Hi all, I believe this issue is fixed. If it's still a problem, please add to this thread. Thanks, Matt

Hi @nitex, The team is reporting this as fixed, but we're waiting on team members to test. Do you mind trying again? Thanks, Matt

Hi @nitex, I believe we got Steam auth working a couple days ago. Amayii submitted an urgent request to the team for us to look at payments. Thanks, Matt

Hi everyone, Only three working days this week, and lots of interruptions with the terrible network move. I don't have a comprehensive update today, but I was told that we are finally "done" with all the major blockers. However we need to make a build and get our test environments put back together, and then we'll see how all the fixes hold together. I'll likely update later this week once I have more information. Thanks, Matt

Hi all, This was a sustained attack against Citadel that exploited part of our infrastructure that wasn’t working properly. The network team has now rebuilt those items, so it shouldn’t be a problem going forward. Thanks, Matt