MattScott

Hi there, We are still working on error -1. It’s a generic error from PSN, but it is only occurring for a small part of the PS4 population. We have sent Sony info, and are waiting for their response to get more details. Thanks, Matt

Hi all, Lots of good progress on tickets, but we're hopping around working both ends of the queue depending on the issue. That means we closed a lot of tickets, but we have some older straggler tickets. As of today we have: 400 unanswered tickets (-115 from last week) 605 total tickets (-112 from last week) A few tickets from 11/20 but mostly we are answering tickets from 11/21. Thanks, Matt

Hi there, @iProkaza Do you want to have this chat here publicly or in private through Customer Support? GMs can only verify and report offensive content they see or that is pointed out to them. Customer Service deleted your graphics library after getting my approval. Thanks, Matt

Hi all, Some days we just can't win. Services are back online. Thanks, Matt

Hi all, We've had two outages today - both deserve some explanation. The first outage was at ~10pm to midnight Pacific time. This was caused by unannounced maintenance with the circuit provider into our East Coast provider. By 12/19, we will have our backup circuit provisioned, so this shouldn't happen any more. The second outage was around 10am to 11am Pacific time. This was caused by a technician with our provider adjusting some DDoS rules that ended up blocking all traffic for a period of about 45 minutes while we tried to figure out what happened. We have made it clear that no changes to our environment can occur outside of our planned maintenance windows. Apologies, Matt

Hi all, The extended time was so we could move everything to a new larger switch in LA. We are very close now. Apologies, Matt

Hi there, We finally replicated this in our environment. The team is looking at it now. Thanks, Matt

Hi all, Many players have complained about latency or status on the Network Migration effort. So I'm going to use this thread to post updates. For context, here is what we did: In May we shutdown an LA datacenter that was only being used for a couple critical services. That long standing contract that was expiring, and it was an opportunity to consolidate network services elsewhere. This datacenter had been in use for years, and as we discovered, there were a lot of buried landmines and hard coded references. In November we migrated services to a new provider because that second long standing contract was expiring. The new provider offered us better hosting locations, more bandwidth, new servers, and more comprehensive DDoS protection. This effort was required to prep the upcoming World Consolidation and local districts. However, again this migration did not go smoothly. A number of new servers had bad SSDs and broken network transceivers. Additionally, the servers meant for our new EU datacenter got held up in customs due to bad paperwork filled out by our vendors. Since our departure was based on an expiring contract, we were forced to migrate and launch on a reduced set of servers. There are 6 "worlds" in our environment: NA PC, PS4, and XB1. EU PC, PS4, and XB1. Before I get a bunch of players asking why we chose to keep PS4 and XB1 worlds up, we are contractually obligated to our console publisher to keep those worlds online. There was no option other than to support them regardless of concurrency. Needless to say, all PS4 and XB1 worlds are running minimal environments. Currently Jericho is operating on mostly Los Angeles-based servers with a couple New York servers. This is why west coast players occasionally see increased latency. We didn't have enough servers to support threat segregation, so all Jericho mission districts are open threat right now. Currently Citadel is operating on New York-based servers. This is why EU players are seeing horrible latency. We also didn't have enough servers to support threat segregation, so all Citadel mission districts are open threat right now. We have been told our EU servers cleared customs today and are enroute to the new datacenter. Little Orbit has no control over this process. When the servers arrive, they will be racked, hooked up, and installed with our basic OS image before being turned over to our techs. Once we get access, then it will take a day or two for our techs to configure everything, load the game servers, and test them. Citadel will have local EU district servers, and the latency will drop down significantly. Then all the LA / NY servers will be re-assigned to Jericho. The dev team is working on exposing locations in the Advanced tab, and adding logic for how players are assigned to LA or NY servers. Thanks, Matt

Hi everyone, Here is a quick rundown of the active blockers: - Lighting Now that we have de-duped all the lights and props, we need to re-run lighting so we can bake the static portions of each district that combine with the dynamic portions at run time. Due to a bug we fixed last week, up until now all the lights were running dynamically, which carries a big performance penalty at night. We started baking lighting for Financial on Wednesday. But Friday it wasn't done. This is unusually long compared to light builds we have run in the past. There are quite literally thousands of lights in Financial that all switched to baked instead of dynamic. So it's possible that the longer build time is normal. However, we also suspect that there was an issue on the build machine during the week due to a network issue. So ultimately we elected to kill the job, make sure all the workstations on the network were on and running the Swarm agent, and then re-run it starting Friday night. We'll see if it finishes by Monday. - Physics This is a great example of unplanned complexity in the Engine Upgrade project. In order to support the latest Microsoft and Sony libraries for console, we did a massive series of upgrades to external libraries. One of those was the physics library. Now it looks like that developer changed a core part of how physics objects are initialized, because SPCT previously reported that interactive props and other physics objects would immediately explode and then respawn as players entered the district. Part of the team spent time this week troubleshooting (and ultimately fixing) several of these physics issues. I'm guessing that this is the sort of frustrating loop that I'm sure Reloaded found themselves in. You upgrade one part of the code to support one platform, and then break other parts of the code. Then to fix it, you end up needing more upgrades elsewhere. And so on. - Memory issue On Monday, we ran our test build and found a massive memory leak that was able to allocate 7GB of ram before even getting into a district. By Friday, we had fixed two possible areas, but we are still investigating. - Console blocker The memory leak blocked all console development, because those platforms can't allocate that much memory. Work there was paused this week. - Other build blockers Making builds is a big part of development. Devs produce local builds all the time, but those have all the optimizations turned off and don't really provide any sort of real performance. We have a number of internal systems that we use to identify specific versions of the code and generate end-user optimized builds. This week we had a number of build issues starting on Tuesday that prevented us from getting a new APB 2.1 build done. Most of the issues were related to the network move and the locations of various resources. The team worked through most of these, but the process takes several hours to get a successful (or unsuccessful) build, so it is extremely tedious. At this point it looks like we have worked out all the kinks. But we still don't have our Test Worlds back online yet, so we can't share anything with SPCT. I'm told our Test Worlds will be back online this week. For next week, hopefully lighting will be done for Financial so that we can get a new end-user build generated to test all the work we have done over the last 3 weeks. We will also need to run lighting for all the other districts during the week. This new build should help us identify remaining tasks and perform critical benchmarks. Thanks, Matt

Hi all, Still catching up from the holiday weekend, and we have lots of network and sale related tickets. Many of these can be cleared because we have fixed the various issues, so I expect these numbers to drop a lot next week. As of today we have: 515 unanswered tickets 717 total tickets We are answering tickets from 11/14. If your ticket is older than November 14th, and we have not reached out, then I recommend updating your ticket and asking for status. Thanks, Matt

Hi all, Unfortunately it looks like a data entry error accidentally pushed the prototype changes for the Showstopper into Live districts. We've already had enough downtime this week, and this change requires a new build. We'll patch this back to prototype districts during our regular maintenance next week. Thanks, Matt

Hi all, I have just confirmed that all platforms (PC, XB1, and PS4) in all regions (NA & EU) are up with all districts online again. Thanks, Matt

Hi all, I have just confirmed that all platforms (PC, XB1, and PS4) in all regions (NA & EU) are up with all districts online again. We have been attempting to correct an ongoing data issue caused during our network migration. At this point, all data is current with nothing lost. The remaining work will be performed during regular maintenance. Thanks, Matt

Hi there, This was totally a mistake on our part. The change was staged for test but the Showstopper stats got copied to Live by accident. We've had a lot of downtime already today, and this requires a new build to fix, so we will have to wait till our regular maintenance window next week to fix it. Apologies, Matt

Hi all, We just became aware that the experimental test changes to the Showstopper accidentally got pushed to Live. This was unintentional. I'll post on some of the other threads. We plan to fix this and move the change to test as soon as possible. However we've already had a lot of downtime today due to another issue, so I'm not going to take the game back down again this weekend. Thanks, Matt

Hi all, We are having some issues with our monitoring systems, and we have been getting hit nonstop with DDoS attacks since we moved datacenters. However I just confirmed that XB1 login and all districts should be up now. Thanks, Matt

Hi all, PS4 login and districts are back online. This is because Nekrova servers are offline now, and we need to update that system to remove those checks. Thanks, Matt

Hi everyone, We just made some changes, and we are seeing players on the PS4 NA districts. Can you try again? If you're getting a login error, then you may need to open a ticket with support, so we can collect more details. EDIT: This just came back up about 30 minutes ago. Thanks, Matt

Hi there, We've made some changes, and we do see players on the servers. Can you retest? Thanks, Matt

To be clear, you should have had Premium since 11/28. There is a display bug on the site that shows the wrong end date. If you're not getting the benefits of Premium, then open a trouble ticket. We are escalating those tickets to figure out if there is something else going on.

Right now Premium is turned on globally, so it should effect everyone. We tried individually assigning Premium months ago for the last compensation period, and it turned into a nightmare of waiting for days to let the database job finish. We are still investigating why some players aren't getting it based on tickets that were sent in. EDIT: Looks like its a display bug that shows the wrong Premium end date for people who already had Premium. We are in the process of fixing this on the site. EDIT: The new ARMAS sale will stack with the Premium for a total of 40% off on select items.

Hi all, We originally activated 7 Days of Premium for all players and all platforms. We have extended this till 12/22 for a total of 25 days while we continue making adjustments and get the EU servers online. Thanks, Matt

Hi all, The team did some work on this today. We will continue to look at it today. Thanks, Matt

Hi there, We are waiting for EU server hardware to clear customs. My expectation is that will be later this week. Thanks, Matt

Hi there, We have been experiencing nonstop DDoS attacks, which disrupt our authentication systems. We are in the process of bringing online new protection systems (from our new provider). So hopefully this will be resolved over the coming days. Please post here if you continue to see this issue. Thanks, Matt