MattScott

NGL, I laughed out loud when I watched this. Well done.

Typo. Unexpected.

Hi all, Lots of good progress on getting the remaining servers up and running in our new LA datacenter, however the EU servers continue to be a black box. I hate excuses, so I’m not going to make any. I haven’t had a direct response about the EU servers since they said they would deliver access on Friday. We scheduled our network team to work the weekend, but nothing has been turned over to us yet. I believe the group we are working with outsourced their EU support, so they are likely waiting on someone else, who is probably waiting on actual datacenter staff to move around hardware. With all these problems, many players may be wondering why we chose this new provider. Simply put, they are experts in defending DDoS attacks, and the end of our previous contract gave us an opportunity to be in the same cage as high-end custom hardware that can protect the game far better than in the past. Since we cutover in November, we have been hit with some of the biggest attacks I have seen since we took over. One attack was 200Gbps. That is 10-20x what were seeing in the past. Our new provider blocked nearly all of the attacks with zero impact to players, and the few that got through were analyzed immediately so our rules could be patched in minutes. I will continue to push for status on the EU servers, and I’ll post here when I have more details. Thanks, Matt

Hi all, I’ve seen some good responses to the OP’s issue. I appreciate the players who took the time to share their view point. I’m going to lock this thread, as it seems to have reached its conclusion. Thanks, Matt

Hi all, I was hoping for more of an update this week, but the holiday got in the way. As of Friday, we believe we have fixed the massive lighting issue that was plaguing us during December. I’m waiting for a build to evaluate performance. For console players, we noticed that both Sony and Microsoft have required upgrades to their latest SDKs before we can resubmit again. The work on the Sony SDK is complete, and we are 50% done with Microsoft. Looking forward to seeing progress next week now that vacations are done. Thanks, Matt

Hi all, This is a quick note to let you all know that Lixil has had an unexpected life situation that requires a change in her role. She is staying with the company but stepping down as the Community Manager. We are actively looking for a replacement. In the meantime Ritual and the rest of the team will be filling in. This is a bit of a scramble for us, but I hope to introduce the new Community Manager soon. EDIT: Unexpected. Thanks, Matt

Hi all, Missed the holiday week while we had reduced staff answering support tickets. Definitely saw a rise in tickets during that time. As of today we have: 497 unanswered tickets (up 159 over the holiday) 677 total tickets (up 137 over the holiday) We are answering tickets from 12/13 putting us back to 21 day turnaround. We have a new CS starting on Monday, and all staff will be back in the office to help bring this back down. Thanks, Matt

Hi all, All of our networking partners are back from the break. It was extremely frustrating to see everyone leave for the holidays with our servers not back online. I'm told my team will have access tomorrow (Friday) and that we can start installing images over the weekend. If everything holds true, then EU servers will come online during next week's patch maintenance. I will be extremely happy to have this put behind us, so we can concentrate on the next phase of world consolidation and local districts. Apologies, Matt

We are working on this issue.

Hi there, We are looking at that issue. There were other more important things to fix first. Thanks, Matt

Hi there, We appreciate you bringing it to our attention. Thanks, Matt

Hi all, Unfortunately we are still working through issues with the EU servers. All of the boxes arrived as expected, however we hit a very frustrating snag - it appears the configuration we laid out was not explicitly followed, and the game server software wouldn’t run on the CPUs due to missing instruction sets that we require. The team has expedited a new batch of servers arriving tomorrow. I’ll keep you posted. Thanks, Matt

Hi all, I missed the update yesterday, and I have been out all week with a nasty cold, so I don’t have a lot of new info. We did release a build to SPCT last week (our first in a while due to the network migration). That build had some problems and ran horribly bad. Looks like it was missing a huge chunk of the work that was completed. The team is organizing a new build for testing, and I’ll post results as soon as I have them. Hopefully all the APB players are gearing up for a nice holiday break. Thanks, Matt

Hi all, As of today we have: 338 unanswered tickets (-62 from last week) 540 total tickets (-65 from last week) We are answering tickets from 12/4 bringing us to 16 day turnaround on new tickets. Thanks, Matt

Hi everyone, We've just put up details for the Holiday Event 2019 including lots of new rewards. You can read them here: https://www.gamersfirst.com/apb/news/2019/12/18/holiday-event-2019 For PC, we're doing a new event called "Holiday Cheer" in Beacon. For Console, we still can't patch, so we'll be running the Elves and 12 Deaths. Holiday Cheer is coming to Beacon! A new drug has hit the streets of San Paro, will you be able to resist the scourge that is… “Holiday Cheer?” Those brave enough to try and resist the hallucinogenic properties of this powerful drug and fight to keep the kids safe from all that is merry can head to Beacon straight away! Be on the lookout for the Prince of Pushers… The Nutcracker Prince, with his “snow ball” launcher full of “Holiday Cheer.” Will you become dosed and join his army of tweaking Nutcrackers, or fight valiantly to keep this menace off the streets? Either way you’ll be earning new rewards! NOTE: We hit a snag yesterday with some incompatible CPUs that were accidentally ordered by our vendor and shipped to the new EU datacenter. Replacements have been organized and shipped locally. I'm working with our team over there to expedite things. At this point we're delaying the Holiday Event till Monday the 23rd in an effort to try and get EU servers online before we start. Thanks, Matt

Hi there, We are definitely running late on this event, but I'm the one holding it back till we can get EU servers online. In my opinion, it makes no sense to waste all the hard work and effort that we put in this year, when a significant amount of the population wont play due to terrible latency. I'm still waiting on an update from our EU hosting vendor to make the final plan. Thanks, Matt

Hi all, Our network provider has informed us that the EU hardware has been delivered to the new datacenter, racked, and they are working on installing OSes. I believe the plan is to give us access tonight, so we can install the game server build. I’m sharing what I’m being told, but honestly everything involving the new hardware has been 3-4 days later than they tell us. For those that have been asking, the hardware was ordered and shipped by our network provider that we lease the boxes from. Unfortunately, we had no input or window into any of the customs paperwork or issues, so I can’t shed any light on what happened there. Thanks, Matt

Hi everyone, I'm a little late getting this posted today, but here is the update. It's easier to start with what we fixed: - Baking Lighting Not sure what happened the first time around, but Financial ran in 3 hours over last weekend once we restarted it. All major districts are done now. - Memory issue This is mostly fixed. We're operating within proper limits on all platforms now which unblocks console development. There will likely be some ongoing work to reduce the memory footprint. - Performance Once proper lighting was baked, we did uncover an underlying problem with the way we re-wrote interactive features. Many of them include lights, and previously those lights were destroyed and rebuilt every time a player would go in and out of range of the feature. That had the upside of making sure the game was only processing and rendering lights within a specific distance, but the horrible downside of invalidating all the baked lighting forcing those lights to run as dynamic lights. Consoles got away with this, because they are locked at 1pm, so that version never runs at night. We had already fixed the code so that baked lighting is used all the time, but then we realized that the game was processing and rendering every light attached to a feature... every frame... regardless of how far away it was. This new issue got fixed by Friday, and the subsequent performance tests were very very promising. What is left: - Still missing our APB 2.1 Test World environment We can't deploy anything for SPCT to test yet. - Pedestrians popping through raised floors We didn't have quite enough time last week to look at this. - Console specific updates I have previous listed a number of console SDK issues that need to be fixed so we can submit to Sony and Microsoft again. - Lots of minor issues through out the levels These are gameplay bugs reported by SPCT and our internal QA such as bad collisions on specific rooftops. As I have previous stated, I wont hold up the release to OTW for these kinds of issues. - New localization Lots and lots of text in the game is new or changed, and it needs to be translated to French, Italian, German, Spanish and Russian. Again, I wont hold up the release to OTW waiting on localization. I may not even hold up the full public release. We'll fill this in as soon as we can, but I want the game to settle down first, so we can make one big pass. Thanks, Matt

Hi there, We are still working on error -1. It’s a generic error from PSN, but it is only occurring for a small part of the PS4 population. We have sent Sony info, and are waiting for their response to get more details. Thanks, Matt

Hi all, Lots of good progress on tickets, but we're hopping around working both ends of the queue depending on the issue. That means we closed a lot of tickets, but we have some older straggler tickets. As of today we have: 400 unanswered tickets (-115 from last week) 605 total tickets (-112 from last week) A few tickets from 11/20 but mostly we are answering tickets from 11/21. Thanks, Matt

Hi there, @iProkaza Do you want to have this chat here publicly or in private through Customer Support? GMs can only verify and report offensive content they see or that is pointed out to them. Customer Service deleted your graphics library after getting my approval. Thanks, Matt

Hi all, Some days we just can't win. Services are back online. Thanks, Matt

Hi all, We've had two outages today - both deserve some explanation. The first outage was at ~10pm to midnight Pacific time. This was caused by unannounced maintenance with the circuit provider into our East Coast provider. By 12/19, we will have our backup circuit provisioned, so this shouldn't happen any more. The second outage was around 10am to 11am Pacific time. This was caused by a technician with our provider adjusting some DDoS rules that ended up blocking all traffic for a period of about 45 minutes while we tried to figure out what happened. We have made it clear that no changes to our environment can occur outside of our planned maintenance windows. Apologies, Matt

Hi all, The extended time was so we could move everything to a new larger switch in LA. We are very close now. Apologies, Matt

Hi there, We finally replicated this in our environment. The team is looking at it now. Thanks, Matt