MattScott

Hi there, We are waiting for EU server hardware to clear customs. My expectation is that will be later this week. Thanks, Matt

Hi there, We have been experiencing nonstop DDoS attacks, which disrupt our authentication systems. We are in the process of bringing online new protection systems (from our new provider). So hopefully this will be resolved over the coming days. Please post here if you continue to see this issue. Thanks, Matt

Hi there, We have been getting hit with constant DDoS attacks since EU came online. Due to the rush to get back online, we hadn't yet fully implemented all the protection systems or finished training the new hardware. Hopefully this will smooth out over the coming days. Please post here if it continues to be an issue. Thanks, Matt

Hi there, Can you verify this is still happening? We have been getting a strong series of DDoS attacks all day, and we had one earlier during this period when you were having trouble. Thanks, Matt

Excellent. I'm going to lock this thread and mark it as fixed.

Hi all, I believe this issue is fixed. If it's still a problem, please add to this thread. Thanks, Matt

Hi @nitex, The team is reporting this as fixed, but we're waiting on team members to test. Do you mind trying again? Thanks, Matt

Hi @nitex, I believe we got Steam auth working a couple days ago. Amayii submitted an urgent request to the team for us to look at payments. Thanks, Matt

Hi everyone, Only three working days this week, and lots of interruptions with the terrible network move. I don't have a comprehensive update today, but I was told that we are finally "done" with all the major blockers. However we need to make a build and get our test environments put back together, and then we'll see how all the fixes hold together. I'll likely update later this week once I have more information. Thanks, Matt

Hi all, This was a sustained attack against Citadel that exploited part of our infrastructure that wasn’t working properly. The network team has now rebuilt those items, so it shouldn’t be a problem going forward. Thanks, Matt

Hi all, I missed these stats from yesterday, and its a short holiday week for us, so customer support was mostly shutdown Thurs and Fri. As of this morning we have: 434 unanswered tickets 683 total tickets We are answering tickets from 11/11. If your ticket is older than November 11th, and we have not reached out, then I recommend updating your ticket and asking for status. Thanks, Matt

If you're playing on Jericho, I think you're playing on LA servers for now. If you're playing on Citadel, then you're playing on NY servers for now. As soon as our EU servers clear customs, then we'll do some moving around.

Hi all, EDIT: We originally activated 7 Days of Premium for all players and all platforms. We have extended this till 12/22 for a total of 25 days while we continue making adjustments and get the EU servers online. Thanks, Matt

We are working on new server locations. Currently only LA and NY are online.

Hi everyone, All services are back online. We are fine tuning district server assignments to get us through the next couple days. Please note that we are missing all EU-based servers, so Citadel is temporarily running off the new East Coast servers. I'm going to lock this thread for now. Thanks, Matt

Hi all, This is taking agonizingly too long. We are waiting on one more fix to get PC back online. Sorry, Matt

Hi all, XB1 NA/EU and PS4 NA/EU are back online. We're working on bringing up PC. Thanks, Matt

Hi all, The team is troubleshooting an issue with connecting new district servers to the existing worlds. We're close, but I don't think the PC, XB1, or PS4 environments for North America will be ready for another couple hours. Thanks, Matt

Hi all, We are getting closer. The databases are all restored. The network team has been working for 72 hours straight now. The main issue now is some bad hardware that had to be replaced. I hate to throw out dates, but I feel like you have all been in limbo too long. So if I had to start making some guesses, here is how I think the various platforms will come online: PS4 and PC NA = midnight tonight Pacific time XB1, PS4, and PC EU = early morning (7am-ish) tomorrow Pacific time A couple caveats: We are missing some network equipment and a bunch of district boxes that got stuck in customs for the new EU datacenter. This means all of EU will be temporarily running on districts in New York for a couple days till we get the new EU location online properly. And all platforms will be running Financial and Waterfront with no threat segregation for a couple days. Sorry, Matt

Hi all, We’re fixing one internal issue that is still preventing the EU world from being available. I don’t have an eta, but the team is working very hard. The other database restores all started at the same time. Each platform will come online as they finish. Thanks, Matt

Hi all, I have posted an update here: Thanks, Matt

Hi all, I have updated Lixil's original post with more details. These are large databases, so they are taking quite a bit of time to backup, copy, and then restore. We have XB1 North America up. Logging in is very slow for some reason, and we're troubleshooting that. There are also only 5 districts with no threat limitations in Financial and Waterfront because the XB1 population is so small. I think PS4 North America will be next up. We are waiting on the database to finish restoring. All EU environments had to be recreated from scratch, so those are taking a bit longer than expected. PC databases are being restored right now. Thanks, Matt

Hi all, I know this is garbage timing. The team is working as fast as they can to get things cut over. We will do our best to communicate timing. As of right now I don't have any extra information. Sorry, Matt

I should have qualified my statements a little better. Each event requires dev work to integrate achievements and rewards. There is a minor tweak this year to an existing mini-game that required some additional work, but a couple devs spending a week to prep out an event is actually quite short for us.

This is exactly correct. This is a fair criticism. Unfortunately we have to coordinate these larger moves around rigid contract end dates. Pushing this off would require us to wait another year. These will all be brand new servers, but many of the stability/performance issues are related to the 32-bit single threaded server code. While the existing hardware is quite old, in the year and a half since we took over, I have not seen any major issues with the power of the servers. Yes. The “loan” model is the end goal now. Citadel’s database is already pushing max capacity for performance. Our initial tests showed that consolidating worlds further would seriously bottleneck things. The loan model opens up some very cool functionality downstream, while keeping each world small and easier to run. Thanks, Matt