Network migration status

[MattScott 15242]

Hi all,

 

Many players have complained about latency or status on the Network Migration effort.

So I'm going to use this thread to post updates.

 

For context, here is what we did:

 

• In May we shutdown an LA datacenter that was only being used for a couple critical services. That long standing contract that was expiring, and it was an opportunity to consolidate network services elsewhere. This datacenter had been in use for years, and as we discovered, there were a lot of buried landmines and hard coded references.
• In November we migrated services to a new provider because that second long standing contract was expiring. The new provider offered us better hosting locations, more bandwidth, new servers, and more comprehensive DDoS protection. This effort was required to prep the upcoming World Consolidation and local districts. However, again this migration did not go smoothly. A number of new servers had bad SSDs and broken network transceivers. Additionally, the servers meant for our new EU datacenter got held up in customs due to bad paperwork filled out by our vendors.

 

Since our departure was based on an expiring contract, we were forced to migrate and launch on a reduced set of servers.

 

There are 6 "worlds" in our environment: NA PC, PS4, and XB1. EU PC, PS4, and XB1. Before I get a bunch of players asking why we chose to keep PS4 and XB1 worlds up, we are contractually obligated to our console publisher to keep those worlds online. There was no option other than to support them regardless of concurrency. Needless to say, all PS4 and XB1 worlds are running minimal environments.

 

• Currently Jericho is operating on mostly Los Angeles-based servers with a couple New York servers. This is why west coast players occasionally see increased latency. We didn't have enough servers to support threat segregation, so all Jericho mission districts are open threat right now.
• Currently Citadel is operating on New York-based servers. This is why EU players are seeing horrible latency. We also didn't have enough servers to support threat segregation, so all Citadel mission districts are open threat right now.

 

We have been told our EU servers cleared customs today and are enroute to the new datacenter. Little Orbit has no control over this process.

When the servers arrive, they will be racked, hooked up, and installed with our basic OS image before being turned over to our techs.

Once we get access, then it will take a day or two for our techs to configure everything, load the game servers, and test them.

 

Citadel will have local EU district servers, and the latency will drop down significantly.

Then all the LA / NY servers will be re-assigned to Jericho.
The dev team is working on exposing locations in the Advanced tab, and adding logic for how players are assigned to LA or NY servers.

 

Thanks,

Matt

[MattScott 15242]

Hi all,

 

Our network provider has informed us that the EU hardware has been delivered to the new datacenter, racked, and they are working on installing OSes. I believe the plan is to give us access tonight, so we can install the game server build.

 

I’m sharing what I’m being told, but honestly everything involving the new hardware has been 3-4 days later than they tell us.

 

For those that have been asking, the hardware was ordered and shipped by our network provider that we lease the boxes from. Unfortunately, we had no input or window into any of the customs paperwork or issues, so I can’t shed any light on what happened there.

 

Thanks,

Matt

[MattScott 15242]

Hi all,

 

Unfortunately we are still working through issues with the EU servers. All of the boxes arrived as expected, however we hit a very frustrating snag - it appears the configuration we laid out was not explicitly followed, and the game server software wouldn’t run on the CPUs due to missing instruction sets that we require.

 

The team has expedited a new batch of servers arriving tomorrow. I’ll keep you posted.

 

Thanks,

Matt

[MattScott 15242]

Hi all,

 

All of our networking partners are back from the break.

It was extremely frustrating to see everyone leave for the holidays with our servers not back online.

I'm told my team will have access tomorrow (Friday) and that we can start installing images over the weekend.

 

If everything holds true, then EU servers will come online during next week's patch maintenance.

I will be extremely happy to have this put behind us, so we can concentrate on the next phase of world consolidation and local districts.

 

Apologies,
Matt

[MattScott 15242]

Hi all,

 

Lots of good progress on getting the remaining servers up and running in our new LA datacenter, however the EU servers continue to be a black box.

 

I hate excuses, so I’m not going to make any.
 

I haven’t had a direct response about the EU servers since they said they would deliver access on Friday. We scheduled our network team to work the weekend, but nothing has been turned over to us yet. I believe the group we are working with outsourced their EU support, so they are likely waiting on someone else, who is probably waiting on actual datacenter staff to move around hardware.

 

With all these problems, many players may be wondering why we chose this new provider.

Simply put, they are experts in defending DDoS attacks, and the end of our previous contract gave us an opportunity to be in the same cage as high-end custom hardware that can protect the game far better than in the past.

 

Since we cutover in November, we have been hit with some of the biggest attacks I have seen since we took over. One attack was 200Gbps. That is 10-20x what were seeing in the past. Our new provider blocked nearly all of the attacks with zero impact to players, and the few that got through were analyzed immediately so our rules could be patched in minutes.

 

I will continue to push for status on the EU servers, and I’ll post here when I have more details.

 

Thanks,

Matt

[MattScott 15242]

Hi all,

 

A portion of the EU servers were turned over to my team last night. We are getting game server software and testing done immediately.

There are enough servers that we should be able to move all action districts and fight club districts locally to Europe.

The plan will be to do a short maintenance on Citadel on Thursday night to take these live.

 

The remaining servers should all be racked early next week. I'll be watching the tracking numbers and providing status as I get it.

 

Thanks,
Matt

[MattScott 15242]

Hi all,

 

We have 6 new servers on the West Coast and 7 new servers in Europe coming online today. That's still only a fraction of the total servers that will eventually be in place, but it's enough to host the current configuration locally in Europe with the possible exception of Social districts which will stay on the East Coast. I'll be posting the maintenance announcement soon. The plan is to stop one world at a time, change out districts, do some testing, and then open that world to players. So there will be rolling availability during the down time.

Thanks,
Matt

[MattScott 15242]

Hi all,

 

Quick update. We have replacement CPUs coming in for the Europe servers which should address the lag issues. We're hoping those will get swapped out during tonight's maintenance window.

More new Europe servers are coming in on Thursday. They were due today, but some components failed testing before shipping which delay them by 2 days.

As soon as we get the next batch of Europe servers up, I think we can re-enable threat across all worlds.

 

We are working on a patch that will help players identify the fastest / closest servers much easier.

We are also spinning up more East Coast servers to help those players get better latency.

Thanks,
Matt

[MattScott 15242]

Hi all,

We're now very close to finally being completed.

The remaining Europe servers will be delivered on Monday, and it is my hope to have them up and running by the end of next week.

As soon as all servers are online, then we can re-activate threat.

 

Thanks,
Matt