The plan to fix the lag

[MattScott 15242]

Hi everyone,

 

Several posts around the forums have mentioned network issues and especially bad lag over the weekend, so I thought I would address this publicly.

 

First, I don't want to get in the habit of blaming "DDOS attacks" every time there is a problem. Sometimes that's the case. And sometimes it's not. It's important that G1 and LO be honest with the community when we mess up, and when our systems are being effected by external factors. 

 

We did get an especially heavy DDOS attack on Friday a few hours after the Q&A that took a bit to recover from. We switched on mitigation, and we were able to weather the storm. While we are on mitigation, our servers simply do not perform well. However, the attacked caused a side effect issue where some servers were "up" but disconnected from each other in a way that prevented new districts from spinning up and existing districts from responding. It took the better part of the weekend for us to both discover and resolve all of these issues.

 

Simply put, the initial lag was out of our control, and we have been on mitigation ever since which creates lag, but we also messed up and made the situation worse over the weekend. Please accept my apologies. We're a little new a this.

 

Second, we take this game and our players very seriously. An apology would be meaningless if we didn't describe how we are going to fix the issues. So here is how we're moving forward.

 

• We now have a new DDOS solution that is being tested, and if we find that the results are solid, then the new solution will all but eliminate the kind of lag caused by the existing mitigation solution. I'm already sick and tired of talking about how DDOS mitigation causes lag. I never want that to be an excuse for laggy servers again.
• We did some brainstorming, and we started on some new networking layer changes that will help with DDOS vulnerabilities. These sorts of changes need to be thoroughly tested, but we are committed to making this more difficult.
• We have provisioned several new servers. The player concurrency has doubled since we took over, and we are expanding our ability to host districts and provide a smoother in-game experience.
• We are altering our monitoring to not only determine whether servers are up, but also whether they care actively increasing their concurrent player count and other important metrics that let us know things are running correctly.

Thanks,
Matt

[Bloo 43]

Glad you guys are keeping the players updated with everything that you plan to do and care currently doing, much respect. Can't wait to see everything completed.

[Similarities 226]

Thanks a lot for the update, is there an ETA on when we can see this new system in place, or does the player base just need to keep our heads above the tide for now? Are these new servers that are being rented out going to help with the population, or help to mitigate the current lag? Or is that just preparation for the future?

Edited May 30, 2018 by Similarities

[Tessar 24]

our lord and savior. 

[vsb 6170]

am i dreaming

[ThatOneCripGirl 6]

Thanks for the update! Although, maybe next time, could you make an announcement in game letting players know that there will be server maintenance within the hour or so? But anyways, its good to finally hear that there'll be maintenance for once. Keep up the good work!

 

[itsExo 3]

Thanks for info! 

[speee 90]

Glad to see the communication. Thanks for updating us.

[Kurby 9]

Appreciation levels are over 9000.

[MattScott 15242]

3 minutes ago, Similarities said:

Thanks a lot for the update, is there an ETA on when we can see this new system in place, or does the player base just need to keep our heads above the tide for now? Are these new servers that are being rented out going to help with the population, or help to mitigate the current lag? Or is that just preparation for the future?

The new servers allow us to offload an instance here and there to help the servers perform better. But mostly we just need to keep pace with the growth.

[Cocoonutz 11]

Good to hear that the lag will be over soon enough

Are the new servers for hosting Jericho, Citadel, Nekrova and Han better or are they going to be like whole new server worlds separate from the others?

Edited May 30, 2018 by Cocoonutz

[Neuer 2]

thnx matty love u brodie

[SandyBitch 31]

Thanks Mr. Scott for the Info, guess we all can imagine the heavy workload and tasks you guys been busy with. I hope you guys can takeout all the issues you guys have to deal with. best of luck

 

have a nice day / night Little Orbit

 

/regards

[MattScott 15242]

Just now, Cocoonutz said:

Good to hear that the lag will be over soon enough

Are the new servers for hosting Jericho, Citadel, Nekrova and Han better or are they going to be like whole new server worlds separate from the others?

They are new district servers spread across existing worlds to help host more in-game players.

[dett2 64]

Thanks so much.

Jericho was almost unplayable , I'm really glad to hear that.

 

[Cocoonutz 11]

Just now, MattScott said:

They are new district servers spread across existing worlds to help host more in-game players.

Thanks for the clarification :)

[WhiskeyTangoFoxX 280]

Thank you, Matt! Just this communication alone is amazing, really looking forward to the changes in the coming months!

[Couth 5]

 

YES

Edited May 30, 2018 by Couth

[xnyuu 23]

@MattScottThanks for the communication, appreciated!♥

[Nevv 49]

Putting in that work

[T0asty 4]

I love the transparency of the new leadership. You guys rock! Can't wait for the future of APB and a fix for this lag/server outtages.

[Froztzen 0]

bless up ♥

[Zoehh 33]

3 minutes ago, Couth said:

YES

yes, but in ALL CAPITALS

[SiiG 1]

Just now, MattScott said:

Hi everyone,

 

Several posts around the forums have mentioned network issues and especially bad lag over the weekend, so I thought I would address this publicly.

 

First, I don't want to get in the habit of blaming "DDOS attacks" every time there is a problem. Sometimes that's the case. And sometimes it's not. It's important that G1 and LO be honest with the community when we mess up, and when our systems are being effected by external factors. 

 

We did get an especially heavy DDOS attack on Friday a few hours after the Q&A that took a bit to recover from. We switched on mitigation, and we were able to weather the storm. While we are on mitigation, our servers simply do not perform well. However, the attacked caused a side effect issue where some servers were "up" but disconnected from each other in a way that prevented new districts from spinning up and existing districts from responding. It took the better part of the weekend for us to both discover and resolve all of these issues.

 

Simply put, the initial lag was out of our control, and we have been on mitigation ever since which creates lag, but we also messed up and made the situation worse over the weekend. Please accept my apologies. We're a little new a this.

 

Second, we take this game and our players very seriously. An apology would be meaningless if we didn't describe how we are going to fix the issues. So here is how we're moving forward.

 

• We now have a new DDOS solution that is being tested, and if we find that the results are solid, then the new solution will all but eliminate the kind of lag caused by the existing mitigation solution. I'm already sick and tired of talking about how DDOS mitigation causes lag. I never want that to be an excuse for laggy servers again.
• We did some brainstorming, and we started on some new networking layer changes that will help with DDOS vulnerabilities. These sorts of changes need to be thoroughly tested, but we are committed to making this more difficult.
• We have provisioned several new servers. The player concurrency has doubled since we took over, and we are expanding our ability to host districts and provide a smoother in-game experience.
• We are altering our monitoring to not only determine whether servers are up, but also whether they care actively increasing their concurrent player count and other important metrics that let us know things are running correctly.

Thanks,
Matt

Matt Scott his method of work and transparency is really admirable my cincers congratulations, It's good to see that someone is really striving and giving their best for improvements.

[Ohshii 215]

yay ty!