Jump to content
MattScott

The plan to fix the lag

Recommended Posts

Hi everyone,

 

Several posts around the forums have mentioned network issues and especially bad lag over the weekend, so I thought I would address this publicly.

 

First, I don't want to get in the habit of blaming "DDOS attacks" every time there is a problem. Sometimes that's the case. And sometimes it's not. It's important that G1 and LO be honest with the community when we mess up, and when our systems are being effected by external factors. 

 

We did get an especially heavy DDOS attack on Friday a few hours after the Q&A that took a bit to recover from. We switched on mitigation, and we were able to weather the storm. While we are on mitigation, our servers simply do not perform well. However, the attacked caused a side effect issue where some servers were "up" but disconnected from each other in a way that prevented new districts from spinning up and existing districts from responding. It took the better part of the weekend for us to both discover and resolve all of these issues.

 

Simply put, the initial lag was out of our control, and we have been on mitigation ever since which creates lag, but we also messed up and made the situation worse over the weekend. Please accept my apologies. We're a little new a this.

 

Second, we take this game and our players very seriously. An apology would be meaningless if we didn't describe how we are going to fix the issues. So here is how we're moving forward.

 

  • We now have a new DDOS solution that is being tested, and if we find that the results are solid, then the new solution will all but eliminate the kind of lag caused by the existing mitigation solution. I'm already sick and tired of talking about how DDOS mitigation causes lag. I never want that to be an excuse for laggy servers again.
  • We did some brainstorming, and we started on some new networking layer changes that will help with DDOS vulnerabilities. These sorts of changes need to be thoroughly tested, but we are committed to making this more difficult.
  • We have provisioned several new servers. The player concurrency has doubled since we took over, and we are expanding our ability to host districts and provide a smoother in-game experience.
  • We are altering our monitoring to not only determine whether servers are up, but also whether they care actively increasing their concurrent player count and other important metrics that let us know things are running correctly.


Thanks,
Matt

  • Like 101
  • Thanks 46

Share this post


Link to post
Share on other sites

Glad you guys are keeping the players updated with everything that you plan to do and care currently doing, much respect. Can't wait to see everything completed.

  • Like 3

Share this post


Link to post
Share on other sites

Thanks a lot for the update, is there an ETA on when we can see this new system in place, or does the player base just need to keep our heads above the tide for now? Are these new servers that are being rented out going to help with the population, or help to mitigate the current lag? Or is that just preparation for the future?

Edited by Similarities

Share this post


Link to post
Share on other sites

am i dreaming

  • Like 3

Share this post


Link to post
Share on other sites

Thanks for the update! Although, maybe next time, could you make an announcement in game letting players know that there will be server maintenance within the hour or so? But anyways, its good to finally hear that there'll be maintenance for once. Keep up the good work!

 

Share this post


Link to post
Share on other sites

Glad to see the communication. Thanks for updating us.

Share this post


Link to post
Share on other sites
3 minutes ago, Similarities said:

Thanks a lot for the update, is there an ETA on when we can see this new system in place, or does the player base just need to keep our heads above the tide for now? Are these new servers that are being rented out going to help with the population, or help to mitigate the current lag? Or is that just preparation for the future?

The new servers allow us to offload an instance here and there to help the servers perform better. But mostly we just need to keep pace with the growth.

  • Like 3

Share this post


Link to post
Share on other sites

Good to hear that the lag will be over soon enough 😄

Are the new servers for hosting Jericho, Citadel, Nekrova and Han better or are they going to be like whole new server worlds separate from the others?

Edited by Cocoonutz

Share this post


Link to post
Share on other sites

Thanks Mr. Scott for the Info, guess we all can imagine the heavy workload and tasks you guys been busy with. I hope you guys can takeout all the issues you guys have to deal with. best of luck 🙂

 

have a nice day / night Little Orbit

 

/regards

Share this post


Link to post
Share on other sites
Just now, Cocoonutz said:

Good to hear that the lag will be over soon enough 😄

Are the new servers for hosting Jericho, Citadel, Nekrova and Han better or are they going to be like whole new server worlds separate from the others?

They are new district servers spread across existing worlds to help host more in-game players.

  • Like 4

Share this post


Link to post
Share on other sites

Thanks so much.

Jericho was almost unplayable , I'm really glad to hear that.

 

Share this post


Link to post
Share on other sites
Just now, MattScott said:

They are new district servers spread across existing worlds to help host more in-game players.

Thanks for the clarification :)

Share this post


Link to post
Share on other sites

Thank you, Matt! Just this communication alone is amazing, really looking forward to the changes in the coming months!

  • Like 1

Share this post


Link to post
Share on other sites

skeletor-gif-11.gif

 

YES

Edited by Couth
  • Like 1

Share this post


Link to post
Share on other sites

Putting in that work

Share this post


Link to post
Share on other sites

I love the transparency of the new leadership. You guys rock! Can't wait for the future of APB and a fix for this lag/server outtages.

Share this post


Link to post
Share on other sites
3 minutes ago, Couth said:

YES

yes, but in ALL CAPITALS

  • Like 1

Share this post


Link to post
Share on other sites
Just now, MattScott said:

Hi everyone,

 

Several posts around the forums have mentioned network issues and especially bad lag over the weekend, so I thought I would address this publicly.

 

First, I don't want to get in the habit of blaming "DDOS attacks" every time there is a problem. Sometimes that's the case. And sometimes it's not. It's important that G1 and LO be honest with the community when we mess up, and when our systems are being effected by external factors. 

 

We did get an especially heavy DDOS attack on Friday a few hours after the Q&A that took a bit to recover from. We switched on mitigation, and we were able to weather the storm. While we are on mitigation, our servers simply do not perform well. However, the attacked caused a side effect issue where some servers were "up" but disconnected from each other in a way that prevented new districts from spinning up and existing districts from responding. It took the better part of the weekend for us to both discover and resolve all of these issues.

 

Simply put, the initial lag was out of our control, and we have been on mitigation ever since which creates lag, but we also messed up and made the situation worse over the weekend. Please accept my apologies. We're a little new a this.

 

Second, we take this game and our players very seriously. An apology would be meaningless if we didn't describe how we are going to fix the issues. So here is how we're moving forward.

 

  • We now have a new DDOS solution that is being tested, and if we find that the results are solid, then the new solution will all but eliminate the kind of lag caused by the existing mitigation solution. I'm already sick and tired of talking about how DDOS mitigation causes lag. I never want that to be an excuse for laggy servers again.
  • We did some brainstorming, and we started on some new networking layer changes that will help with DDOS vulnerabilities. These sorts of changes need to be thoroughly tested, but we are committed to making this more difficult.
  • We have provisioned several new servers. The player concurrency has doubled since we took over, and we are expanding our ability to host districts and provide a smoother in-game experience.
  • We are altering our monitoring to not only determine whether servers are up, but also whether they care actively increasing their concurrent player count and other important metrics that let us know things are running correctly.


Thanks,
Matt

Matt Scott his method of work and transparency is really admirable my cincers congratulations, It's good to see that someone is really striving and giving their best for improvements.

Share this post


Link to post
Share on other sites

yay ty!

 

giphy.gif

Share this post


Link to post
Share on other sites

  • Recently Browsing   0 members

    No registered users viewing this page.

×
×
  • Create New...