Jump to content
MattScott

Server lag update

Recommended Posts

Hi everyone,

 

Let's talk about yesterday a little.

 

I just got back from Melbourne on Friday, and finally had a chance to get back in the game yesterday. 

A player alerted me that the game was running poorly, but we had been on mitigation, so I got on but wasn't sure what to expect.

I started with Financial and started running around with the other player.

 

You'll have to excuse my ignorance, but I had not actually experienced rubberbanding. Until yesterday.

 

The game was literally unplayable, and I'm sorry for that. It's inexcusable.

I'm sure many of you experienced the same frustration and terrible issues during your missions and attempts to play.

 

I spent the better part of the day doing latency tests and pulling my dev and network team in on a Saturday to look at the problem.

All total I played on 4 different servers across PC and PS4, and at one point I was on a server entirely by myself. The rubberbanding was just as bad.

We finally decided it was better to quickly reboot the Jericho servers to continue testing.

 

After the reboot, I got on again. I even played in a couple missions in green.

(Yes. You may have seen me running around Financial and Waterfront. I got my patootie handed to me. I think I got one kill the entire time.)

 

Here are a couple things I want to acknowledge from yesterday's research:

  • Mitigation for DDOS causes latency.. maybe 20-30ms extra on Jericho. Less on Citadel. It certainly affects the game, but nothing like what I saw.
  • We run real hardware not cloud servers. And while it's entirely possible that our servers aren't strong enough, after rebooting, the servers I played on ran great.
  • In my opinion, this is clearly an issue in the server code, and my team is working through the server logs from yesterday to determine what happened so we can fix it.

 

Unfortunately in our efforts to try and clear more support tickets, we had more of our GMs handling tickets.

Going forward Lixil has re-tasked them all with staying in-game to help report latency issues, so we can quickly reboot servers is there are issues.

 

I had something of a celebration planned for this week, and we're still going to run it, but given the circumstances I'm going to change that theme to something else. Ultimately we appreciate you guys for hanging in there. I'll post tomorrow with more details.

 

Thanks,
Matt

  • Like 35
  • Thanks 45

Share this post


Link to post
Share on other sites

Thanks for the update! Hopefully this can be figured out sooner rather than later, and I'm looking forward to the "celebration"!

Share this post


Link to post
Share on other sites

thumbs up homie, communication strong

  • Like 4

Share this post


Link to post
Share on other sites
11 minutes ago, BXNNXD said:

thumbs up homie, communication strong

and G1's equivalent? 

 

 

wait about half a year to a full year after a noticable uprising about something. make a vaguely stated blog post. dust off hands. 

 

 

rinse and repeat. 

Edited by Aeronaut
  • Like 5

Share this post


Link to post
Share on other sites

Can't wait for that celebration!

Share this post


Link to post
Share on other sites
1 minute ago, Aeronaut said:

and gG1's equivalent? 

 

 

wait about half a year to a full year after a not I able uprising about something. make a vaguely stated blog post. dust off hands. 

 

 

rinse and repeat. 

30 days is a bit aggressive, 60 days seems far more likely

 

 

edit: are you drunk

Edited by BXNNXD

Share this post


Link to post
Share on other sites

thx for the update matt :')

Share this post


Link to post
Share on other sites

Thx for keeping us up to date and all the hard work! =D Can't wait to see what that celebration is going to be.

Share this post


Link to post
Share on other sites
4 minutes ago, BXNNXD said:

30 days is a bit aggressive, 60 days seems far more likely

 

 

edit: are you drunk

oh god autocorrect

Share this post


Link to post
Share on other sites

I hope you make APB stable 😞

 

Thanks for the news

Edited by DopeSV
  • Like 2

Share this post


Link to post
Share on other sites

Awesome update, I've been closing the game in the middle of my first mission in the last few days I attempted to play because this. Glad to know you know now.

Share this post


Link to post
Share on other sites

Appreciate the updates!

 

45 minutes ago, MattScott said:

 

  • We run real hardware not cloud servers. And while it's entirely possible that our servers aren't strong enough, after rebooting, the servers I played on ran great.

 

So this is probably a good time for me to ask - given the steady improvements being touted by the major cloud providers these days, would migrating to a large cloud service (AWS/Azure/Google) be something worth considering in future? Possibly as a post-engine upgrade endeavor? How big a technical hurdle would that be to surmount?

  • Like 1

Share this post


Link to post
Share on other sites

What would be the feasibility of setting up the servers to automatically restart at a set time every so often (24/48/72 hours, as examples)? Since that apparently fixes this issue in the short term.

  • Like 1

Share this post


Link to post
Share on other sites

do you mean community volunteer GM's? I didn't realize the GM's would be handling support tickets...i thought they would just be helping players in-game?

Share this post


Link to post
Share on other sites
1 hour ago, MattScott said:

 

You'll have to excuse my ignorance, but I had not actually experienced rubberbanding. Until yesterday.

 

 

thx for your communication but how is this even possible. It's been happening maybe like 2 weeks since you guys took over. and it's been happening pretty much EVERYDAY. And according to the other thread your CM has been in-game and apparently she had "fun". None of your team reported this issue to you that warrantied some kind of investigation either on your part or your engineer's part? I play maybe like 1 or 2 times in a week and it doesn't matter when I'm on, it lags, and often the district ends up crashing. And I've seen this in early morning with no /pop, afternoon with some /pop , and evening with some decent /pop. Also it happens everywhere including fight club.

 

and what can you say about packet loss? how come never had packet loss literally since G1 revived this game 2012 until you guys took over?

  • Like 3

Share this post


Link to post
Share on other sites
2 minutes ago, ilikecake said:

 

and what can you say about packet loss? how come never had packet loss literally since G1 revived this game 2012 until you guys took over?

I remember plenty of packet loss going on in the post-NA merge days though

  • Like 1

Share this post


Link to post
Share on other sites
43 minutes ago, Deadliest said:

hows Melbourne 

 

 

thanks for visiting.

 

 

 

Full of too many hipsters and people that think it is better than Sydney.

Edited by NotZombieBiscuit
  • Like 1

Share this post


Link to post
Share on other sites

Hooray! Lag's being looked into!

 

Hooray! Celebration!

 

 

Dis gon be a good week! 😄

Share this post


Link to post
Share on other sites
34 minutes ago, iodyne_ said:

do you mean community volunteer GM's? I didn't realize the GM's would be handling support tickets...i thought they would just be helping players in-game?

 

I also thought of this, but I'm quite sure he means that the current LO GM's, the ones that used to be managed by Tiggs and are part of the company I think, were put to do some sort of "support" tasks by assisting ingame with the latency issues and helping players to report their /latencytest results to contribute with this.

 

Still not sure though, but I believe a player or anyone outside of the company handling support tickets would be simply a no no.

Share this post


Link to post
Share on other sites

  • Recently Browsing   0 members

    No registered users viewing this page.

×
×
  • Create New...