Server lag update

[MattScott 15242]

Hi everyone,

 

Let's talk about yesterday a little.

 

I just got back from Melbourne on Friday, and finally had a chance to get back in the game yesterday. 

A player alerted me that the game was running poorly, but we had been on mitigation, so I got on but wasn't sure what to expect.

I started with Financial and started running around with the other player.

 

You'll have to excuse my ignorance, but I had not actually experienced rubberbanding. Until yesterday.

 

The game was literally unplayable, and I'm sorry for that. It's inexcusable.

I'm sure many of you experienced the same frustration and terrible issues during your missions and attempts to play.

 

I spent the better part of the day doing latency tests and pulling my dev and network team in on a Saturday to look at the problem.

All total I played on 4 different servers across PC and PS4, and at one point I was on a server entirely by myself. The rubberbanding was just as bad.

We finally decided it was better to quickly reboot the Jericho servers to continue testing.

 

After the reboot, I got on again. I even played in a couple missions in green.

(Yes. You may have seen me running around Financial and Waterfront. I got my patootie handed to me. I think I got one kill the entire time.)

 

Here are a couple things I want to acknowledge from yesterday's research:

• Mitigation for DDOS causes latency.. maybe 20-30ms extra on Jericho. Less on Citadel. It certainly affects the game, but nothing like what I saw.
• We run real hardware not cloud servers. And while it's entirely possible that our servers aren't strong enough, after rebooting, the servers I played on ran great.
• In my opinion, this is clearly an issue in the server code, and my team is working through the server logs from yesterday to determine what happened so we can fix it.

 

Unfortunately in our efforts to try and clear more support tickets, we had more of our GMs handling tickets.

Going forward Lixil has re-tasked them all with staying in-game to help report latency issues, so we can quickly reboot servers is there are issues.

 

I had something of a celebration planned for this week, and we're still going to run it, but given the circumstances I'm going to change that theme to something else. Ultimately we appreciate you guys for hanging in there. I'll post tomorrow with more details.

 

Thanks,
Matt

[Ashley 19]

Thanks for the update! Hopefully this can be figured out sooner rather than later, and I'm looking forward to the "celebration"!

[vsb 6171]

thumbs up homie, communication strong

[Aeronaut 647]

11 minutes ago, BXNNXD said:

thumbs up homie, communication strong

and G1's equivalent? 

 

 

wait about half a year to a full year after a noticable uprising about something. make a vaguely stated blog post. dust off hands. 

 

 

rinse and repeat. 

Edited July 2, 2018 by Aeronaut

[Sadira 292]

Can't wait for that celebration!

[vsb 6171]

1 minute ago, Aeronaut said:

and gG1's equivalent? 

 

 

wait about half a year to a full year after a not I able uprising about something. make a vaguely stated blog post. dust off hands. 

 

 

rinse and repeat. 

30 days is a bit aggressive, 60 days seems far more likely

 

 

edit: are you drunk

Edited July 2, 2018 by BXNNXD

[CookiePuss 5378]

GG homie... GG

[Keshi 436]

thx for the update matt :')

[Senodus 49]

Thx for keeping us up to date and all the hard work! =D Can't wait to see what that celebration is going to be.

[Aeronaut 647]

4 minutes ago, BXNNXD said:

30 days is a bit aggressive, 60 days seems far more likely

 

 

edit: are you drunk

oh god autocorrect

[Queen of Love 453]

@boss

please disable BE in social district

[DopeSV 9]

I hope you make APB stable

 

Thanks for the news

Edited July 2, 2018 by DopeSV

[BriannaMae 0]

[Deadliest 385]

hows Melbourne 

 

 

thanks for visiting.

 

 

 

[Salvick 248]

Awesome update, I've been closing the game in the middle of my first mission in the last few days I attempted to play because this. Glad to know you know now.

[Nite 261]

Appreciate the updates!

 

45 minutes ago, MattScott said:

 

• We run real hardware not cloud servers. And while it's entirely possible that our servers aren't strong enough, after rebooting, the servers I played on ran great.

 

So this is probably a good time for me to ask - given the steady improvements being touted by the major cloud providers these days, would migrating to a large cloud service (AWS/Azure/Google) be something worth considering in future? Possibly as a post-engine upgrade endeavor? How big a technical hurdle would that be to surmount?

[Hexerin 1140]

What would be the feasibility of setting up the servers to automatically restart at a set time every so often (24/48/72 hours, as examples)? Since that apparently fixes this issue in the short term.

[iodyne_ 127]

do you mean community volunteer GM's? I didn't realize the GM's would be handling support tickets...i thought they would just be helping players in-game?

[professionalgamer 32]

apb reloaded more like apb relagged

[ilikecake 13]

1 hour ago, MattScott said:

 

You'll have to excuse my ignorance, but I had not actually experienced rubberbanding. Until yesterday.

 

 

thx for your communication but how is this even possible. It's been happening maybe like 2 weeks since you guys took over. and it's been happening pretty much EVERYDAY. And according to the other thread your CM has been in-game and apparently she had "fun". None of your team reported this issue to you that warrantied some kind of investigation either on your part or your engineer's part? I play maybe like 1 or 2 times in a week and it doesn't matter when I'm on, it lags, and often the district ends up crashing. And I've seen this in early morning with no /pop, afternoon with some /pop , and evening with some decent /pop. Also it happens everywhere including fight club.

 

and what can you say about packet loss? how come never had packet loss literally since G1 revived this game 2012 until you guys took over?

[Nite 261]

2 minutes ago, ilikecake said:

 

and what can you say about packet loss? how come never had packet loss literally since G1 revived this game 2012 until you guys took over?

I remember plenty of packet loss going on in the post-NA merge days though

[NotZombieBiscuit 3146]

43 minutes ago, Deadliest said:

hows Melbourne 

 

 

thanks for visiting.

 

 

 

Full of too many hipsters and people that think it is better than Sydney.

Edited July 2, 2018 by NotZombieBiscuit

[Excalibur! 207]

I wish your team finds a way to solve this issue... for me 30-40ms gets really bad (from 150 + those, my hit reg goes literally broken).

[Shiniko 5]

Hooray! Lag's being looked into!

 

Hooray! Celebration!

 

 

Dis gon be a good week!

[Salvick 248]

34 minutes ago, iodyne_ said:

do you mean community volunteer GM's? I didn't realize the GM's would be handling support tickets...i thought they would just be helping players in-game?

 

I also thought of this, but I'm quite sure he means that the current LO GM's, the ones that used to be managed by Tiggs and are part of the company I think, were put to do some sort of "support" tasks by assisting ingame with the latency issues and helping players to report their /latencytest results to contribute with this.

 

Still not sure though, but I believe a player or anyone outside of the company handling support tickets would be simply a no no.