Apology to the community for this last week

[BrandonBranderson 672]

10 hours ago, Rubbermade said:

how long does it usually take for FE to Initialize after it crashes??

 

I think Matt said recently that it takes a little over an hour for a FE server to boot up.

[PersianTiger 28]

Hi. i just wanted to let you moderators know i am still having issue with login server, it wont let me in, maybe another issue ? i am asking because there is no one referencing it in the forums. 

[Keshi 436]

thank you for the clarification @MattScott i got home from work yesterday and played fine. Only to open the forums today to see that things wasnt going well the past few days i wasnt home xD
Good job tho!

[ggbantz 2]

After 18+ hours ( ͡° ͜ʖ ͡°)

 

 

[xDoomGuardx 22]

The answer concerning the solution of technical problems is a great pity that the moderators have the incentive not capable of properly articulate and convey to the audience. But I have a question for Matt or the person in charge of the technical support solution:

when will the technical support start to answer the questions normally and in a timely manner. Waiting half a year is bullying.

The next question is: when you start to seriously address the cheaters in the game. Your vaunted anti-cheat does nothing but complicate entering the game. Cheaters play freely, nothing and no fear.

Game masters don't do their job.

[Westford 170]

47 minutes ago, xDoomGuardx said:

The answer concerning the solution of technical problems is a great pity that the moderators have the incentive not capable of properly articulate and convey to the audience.

I believe that if Matt Scott wanted the moderators to communicate the technical issues related to the servers, than it would have been Matt Scott drafting the verbiage with which the moderators would have disseminated to the player base.

 

I also believe that because of the severity of these recent technical issues, that Matt Scott thought it would be best for "that" message to come from him directly.

 

Although my beliefs could be wrong.

 

 

 

Edited May 8, 2019 by Westford

[Arianna 82]

Thank you 

[YeOldLegends 51]

16 hours ago, MattScott said:

Hi all,

 

I'm not interested in sugarcoating or spinning this last week.

I am sorry for all the downtime.

 

Many players have asked what happened, so I'm going to do my best to summarize the events.

 

On Monday the 29th, we took the servers down to move data centers and migrate to all new hardware for our backend systems. This move was necessary for a couple reasons, but mainly because a lot of the hardware was more than 5 years old, and we were already seeing failures and system performance problems. It was only a matter of time before something critical failed. The timing also happened to line up with the end of our last legacy (ludicrously overpriced) hosting contract, and we needed to make some network architecture changes to facilitate some of the features coming after the Engine Upgrade. 

 

I figured we could kill three birds with one stone.

 

The core challenge was managing all the information needed to drive APB since all the hardware was new. That meant backing everything up and then hand carrying a series of large hard drives from one location to another. The hardware had been prepped and configured in advance, so while we knew this would be challenging, but we had a fairly detailed plan and felt we could manage any issues that popped up.

 

Problem #1: Unfortunately, during the move we unearthed some buried issues that delayed our ability to bring servers back online for quite a while. The team did a solid job of working through those problems and even recompiling code in a couple places to remove landmines that we stumbled over. But once we got the servers back online, I felt we did a decent job.

 

Problem #2: Shortly after we went live, the brand new RAID controller in our new primary database server quickly degraded in performance. To make the situation worse, we rushed to get servers back online and decided it would be okay to let players back in while the secondary database server finished syncing. The hardware failure hit so quickly that the secondary wasn't ready, so we couldn't failover. At this point we made an effort to keep the servers online through the weekend, and while our jury-rigged fix allowed some players to get on, it also lead to many other players being unable to login (Error 9). The team decided the quickest way to fix the issue would be to build an entirely new primary database server and then swapping everything over on Monday. We didn't want to risk moving damaged drives to the new servers, so we needed a complete backup to make sure we didn't lose anything.

 

Problem #3: Once we shutoff all the servers and started the backup, we found that the faulty RAID controller could only copy files at the rate of 1GB per minute. After 18+ hours, we were finally able to complete the backup, and then finally get the new server finalized and back online.

 

There are a lot of things that went wrong, but in the end, I should have planned better. With that much new hardware, we were bound to have an issue somewhere.

 

To make it up to everyone, on Friday we will be turning on 2 weeks of free Premium for all players. For anyone who has existing Premium, this will add to it.

 

I never want to have to make this kind of apology again.

Little Orbit can and will do better in the future.

 

Sorry,

Matt

I work in the industry and I get it but I still want those pictures of UE3.5

[Sayori 311]

It's not that bad, seen worse tech sh.tstorms.

And no need to apologize, got new servers, (man-)children should be happy now. 

[Ohshii 215]

This community is aids and the vast majority of them including myself don't deserve anything from you on a personal level, with that being said, I appreciate the explanation and more so the apology to all of us.

 

Real rap, good looking out. I hope the community realizes how different you and your company is from G1/K2 Networks. If they don't, they'll regret it later.

[gypexa cynepcmap 11]

being Russian I got used to difficulties, and, to tell the truth, entertaining Internet projects like online games are far from perfect here, most are made somehow and are designed solely for receiving money. In many of those in which I participated, it is only interesting at first, and then people .. the players already assigned to them are forced to donate, often not justified.
One day, having become acquainted with this game, I saw in it a future of interestingness for an RPG player ..
.. Donat here almost by 90% coincides with my ideas about a good donate - this is a payment for visual effects. Of course, there are 10% of those that do not affect the gameplay, but there is also the possibility to resist within the game, without the participation of money, which is more interesting, and, of course, even more successful for the growing and evolving player ..

being Russian, I don’t strive for compensation, for me the best reward is the performance of the game, the stability and inviolability of the servers ..

thank you for your work, Matt

[Hausman 34]

rip to all the freetime u guys spend / wasted for this unwanted issue
but good job to whole LO team that all finally works again, ty 

[koenyboy500 38]

Between you and me Matt, I can never really get angry at you guys : p I got a ton'o'games on other things as well, APB is just a hobby of mine (altough it has been made abundantly clear that SOME people have ABSOLUTELY nothing else to do besides playing APB:R and start complaining without end when something happens \_/^^\_/

 

But I do have an IMPORTANT QUESTION: Will the RIOT-playtest be delayed due to the technical issues? I would love to know if I can enjoy a good ol' let-em-eat-lead trick in a brand new gamemode this friday (saturday for me actually).

[doorkicker 46]

21 hours ago, MattScott said:

I never want to have to make this kind of apology again.

Little Orbit can and will do better in the future.

 

I feel like if anything, it is all of us who should be thankful for what you and your team have done over the past year or so for this game.  Let us not forget that barely a year ago, APB looked certain to die a slow, painful and silent death. 

 

Sure, things may not exactly be fantastic at present and people might've had bigger (if not unrealistic) expectations but I don't think any of that should take away from the work that you've all done and the actual, demonstrable progress that has been made.  I don't even think that the RIOT update is something that will interest me too much (at least not compared to the engine update), but it is certain to breathe new life into the game and really, what else can anyone ask for at this point?

 

Huge props to you and your team for not only owning your failures, but also pushing on nonetheless.  That's a very, very welcome change from a year ago.

Edited May 8, 2019 by doorkicker

[Aeronaut 647]

i feel like apb is the only game that would be understood and deserving of a few weeks of downtime so the team behind it all can have a non-rushed environment to work out the kinks. of course this isn't reasonable because downtime is a big no-no regardless of the reason...

[Guest]

6 hours ago, Aeronaut said:

i feel like apb is the only game that would be understood and deserving of a few weeks of downtime so the team behind it all can have a non-rushed environment to work out the kinks. of course this isn't reasonable because downtime is a big no-no regardless of the reason...

[Deadliest 388]

population's foking dead m8

[Mitne 724]

17 hours ago, Ohshii said:

This community is aids and the vast majority of them including myself don't deserve anything from you on a personal level, with that being said, I appreciate the explanation and more so the apology to all of us.

 

Real rap, good looking out. I hope the community realizes how different you and your company is from G1/K2 Networks. If they don't, they'll regret it later.

Meh. Screw them, most of them is lost cause anyway.

They enter for week, complaing about everything, quit for month, to go back and then moan how there is not any major change, continue to complain, repeat and go.

You either stay to end or give up - dunno why this people don't give up and let normal people step in. Saw a lot of normal newbies who geniunely enjoy game but are destroyed because constant moaners need their kill.

It's insane.

[yood 345]

1 hour ago, Mitne said:

 

It's insane.

Yes. the internal state of the game is CHAOS .

 

Merged.

 

Boss ! I would never want such posts again . 

 

there is a proverb : For two hares chase - and not one will not catch . Boss you made a choice - the team followed your instructions . who is the weak link in the team ?

 

[Cr0 328]

Question:

On the FE forum you wrote that you tried to extend the hosting contract but got declined, thus forced to move. Here it sounds more like it was your own choice to move and that the expired contract just happened to line up with this choice. Could you clarify?

Edited May 9, 2019 by SilverCrow

[LAPDAlonso 41]

2 Weeks of premium, hmmm not bad.

But what about sharing some new engine video and photos? can't you guys do this for the community.

[CookiePuss 5382]

1 hour ago, LAPDAlonso said:

2 Weeks of premium, hmmm not bad.

But what about sharing some new engine video and photos? can't you guys do this for the community.

I think Matt has said several times now, when they are ready, he will.

[MattScott 15245]

12 hours ago, SilverCrow said:

Question:

On the FE forum you wrote that you tried to extend the hosting contract but got declined, thus forced to move. Here it sounds more like it was your own choice to move and that the expired contract just happened to line up with this choice. Could you clarify?

Sure thing. I can see the confusion.

 

It was both.

 

The contract was up, and we had already put our notice in with the old hosting facility.

However at the very end there was a delay with the new hardware arriving and getting setup on-time at the new facility.

My preference would have been to get 2 more weeks at the old facility, but they denied our request to stay a bit longer.

[Լіe 12]

Thanks. Although, 2 weeks premium is nice. Please add in a free character slot if possible.  It's something that's been done in the past for compensations. Would really like seeing a return on it. I asked members of the community and they agreed.
  
Plus would like to ask when Nekrova is being migrated to Citadel?

Edited May 10, 2019 by Լіe

[vsb 6174]

5 minutes ago, Լіe said:

Thanks. Although, 2 weeks premium is nice. Please add in a free character slot if possible.  It's something that's been done in the past for compensations. Would really like seeing a return on it.

free character slot were given out in the past because specific servers were unplayable, doesnt really apply here