MattScott

Hi there, We're about 1/2 done with it. The server issues did put us back a couple weeks but not that far. Thanks, Matt

We are still working on the issue, but I have been told Jericho is back up. I dont feel comfortable giving out any more information except that we are in fact operating on the G1 hardware that we got with the game. Some of that hardware has already been replaced or upgraded. We will be getting rid of more of it this week.

This is interesting, but there is no indication that this is related to issues we’ve been seeing. I suppose time will tell once everything is replaced.

Hi all, We have resolved the issue and the servers are open again. I have posted a more complete update here: Thanks, Matt

Hi everyone, Sorry for the unplanned outage this morning. We had a significant hardware failure in the middle of the night that took down the ability to login for all players. It took longer than expected, but the network team was able to recover the piece of equipment and bring the game back online. They are still looking at the issue so we can implement a more permenant fix during normal maintenance hours. Thanks, Matt

Hi everyone, This is a great list. I appreciate the time and effort collecting all the items. I’ve bookmarked it and passed it to the devs, so we can review and prioritize the issues. Once we get PC on Unreal 3.5, it will be on the same codebase as console, so we’ll be fixing most of these issues for everyone. Thanks, Matt

Hi there, Were you able to get your original premium back? If not, please open a ticket and we’ll get that resolved. Thanks, Matt

Hi there, We have made a round of fixes. We hope the high latency is better. Thanks, Matt

Hi there, I made a post in the Social discussion forum apologizing for these issues. We have made some recent lag improvements. We hope you’ll try again. Thanks, Matt

Hi there, We have made some recent adjustments for lag. We hope you’ll try the servers again. Thanks, Matt

Hi there, Can you try again? We have put up several fixes for this issue. Thanks, Matt

Hi everyone, I appreciate everyone's feedback. We're going to keep monitoring things. For those of you reporting hit reg issues, we have managed to recreate that in QA. We're going to try and get a fix in for those ASAP. Thanks, Matt

Hi everyone, We made several backend adjustments during last night's maintenance to help improve lag. I would be curious to get anyone's feedback on Jericho and Citadel today. Thanks, Matt

Hi there, Yes. We are getting organized on the HAN migration now. Expect a forum post soon. That will happen in advance of launching Unreal 3.5. Thanks, Matt

Hi there, We tested network code last week, and it still needs some work, so it wont be ready for Tuesday. Believe me, we are pushing as hard as we can. BTW - in publicly acknowledging and apologizing for the DDOS attacks (over Twitter, Facebook, and this forum), we got two more attacks within minutes. I often debate exactly how much information to share with the community due to this exact problem. LO has not interest in hiding anything, but every time we give information out, it also goes directly to the people initiating these attacks. Fortunately we're on mitigation right now, and those attackers wasted their efforts. They can't make it any worse. Rest assured, we're not giving up. Stay tuned. Thanks, Matt

We are reaching out to a couple other MMO/online game companies now.

Hi everyone, This is just a brief post to acknowledge and apologize for the horrible networks issues today. Our servers were hit with a series of nonstop DDOS attacks across both regions for about 5 hours between 11am and 4pm PDT. This is by far the worst series of collective attacks we’ve seen. We switched to mitigation and tripped another network issue with our provider that made the issue worse. We will do better in the future. Sorry, Matt

Hi there, We are working on the issue. (I know that sounds like a broken record, but it our #1 priority right now). Thanks, Matt

Hi everyone, This week had an unusually high volume of tickets +721. We got a large influx of account recovery requests, which indicates that many old players are still discovering the game again. We closed less tickets than last week. Some of them required a lot of research. Several tickets took more than 4 hours to resolve. Overall we've been working the oldest tickets to try and get our general wait time down, but those are also the most complex. We were able to start splitting tickets up to various internal specialists that have gotten better at handle specific types. Hopefully that will help. And we've created several new queues with various priorities. Be sure to correctly identify what kind of issue you're having. Here are the current CS stats this week: - 7,809 total tickets have been created for APB since the new portal went up on 5/25 - 4,702 of them have been closed. - 19,662 interactions back and forth on tickets. - 1,641 of them are untouched. Thanks, Matt

I can confirm this was a DDOS attack. *sigh* A little salt in the wound for this morning’s outage I suppose.

Hi guys, We are having a rough morning. We had an outage this morning with our European provider. My team is looking into this one. Thanks, Matt

Hi guys, I haven't responded to this in a while, so here is an update. For context, this morning's outage had nothing to do with DDOSing or lag. Our European provider had a problem, and it took our servers down. That caused another problem, so we had to restart Jericho. My apologies to the players - especially in Europe - who had to wait. We have another round of server code updates that we're hoping will address the lag. One of the big blockers is how to get that code properly tested. Most servers run fine with 20, 30, or even 50 players. We don't generally run into problems until we have a nearly full district at 80 players with a bunch of missions going on. Yesterday we were finally able to get an environment up and organize a regular group of employees to run a decent test. The results were mixed but very informative. I'm hoping we'll be able to do this now a couple times a week until we have a build that we feel is stable and addresses the issues. For now, I'm sure you've noticed that we are rebooting the servers a little more often to help alleviate the problem. I've setup a schedule with our NOC to try and do that during low concurrency times, and I apologize for the disruption that causes, but I feel it's better than leaving the servers in an unplayable state. Thanks, Matt

Hi all, Lots of good discussion here. Per the request of the OP, I'm going to lock the thread. Thanks, Matt

Hi there, This is exactly why we normally do not delete threads unless they violate our ToS. Deleting a thread removes the context for readers like yourself aftert the fact, and it leads to assumptions like ‘censorship’ and ‘abuse of power’. In this case, the OP of that thread asked us to remove it. I started a new thread. Let that get discussed for several hours, and then removed the OP’s thread. It wasn’t ideal. I felt she suffered quite a bit at the hands of the community, and I don’t like all that toxicity. Thanks, Matt

Hi everyone, I'm going to weigh in again here, and then I'm going to lock this thread. First, I've read every response so far. Rest assured I am 110% committed to continue trying new ideas on how to interact with our community. That's the only way this game will grow. Second, I still believe there were problems in our approach in this case. If we are going to stream, we'll do it on an official channel - as representatives of the company. If we play against players, they need to know and agree to that possibility before they join the match. And we can't use anything we haven't earned as a regular player - unless it is a special event setup that way. I don't view our response as letting a vocal minority win. I'm not taking anything away from the community. I view it as a learning opportunity. We're going to try stuff like this again. But next time, we'll do it better. Lastly, here is my opinion on the subject of Lixil's team members: Everyone in the game right now has started with zero strikes against them. I'm not interested in toxicity, finger pointing, or treating any one any differently for past accusations or bans. We only unbanned people where there wasn't any evidence to support the original ban. However, that is not the same thing as condoning cheating. If we have proof of cheating or breaking our ToS, we will ban you. Period. That's a shame, and every ban feels like a failure on our part to salvage that player. But APB is a challenging game that takes a lot of practice to master. Using cheats to gain an advantage destroys that fun challenge. If we don't get it under control, it will destroy the game. It's our job to make sure that doesn't happen. Thanks, Matt