Jump to content
Forum rules

MattScott

CEO
 View Profile  See their activity

  • Content Count

    1299

  • Joined

  • Last visited

Posts posted by MattScott

  1. Posted

    21 minutes ago, Lelouche said:

    I'm curious about this too as it said for All players. My buddy who hasn't been on for months got his but i still haven't gotten it 😕

     

    48 minutes ago, SpeedyCat said:

    What is with Players who havent logged in for more than 60 days on theyr accounts?

    Because you sayd that every player will get the compensation:

    (snip)

    I only ask about curiousity, it confused me a bit. 🤔

    Before somebody think something wrong....: “No, i dont need Premium, i have around 3k days.“ and atm i am happy with that. 👍

    Thx & greetz,

    Speedy^^

    My original intention had been to award Premium to everyone.

     

    However once we figured out that we couldn't release a code due to problems with consoles players, that meant we had individually award the Premium.

    Even doing this programmatically took an obscene amount of time. So at that point, I had to change the plan and limit the compensation to just players who have logged in within 60 days.

    • Like 1

  4. Posted

    Hi everyone,

     

    I just checked with the team, and it looks like all players who logged in within the last 60 days should now have 2 weeks of Premium added to their account.

    This includes players on all worlds, on PC, XB1, and PS4.

     

    Several players asked me why this took so long, and why we didn't just issue a code like last time.

    I would have preferred to issue a code, because it's much easier to do, but console players can't redeem them.

    There was one other way, but it would have involved waiting till Wednesday's patch to implement.

     

    The easiest way to check for Premium is to log into the APB Marketplace here: https://www.gamersfirst.com/marketplace/ingame/index.php?gameID=20

     

    If you logged in during the last 60 days, but for some reason your account didn't get the compensation, please open a Support ticket, and we will do our best to escalate getting it out to you.

     

    Thanks,

    Matt

    • Like 10
    • Thanks 7

  8. Posted

    Hi all,

     

    It looks like several things happened to make this situation fairly bad.

     

    First, our XB1 monitoring got screwed up during the data center move. So it appears there were several DDoS attacks that none of the team were aware of.

    Second, the DDoS attacks were so large that our ISP decided to block all traffic to the Xbox district for a period of time.

    Third, we're short handed this weekend, because many of my engineers have been working 7 day weeks since the end of April.

     

    We're working on the situation now with our ISP to get better notifications from them when they block IPs.

    And we're rebuilding our monitoring for XB1.

     

    Sorry,
    Matt

    • Like 4

  9. Posted

    7 hours ago, Gamma64c said:

    I also suffer under the __no premium syndrome__. If it were Tiggs she would have just run one code for all. It expires in a month, you enter it via armas and done! I wonder why Matt chose this way, which misses "a lot" of people

    Hi all,

     

    We didn't do a code, because the console players can't redeem codes, and we promised 2 weeks of Premium to everyone.

    For those that don't have it, we are still distributing the compensation.


    Thanks,
    Matt

    • Thanks 6

  14. Posted

    Hi all,


    We just started awarding the Premium (4pm Friday Pacific time).

    There are a lot of players to gift, so we're doing this in waves. To start we will hit everyone who has logged in within the last 30 days sorted by most recent login first.

    Then we'll go back 30 days, etc.

     

    Your Premium wont start till it is awarded to the account.

    I'll keep you posted.

     

    Thanks,
    Matt

    • Like 6
    • Thanks 3

  15. Posted

    36 minutes ago, Wypsnchanz said:

    Personally I'd like to see the game shut down for a time (a month?) to allow the LO team to work on issues without having to deal with all this pressure.  I also think they should wipe the current (problematic) character database and have everyone start fresh and perhaps do a new beta test period.  Once the game is "live" again, current players who spent real money can be reimbursed (in-game) based on their transaction history.  Not only will this serve to get rid of what is essentially a massive amount of "junk" data (much of which is probably corrupt), it'll free up a lot of character names for players to use.

    Just my opinion.

    I can appreciate your position, but I've already gone on record to say we are not wiping data.

    We can and will get the server back up. The requirements are just a little tricky to nail down for something like this.

    Having a server up for years collecting and indexing data over time is a different spec from starting a server empty and loading it with years of data.

     

    Once we get stabilized then we have a different way of moving old data out of the active tables dynamically without wiping anything.

    This just requires some coding and testing in our PTS environment, which we don't have time for right now.

     

    Thanks,
    Matt

    • Like 3
    • Thanks 1

  16. Posted

    5 hours ago, zefcool said:

    you mean a new server (with 2019 hardware) with a copy of the old database of the crappy server, or is it the same shity server with maxed "dirty tricks" skill used by engineers to make it stable somehow ?

    We can't change the software OS, the size or structure of the structure of the database. There is no way to avoid some of the "dirty tricks" to keep the game online till we re-write the backend.

    However we are running on new hardware across the board.

     

    The issue we are solving right now relates to the problematic database server (used for pulling all character data).

    Originally we got everything restored, but clearly players were having trouble retrieving characters on login.

     

    At this point, we have determined that this specific database is so large (and poorly structured) that we can no longer simply restore it. Loading that much data results in unusable table indexes which generates the problems we were seeing before we took it down. It takes 6 hours each attempt to restore and then 7 hours each attempt to repair. We have tried 3 different ways to get the database setup, and all have yielded the same result.

     

    We are shifting tactics and moving to a different set of expanded hardware. Then we'll give it a 4th attempt.

    Everyone is working the weekend to get this back online asap.

     

    Thanks,
    Matt

    • Thanks 7

  17. Posted

    Hi all,

     

    This is my last update for the evening. I'm managing the hand off between engineers and then going to call it a night.

     

    We got entirely through the restore and most of the repair and then the database errored.

    It cost us hours of work. Instead of proceeding on a shaky server, I'm having the team tear it down and rebuild it, so we know its stable.

    Unfortunately there is a lot more work to do. I don't have a good ETA, but likely 12 more hours.

    I'll update everyone where we are at in the morning.

     

    Sorry,
    Matt

    • Like 2
    • Thanks 5

  18. Posted

    Hi all,

     

    We're nearly done with the partial restore to get back to a position to repair things.
    The backup was done right after we took the servers offline, so there should be no rollback.

     

    Restore should be done around 1am, and then I'll have a better estimate for the repair time.

    Unfortunately we wont be back online at 1am like I had hoped.

     

    Sorry,

    Matt

    • Like 1
    • Thanks 5

  19. Posted

    12 hours ago, SilverCrow said:

    Question:

    On the FE forum you wrote that you tried to extend the hosting contract but got declined, thus forced to move. Here it sounds more like it was your own choice to move and that the expired contract just happened to line up with this choice. Could you clarify?

    Sure thing. I can see the confusion.

     

    It was both.

     

    The contract was up, and we had already put our notice in with the old hosting facility.

    However at the very end there was a delay with the new hardware arriving and getting setup on-time at the new facility.

    My preference would have been to get 2 more weeks at the old facility, but they denied our request to stay a bit longer.

    • Thanks 6

  21. Posted

     

    Hi all,

     

    We spent some time yesterday looking at the character login issues and found that 6 tables across our various databases needed heavy re-indexing after they got restored during the move. The tables are all over 5GB in size with the largest being 550MM rows at 55GB. This largest table contains all character attributes for the more than 2.5MM characters made in Fallen Earth to date.

     

    The short term fix is to re-index and stabilize things.

    The long term fix will be to move old character data to an archive database where it can be restored on the fly if those older players choose to log back in.

     

    Since 1am we spent a couple hours backing everything up, and then about 8 hours re-indexing. We are down to the last large table. It appears that it will take about 5 hours more and then we need about another hour to turn on servers and run through a quick QA test. That puts our earliest ETA for letting players back in at ~6pm Pacific time. With a little luck we should have things running much better for the weekend.

     

    I'll keep monitoring this and updating the community.

     

    EDIT: We hit a serious snag and ran out of drive space on the volume while doing the repair. The team is working around it, but we are going to have to extend the maintenance.

    EDIT: 5/10 1am Pacific. Nearly done restoring and adding more space for the repair. No ETA yet.

    EDIT: 5/10 3:20am Pacific. The server dumped. We've decided to scrap it and replace it due to ongoing issues with it. The team is working on replacing it now.

    EDIT: 5/10 12:00pm Pacific. New hardware is online. Reinstalling everything and then we'll start the database restore / repair process.

    EDIT: 5/10 3:20pm Pacific. Current estimate is 9:30pm for the restore to finish. Then the repair will start.

    EDIT: 5/11 3:30am Pacific. Still waiting on various parts to finish.

    EDIT: 5/11 12pm Pacific. Finishing one last task and then we'll be bringing servers online for QA internal testing.

     

    To be clear, backups are intact. No rollback is expected.

     

    Thanks,
    Matt

    • Like 10
    • Thanks 10

  22. Posted

    1 hour ago, grayfox_18 said:

    Since the server update, i am no longer able to log through steam.

    Still being stuck at splash screen telling me that i'm logged in and with entering my password i coud force a disconnect !

     

    Tried it several times, with a forced disconnect and relogging with no succes at all !

     

     

    Any help on this ?

     

    Thx

    Out of curiosity, did you create your account in April?

  23. Posted

    1 hour ago, ColeTrane said:

    i realize they are trying to change a bunch of things which making this game " better " i think its great and alot of this game is already awesome... i have a feeling they will ruin this game making it First Person only then this game will have the same demise as star wars galaxies... when they made it first person it ruined it and they killed the game for good.

     

    There is no intention to change the core mechanics to the game. Specifically we are not turning Fallen Earth into a first person game.

    Most of this effort is just to upgrade the tech so it is more supportable.

  24. Posted

    48 minutes ago, Kaylee Fryes said:

    Any chance will the achievements we had come back?  8.5 years and I was still working on them.

    Just wanted to make sure someone knew this was part of the loss.  

    Hmm. Did you lose all your achievements?

    According to my understanding, we only lost about 2 weeks of progression.

    So the large majority of your 8.5 year progress should be preserved.

×
×
  • Create New...