5/9/2019 Maintenance Update

[Sang 66]

While I am heartbroken about missing the wastelands I am very relieved we are lucky enough to have someone that keeps us in the loop and works so diligently on bringing it back.

[Warg86 38]

Im glad to see, LO doing much better work than G1 ever did, respect.

[CommanderDeacon 4]

1 hour ago, kittykat bigpaws said:

How  'The Kitty' fixes  servers. ^.^

 

 

One big bowl of kitty food and i will help you ^.^

we all need a little help from kitty

in respect to G1 they were hardcore into the mechanics and really wanted to pump more content and thus more money.  

 

[CommanderDeacon 4]

did ya'll tell them to remove patch 2.0?

 

at this point I know this new version is going to be fricken sweet and I cant wait to run up junk fortress and swoop around for lost creeper DNA.

[ZenRoyal 47]

4 hours ago, CommanderDeacon said:

this game just really needed new tech im gonna say that I would never be unruly or unfaithful. I understand I can be rude and ignorant. it is my own complete stupidity by my own design. a lot of times I just have to be quiet and just smile and like w/e. u guys do the work as I just consume, like a growing bacteria I will consume. 

Such as the way of human nature, friend. Same reason it's taught as consumer economics.

 

-Figment

[CommanderDeacon 4]

36 minutes ago, TotallyNotKodaro said:

Such as the way of human nature, friend. Same reason it's taught as consumer economics.

 

-Figment 

Indeed and thank you.

[FakyouArtemiSix 9]

less (crappy) philosophy and more FE updates pls

Edited May 10, 2019 by zefcool

[MattScott 15245]

5 hours ago, zefcool said:

you mean a new server (with 2019 hardware) with a copy of the old database of the crappy server, or is it the same shity server with maxed "dirty tricks" skill used by engineers to make it stable somehow ?

We can't change the software OS, the size or structure of the structure of the database. There is no way to avoid some of the "dirty tricks" to keep the game online till we re-write the backend.

However we are running on new hardware across the board.

 

The issue we are solving right now relates to the problematic database server (used for pulling all character data).

Originally we got everything restored, but clearly players were having trouble retrieving characters on login.

 

At this point, we have determined that this specific database is so large (and poorly structured) that we can no longer simply restore it. Loading that much data results in unusable table indexes which generates the problems we were seeing before we took it down. It takes 6 hours each attempt to restore and then 7 hours each attempt to repair. We have tried 3 different ways to get the database setup, and all have yielded the same result.

 

We are shifting tactics and moving to a different set of expanded hardware. Then we'll give it a 4th attempt.

Everyone is working the weekend to get this back online asap.

 

Thanks,
Matt

[FakyouArtemiSix 9]

17 minutes ago, MattScott said:

We are shifting tactics and moving to a different set of expanded hardware. Then we'll give it a 4th attempt.

Everyone is working the weekend to get this back online asap.

thanks for the update and for the usage of the weekend "group tactics" skill in addition to "dirty tricks" one

 

Edited May 10, 2019 by zefcool

[Rubbermade 36]

25 minutes ago, MattScott said:

We can't change the software OS, the size or structure of the structure of the database. There is no way to avoid some of the "dirty tricks" to keep the game online till we re-write the backend.

However we are running on new hardware across the board.

 

The issue we are solving right now relates to the problematic database server (used for pulling all character data).

Originally we got everything restored, but clearly players were having trouble retrieving characters on login.

 

At this point, we have determined that this specific database is so large (and poorly structured) that we can no longer simply restore it. Loading that much data results in unusable table indexes which generates the problems we were seeing before we took it down. It takes 6 hours each attempt to restore and then 7 hours each attempt to repair. We have tried 3 different ways to get the database setup, and all have yielded the same result.

 

We are shifting tactics and moving to a different set of expanded hardware. Then we'll give it a 4th attempt.

Everyone is working the weekend to get this back online asap.

 

Thanks,
Matt

Thank you for the details it helps ease the waiting knowing the issues involved

[Kristia Velzo 3]

Thanks, good luck ! I can't wait to play again after all those months 

[Killenkittens 3]

So take it game won't be up today probly

[ColeTrane 2]

4 days of not playing and there is no other game at this moment id rather play.... 

[Wypsnchanz 0]

Personally I'd like to see the game shut down for a time (a month?) to allow the LO team to work on issues without having to deal with all this pressure.  I also think they should wipe the current (problematic) character database and have everyone start fresh and perhaps do a new beta test period.  Once the game is "live" again, current players who spent real money can be reimbursed (in-game) based on their transaction history.  Not only will this serve to get rid of what is essentially a massive amount of "junk" data (much of which is probably corrupt), it'll free up a lot of character names for players to use.

Just my opinion.

[MattScott 15245]

36 minutes ago, Wypsnchanz said:

Personally I'd like to see the game shut down for a time (a month?) to allow the LO team to work on issues without having to deal with all this pressure.  I also think they should wipe the current (problematic) character database and have everyone start fresh and perhaps do a new beta test period.  Once the game is "live" again, current players who spent real money can be reimbursed (in-game) based on their transaction history.  Not only will this serve to get rid of what is essentially a massive amount of "junk" data (much of which is probably corrupt), it'll free up a lot of character names for players to use.

Just my opinion.

I can appreciate your position, but I've already gone on record to say we are not wiping data.

We can and will get the server back up. The requirements are just a little tricky to nail down for something like this.

Having a server up for years collecting and indexing data over time is a different spec from starting a server empty and loading it with years of data.

 

Once we get stabilized then we have a different way of moving old data out of the active tables dynamically without wiping anything.

This just requires some coding and testing in our PTS environment, which we don't have time for right now.

 

Thanks,
Matt

[Rubbermade 36]

45 minutes ago, Wypsnchanz said:

Personally I'd like to see the game shut down for a time (a month?) to allow the LO team to work on issues without having to deal with all this pressure.  I also think they should wipe the current (problematic) character database and have everyone start fresh and perhaps do a new beta test period.  Once the game is "live" again, current players who spent real money can be reimbursed (in-game) based on their transaction history.  Not only will this serve to get rid of what is essentially a massive amount of "junk" data (much of which is probably corrupt), it'll free up a lot of character names for players to use.

Just my opinion.

as a player of 8 years and 7 accounts that would cause me to quit the game forever . and i log into each often

8 minutes ago, MattScott said:

I can appreciate your position, but I've already gone on record to say we are not wiping data.

We can and will get the server back up. The requirements are just a little tricky to nail down for something like this.

Having a server up for years collecting and indexing data over time is a different spec from starting a server empty and loading it with years of data.

 

Once we get stabilized then we have a different way of moving old data out of the active tables dynamically without wiping anything.

This just requires some coding and testing in our PTS environment, which we don't have time for right now.

 

Thanks,
Matt

thanks Matt

[Rubbermade 36]

if you need pts testers a few of us older players have the test server login info installed from gt testing

 

Edited May 10, 2019 by Rubbermade
spelling

[theemole 13]

I had to switch laptops, so count me out

[robbed 0]

Thanks matt for the continued updates

[Rubbermade 36]

1 minute ago, theemole said:

I had to switch laptops, so count me out

not even sure the link i have still works its been years since it was open for build testing

 

[theemole 13]

I had it for dome testing

[Sang 66]

I'm also willing to any testing if needed, I was there for Alpha and closed/open beta so definitely have the patience for it. 

 

[Sooney 7]

Much apreciated yours and your team work, Matt.

Let me guess and idea...maybe, even if not dinamically yet, you might need to backup the database, leaving online only the last year's characters data (enough to be handle by the diagnostics you are doing). If someone from far beyond that time need his/her character can use a ticket, at least untill you clean up the rest of the database and make it dinamically at last.

Only keep in mind that you have to acces all records when naming a character, maybe leaving only names table and cleaning the rest would be enough... otherwise this will be repeating once and over again... such mass of data... i feel it is unbareable to make good maintenance of that.

Anyway, you have my full support. Hope next time fixing is a charm. If not, Matt, slice it!   

 

[Grelko 33]

23 minutes ago, Sang said:

I'm also willing to any testing if needed, I was there for Alpha and closed/open beta so definitely have the patience for it. 

 

Count me in if needed also, got about 20 years of Hobby bug testing. Couple games I've helped,

 

Vantage - A Prehistoric Simulation
Neo Scavenger
Cataclysm DDA
Nodiatis
A Drow's Life (Also wrote a walkthrough for this)
Atomiclick

 

Even made a couple small games of my own, but they were on RPGmaker for Playstation.

 

 

Edit - If you removed all the characters that haven't been logged in, in the last year or 2, that would free up quite a bit of space. You may want to keep the characters that people made purchases on, because some people come back every so often just to pvp etc.

Edited May 10, 2019 by Grelko

[FakyouArtemiSix 9]

1 hour ago, MattScott said:

Once we get stabilized then we have a different way of moving old data out of the active tables dynamically without wiping anything.

This just requires some coding and testing in our PTS environment, which we don't have time for right now.

i'm looking forward to this, the sooner the better, that should also make the daily maintenance duration shorter right ? and should give you the time to rewrite the backend to finally replace that old piece of junk you bought from G1 at the next scrap facility.

 

Edited May 10, 2019 by zefcool