[Burnett]

Hello Community,

this announcement is to inform you about a recent database corruption that took place on one of our database servers.

On April 23rd 2020 around 11:16 pm GMT+1 we received reports from players about inconsistencies of in-game data.
Meanwhile the entire forum was non-operative, causing it to display random characters.
We immediately started investigation into this matter at 11:19 pm and found that one of our databases crashed.
Further analysis lead to the conclusion that our backups system - responsible for offloading our backups - did not function the way we intended. 

On April 15th 2020 we reconfigured that system to ensure future backup reliability in response to the 
disastrous database corruption that took place at another community.
Instead of offloading our backups as it's being successfully done for almost 5 years, the system did not mount our storages.
This little but severe incident lead to overflowing our disk which eventually resulted in the loss of some of our data in the database.

Luckily we were able to obtain the last functional backup from about a week ago (15th April 2020), before the re-configuration of our backup system.

In addition, we've been keeping separate file-based logs on our disks that function as backups as well.
This allows allowed us to re-instantiate your data (except for meska metrics) to where it was before the incident happened.

---

Here is what you need to know:

+ We have backups - The most recent (functional) one was created on April 15th (one week ago)
+ We applied that backup within 5 minutes at 11:24 pm
+ We re-added bans, blacklists, warnings that were commited via ACP and Forums
+ We have additional file-based logs that function as backups
+ 329 affected players

---

Your member-status, inventory, money, reps:

We have separate file-based logs for member-status, inventory, money, reps, and others.
@Temar is currently working on re-importing has restored your stats.

Update 1:  Inventory + Money restored

Update 2: Game logs restored

Update 3: [font=Arial]Stats + Hours + Membership restored  (https://limelightgaming.net/forums/threa...#pid296981)[/font]

Update 4: Bans restored (https://limelightgaming.net/forums/threa...#pid296986)

[font=Arial]Update 5: Blacklists, Warnings, Reps restored[/font]

[font=Roboto, sans-serif]To quote Temars post:[/font]
[font=Roboto, sans-serif]Anything substantially wrong that you would like to be checked post a thread here:
https://limelightgaming.net/forums/forum-203.html[/font]

---

Please note that your stats will be were reset to its original state right before the incident happened!
This does not apply to your bans, blacklist, warnings and other punishments you received during/after the incident.
A server incident does not excempt players from breaking the rules.

---

Meska metrics:

Currently there is no way for us to restore the Meska metrics that were tracked after April 15th.
If we find a way we will let you know but best assume metrics collected within the last 8 days are not available.
This does not affect products obtained with a skill, for instance the fishing skill.
Collected fish is back in your inventory to where it was before the incident happened (see above).
You just don't have metrics on that fish, hence it's not counted towards your exp and level.

---

TL;DR:

- Parts of our database were corruped
- Backup from April 15th was applied
- We have file-based up-to-date logs 
- You will get got back your member-status, inventory, money, reps, and related stats.
- Meska metrics (for fishing/achievements) were set back to a week ago (April 15th)
- We are working on additional backup strategies (eg. re-implementing Fausties backup strategy, buying external servers solely for backups, keeping more file-based db dumps/logs)


We thank you for your patience and we are working hard to let this not affect your upcoming weekend.
Those incidents (Attacks, Server failure) always show how vital your support is. Thank you.

Limelight Gaming Security Team
Limelight Gaming Human Resources

---

Overview:

Restored  (April 15th - April 23rd):
+ All your stats
+ Logs, Bans, Blacklists & Co

Most likely not to be restored  (April 15th - April 23rd):
- Meska metrics collected during the above timespan

[Marvin Hampton]

Thank you for the quick response and update to this.

[Duckie]

Thanks guys. Glad ya'll were on top of it. Sorry that this happened.

[brave soldier]

Thank you.

[Temar]

Took about 5 hours but I managed to restore most data
your inventory and money should be pretty recent snap shots
missing playtime has been calculated and added
rep point list has been passed onto staff to review and re-add

Membership purchases during last week have been fixed, ingame money etc from them would of already been restored with money/inv restoring
 
there is most likely going some slight discrepancies
anything substantially wrong that you would like to be checked post a thread here:
https://limelightgaming.net/forums/forum-203.html

[Burnett]

+++ Updated the OP - @all please re-read

[Tom*]

Deadset burnett responded in 9 minutes to the initial report, anyway good job lads

[Mr.Sir]

Pretty damn quick response. Thanks!

[Burnett]

I think we can all appreciate the fact that Temar worked his ass off, writing multiple parsers/scripts to restore vital data such as player stats, logs, memberships, reps etc.
In addition a huge thank you to Doctor Internet who provided additional help in restoring the bans and blacklists.
There is still some data that we must manually restore, but your most vital data are back.

This incident will help us further navigate how we want to deal with BigData in the future.
Turns out having additional file-based logs (2nd class backups) are much more vital than I expected.

From here on now we must allocate more money into storage and enterprise backup solution, eg. replication/db syncs.


Edit:  Additional thanks to Bambo and the Mod/Admin team for dealing with the warnings and reps.

[Decay]

You guys are amazing!

[Zaidplays]

LL ON TOP NON-STOP

[Burnett]

Process completed.

Additional backup strategies have been put in place, 
including a physical first-class backup (Temar).

We'll try and allocate more resources to further maximize data reliability.

Thank you for your support.