Apr 24, 2020, 05:02 AM
Hello Community,
this announcement is to inform you about a recent database corruption that took place on one of our database servers.
On April 23rd 2020 around 11:16 pm GMT+1 we received reports from players about inconsistencies of in-game data.
Meanwhile the entire forum was non-operative, causing it to display random characters.
We immediately started investigation into this matter at 11:19 pm and found that one of our databases crashed.
Further analysis lead to the conclusion that our backups system - responsible for offloading our backups - did not function the way we intended.
On April 15th 2020 we reconfigured that system to ensure future backup reliability in response to the
disastrous database corruption that took place at another community.
Instead of offloading our backups as it's being successfully done for almost 5 years, the system did not mount our storages.
This little but severe incident lead to overflowing our disk which eventually resulted in the loss of some of our data in the database.
Luckily we were able to obtain the last functional backup from about a week ago (15th April 2020), before the re-configuration of our backup system.
In addition, we've been keeping separate file-based logs on our disks that function as backups as well.
This allows allowed us to re-instantiate your data (except for meska metrics) to where it was before the incident happened.
Here is what you need to know:
+ We have backups - The most recent (functional) one was created on April 15th (one week ago)
+ We applied that backup within 5 minutes at 11:24 pm
+ We re-added bans, blacklists, warnings that were commited via ACP and Forums
+ We have additional file-based logs that function as backups
+ 329 affected players
Your member-status, inventory, money, reps:
We have separate file-based logs for member-status, inventory, money, reps, and others.
is currently working on re-importing has restored your stats.
Update 1: Inventory + Money restored
Update 2: Game logs restored
Update 3: [font=Arial]Stats + Hours + Membership restored (https://limelightgaming.net/forums/threa...#pid296981)[/font]
Update 4: Bans restored (https://limelightgaming.net/forums/threa...#pid296986)
[font=Arial]Update 5: Blacklists, Warnings, Reps restored[/font]
[font=Roboto, sans-serif]To quote Temars post:[/font]
[font=Roboto, sans-serif]Anything substantially wrong that you would like to be checked post a thread here:
https://limelightgaming.net/forums/forum-203.html[/font]
Please note that your stats will be were reset to its original state right before the incident happened!
This does not apply to your bans, blacklist, warnings and other punishments you received during/after the incident.
A server incident does not excempt players from breaking the rules.
Meska metrics:
Currently there is no way for us to restore the Meska metrics that were tracked after April 15th.
If we find a way we will let you know but best assume metrics collected within the last 8 days are not available.
This does not affect products obtained with a skill, for instance the fishing skill.
Collected fish is back in your inventory to where it was before the incident happened (see above).
You just don't have metrics on that fish, hence it's not counted towards your exp and level.
TL;DR:
- Parts of our database were corruped
- Backup from April 15th was applied
- We have file-based up-to-date logs
- You will get got back your member-status, inventory, money, reps, and related stats.
- Meska metrics (for fishing/achievements) were set back to a week ago (April 15th)
- We are working on additional backup strategies (eg. re-implementing Fausties backup strategy, buying external servers solely for backups, keeping more file-based db dumps/logs)
We thank you for your patience and we are working hard to let this not affect your upcoming weekend.
Those incidents (Attacks, Server failure) always show how vital your support is. Thank you.
Limelight Gaming Security Team
Limelight Gaming Human Resources
Overview:
Restored (April 15th - April 23rd):
+ All your stats
+ Logs, Bans, Blacklists & Co
Most likely not to be restored (April 15th - April 23rd):
- Meska metrics collected during the above timespan
this announcement is to inform you about a recent database corruption that took place on one of our database servers.
On April 23rd 2020 around 11:16 pm GMT+1 we received reports from players about inconsistencies of in-game data.
Meanwhile the entire forum was non-operative, causing it to display random characters.
We immediately started investigation into this matter at 11:19 pm and found that one of our databases crashed.
Further analysis lead to the conclusion that our backups system - responsible for offloading our backups - did not function the way we intended.
On April 15th 2020 we reconfigured that system to ensure future backup reliability in response to the
disastrous database corruption that took place at another community.
Instead of offloading our backups as it's being successfully done for almost 5 years, the system did not mount our storages.
This little but severe incident lead to overflowing our disk which eventually resulted in the loss of some of our data in the database.
Luckily we were able to obtain the last functional backup from about a week ago (15th April 2020), before the re-configuration of our backup system.
In addition, we've been keeping separate file-based logs on our disks that function as backups as well.
This allows allowed us to re-instantiate your data (except for meska metrics) to where it was before the incident happened.
Here is what you need to know:
+ We have backups - The most recent (functional) one was created on April 15th (one week ago)
+ We applied that backup within 5 minutes at 11:24 pm
+ We re-added bans, blacklists, warnings that were commited via ACP and Forums
+ We have additional file-based logs that function as backups
+ 329 affected players
Your member-status, inventory, money, reps:
We have separate file-based logs for member-status, inventory, money, reps, and others.
Update 1: Inventory + Money restored
Update 2: Game logs restored
Update 3: [font=Arial]Stats + Hours + Membership restored (https://limelightgaming.net/forums/threa...#pid296981)[/font]
Update 4: Bans restored (https://limelightgaming.net/forums/threa...#pid296986)
[font=Arial]Update 5: Blacklists, Warnings, Reps restored[/font]
[font=Roboto, sans-serif]To quote Temars post:[/font]
[font=Roboto, sans-serif]Anything substantially wrong that you would like to be checked post a thread here:
https://limelightgaming.net/forums/forum-203.html[/font]
Please note that your stats will be were reset to its original state right before the incident happened!
This does not apply to your bans, blacklist, warnings and other punishments you received during/after the incident.
A server incident does not excempt players from breaking the rules.
Meska metrics:
Currently there is no way for us to restore the Meska metrics that were tracked after April 15th.
If we find a way we will let you know but best assume metrics collected within the last 8 days are not available.
This does not affect products obtained with a skill, for instance the fishing skill.
Collected fish is back in your inventory to where it was before the incident happened (see above).
You just don't have metrics on that fish, hence it's not counted towards your exp and level.
TL;DR:
- Parts of our database were corruped
- Backup from April 15th was applied
- We have file-based up-to-date logs
- You will get got back your member-status, inventory, money, reps, and related stats.
- Meska metrics (for fishing/achievements) were set back to a week ago (April 15th)
- We are working on additional backup strategies (eg. re-implementing Fausties backup strategy, buying external servers solely for backups, keeping more file-based db dumps/logs)
We thank you for your patience and we are working hard to let this not affect your upcoming weekend.
Those incidents (Attacks, Server failure) always show how vital your support is. Thank you.
Limelight Gaming Security Team
Limelight Gaming Human Resources
Overview:
Restored (April 15th - April 23rd):
+ All your stats
+ Logs, Bans, Blacklists & Co
Most likely not to be restored (April 15th - April 23rd):
- Meska metrics collected during the above timespan