[Burnett]

Hello community, on Monday 11th of November 2019 - 11am MESZ (9am UTC),
we will be upgrading our entire infrastructure. 
If you have been with us for some years you might be aware of our yearly maintenance procedure.

Actions included in this maintenance are vital in order to keep safe
our services and your data.

Scheduled actions:


:: Full upgrade of our server infrastructure (DS1(EU)- Gameservers)
   We'll be updating various software including the operating system.
   [font=Verdana]Our Teamspeak, UCP and several services that rely on DS1 (Gameservers) won't be available.[/font]
   Expected downtime:  6-10 hours
   [font=Verdana]Alternative services:  Discord, Forums (You can join the forums while gameservers are offline.)  COMPLETED[/font]

[font=Verdana]:: Full upgrade of our server infrastructure (VPS 1(Web))
   We'll be updating various software including the operating system.[/font]
[font=Verdana]   [font=Verdana]Expected downtime:  4-8 hours[/font][/font]
[font=Verdana][font=Verdana]   [font=Verdana]Alternative services:  Discord, Gameservers [font=Verdana](You can join the gameservers while Webserver is offline.) [b][font=Verdana][font=Verdana]COMPLETED[/font][/font][/b][/font][/font][/font][/font]

[font=Verdana][font=Verdana]:: Full maintenance of our Forums
   We'll be updating various software.[/font][/font]
[font=Verdana][font=Verdana]   [font=Verdana]Expected downtime:  30 minutes[/font][/font][/font]
[font=Verdana][font=Verdana][font=Verdana]   Alternative services:  Discord[font=Verdana][font=Verdana][font=Verdana][font=Verdana], Gameservers [font=Verdana](You can join the gameservers while forum is offline.) [b][font=Verdana][font=Verdana]COMPLETED[/font][/font][/b][/font][/font][/font][/font][/font][/font][/font][/font]

:: Full maintenance of our developer services ([font=Verdana]Git Migration)[/font]
   [font=Verdana]Expected downtime:  1 hour[/font]
[font=Verdana]   [font=Verdana]Alternative services:  None  [font=Verdana]COMPLETED[/font][/font][/font]



Best regards,
STAFF

[Soviethooves]

You gonna need any gear for that? Home Depot's got a killer clearance on wrench sets right now.

[Judge Rage]

You gonna need any gear for that? Home Depot's got a killer clearance on wrench sets right now.

Think we might need to go to B&Q, that's gonna be some hella shipping

[Burnett]

// Reminder

[Doctor Internet]

A reminder that some other services will be down for the entire of the maintenance period.

Minimum Effected List

• OOC -> discord relay (cross server chat should still work).
  
• Interbot.
  
• Discord Auth.
  
• Boost4Boost sync (new / removed boosts, current boosts still work)
  
• Loading page cache clearing.
  
• Inventory value cron updating.
  
• Teacher applications service (internal posts may be delayed, new posts can still be made.)
  
• Suggestions management systems (new suggestions can be posted, but won't be moved for staff review).

[Burnett]

Current status update:

Our secondary network interface is not responding and we are working on fixing this issue.

[Burnett]

Update:

The issue was fixed and we are now transferring our gameserver backups to the server.

[Burnett]

Update:

+ Teamspeak online     (Edit:  Offline again - maintenance)
+ Dev FTP online
+ Databases online

[Burnett]

The gamerservers are back online. 

Not 10 hours, but after 18 hours you can finally play again.

We have upgraded the DS1 to a brand new operating system, and that caused some (un)expected problems.
Our primary NIC (Networking Interface) went completely offline for hours, after the system refused to bring up one of its Failover IPs. 
Re-configuraton was required - while accessing a rescue system - to gain back control over our pirmary NIC.

Later, the new database server was unable to read our file backups so we had to re-configure that one too.

Nonetheless it was worth it and we are now better prepared for the remaining upgrades.


Thank you for your patience.


Please note:  

Maintenance is still in progress for other areas (check op above). Gameservers however are not affected by this.

[Burnett]

Tomorrow, Nov 15th we will conclude the maintenance by running the
Full upgrade of VPS 1(Web) (see above).

This upgrade shouldn't take long due to the experience gained on upgrading the much bigger server (D1).

[Burnett]

Maintenance completed.

That was a very tough week and we are glad things are back in order.
A fully detailed report on this maintenance will be released.

Happy Weekend.

[Doctor Internet]

*almost* complete, there's still a few minor tasks.

A couple of migrations over to new systems are going to be ran. Bug reports might be affected, and the LL API may go down.
The suggestions system is still down because it decided to crash instead of working.

We're on the home straight.

[Doctor Internet]

Bug reports are up and running fully.

The API is up in a limited capacity, and we're now on endpoint version 3. If one of the endpoints you use in your application isn't available, please let me know and we'll try and get them back up as soon as possible.