Debrief: Server Crashes 5th December

You may have noticed that the server was starting and stopping constantly earlier today.

So what happened?

There was a tile entity in the mining world that was causing the server to crash when it was started. Our monitoring services picked this up and kept trying to restart the server.

Fixing the error

The erroring tile entity was removed from the server once it was found.

Forestry blocks, what happened to them?

In the process of trying to find, and fix, the cause of the server crash a configuration option was enabled to allow the server to replace erroring tile entities automatically, this did not work however.
I am of the opinion that this option replaced the forestry blocks

Please generate new terrain to find new forestry blocks

But wheres your backups?

Due to the recent server move, there has been some issues with the backups due to using a different backup storage location. We are currently in the process of sorting these issues out.

What have we learnt from this?

  1. Our monitoring service works to restart the server when it is down
  2. We need to have a way to notify the relevant people when the monitors fail to restart the server
  3. The config option will never be used again
  4. We need to focus on getting backups working properly again.