Game Server impacted for 30 minutes on December 30th

The server hamsters were asked to stop working for about 45 minutes while we negotiated with the server gerbils when they decided to go on strike. (The server gerbils transport the data updates to the database servers - because everyone knows that gerbils are too small to run the server itself)

Last week we experienced an issue with the Database backup that locked the DB against writes when the backup process failed to initialize correctly. We were able to clear that issue, restart the backup, and restart the game server in just a few minutes over an hour.

We had then planned to reboot the DB servers as part of preventative maintenance during the R122 release deployment to ensure all was clean. As it turns out, one of the DBservers in the replication set was not happy and errors started to appear in the logs. We executed a command to reset the secondary DB server and re-establish the links to the Primary DB (which we have done before with no impact to production), this time it impacted any players changing scenes.

Since this was the case, and we wanted to ensure that the DB remains completely stable, we opted to take the game server offline and perform the maintenance scheduled for the day of R122's deployment, at that time.

This work allowed us to reset all three DB servers and ensure the two secondary servers were completely re-synced before we brought the game server back online.

We will be monitoring the DB server logs to ensure that nothing unusual appears. Chris and I were successful in adding new drives to the servers at the DataCenter earlier in the week, which will allow us to move forward with server migrations (including the DB servers), which will increase our resiliency in 2024.

Apologies for interrupting game play; no gerbils (or hamsters) were harmed during this event.