Devlog Migration/Attack Progress Update

Hey Everyone, update on the migration and attack situation.

Since the migration we've had to cram most of our infrastructure into one machine; this has (obviously) put a lot of strain on that machine. We'll be getting access to our old hardware again soon, and that should help plenty. Once that happens we'll move everything towards a clustered setup. This setup will be more performant as well as more fault tolerant.

In the meantime, we've been experiencing API instability, causing some servers to not be able to register and generally making "weird things" happen. We've been as frustrated by this as you guys.

To fix this we tried a few things, including moving the API out from behind the DDoS protection in case the extra latency was the issue. This resulted in the API being hit again by a DDoS attack, and being moved back behind the protection so that people could continue to play the game. Whoever is behind the attack is out for blood, and we'll have to deal with the extra latency for the time being.

However, on looking deeper (special thanks to Joe here), database performance has been particularly bad with everything crammed onto one machine. This has been drastically optimised, with a lot of old code and configurations getting a makeover, and a lot more transient data sitting in memory instead of having to hit the disk. The disk was also getting periodically "choked" by builds getting run as well as we try to get the build infrastructure ready for production again. The periodic spike in activity was interfering with other services, and properly testing the build system has been put on hold til we have more hardware to work with.

The result of this optimisation is that the THD forums, Soldat forums, and API should all load a lot faster! Stability should also be improved - at least compared to the last few weeks, if not even better than before. Let us know if you notice any issues, and please provide as much information as possible when reporting anything.

Thanks so much to you the community for riding with us through this,
and to our lovely sysadmins for keeping us in the loop and taking an axe to the skeletons in our closet,
Max, Michal & Everyone at THD

 

@Geti, thanks for this update.

 

I'll improve this later to actually include service disrupted if response times are to slow, and timeline graph, but for now here's something for antsy players if they need a quick answer for if kag services are really down. http://verra.xyz/thdstatus/

 

12/10 many thanks for that site Verra. Will look at putting it in the news feed maybe but if thd.vg goes down then so does it anyway, haha.

 

Rather than starting another thread - yep we're aware auth is down at the moment, working on getting it back up asap. Thanks to @Verrazano for the ping about it.

E: fixed! Big thanks to @Furai and @jrgp as usual.