In an effort to reduce memory, traffic usage, and Garbage Collector allocations, we recently switched from using JSON to a binary serialization library called BinaryPack. For a few weeks, life was great, memory usage was down, GC wait time was down, CPU usage was down, until it all went down.
Turns out, the corrupted file caused the BinaryPack serialization library to immediately execute itself with a stack overflow exception, bypassing any exception handling and just exiting the process. Within minutes, our global DNS server fleet of close to a 100 servers was practically dead.
<sponge bob style: a few moments later>
Hey, I'm the author of BinaryPack! This was very interesting to read, though I'm sad to hear about the issue. I can say that the issue could've easily been avoided if you had contacted me about the issue though. I haven't updated the library in 3 years now and I didn't really claim it was production-ready (and I would've told you so).
https://bunny.net/blog/the-stack-overflow-of-death-dns-collapse/https://bunny.net/blog/the-stack-overflow-of-death-dns-collapse/#comment-5431379144
