Who runs out of disk space these days
Late last night, one of my Rust servers ran out of disk space. Putting aside that this should have never happened to begin with, and that I’m bad at monitoring my infrastructure, I woke up with over 12 hours of player data not being saved to disk. Rust typically saves every 5-20 minutes, depending on your personal preference and configuration. This meant if I restarted the servers, or the server crashed, players would lose 12 hours worth of work.
As soon as I noticed, I went on the system to cleaned up a bunch of old backup files and other cruft that resulted in the disk filling up, and then issued a server.save to manually save before something bad happened. Only, it didn’t work. This lead me to dig around the decompiled source for the Rust server to figure out why.
I could go into a lot more detail, going over the coroutines used in saving, but to sum it up the offending bit of code here is
if (IsSaving)
{
return false;
}
When I ran out of disk space, Rust didn’t handle that error gracefully. The automated coroutine that regularly saves, aptly named SaveRegularly, threw an exception and stopped working. It did not reset the IsSaving flag back to false, leaving the ability to perform even manual saves impossible. So, I’m screwed right?
Oxide, Harmony, and black magic..
Those of you following along might yell out to write a simple Oxide plugin to set the flag back to false. Unfortunately, this is my non-modded, Oxide-free server. Alternatively, if I had planned ahead for such occasions, I could have had a Harmoy mod ready in waiting to do such a thing. Also no luck. Both installing Oxide, or new Harmony mods requires restarting the server. Which is precisely what I can’t have happen.
What I’m left with is attempting to modify the running application to get that flag reset to false.
Enter DLL injection. A technique very common among cheaters, or more specifically cheat tool developers, you can by a variety of methods force a running process to load a dynamic library, and initialize it. All I needed to do is find a DLL injector for Mono games, and have it do one simple thing:
SaveRestore.IsSaving = false;
That’s it. After a bit of browsing game hacking forums and thoroughly washing my hands afterwards, I found a ready made DLL injector for Mono games called SharpMonoInjector.
Now I just needed a DLL. Quickly opened Rider, and a few minutes later:
using UnityEngine;
namespace SaveFixer
{
public class Loader
{
public static void Init()
{
DebugEx.Log("Fixing isSaving flag, currently set to " + SaveRestore.IsSaving);
SaveRestore.IsSaving = false;
DebugEx.Log("isSaving reset to false, enjoy.");
}
}
}
Crisis Adverted
Now we can simply run the injector, find our RustDedicated.exe process, fill out the namespace, class, and init method we want to call when it’s injected.
..and success. We can see the log output running our injected DLL’s code and fixing the IsSaving flag. We can run server.save again! Players won’t hunt me down with pitchforks.