Cloud-based RAID 1 with rclone

I consider myself to be a data hoarder. It’s a debilitating illness that’s filled many hard drives and made some disk migrations Years ago, on a Windows 7 install, I happened to buy and add another hard drive to my PC. I roughly remember Windows asking to set it up for me, which I naively accepted. After months of experiencing Windows simply stopping for a minute or so, randomly (although a running instance of Plants vs Zombies worked, it was like a pomodoro timer), I eventually moved on to another computer.

A year or so later, when inspecting those hard disks again, I noticed that the new disk I’d bought was entirely in an extended partition, with all other partitions living in it (remember, MBR days). So, Windows was bugging out because it was seeking a terabyte of data every time it needed to find a partition.

What did I do with that data? I could have just copied the few files I’d maybe liked to save and formatted. Instead, I slowly copied over partition by partition onto an external hard drive (only thing I had), shrank the extended partition, and copied it back.
painfully and unnecessarily long. Although I’ve cut back on it quite a bit, mainly by being too lazy to download and catalog stuff, deleting folders is hard, and losing data is a hellish nightmare I wouldn’t wish on anyone.

All my passwords are at least 100, though usually around 500, characters, randomly generated. You might say it’s overkill, but it’s not like I remember them in the first place, so why not use what you’re paying for? To help me manage them, I store them in a KeePass database file.

Losing your passwords is known to be unwanted, and naturally, losing them in a freak data-loss catastrophe was a scenario that kept me up at night, flushing my disk buffers with sync and looking at SMART data. Worse yet, a friend’s SSD fell ill and let out its force de vie, the magic smoke powering it.

Keeping Copies #

Obviously, we want some redundancies on this file. Sadly, making a copy right next to it won’t save you from a neutrino glassing the inside of your PC case, or whatever causes hard drives and SSDs to fail.

The obvious method is to litter your .kdbx wherever you can, across as many devices as you can. The immediate downside to this happens, well, immediately when you forget where all your copies are, or when you add a new password. Maybe you should stop making accounts, the internet’s bad for you anyways.

The great thing about storing KeePass databases is that they’re encrypted already, so you don’t have to worry about keeping them too secure. In fact, you can use this to your advantage and register accounts for services like OneDrive, Dropbox, Google Drive, or even shady services.

Still, you have to manually copy over the new version of the file to every service. Dropbox and OneDrive have their own clients that automatically sync your files, but running them over the same folder might cause some funny side effects, like breaking the sync.

rclone As the Universal Client #

rclone is a command line tool for using cloud storage services. You can think of it as rsync, but for cloud. It supports a plethora of services, from generic (S)FTP to specific host providers.

You add all these remotes by running rclone config, and following the onscreen interactive instructions. For most services you’ll have to (or should, at least) register your own OAuth app, as it’ll probably be less limited than the generic one rclone has registered.

Also note that the name you give your remote is important and will be what you use to refer to the remote thereupon. Spaces are, interestingly enough, allowed, so if you name a remote google drive, then rsync ls 'google drive':/ will work.

Right, we’ve got a way to manually sync every service (and we can just check the config for a list of them). But rclone also supports a union mount of one or more remotes (also note that remotes can be local folders, like /media/storage/rclone/). You can set up a new remote of type union, and specify everything you want to merge. By default, the r/w policies won’t be ensuring data is written to every remote; for that you want to specify the all policy. More info on that here.

After you’ve got your remote union set up, you can rclone -i sync ./important.kdbx everything:/ and upload to everywhere at once.

In fact, rclone will let you mount your remotes, including the union. You have to specify some cache options, but it works – though how much you trust the mount is up to you. You can specify it as a systemd mount unit, or in your fstab if you don’t have systemdick.

Bonus: Syncing to Your Phone #

Although I wouldn’t recommend unlocking your database on your phone, as your phone should be considered a fully compromised bad actor (in fact, I’d suggest you isolate your wireless from the physical network), you can still keep a copy there.

rclone can serve a remote over (S)FTP, WebDAV, HTTP(S), and you can find a client (or write your own) that will download the files from your computer every so often. You can use something like avahi to autodiscover your IPs on the network.

This is bit of a hassle, so instead you can use something like Syncthing for the same thing. Yes, they’ve got a cringe anti-racism banner, but it’s probably just €1500 for alternative marketing that they spent. You run it on your computer, disable the NAT traversal and global discover features which are on by default, pair with your phone and select the folder you want to sync.

A more promising alternative is Librevault whose site is now a teaser for v1.0, but they had some software on there at some point.