put SQLite DB on a separate EBS volume

[cdepillabout]

We should probably make sure our SQLite DB is put on a separate EBS volume, so that when we redeploy our EC2 instance, we don't have to worry about our SQLite DB getting destroyed.

We might also want to put /nix on the EC2 instance on a separate EBS volume as well.

We probably also want to have occasional backups of the EBS volume with the SQLite DB.

[cdepillabout]

@jonascarpay pointed out that we also have the problem that the EC2 server gets created with only 4GB of space, so it very quickly runs out of space. (This may be partially solved by putting the SQLite DB and /nix on separate paritions.)

[cdepillabout]

I'm going to attempt this, since trying to build rustc on the AWS instance will almost immediately run out of space.

[cdepillabout]

I started looking into this, but I ran into some problems.

The main problem we're seeing with the current AMI is that the filesystem is too small, so /nix/store fills up really quickly. Then the binplz server becomes unusable. My original idea to work around this was to mount an EBS volume at /nix or /nix/store to hold all the derivation outputs.

There are a couple problems with this approach:

1. The AMI we produce already has a populated /nix directory. It is not possible to easily mount a blank filesystem on /nix or /nix/store, because all the running software on the machine lives in /nix.

2. It may be possible to do something hacky like mount the EBS volume to /mnt/tmp-nix, copy over everything from /nix to /mnt/tmp-nix, then re-mount the EBS volume to /nix. The problem with this approach is that on subsequent deploys with a new AMI, the EC2 instance will get destroyed and re-created, so you have a non-empty EBS volume that needs to be mounted on /nix, but you'd some how have to merge the contents (and Nix database state) of the new AMI's /nix directory with the current contents of the EBS volume.

   In theory there may be some way to do this, but in practice I couldn't find a way.

3. When running terraform apply, first the EC2 instance is brought up, and only after it has booted and been running does the EBS volume get attached to the instance. (I'm not sure exactly what happens when you restart the instance, and at what point you can see the EBS volume from the instance.)

   It doesn't appear there is any way to say to EC2, "just give me an instance where /nix is actually backed by an EBS volume". Since /nix is really critical during our boot process and for running of the system, what we're trying to do seems a little difficult.

I have a few ideas for a solution:

1. Try to resize the AMI or initial root disk to be much larger. Ideally the AMI would be only as small as necessary, but then when the machine booted it would have / be like 100 gb or 200 gb or something.

   I'm pretty sure that the Amazon Linux and Ubuntu AMIs provided by AWS are able to do this, but I don't know anything about this. I'd have to do some research here. I feel like I remember that @jonascarpay was saying that resizing the AMI means that the actual thing you're uploading to S3 needs to be 100 gb or 200 gb or whatever. I'm pretty sure this wouldn't work for us.

2. Mount an EBS volume (or fast instance volume) to /nix as an overlayfs: https://askubuntu.com/questions/1222052/overlayfs-to-an-existing-directory. That way /nix would effectively be really big, but we wouldn't have to worry about re-sizing the AMI.

The problem with these approaches is that when re-deploying a new AMI, we'd lose everything in the /nix/store of the previous EC2 instance. This isn't really a problem, since /nix/store was just acting as a cache, but it would be nice if this didn't happen. I'm imagining that we'd have nixbuild.net as a cache, as well as our actual S3 Nix cache, so they should be doing most of the work.

The problem with an overlayfs is that it might be slightly slower than an actual filesystem, but this is probably negligible compared to network latencies.