NodeOS: How We Stopped Worrying about Disks and Started Booting from RAM

“Where are the keys?”, it's not just a question to ask yourself as you drunkenly fumble through your pockets while swaying outside your front door at 2am; it’s also an extremely poignant question to ask in the context of a VPN, proxy, or any other protocols where data is encrypted in transit.

No More Secrets (An Introduction to Our Diskless OS Stack)

The lackluster storage of private keys (and other secrets) by ostensible providers of privacy services pose a common target for would-be attackers looking to compromise a system, which puts sensitive information at risk of being leaked. Over the years we've had our own run in with this sort of problem and have taken steps to ensure that no secrets are stored on non-volatile storage (i.e. Hard disks, SSDs, Floppy Disks).

What Does This Mean?

As of 2022 (last year) 100% of our VPN node fleet now runs entirely in RAM. Not only that, but we’ve implemented features to actively detect evidence of hardware tampering.

What Happens When a Node Is Rebooted?

Once one of our nodes is turned off or rebooted, all data on this ephemeral “RAM disk” is wiped.

All of our VPN nodes run on bare-metal hardware, except for a small number of VMs that run in our own private infrastructure. This mitigates a wide variety of physical attacks on our infrastructure, primarily cold storage tampering or theft of hardware.

In regards to device tampering, we’re able to track both when and what hardware has been modified and pull potentially compromised VPN nodes out of service to be investigated.

As we've had our diskless OS stack out in the wild for well over a year now, we want to give you some insight on how we’ve implemented this and elaborate on how this technology works.

The Problem: Physical Access Attacks

On a normal bare-metal GNU/Linux server setup, a fresh install might go something like this:

1. Insert installer boot disk
2. Partition your disk with a boot and root partition
3. Install OS to disk
4. Install a bootloader to the disk

Now, let's say you wanted to set up a VPN on this machine, for example, WireGuard. There are many straightforward guides to do so but, for brevity, I'll pick the WireGuard Quick Start guide.

At the Key Generation portion of the document, we get this line:

$ wg genkey > privatekey

In operation, this private key is - along with the public key of the peer - used to asymmetrically encrypt VPN traffic.

In a scenario where an attacker gains physical access to this server, disk access could potentially provide a vector for retrieving this key and other secrets. Upon restarting the server, this attacker could then decrypt the traffic in transit, potentially unbeknownst to now-compromised users.

Disk encryption could be used to partially mitigate this sort of attack, but what if we can avoid ever committing secrets outside of RAM entirely? What if we can make the entire OS disappear on a reboot?

The Theory: Booting from RAM

OK, so we've covered the basics. Running our OS from RAM will help ensure all our data is ephemeral, but how do we do that?

Chances are, if you've *ever* run a modern UNIX clone or derivative, you're already doing this, or at least briefly. Take a look at this sample GRUB2 boot option for Linux.

menuentry "Loser's Unix" {
    set root=(hd0,2)
    linux /boot/vmlinuz # Our kernel
    initrd /boot/initrd.img # Our initial ramdisk
}

The file at /boot/initrd.img is the filesystem the kernel (itself located at /boot/vmlinuz) initially loads into on boot, or "Initial ramdisk". Inside this initrd.img is an archived root filesystem, containing the necessary binaries, scripts, and configuration.

From here we perform initial system config, mount filesystems, and change our root mount (specified by set root= in this example to a partition on a physical disk) to our full root filesystem to continue booting.

That’s cool, but, again, how does it get us booting into RAM?

Since we've already demonstrated we begin our boot process in RAM and that we can change the root filesystem we point to during our boot process, what's stopping us from just booting into a root filesystem that's stored in RAM?

Well, nothing.

The Solution: pivot_root

"Well, it works on my machine" – Ancient Klingon Proverb

The mechanism that enables both our ramdisk and virtually all modern GNU/Linux distributions to boot into their root (/) mounts is called pivot_root(2).

When the OS is on disk, booting into our root filesystem from the initramfs is straightforward, as we have a physical block device we can mount, and call pivot_root on. How does this work when we want to use a ramdisk?

zram, zstd, curl, tar, and a Little Bit of Duct Tape

In our example with Ubuntu, and specifying a boot script with initramfs-tools, we can mount whatever filesystem we want onto our new root directory using the boot= kernel command-line option, and the system will continue booting from that new root.

How do we do that? All we need is to set up our ramdisk and fetch the filesystem contents.

Here's a simplified boot script example, using zram, curl, and tar, that creates an 8GiB ext4 root partition:

#!/bin/sh
mountroot() {        
	# Set up our ramdisk with 8GiB of storage utilizing zstandard compression       # You'll want to make sure you have enough RAM for your root filesystem here!
    #
    zramctl -a zstd -f -s 8GiB
    mkfs.ext4 /dev/zram0
    	mount -t ext4 /dev/zram0 "${rootmnt}"
    
    # Download our rootfs
    #
    curl https://www69.linux-isos.biz/only_legitimate_files/rootfs.tar.gz -o /tmp/rootfs.tar.gz        
    curl https://www69.linux-isos.biz/only_legitimate_files/rootfs.tar.gz -o /tmp/rootfs.tar.gz.sig
    
    # Do some sort of signature verification, ideally
    #
    # gpgv2 ...
    # ????
    # Profit
    
    # Untar our rootfs
    tar -xzf /tmp/rootfs.tar.gz "$rootmnt"
    
    # Clean up
    rm -rf /tmp/rootfs.tar.gz
    rm -rf /tmp/rootfs.tar.gz.sig
}

Once you have a boot script that mounts a ramdisk, formats it, and extracts your rootfs onto your newly formatted ramdisk we can use update-initramfs or mkinitramfs to generate our initramfs. Other distributions will have similar mechanisms.

By placing this kernel, our newly created initial ramdisk, alongside a bootloader on to the boot partition on any disk, we can now boot into our ramdisk with the rootfs specified in the script.

In practice, you'd likely combine this technique with signature verification of your rootfs, and any other configuration/authentication required to download it. In addition to signature verification in the initial ramdisk, we also revert the bootloader configuration and remove our boot files from the disk.

Node Provisioning and Maintenance

Our root filesystem tarball is signed and verified, containing a minimal, almost blank root filesystem. On top of this, we deploy our software stack and dependencies manually, using a bespoke, state-of-the-art provisioning system.

Upon booting into our ramdisk rootfs, our nodes are in an unprovisioned state, containing an agent process that:

1. Starts a listener
2. Accepts a connection from our provisioning daemon
3. Waits on jobs sent by our provisioning daemon via signed messages.
4. Runs jobs to install our software and configurations onto the ephemeral disk

Previously, ansible had been used for this task, but over the years we realized we needed something easier to use and simpler to configure, and more responsive upon failures.

Device Trust

Early on in the article, we mentioned our ability to track physical hardware changes, and quickly pull compromised nodes out of service. This includes things such as but not limited to:

• Physical device ordering
• PCI devices
• USB devices (including HID)
• Various classes of storage devices

The information gathered by our agent on the node is sent regularly to our provisioning daemon, which allows us to act quickly upon detecting any potential device compromise, as well as ensure users do not connect to these nodes.

That’s Not All, Folks

We're very pleased to officially announce this milestone as part of our ongoing mission to keep your activity on the web private. We hope you enjoyed the deeper dive into the technical portions of this, just as much as we love keeping y’all safe.

We’re not done with this topic, though, and we’ll be covering our node provisioning stack in more detail in a later blog post, so stay tuned!