Investigate providing a Firecracker task driver option

[picatz]

It would be interesting to provide an option to enable running a Firecracker task driver since GCP supports nested virtualization.

Supporting Documentation

• https://github.com/firecracker-microvm/firecracker/blob/main/docs/dev-machine-setup.md#gcp
• https://cloud.google.com/compute/docs/instances/nested-virtualization/overview
• https://www.nomadproject.io/docs/drivers/external/firecracker-task-driver

[picatz]

Spent a little bit of time investigating this. Starting with the basics. Currently able to enable nested virtualization, install Firecracker, and setup basic network configuration manually for a single VM. Including an iptables rule to prevent metadata access from the VM. Also able to build a custom root filesystem, but haven't figured out the right kernel options to build vmlinuxcorrectly. It will panic when booting, for some reason. Can use the hello-vmlinux.bin from the tutorial though.

Requires updating the vm module to enable nested virtualization:

 advanced_machine_features {
    enable_nested_virtualization = true
  }

And configuring the Packer template to install Firecracker for the client image:

{
	"only": ["client"],
	"type": "shell",
	"scripts": [
		"scripts/install_firecracker.sh"
	]
},

Start of an install script with some other notes:

#!/bin/bash

set -ex

# Documentation:
# * https://github.com/firecracker-microvm/firecracker/blob/main/docs/getting-started.md
# * https://github.com/firecracker-microvm/firecracker/blob/main/docs/network-setup.md
# * https://github.com/firecracker-microvm/firecracker/blob/main/docs/rootfs-and-kernel-setup.md

# install KVM and any nessecary configuration
DEBIAN_FRONTEND=noninteractive sudo apt-get update
DEBIAN_FRONTEND=noninteractive sudo apt-get install qemu-kvm -y
# sudo setfacl -m u:root:rw /dev/kvm

# install firecracker
release_url="https://github.com/firecracker-microvm/firecracker/releases"
latest=$(basename $(curl -fsSLI -o /dev/null -w  %{url_effective} ${release_url}/latest))
arch=`uname -m`
curl -L ${release_url}/download/${latest}/firecracker-${latest}-${arch}.tgz | tar -xz
cd release-${latest}
sudo mv firecracker-${latest}-$(uname -m) /usr/local/bin/firecracker
sudo rm -f /tmp/firecracker.socket

# setup networking (for single VM)
# https://github.com/firecracker-microvm/firecracker/blob/main/docs/network-setup.md
INTERFACE="ens4"
sudo ip tuntap add tap0 mode tap
sudo ip addr add 172.16.0.1/24 dev tap0
sudo ip link set tap0 up
sudo sh -c "echo 1 > /proc/sys/net/ipv4/ip_forward"
sudo iptables -t nat -A POSTROUTING -o $INTERFACE -j MASQUERADE
sudo iptables -A FORWARD -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
sudo iptables -A FORWARD -i tap0 -o $INTERFACE -j ACCEPT

# prevent access to metadata API from VM
# sudo iptables --insert FORWARD 1 --in-interface tap0 --destination 169.254.169.254/32 --jump DROP

# NOTE: Before starting the guest, configure the network interface using Firecracker's API:
# sudo curl --unix-socket /tmp/firecracker.socket -i \
#   -X PUT 'http://localhost/network-interfaces/ens4' \
#   -H 'Accept: application/json' \
#   -H 'Content-Type: application/json' \
#   -d '{
#       "iface_id": "ens4",
#       "guest_mac": "AA:FC:00:00:00:01",
#       "host_dev_name": "tap0"
#     }'

# Once you have booted the guest, bring up networking within the guest:
# ip addr add 172.16.0.2/24 dev eth0
# ip link set eth0 up
# ip route add default via 172.16.0.1 dev eth0

# you can add a public DNS server to /etc/resolv.conf by adding a line like this:
# nameserver 8.8.8.8


####

# Start Firecracker VM

# 1. in first terminal:
# sudo rm -rf /tmp/firecracker.socket
# sudo firecracker --api-sock /tmp/firecracker.socket

# 2. in second terminal:
# mkidr -p /tmp/firecracker-imgage
# cd /tmp/firecracker-imgage
# arch=`uname -m`
# dest_kernel="hello-vmlinux.bin"
# dest_rootfs="hello-rootfs.ext4"
# image_bucket_url="https://s3.amazonaws.com/spec.ccfc.min/img"
# 
# if [ ${arch} = "x86_64" ]; then
#     kernel="${image_bucket_url}/quickstart_guide/x86_64/kernels/vmlinux.bin"
#     rootfs="${image_bucket_url}/hello/fsfiles/hello-rootfs.ext4"
# elif [ ${arch} = "aarch64" ]; then
#     kernel="${image_bucket_url}/quickstart_guide/aarch64/kernels/vmlinux.bin"
#     rootfs="${image_bucket_url}/aarch64/ubuntu_with_ssh/fsfiles/xenial.rootfs.ext4"
# else
#     echo "Cannot run firecracker on $arch architecture!"
#     exit 1
# fi
# 
# echo "Downloading $kernel..."
# curl -fsSL -o $dest_kernel $kernel
# 
# echo "Downloading $rootfs..."
# curl -fsSL -o $dest_rootfs $rootfs
# 
# echo "Saved kernel file to $dest_kernel and root block device to $dest_rootfs."
# 
# sudo su root
#
# arch=`uname -m`
# kernel_path=$(pwd)"/hello-vmlinux.bin"
# 
# if [ ${arch} = "x86_64" ]; then
#     curl --unix-socket /tmp/firecracker.socket -i \
#       -X PUT 'http://localhost/boot-source'   \
#       -H 'Accept: application/json'           \
#       -H 'Content-Type: application/json'     \
#       -d "{
#             \"kernel_image_path\": \"${kernel_path}\",
#             \"boot_args\": \"console=ttyS0 reboot=k panic=1 pci=off\"
#        }"
# elif [ ${arch} = "aarch64" ]; then
#     curl --unix-socket /tmp/firecracker.socket -i \
#       -X PUT 'http://localhost/boot-source'   \
#       -H 'Accept: application/json'           \
#       -H 'Content-Type: application/json'     \
#       -d "{
#             \"kernel_image_path\": \"${kernel_path}\",
#             \"boot_args\": \"keep_bootcon console=ttyS0 reboot=k panic=1 pci=off\"
#        }"
# else
#     echo "Cannot run firecracker on $arch architecture!"
#     exit 1
# fi
# 
# rootfs_path=$(pwd)"/hello-rootfs.ext4"
# curl --unix-socket /tmp/firecracker.socket -i \
#   -X PUT 'http://localhost/drives/rootfs' \
#   -H 'Accept: application/json'           \
#   -H 'Content-Type: application/json'     \
#   -d "{
#         \"drive_id\": \"rootfs\",
#         \"path_on_host\": \"${rootfs_path}\",
#         \"is_root_device\": true,
#         \"is_read_only\": false
#    }"
# 
# curl --unix-socket /tmp/firecracker.socket -i  \
#   -X PUT 'http://localhost/machine-config' \
#   -H 'Accept: application/json'            \
#   -H 'Content-Type: application/json'      \
#   -d '{
#       "vcpu_count": 2,
#       "mem_size_mib": 1024,
#       "ht_enabled": false
#   }'
# 
# curl --unix-socket /tmp/firecracker.socket -i \
#   -X PUT 'http://localhost/network-interfaces/ens4' \
#   -H 'Accept: application/json' \
#   -H 'Content-Type: application/json' \
#   -d '{
#       "iface_id": "ens4",
#       "guest_mac": "AA:FC:00:00:00:01",
#       "host_dev_name": "tap0"
#     }'
# 
# curl --unix-socket /tmp/firecracker.socket -i \
#   -X PUT 'http://localhost/actions'       \
#   -H  'Accept: application/json'          \
#   -H  'Content-Type: application/json'    \
#   -d '{
#       "action_type": "InstanceStart"
#    }'
# 

###

#!/bin/bash

# https://github.com/firecracker-microvm/firecracker/blob/main/docs/rootfs-and-kernel-setup.md

set -ex

# install some deps
sudo apt-get install libncurses-dev -y
sudo apt-get install bison -y
sudo apt-get install flex -y
sudo apt-get install libssl-dev -y

# clone linux using v4.20
git clone https://github.com/torvalds/linux.git linux.git
cd linux.git
git checkout v4.20

# interactive configuration
make menuconfig

# to fix broken x509 cert stuff (probs not great?)
sed -ri '/CONFIG_SYSTEM_TRUSTED_KEYS/s/=.+/=""/g' .config

# build kernel image
make vmlinux

# https://github.com/firecracker-microvm/firecracker/blob/main/docs/rootfs-and-kernel-setup.md#creating-a-rootfs-image

# build rootfs
dd if=/dev/zero of=rootfs.ext4 bs=1M count=5000
mkfs.ext4 rootfs.ext4
mkdir /tmp/my-rootfs
sudo mount rootfs.ext4 /tmp/my-rootfs

# start the Alpine container, bind-mounting the EXT4 image created earlier, to /my-rootfs
sudo docker run -it --rm -v /tmp/my-rootfs:/my-rootfs alpine

# exit docker when done copying stuff into rootfs from linked tutorial

# Start VM

# 1. in first shell
# sudo m -f /tmp/firecracker.socket
# sudo firecracker --api-sock /tmp/firecracker.socket

# 2. in second shell (in the path with build vmlinux kernel image)
# kernel_path=$(pwd)"/vmlinux"
# sudo curl --unix-socket /tmp/firecracker.socket -i \
#       -X PUT 'http://localhost/boot-source'   \
#       -H 'Accept: application/json'           \
#       -H 'Content-Type: application/json'     \
#       -d "{
#             \"kernel_image_path\": \"${kernel_path}\",
#             \"boot_args\": \"console=ttyS0 reboot=k panic=1 pci=off\"
#        }"
#
# rootfs_path=$(pwd)"/rootfs.ext4"
# sudo curl --unix-socket /tmp/firecracker.socket -i \
#   -X PUT 'http://localhost/drives/rootfs' \
#   -H 'Accept: application/json'           \
#   -H 'Content-Type: application/json'     \
#   -d "{
#         \"drive_id\": \"rootfs\",
#         \"path_on_host\": \"${rootfs_path}\",
#         \"is_root_device\": true,
#         \"is_read_only\": false
#    }"
#
# sudo curl --unix-socket /tmp/firecracker.socket -i  \
#   -X PUT 'http://localhost/machine-config' \
#   -H 'Accept: application/json'            \
#   -H 'Content-Type: application/json'      \
#   -d '{
#       "vcpu_count": 2,
#       "mem_size_mib": 1024,
#       "ht_enabled": false
#   }'
#
# sudo curl --unix-socket /tmp/firecracker.socket -i \
#   -X PUT 'http://localhost/network-interfaces/ens4' \
#   -H 'Accept: application/json' \
#   -H 'Content-Type: application/json' \
#   -d '{
#       "iface_id": "ens4",
#       "guest_mac": "AA:FC:00:00:00:01",
#       "host_dev_name": "tap0"
#     }'
#
# sudo curl --unix-socket /tmp/firecracker.socket -i \
#  -X PUT 'http://localhost/actions'       \
#  -H  'Accept: application/json'          \
#  -H  'Content-Type: application/json'    \
#  -d '{
#      "action_type": "InstanceStart"
#   }'