VyOS on an Asus Chromebox M004U

August 04, 2016

This is a bit of a follow up from my last post. I’ve still got my Asus Chromebox. I’m not running ESXi any more… but something I do want to try out is VyOS. Why? Well, my original intent when purchasing the Chromebox was to play around with it, but then I really did expect it to replace my home router. I’ve used pfSense for a long time, but I’m curious about VyOS because I know several people using EdgeRouter Lite which uses EdgeOS, a fork of VyOS. So I wanted to play around on existing hardware.

Note: everything below assumes your Chromebox has custom firmware for installing any OS you choose (see last post).

Turns out though, VyOS isn’t quite ready for installing on UEFI systems. And the Chromebox is… well… sort of UEFI, even though it uses SeaBIOS. Also, while most UEFI systems have a management screen where you can configure legacy mode, the Chromebox is a bit more limited in how we can work around its oddities.

Bottom line, VyOS installer would not boot directly on the Chromebox. I wasn’t thinking “UEFI is the problem” when I first hit the boot issue; I assumed it was an issue with USB 3. After much poking and prodding, I was able to get it running. But it was tricky enough, I figured I should document the process.

Install VyOS

First things first, you’ll need another system, a BIOS system, on which you can install VyOS. This doesn’t have to be a physical computer. VyOS is very often used in virtualized environments. In my case, all initial testing was done on a physical BIOS machine, but for proving out a second round of tests, and to aid in documentation for this blog, I did another install using a VMware machine for my starting system.

I used both the 1.1.7 stable and 1.2.0-beta1 releases for my testing, but always 64-bit ISO installations. I didn’t use any pre-made images.

Using VMware (etc), it’s super easy to install. I used a VM with 1GB of RAM and 10GB of hard disk. I removed most extra devices, as we don’t need them, but I did keep the USB controller. Note: 10GB hard drive was chosen to be sure it is smaller than the 16GB SSD in the Chromebox; it may not matter in the long run but makes me feel safer.

Once you have a VM created, and the ISO downloaded, boot and follow the VyOS install guide.

Quick summary:

boot
login as vyos
run install image
use default config
set password for vyos user to something
auto partition disk sda using whole 10GB of disk
install grub to sda
DONE

Reboot to make sure it boots. Huzzah!

SystemRescueCD

We aren’t going to clone in a traditional sense, no Ghost or Clonezilla… nope. We’re just going to take the bits we need. First, we need a rescue disk to boot from. I’m a long time fan of SystemRescueCD as it has lots and lots of useful tools, can cache itself 100% into RAM, and has a decent X Windows setup available if you need it.

So, download the latest ISO of SystemRescueCD. Before we go any further, make a bootable USB thumb drive from this. The easiest way is to be running Linux already, mount the ISO image, and run the usb_inst.sh script. On Windows you can try using Rufus, which I use when on Windows. On OSX, well… dd might work.

OK, set the SystemRescueUSB aside for now…

Copy the Disk

Now we are ready to clone the disk. Configure the ISO image as your bootable drive in your VyOS VM, then boot it up. You may need to be quick on the “ESC” key to get into the boot menu. The default SystemRescueCD boot option should be fine.

Once you are sitting at a shell prompt, insert that SystemRescueUSB drive you made. You’ll want to make sure it’s connected to the VM, not your host system.

root@sysresccd /root % fdisk -l /dev/sdb
Disk /dev/sdb: 15 GiB, 16079781888 bytes, 31405824 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x0002f767

Device Boot Start End Sectors Size Id Type
/dev/sdb1 * 1 31405823 31405823 15G c W95 FAT32 (LBA)

It’s connected, so mount it up, and create a place to store our VyOS stuff

mkdir /mnt/usb && \
mount /dev/sdb1 /mnt/usb && \
mkdir /mnt/usb/VYOS && \
cd /mnt/usb && \
ls
boot bootprog ntpasswd syslinux sysrcd.md5 usb_inst.sh version
bootdisk efi readme.txt sysrcd.dat usb_inst usbstick.htm VYOS

Now, copy out the MBR, which holds both the partition table and the bootloader.

dd if=/dev/sda of=/mnt/usb/VYOS/mbr.img bs=512 count=1
1+0 records in
1+0 records out
512 bytes (512 B) copied, 0.00109355 s, 468 kB/s

Sweet… now make a copy of the filesystem.

mkdir /mnt/disk && \
mount /dev/sda1 /mnt/disk && \
cd /mnt/disk && \
tar -zcf /mnt/usb/VYOS/1.1.7/rootfs.tar.gz boot && \
ls -lh /mnt/usb/VYOS 
total 233M
-rwxr-xr-x 1 root root 512 Aug 4 04:47 mbr.img
-rwxr-xr-x 1 root root 232M Aug 4 04:50 rootfs.tar.gz

Excellent! sync for good measure, then unmount and remove the USB drive.

sync && umount /mnt/usb

We are done with the VM for now, but it doesn’t hurt to leave it around in case you want to inspect something.

Paste the Disk

Now, lets pull out the physical Chromebox.

Insert the SystemRescueUSB drive and power it up. Again the the default SystemRescueCD boot option is fine, but if you choose the second (docache), you’ll need to manually mount the USB drive. Using the default, our USB drive is now auto-mounted to /livemnt/boot .

We can verify, since a stock SystemRescueCD won’t have our VYOS stuff.

ls -lh /livemnt/boot/VYOS
total 233M
-rw-r--r-- 1 root root 512 Aug 4 04:47 mbr.img
-rw-r--r-- 1 root root 232M Aug 4 04:50 rootfs.tar.gz

ALERT: NEXT STEP WILL ERASE EVERYTHING ON YOUR CHROMEBOX SSD!

First, nuke existing partition maps. You’ll have to tell it you want a DOS disk label, then arrow over to “Write” to save the partition map.

cfdisk -z /dev/sda

Restore the VM’s MBR.

dd if=/livemnt/boot/VYOS/mbr.img of=/dev/sda bs=512 count=1
1+0 records in
1+0 records out
512 bytes (512 B) copied, 0.00135381 s, 378 kB/s
fdisk -l /dev/sda
Disk /dev/sda: 14.9 GiB, 16013942784 bytes, 31277232 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x000ce790

Device Boot Start End Sectors Size Id Type
/dev/sda1 * 2048 20971519 20969472 10G 83 Linux

Create a filesystem on this partition. Note: I had some trouble with SystemRescueCD’s ext4 fs utils creating a version of ext4 that was too new for the older utils/kernel of VyOS, so I went simple here and used ext3.

mkfs.ext3 -j /dev/sda1
mke2fs 1.42.13 (17-May-2015)
/dev/sda1 contains a file system
Proceed anyway? (y,n) y
Discarding device blocks: done 
Creating filesystem with 2621184 4k blocks and 655360 inodes
Filesystem UUID: d2e311fc-bf9e-4f40-b88a-a9047c969255
Superblock backups stored on blocks: 
 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632

Allocating group tables: done 
Writing inode tables: done 
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done

Lay down our copy of the VM filesystem.

mkdir /mnt/disk && \
mount /dev/sda1 /mnt/disk && \
cd /mnt/disk && \
tar zxf /livemnt/boot/VYOS/rootfs.tar.gz && \
ls
boot lost+found

So far this has been pretty simple way to copy any Linux file system and partition map. Now its time to do the magic required to get this to boot on the Chromebox.

If you were to reboot right now, you’d get a lovely grub rescue> console, and you wouldn’t be able to boot VyOS.

Convert to GPT Magic

First, unmount that disk again, so we are able to work with it.

cd /root && umount /mnt/disk

Use gdisk to write GPT version of the MBR data we currently have. Choose “1” to use MBR as authoritative ( you MAY not be asked this ), then “w” to write the new GPT data.

gdisk /dev/sda
GPT fdisk (gdisk) version 1.0.1

Partition table scan:
 MBR: MBR only
 BSD: not present
 APM: not present
 GPT: present

Found valid MBR and GPT. Which do you want to use?
 1 - MBR
 2 - GPT
 3 - Create blank GPT

Your answer: 1

Command (? for help): w

Final checks complete. About to write GPT data. THIS WILL OVERWRITE EXISTING
PARTITIONS!!

Do you want to proceed? (Y/N): y
OK; writing new GUID partition table (GPT) to /dev/sda.
The operation has completed successfully.

Now we need to add an ESP (EFI System Partition) and BIOS compatibility boot/grub partition.

parted /dev/sda unit MiB print
Model: ATA SanDisk SSD U110 (scsi)
Disk /dev/sda: 15272MiB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:

Number Start End Size File system Name Flags
 1 1.00MiB 10240MiB 10239MiB ext3 Linux filesystem

This shows us where we can start adding partitions, 10239MiB. So we add to next mega byte location the ESP, then one more for our BIOS boot part.

parted /dev/sda mkpart primary fat32 10240MiB 10241MiB 
Information: You may need to update /etc/fstab.

parted /dev/sda mkpart primary fat32 10241MiB 10441MiB 
Information: You may need to update /etc/fstab.

parted /dev/sda set 2 bios_grub on set 2 msftdata off 
Information: You may need to update /etc/fstab.

parted /dev/sda set 3 boot on set 3 msftdata off set 3 esp on
Information: You may need to update /etc/fstab.

parted /dev/sda unit MiB print 
Model: ATA SanDisk SSD U110 (scsi)
Disk /dev/sda: 15272MiB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags: 

Number Start End Size File system Name Flags
 1 1.00MiB  10240MiB 10239MiB ext3  Linux filesystem
 2 10240MiB 10241MiB 1.00MiB        primary    bios_grub
 3 10241MiB 10442MiB 201MiB   fat16 primary    boot, esp

mkfs.vfat /dev/sda2
mkfs.fat 3.0.28 (2015-05-16)

mkfs.vfat /dev/sda3
mkfs.fat 3.0.28 (2015-05-16)

Install Grub

Now we need to unsquash the filesystem image used by VyOS so we can re-install the right version of grub.

mount /dev/sda1 /mnt/disk && \
cd /mnt/disk && \
unsquashfs boot/1.1.7/1.1.7.squashfs 
Parallel unsquashfs: Using 2 processors
32494 inodes (34238 blocks) to write

[===============================================================================================================================-] 34238/34238 100%

created 30398 files
created 9824 directories
created 1457 symlinks
created 37 devices
created 1 fifos

ls squashfs-root
bin boot dev etc home lib lib64 media mnt opt proc root sbin selinux srv sys tmp usr var

Let’s bind some system dirs and install grub (the version from VyOS, not SystemRescueCD)!

mount -o bind /dev /mnt/disk/squashfs-root/dev
mount -o bind /proc /mnt/disk/squashfs-root/proc
mount -o bind /mnt/disk/boot /mnt/disk/squashfs-root/boot 
chroot /mnt/disk/squashfs-root /bin/bash
# grub-install /dev/sda
Installation finished. No error reported.

PHEW! That wasn’t so hard, was it? 🙂

Let’s cleanup the mounts, reboot, and see what happens.

umount squashfs-root/boot
umount squashfs-root/proc
umount squashfs-root/dev 
cd .. && umount disk
reboot

SUCCESS!

Don’t forget you’ll need to remove the USB drive before rebooting.

When booting the Chromebox I now get a VyOS boot and login.