Recently, I had a couple of uses for an illumos-based system that does not use ZFS for the root file system. The first time around, I was trying to test a ZFS code change and it is significantly easier to do when the root file system is not the same as the file system likely broken by my changes. The second time around, I did not want to deal with the ARC eating up all the RAM on the test system.
It is still possible to use UFS for the root file system, it just requires a bit of manual labor as illumos distros do not let you install directly to a UFS volume these days. I am using this post to write down the steps necessary to get a VM with UFS root file system up and running.
First of all, let’s talk about the virtual hardware necessary. We will need a NIC that we can dedicate to the VM so it can reach the internet. Second, we will need two disks. Here, I am using a VNIC and two zvols to accomplish this. Finally, we will need an ISO with OpenIndiana Hipster. I am using the ISO from October 2015.
Let’s create all the virtual hardware:
host# dladm create-vnic -l e1000g0 ufsroot0 host# zfs create storage/kvm/ufsroot host# zfs create -V 10g -s storage/kvm/ufsroot/root host# zfs create -V 10g -s storage/kvm/ufsroot/swap
The overall process will involve borrowing the swap disk to install Hipster on it normally (with ZFS as the root file system), and then we will copy everything over to the other disk where we will use UFS. Once we are up and running from the UFS disk, we will nuke the borrowed disk contents and set it up as a swap device. So, let’s fire up the VM. To make it easier to distinguish between the two disks, I am setting up the root disk as a virtio device while the other as an IDE device. (We will change the swap device to be a virtio device once we are ready to reclaim it for swap.)
host# /usr/bin/qemu-kvm \ -enable-kvm \ -vnc 0.0.0.0:42 \ -m 2048 \ -no-hpet \ -drive file=/dev/zvol/rdsk/storage/kvm/ufsroot/root,if=virtio,index=0 \ -drive file=/dev/zvol/rdsk/storage/kvm/ufsroot/swap,if=ide,index=1 \ -net nic,vlan=0,name=net0,model=virtio,macaddr=2:8:20:a:46:54 \ -net vnic,vlan=0,name=net0,ifname=ufsroot0,macaddr=2:8:20:a:46:54 \ -cdrom OI-hipster-text-20151003.iso -boot once=d \ -smp 2 \ -vga std \ -serial stdio
I am not going to go step by step through the installation. All I am going to say is that you should install it on the IDE disk. For me it shows up as c0d1. (c2t0d0 is the virtio disk.)
Once the system is installed, boot it. (From this point on, we do not need the ISO, so you can remove the -cdrom from the command line.) After Hipster boots, configure networking and ssh.
Now that we have a very boring stock Hipster install, we should at the very least update it to the latest packages (via pkg update). I am updating to “jeffix” which includes a number of goodies like Toomas Soome’s port of the FreeBSD loader to illumos. If you are using stock Hipster, you will have to figure out how to convince GRUB to do the right thing on your own.
ufsroot# pkg set-publisher --non-sticky openindiana.org ufsroot# pkg set-publisher -P -g http://pkg.31bits.net/jeffix/2015/ \ jeffix.31bits.net ufsroot# pkg update Packages to remove: 31 Packages to install: 14 Packages to update: 518 Mediators to change: 1 Create boot environment: Yes Create backup boot environment: No DOWNLOAD PKGS FILES XFER (MB) SPEED Completed 563/563 8785/8785 239.7/239.7 1.3M/s PHASE ITEMS Removing old actions 7292/7292 Installing new actions 5384/5384 Updating modified actions 10976/10976 Updating package state database Done Updating package cache 549/549 Updating image state Done Creating fast lookup database Done A clone of openindiana exists and has been updated and activated. On the next boot the Boot Environment openindiana-1 will be mounted on '/'. Reboot when ready to switch to this updated BE. --------------------------------------------------------------------------- NOTE: Please review release notes posted at: http://wiki.openindiana.org/display/oi/oi_hipster ---------------------------------------------------------------------------
Reboot into the new boot environment and double check that the update really updated everything it was supposed to.
ufsroot# uname -a SunOS ufsroot 5.11 jeffix-20160219T162922Z i86pc i386 i86pc Solaris
First, we need to partition the virtio disk. Let’s be fancy and use a GPT partition table. The easiest way to create one is to create a whole-disk zfs pool on the virtio disk and immediately destroy it.
ufsroot# zpool create temp-pool c2t0d0 ufsroot# zpool destroy temp-pool
This creates an (almost) empty GPT partition table. We need to add two partitions—one tiny partition for the boot block and one for the UFS file system.
ufsroot# format Searching for disks...done AVAILABLE DISK SELECTIONS: 0. c0d1 <QEMU HARDDISK=QM00002-QM00002-0001-10.00GB> /pci@0,0/pci-ide@1,1/ide@0/cmdk@1,0 1. c2t0d0 <Virtio-Block Device-0000-10.00GB> /pci@0,0/pci1af4,2@4/blkdev@0,0 Specify disk (enter its number): 1 selecting c2t0d0 No defect list found [disk formatted, no defect list found] ... format> partition
I like to align partitions to 1MB boundaries, that is why I specified 2048 as the starting sector. 1MB is plenty of space for the boot block, and it automatically makes the next partition 1MB aligned. It is important to specify “boot” for the partition id tag. Without it, we will end up getting an error when we try to install the loader’s boot block.
partition> 0 Part Tag Flag First Sector Size Last Sector 0 usr wm 256 9.99GB 20955102 Enter partition id tag[usr]: boot Enter partition permission flags[wm]: Enter new starting Sector: 2048 Enter partition size[20954847b, 20956894e, 10231mb, 9gb, 0tb]: 1m
Since I am planning on using a separate disk for swap, I am using the rest of this disk for the root partition.
partition> 1 Part Tag Flag First Sector Size Last Sector 1 unassigned wm 0 0 0 Enter partition id tag[usr]: root Enter partition permission flags[wm]: Enter new starting Sector: Enter partition size[0b, 4095e, 0mb, 0gb, 0tb]: 20955102e partition> print Current partition table (unnamed): Total disk sectors available: 20955069 + 16384 (reserved sectors) Part Tag Flag First Sector Size Last Sector 0 boot wm 2048 1.00MB 4095 1 root wm 4096 9.99GB 20955102 2 unassigned wm 0 0 0 3 unassigned wm 0 0 0 4 unassigned wm 0 0 0 5 unassigned wm 0 0 0 6 unassigned wm 0 0 0 8 reserved wm 20955103 8.00MB 20971486
When done, do not forget to run the label command:
partition> label Ready to label disk, continue? yes
Format and Copy
Now that we have the partitions all set up, we can start using them.
ufsroot# newfs /dev/rdsk/c2t0d0s1 newfs: construct a new file system /dev/rdsk/c2t0d0s1: (y/n)? y Warning: 34 sector(s) in last cylinder unallocated /dev/rdsk/c2t0d0s1: 20951006 sectors in 3410 cylinders of 48 tracks, 128 sectors 10230.0MB in 214 cyl groups (16 c/g, 48.00MB/g, 5824 i/g) super-block backups (for fsck -F ufs -o b=#) at: 32, 98464, 196896, 295328, 393760, 492192, 590624, 689056, 787488, 885920, 20055584, 20154016, 20252448, 20350880, 20449312, 20547744, 20646176, 20744608, 20843040, 20941472 ufsroot# mkdir /a ufsroot# mount /dev/dsk/c2t0d0s1 /a
To get a consistent snapshot of the current (ZFS) root, I just make a new boot environment. I could take a recursive ZFS snapshot manually, but why do that if beadm can do it for me automatically? (The warning about missing menu.lst is a consequence of me using jeffix which includes the FreeBSD loader. It can be safely ignored.)
ufsroot# beadm create tmp WARNING: menu.lst file /rpool/boot/menu.lst does not exist, generating a new menu.lst file Created successfully ufsroot# beadm mount tmp Mounted successfully on: '/tmp/tmp._haOBb'
Now, we need to copy everything to the UFS file system. I use rsync but all that matters is that the program you use will preserve permissions and can cope with all the file types found on the root file system.
In addition to copying the mounted boot environment, we want to copy /export. (Recall that /export is outside of the boot environment, and therefore it will not appear under the temporary mount.)
ufsroot# rsync -a /tmp/tmp._haOBb/ /a/ ufsroot# rsync -a /export/ /a/export/
At this point, we are done with the temporary boot environment. Let’s at least unmount it. We could destroy it too, but it does not matter since we will eventually throw away the entire ZFS root disk anyway.
ufsroot# beadm umount tmp
Before we can use our shiny new UFS root file system to boot the system, we need to do a couple of cleanups.
First, we need to nuke the ZFS zpool.cache:
ufsroot# rm /a/etc/zfs/zpool.cache
Second, we need to modify vfstab to indicate the root file system and comment out the zvol based swap device.
ufsroot# vim /a/etc/vfstab
So, we add this line:
/dev/dsk/c2t0d0s1 - / ufs - yes -
and either remove or comment out this line:
#/dev/zvol/dsk/rpool/swap - - swap - no -
Third, we need to update boot properties file (bootenv.rc) to tell the kernel where to find the root file system (i.e., the boot path) and the type of the root file system. To find the boot path, I like to use good ol’ ls:
ufsroot# ls -lh /dev/dsk/c2t0d0s1 lrwxrwxrwx 1 root root 46 Feb 21 17:43 /dev/dsk/c2t0d0s1 -> ../../devices/pci@0,0/pci1af4,2@4/blkdev@0,0:b
The symlink target is the boot path—well, you need to strip the fluff at the beginning.
So, we need to add these two lines to /a/boot/solaris/bootenv.rc:
setprop fstype 'ufs' setprop bootpath '/pci@0,0/pci1af4,2@4/blkdev@0,0:b'
Ok! Everything is configured properly; we have to rebuild the boot archive. This should result in /a/platform/i86pc/boot_archive getting updated and a /a/platform/i86pc/boot_archive.hash getting created.
ufsroot# bootadm update-archive -R /a ufsroot# ls -lh /a/platform/i86pc/ total 33M drwxr-xr-x 4 root sys 512 Feb 21 18:38 amd64 drwxr-xr-x 6 root root 512 Feb 21 17:44 archive_cache -rw-r--r-- 1 root root 33M Feb 21 18:37 boot_archive -rw-r--r-- 1 root root 41 Feb 21 18:37 boot_archive.hash drwxr-xr-x 10 root sys 512 Feb 21 18:00 kernel -rwxr-xr-x 1 root sys 44K Feb 21 18:00 multiboot drwxr-xr-x 4 root sys 512 Feb 21 17:44 ucode drwxr-xr-x 2 root root 512 Feb 21 18:04 updates
Installing Boot Blocks
We have one step left! We need to install the boot block to the boot partition on the new disk. That one simple-ish command. Note that the device we are giving is the UFS partition device. Further note that the installboot finds the boot partition automatically based on the “boot” partition id tag.
ufsroot# installboot -m /a/boot/pmbr /a/boot/gptzfsboot /dev/rdsk/c2t0d0s1 Updating master boot sector destroys existing boot managers (if any). continue (y/n)? y bootblock written for /dev/rdsk/c2t0d0s0, 214 sectors starting at 1 (abs 2049) stage1 written to slice 0 sector 0 (abs 2048) stage1 written to slice 1 sector 0 (abs 4096) stage1 written to master boot sector
If you are using GRUB instead, you will want to install GRUB on the disk…somehow.
Booting from UFS
Now we can shut down, change the swap disk’s type to virtio and boot back up.
host# /usr/bin/qemu-kvm \ -enable-kvm \ -vnc 0.0.0.0:42 \ -m 2048 \ -no-hpet \ -drive file=/dev/zvol/rdsk/storage/kvm/ufsroot/root,if=virtio,index=0 \ -drive file=/dev/zvol/rdsk/storage/kvm/ufsroot/swap,if=virtio,index=1 \ -net nic,vlan=0,name=net0,model=virtio,macaddr=2:8:20:a:46:54 \ -net vnic,vlan=0,name=net0,ifname=ufsroot0,macaddr=2:8:20:a:46:54 \ -smp 2 \ -vga std \ -serial stdio
Once the VM comes back up, we can add a swap device. The swap disk shows up for me as c3t0d0.
ufsroot# swap -a /dev/dsk/c3t0d0p0 operating system crash dump was previously disabled -- invoking dumpadm(1M) -d swap to select new dump device
We also need to add the description of the swap device to /etc/vfstab. So, fire up vim and add the following line:
/dev/dsk/c3t0d0p0 - - swap - no -
That’s it! Now you can bask in the glory that is UFS root!
ufsroot$ df -h Filesystem Size Used Avail Use% Mounted on /dev/dsk/c2t0d0s1 9.9G 2.9G 6.9G 30% / swap 1.5G 1016K 1.5G 1% /etc/svc/volatile swap 1.5G 4.0K 1.5G 1% /tmp swap 1.5G 52K 1.5G 1% /var/run ufsroot$ zfs list no datasets available
Unfortunately, pkg assumes that the root file system is ZFS. So, updating certain packages (anything that would normally create a new boot environment) will likely fail.