Josef “Jeff” Sipek

Installing Debian under FreeBSD's bhyve

This weekend I tried to move my work dev vm to a slightly beefier vm host. This meant moving the vm from kvm on OmniOS to bhyve on FreeBSD 12.1. After moving the disk images over, I tried to configure vm-bhyve to use them as-is. Unfortunately, grub2-bhyve fails to boot XFS root file systems with CRC support and there was no good way to convert the disk image to get it to boot via UEFI instead (to avoid grub2-bhyve completely). So, I decided to simply reinstall it.

In theory, this shouldn’t have been too difficult since I had the foresight to have /home on a separate virtual disk. In practice, I spent several hours reinstalling Debian Buster over and over, trying to figure out why it’d install and reboot fine, but subsequent boots wouldn’t make it past the EFI firmware.

It turns out that Debian tries to make multi-booting possible and puts its EFI binary into a non-standard location. That combined with bhyve not persisting EFI variables after shutdown results in any boot after the the first poweroff not even trying to look at the Debian-specific path.

This is not a Debian bug, but rather bhyve’s EFI support being incomplete. The easiest way around this is to copy the Debian binary into the standard location immediately after installation. In other words:

# cd /boot/efi/EFI
# mkdir BOOT
# cp debian/grubx64.efi BOOT/bootx64.efi

That’s all that’s needed. The downside to this is that the copy will not get automatically upgraded when grub gets an update.

For completeness, here are the relevant bits of the vm’s config:

loader="uefi"
graphics="yes"
cpu="6"
memory="1G"
network0_type="virtio-net"
network0_switch="public"
zfs_zvol_opts="volblocksize=4096"
disk0_dev="sparse-zvol"
disk0_type="virtio-blk"
disk0_name="disk0"
disk1_dev="sparse-zvol"
disk1_type="virtio-blk"
disk1_name="disk1"

2015-09-23

The Apple ISA — An interesting view of what Apple could aim for instruction set architecture-wise.

Internetová jazyková příručka — A reference book with grammar and dictionary detailing how to conjugate each Czech word.

Java is Magic: the Gathering (or Poker) and Haskell is Go (the game)

An Interview with Brian Kernighan (July 2000)

Booting a Raspberry Pi2, with u-boot and HYP enabled

The SmPL Grammar — Description of the grammar used by Coccinelle.

Netbooting Debian Squeeze — A link I had sitting around for a couple of years when I last set up a NFS-root netbooting Linux system.

Are there any 3 dimensional items wwe can’t print layer by layer — A humorous story about Wikipedia article: Fubini’s theorem and its relation to 3D printing.

The Diagnosis of Mistakes in Programmes on the EDSAC — In some ways, debugging hasn’t changed much since 1951.

Meili

Back in June I got myself a new laptop — Thinkpad T520. As always, it’s a solid design (yes, I know, Thinkpads aren’t what they used to be). The unfortunate news is that the hardware was just a bit too new to be supported well. It came with Windows 7, which of course knew how to deal with all the devices in the system.

Debian

I tried installing Debian, but anything but the latest testing snapshot didn’t recognize my Intel 82579LM ethernet chip. The latest development snapshot installed just fine, but when I tried to boot into the installed system, everything got stuck in the middle of the initramfs. Booting with init=/bin/bash got me a shell, but anything and everything I tried didn’t fix the problem in the end. I searched the bug tracker for similar issues — no luck. In a last ditch effort, I tried to ask #debian. In has been my experience in the past that this channel is useless, but I asked anyway. I got precisely zero responses.

OpenIndiana

Unlike Debian, OpenIndiana installed and booted just fine. Sadly, both my wifi (Intel Wifi Link 1000) and my wired ethernet were not supported. I ended up installing VirtualBox in Windows and OpenIndiana underneath it. It worked reasonably well. At the same time, I started pestering some of the Illumos developers that mentioned that they were working on an update to the e1000g driver — the driver for my wired network interface.

Yesterday, one of them updated the bug related to the driver update with binaries for people to try. Well, guess what? I’m writing this entry in Vim over ssh.

The driver install was a simple matter of overwriting the existing e1000g files, and then running update_drv -a -i ’"pci8086,1502"’ e1000g and then rebooting. (I could have used devfsadm instead, but I wanted to make sure things would come up on boot anyway.)

I still need to switch Xorg to the proprietary NVidia driver.

Windows

I’m pretty sure I’ll end up rebooting into Windows every so often anyway…if only to play Wikipedia article: Age of Mythology. :)

P.S. In case you haven’t guessed it yet, Meili is my laptop’s hostname.

Performance Co-Pilot: Part 1, Introduction

I decided to write a series of posts about one piece of software — Performance Co-Pilot, or PCP for short. (I am aware that the project webpage isn’t the greatest, but don’t let that make you ignore a great piece of software.)

What is PCP

First of all, what is PCP? Let me start by quoting a bit from the PCP webpage:

Performance Co-Pilot (PCP) provides a framework and services to support system-level performance monitoring and management. It presents a unifying abstraction for all of the performance data in a system, and many tools for interrogating, retrieving and processing that data.

If you ever administered a server, you know that there are times when you want to know answers to pretty simple questions: what’s the load average on server X, or what’s the current network utilization on server Y, or memory usage, number of processes, number of logged in users, disk bandwidth usage, filesystem usage, etc., etc. You may have written some scripts that try to answer those questions, or used some of the other monitoring frameworks out there (e.g., Ganglia, Cacti). PCP is rather unique in the way it handles monitoring.

PCP architecture

PCP has a three major parts:

PMDAs
PMDAs (short for Performance Metrics Domain Agents) are metric collectors. Each PMDA knows how to retrieve certain system information. For example, the Linux PMDA knows how to parse all the information in /proc, the PowerDNS PMDA knows how to get statistics from the PowerDNS daemons, the MySQL PMDA knows how to get stats from MySQL, etc. There are 35 different PMDAs installed that cover various often used daemons and system services — Apache, KVM, lmsensors, memcached, MySQL, Bind, PowerDNS, Postfix, Samba, Sendmail, VmWare, Zimbra, and more.
PMCD
PMCD (short for Performance Metric Collector Daemon) is responsible for answering client (see below) requests for metrics, and calls into the various PMDAs to retrieve the requested metrics.
client software
The various pieces of client software contact the PMCD (via TCP/IP) and request one or more metrics.

Why not use Ganglia/Cacti/other rrdtool-based software?

As is stated on the rrdtool website, rrdtool is a high performance data logging and graphing system for time series data. So, why not use it? First of all, rrdtool doesn’t collect data; it merely stores it for later retrieval or graphing. This means that if you want to collect data, you still need something to collect it with — a script that parses /proc files and hands the values to rrdtool for storage.

Cacti and Ganglia are just two examples of the many rrdtool-based system logging solutions. Personally, I find them to be a bit too heavy weight (why do I want PHP and MySQL on my logger server?), cumbersome to use (e.g., clunky web interfaces), and just not flexible enough.

Additionally, PCP supports Linux, Solaris, Mac OS X, and even Microsoft Windows. You can use the same tools to monitor your entire network!

Why not use SNMP?

SNMP is a protocol that exposes a hierarchy of variables that often include some performance metrics (e.g., the number of bytes and packets transfered over a network interface). If, however, you want MySQL statistics, you need to write your own stat extractor that feeds it into the SNMP daemon — not very hard, but it still takes some effort.

In this series of posts, I hope to demonstrate to you that PCP is superior to these solutions.

Installation

Debian is my distro of choice, so I’m going to make some Debian-centric assumptions.

On Debian, PCP is packaged so installing it is as easy as:

# aptitude install pcp pcp-gui

Distros that are Debian derivatives (e.g., Ubuntu) should have a package as well. Fedora, RHEL, and SLES have packages as well.

At the end of installation, the Debian package automatically starts the PMCD.

At this point, we only care about the Linux PMDA. By default, anyone that can make a TCP/IP connection to the host can read and write the metrics. (The PMCD has a couple of “metrics” that can be changed — they affect how PMCD behaves.) My suggestion is to set up an access control list. This is what the default config file (/etc/pmcd/pmcd.conf) looks like:

# Performance Metrics Domain Specifications
# 
# This file is automatically generated during the build
# Name  Id      IPC     IPC Params      File/Cmd
pmcd	2	dso	pmcd_init	/var/lib/pcp/pmdas/pmcd/pmda_pmcd.so
linux	60	dso	linux_init	/var/lib/pcp/pmdas/linux/pmda_linux.so
mmv	70	dso	mmv_init	/var/lib/pcp/pmdas/mmv/pmda_mmv.so

PMCD allows for a simple access control list in the config file (/etc/pmcd/pmcd.conf), and it would be a good idea to set one up to allow only specific hosts to access it. Here’s an example of what the new config file would look like:

# Performance Metrics Domain Specifications
# 
# This file is automatically generated during the build
# Name  Id      IPC     IPC Params      File/Cmd
pmcd	2	dso	pmcd_init	/var/lib/pcp/pmdas/pmcd/pmda_pmcd.so
linux	60	dso	linux_init	/var/lib/pcp/pmdas/linux/pmda_linux.so
mmv	70	dso	mmv_init	/var/lib/pcp/pmdas/mmv/pmda_mmv.so

[access]
allow localhost, 127.0.0.1 : all;
allow 10.1.*, 10.2.* : fetch;
allow home.josefsipek.net : all;
disallow * : all;

It allows reads and writes from localhost and home.josefsipek.net, and only reads from 10.1.* and 10.2.*. You’ll have to restart PCP for the change to take place:

# /etc/init.d/pcp restart

First Test

Now that we have PCP installed and running, let’s query it for the list of metrics. We’ll use the pminfo utility. By default it assumes that we want to query the localhost.

$ pminfo

If you want to query another host, you can specify its name or IP using the -h argument. For example,

$ pminfo -h example.com

On my system it spits out 851 different metrics. Here’s a pruned list that includes the more interesting ones:

kernel.all.nusers
kernel.all.load
kernel.all.cpu.user
kernel.all.cpu.sys
kernel.all.cpu.idle
kernel.percpu.cpu.user
mem.physmem
mem.freemem
network.interface.in.bytes
network.interface.out.bytes
disk.dev.read_bytes
disk.dev.write_bytes
filesys.capacity
filesys.used
proc.nprocs
proc.psinfo.cmd

We can use the -f argument to fetch the current value, and for simplicity, let’s specify the specific metric we are interested in:

$ pminfo -f kernel.all.nusers 

kernel.all.nusers
    value 2

So, at the moment, there are two users logged in. Now, let’s check the load average:

$ pminfo -f kernel.all.load 

kernel.all.load
    inst [1 or "1 minute"] value 0.0099999998
    inst [5 or "5 minute"] value 0.02
    inst [15 or "15 minute"] value 0

We can see that pminfo returned 3 values — called instances in PCP speak.

I should note that some of the metric values don’t seem to make much sense unless you already know what they represent. Have no fear! pminfo has two arguments that display a short (-t) or a long (-T) help text. For example, the following command will tell us what what kernel.all.load keeps track of.

pminfo -T kernel.all.load

You can use pminfo to fetch all the metrics you want, or you can use one of the other utilities which do some additional magic.

pmdumptext is one such utility. Suppose we want to watch how the disk utilization changes over time. Here’s the command:

$ pmdumptext -mu disk.dev.read_bytes
Time	disk.dev.read_bytes["sda"]	disk.dev.read_bytes["hda"]
none	Kbyte / second	Kbyte / second
Fri Apr  1 16:15:36	?	?
Fri Apr  1 16:15:37	0.000	0.000
Fri Apr  1 16:15:38	248.003	0.000
Fri Apr  1 16:15:39	1748.117	0.000
Fri Apr  1 16:15:40	543.985	0.000
Fri Apr  1 16:15:41	536.041	0.000
Fri Apr  1 16:15:42	331.968	0.000
Fri Apr  1 16:15:43	2299.862	0.000
Fri Apr  1 16:15:44	6112.226	0.000
Fri Apr  1 16:15:45	8187.725	0.000
Fri Apr  1 16:15:46	1212.029	0.000
Fri Apr  1 16:15:47	84.002	0.000
Fri Apr  1 16:15:48	143.990	0.000
Fri Apr  1 16:15:49	963.538	0.000
Fri Apr  1 16:15:50	64.003	0.000
Fri Apr  1 16:15:51	0.000	0.000
Fri Apr  1 16:15:52	0.000	0.000

The first column contains the date and time, the remaining column are the time-converted metrics/instances. That is, between each line of output, PCP calculated the average utilization and printed that out. The arguments (-m and -u) print the first two lines of the output — the name of the metric/instance, and the units. pmdumptext gets the units from the PMCD along with information whether or not it should rate convert the values (it will rate convert “bytes” to “bytes/sec”, but if there is a metric representing the current latency it will leave it alone).

You can change the amount of time between updates with the -t option. As before, you can use -h to specify the hostname. For example:

pmdumptext -mu -t 15sec -h example.com disk.dev.read_bytes

GUI

pmdumptext is great, but sometimes you want something more…graphical. Again, there’s a program for that. It’s called pmchart. It’s pretty self-explanatory, so I’ll just show you a screenshot of it in action:

pmchart in action

Next time, I’ll cover PCP’s logging facility.

Extracting RPMs and DEBs

Every so often I needed to extract a .deb package manually. Usually I ended up installing Midnight Commander, and used it to copy the contents out. This time around, I did some search, and found a straight forward description how to do it for .debs and .rpms.

RPM

rpm2cpio mypackage.rpm | cpio -vid

DEB

ar vx mypackage.deb
tar -xzvf data.tar.gz

or

ar p mypackage.deb data.tar.gz | tar zx

IPL This!

I finally managed to find few minutes (well, about 2.5 hours) to write down and test my first howto. It is about…*drumroll*…Installing Debian under Hercules.

For those of you who don’t know, Hercules is an emulator for the System/360, 370, 390, and zSeries IBM mainframes.

Have fun, and let me know if you have any suggestions how I could improve the howto.

Guilt: Taking over the world one repository at a time

It is really interesting how sometimes a bit of luck makes things happen. For example, little over 6 months ago, I wrote a few shell scripts, which called gq, to make my life a little easier. I worked on the for about a week, and then I decided I should share with the community. So I tagged the sources as version 0.10, and announced it on the git mailing list. One of the comments I got was about the fact that there is another project (completely unrelated) that had the name gq for a long time. Oh well, it was time for me to rename it. After some procrastination and hacking, new year rolled around, and I decided to release 6th version (v0.15), but this time it wouldn’t be gq anymore — instead I would call it Guilt. My post from January describes how it got the name. As with every version of gq, I announced Guilt v0.15. I could see that Guilt was getting way better, and so I felt even more motivated to hack on it. v0.16 came out. And then a very unexpected thing happened. I got two patches from a guy on the mailing list. Sweet! I applied them, and release v0.17. Shortly thereafter, during the Linux Storage and Filesystem (LSF) workshop in San Jose, I got a patch from Ted Ts’o (of the ext[234] fame). I couldn’t believe it, but it was true. I decided to release v0.19 the next day. At LSF, I met Brandon Philips, and we talked about Guilt. Rather shortly after LSF, he send me an email saying that he’ll try to get Guilt into Debian. :) Well, about a month ago, he succeeded.

As many of you may already know, I stick around a number of channels on OFTC’s IRC network, and it is rather interesting to see people try Guilt, or people talk about Guilt; generally suggesting that someone use it — and people do!

Anyway, I hope I didn’t bore everyone to death with my little tour of history behind Guilt.

Powered by blahgd