Josef “Jeff” Sipek

Working @ Dovecot

It’s been a hectic couple of weeks, and so this post is a bit delayed. Oh well.

A couple of months ago, I decided that it was time for me to move on work-wise. As a result, four weeks ago, I joined Dovecot Oy (a part of Open-Xchange).

As you may have guessed from the name of the company, I get to spend my time making the Dovecot email server code better, more featureful, and otherwise more excellent. It is certainly a significant but fun change—going from kernel hacking on a fairly unknown operating system to hacking on the world’s most popular IMAP server. Not a day goes by where I’m not surprised just how much functionality is in the Dovecot codebase, or when I get to consult an RFC related to some IMAP extension I didn’t even know existed.

So, with this said, you should expect to see some posts related to Dovecot, Dovecot code, and email in general.

Grub Composite Console

In the past, I’ve described how to get a serial console going on Illumos based systems. If you ever used a serial console in Grub (regardless of the OS you ended up booting), you probably know that telling Grub to output to a serial port causes the VGA console to become totally useless — it’s blank.

Well, if you are using Illumos, you are in luck. About 5 months ago, Joyent integrated a “composite console” in Grub. You can read the full description in the bug report/feature request. The short version is: all grub output can be sent to both the VGA console as well as over a serial port.

It is very easy to configure. In your menu.lst, change the terminal to composite. For example, this comes from my test box’s config file (omitting the uninteresting bits):

serial --unit=0 --speed=115200
terminal composite

Note the use of composite instead of serial. That’s all there is to it.

Task Spooler

For a couple of years now, I wished that I could have a mini-batch system on my computers that’d let me submit jobs and they’d execute when the resources became available. This would let me queue up large amount of work and it’d eventually all get processed. I even tried to hack up a dumb little Python script that’d loop over a file executing no more than one per core.

Then, yesterday, I stumbled across Task Spooler. It’s exactly what I was looking for! It lets me queue jobs, supports dependencies between jobs, etc.

I’m hoping to experiment with it in the next couple of days. I’ll let you know how it turns out.

Netflix Chaos Monkey

Somehow, I managed to miss that about two years ago Netflix open sourced their chaos monkey.

Based on my quick look over the code, it appears to be written in Java. Meh. Regardless of the language, it’s great to see large companies open source their code.

Greetings from Nexenta

In case you missed it, back in mid-2011 I discovered Illumos and OpenIndiana. At that point, I already missed hacking on the (Linux) kernel. Based on my blahg posts [1,2], it shouldn’t surprise you that it didn’t take long before I wanted to hack on the Illumos kernel…and so I did.

If you ever contributed to an open source project in your free time while employed full-time, you understand that there’s only so much time you can devote to the open source project and therefore there is only so much you can do.

A couple of months ago, I decided to explore the possibility of working full-time on Illumos. There are only a handful of companies that visibly participate in the Illumos ecosystem, but their use of Illumos is pretty varied (from public clouds to virtualized databases to SAN/NAS appliances). As of this past Tuesday (Monday was a holiday), I’m at Nexenta. At least for now, I’m working remotely (from Ann Arbor) with the fine folks in the Wikipedia article: Lowell office. It feels great to work on open source again.

MongoDB - First Impressions

I just ssh’ed into a server that’s running MongoDB to see if it was doing anything and if there were any cool commands to get stats from it. (I’ve never used MongoDB.) I accidentally ran mongod as non-root. It died because it didn’t have the right permissions — good. However, something in the output seemed completely and utterly retarded.

warning: 32-bit servers don't have journaling enabled by default. Please use
	--journal if you want durability.

[initandlisten] MongoDB starting : pid=12682 port=27017 dbpath=/data/db/
	32-bit host=hkn2012
[initandlisten] ** NOTE: when using MongoDB 32 bit, you are limited to about
	2 gigabytes of data
[initandlisten] **       see
[initandlisten] **       with --journal, the limit is lower

Was it the 2 GB data limit? Nope! Was it the fact that 32-bit systems default to non-durable mode while 64-bit default to durable? Bingo! Seriously, that is retarded. Why would anyone think that it is a good idea to cripple 32-bit systems like that? Was performance so bad with journaling that to avoid gaining a reputation for being slow the developers decided to keep the data unsafe by default? I doubt it. Was it to squeeze in a couple more MB of addressable data? Probably. Good job, folks.

For what it is worth, I decided to see if the 2 GB limit was something silly (i.e., using an int in some on-disk structure). It turns out that mongod likes to mmap the data for fast access. This is reasonable. (More so than using architecture specific integral types in on-disk structures.)

I wonder what other gems are hiding in this software.

OpenIndiana The What and Why

You have seen me publish two posts about OpenIndiana, but neither of them really says what it is and why you should use it.

The What

OpenIndiana started off as a fork of OpenSolaris. At first, its aim was to provide an alternative to Oracle’s soon-to-be-released Solaris 11, but lately its aim shifted to “an enterprise-quality OS alternative to Linux.”

OpenIndiana is much like a distro in the Linux world. It relies on the Illumos project for the kernel and basic userspace utilities (the shell, etc.). In September 2010, Illumos forked the OpenSolaris kernel and utilities, and OpenIndiana forked the surrounding userspace (the build system for all the packages that make the system usable).

The Why

It is the technology that is the reason I started using OI. Here are some of the features that either drew me in to try OI, or made me stay.

Crossbow was the name of the project that consisted of a major revamp of the network stack. With this revamp (which was available in OpenSolaris), you can create virtual network interfaces, vlans, bridges, switches (called etherstubs), as well as aggregate links with simple commands — quickly, and all the configuration is persistent. You can dedicate both physical and virtual links to zones (see below) to create entire network topologies within one computer. (see dladm(1M) and ipdam(1M))
These days, everyone is happily setting up virtual machines whenever they need an environment they can tweak without affecting stability of other services. Solaris zones are a great virtualization technology. They allow you to set up multiple Solaris instances (called zones) that have a separate root filesystem (much like chroot). Unlike chrooted environments, having root access in a zone does not give you unrestricted access to the kernel. Zones combined with crossbow is a great combination to consolidate separate systems onto a single Solaris host. (I am currently writing a post about using zones and crossbow on a home server/router.)
Boot Environments (BE) & IPS
Long story short, if the package manager (IPS) detects that a potentially major change is going to occur during an update (e.g., a driver or kernel upgrade), it clones the current root filesystem (easy to do thanks to ZFS) and applies the updates there. It then adds a menu entry to grub to boot into this new environment. The current environment is unchanged. At your leisure, you just reboot into the new environment. If everything works — great. If, however, things break, you can just reboot into the previous BE, and mount the new BE’s root and fix things up. This means that the only downtime the system sees is the reboot or two.
There’s plenty of ZFS discussion elsewhere. My favorite features about it are (in no particular order): snapshots, deduplication, integrated volume management, and checksumming.

So there you have it. Sure, many of Solaris’s features are available in some shape or form on Linux, but they tend to be either horribly crippled, or if you are “lucky,” lacking sane management interface.

If you want to see what all this fuss is about, I suggest you grab the Live DVD (or Live USB) image on the download page and give it a try.


Back in June I got myself a new laptop — Thinkpad T520. As always, it’s a solid design (yes, I know, Thinkpads aren’t what they used to be). The unfortunate news is that the hardware was just a bit too new to be supported well. It came with Windows 7, which of course knew how to deal with all the devices in the system.


I tried installing Debian, but anything but the latest testing snapshot didn’t recognize my Intel 82579LM ethernet chip. The latest development snapshot installed just fine, but when I tried to boot into the installed system, everything got stuck in the middle of the initramfs. Booting with init=/bin/bash got me a shell, but anything and everything I tried didn’t fix the problem in the end. I searched the bug tracker for similar issues — no luck. In a last ditch effort, I tried to ask #debian. In has been my experience in the past that this channel is useless, but I asked anyway. I got precisely zero responses.


Unlike Debian, OpenIndiana installed and booted just fine. Sadly, both my wifi (Intel Wifi Link 1000) and my wired ethernet were not supported. I ended up installing VirtualBox in Windows and OpenIndiana underneath it. It worked reasonably well. At the same time, I started pestering some of the Illumos developers that mentioned that they were working on an update to the e1000g driver — the driver for my wired network interface.

Yesterday, one of them updated the bug related to the driver update with binaries for people to try. Well, guess what? I’m writing this entry in Vim over ssh.

The driver install was a simple matter of overwriting the existing e1000g files, and then running update_drv -a -i ’"pci8086,1502"’ e1000g and then rebooting. (I could have used devfsadm instead, but I wanted to make sure things would come up on boot anyway.)

I still need to switch Xorg to the proprietary NVidia driver.


I’m pretty sure I’ll end up rebooting into Windows every so often anyway…if only to play Wikipedia article: Age of Mythology. :)

P.S. In case you haven’t guessed it yet, Meili is my laptop’s hostname.

OpenIndiana (build 151a)

Over the past few months, I’ve played with Solaris — specifically, OpenIndiana, or OI for short. OI is a fork of OpenSolaris. OI’s first release happened on September 14, 2010. Today, exactly a year later, the OI community is proud to annouce the release of build 151a. The release notes say it all.

Personally, I find the KVM port to Illumos (the project that forked the core libs, programs, and OpenSolaris kernel) the most interesting. It’ll let me run (and manage!) virtual machines a bit more easily than what I get with VirtualBox. (Since OI now uses Illumos as the core Solaris upstream, it benefits from all the great work done by companies and individuals that contribute to Illumos.)

In case you are a bit confused, OI aims to be the defacto community Solaris distribution.

Oh, I almost forgot… 151a includes a package with Guilt (developer/versioning/guilt). :)

Audacity UI feature

Although I’m filing this under the “rants” category, don’t get fooled. The rant is about UIs in general, with Audacity being the exception.

Here’s what happened…I was going to save the recordings of my radio show to my computer, and I noticed that the first hour recording started about 4 minutes after I took over aether. That meant that I needed to get the previous hour, and cut whatever short portion into a small file and keep it along the 3 1-hour long mp3s.

For audio editing, I tend to use Audacity. It works well, it’s rather intuitive, etc., etc. I did the cut, and I was going to export it as an mp3 (to keep the file format consistent with the other 3 hours of audio, otherwise I’d make it an ogg/vorbis). Audacity let me chose the new file name, the new format, but then when it was about to start the actual encoding, this dialog popped up:

Audacity needs libmp3lame

This is absolutely brilliant! And I mean it; I’m not being sarcastic as I usually am. Normally, one of these scenarios happens…

  • …the application gives you a “I can’t find the encoder” at start (if at all), and disables export to that file format
  • …the application gives you a “I can’t find the encoder” at the start of the encoding process, forcing you to abort the encoding, potentially closing the application, to installed the codec, and redoing whatever you did and trying to re-export
  • …the application gives you a “I can’t find the encoder” at the start of the encoding process, making you look through numerous preferences windows to find the one you care about - if it even exists
  • …the application gives you a “I can’t find the encoder” at the start of the encoding process, making you trying to figure out which environment variable (LD_LIBRARY_PATH, LD_PRELOAD, etc., etc.) will make the linker do the right thing, and make the application find the .so

All are sub-optimal. Asking the user for the path to the .so, while not the newbie-friendliest of things, is really the best thing the application could do. This way, if the .so isn’t installed, the user can install it anywhere - system wide or in one’s $HOME - and then point Audacity to it. If the .so is installed but Audacity couldn’t find it, you can manually point it in the right place.

I use Debian, so installing libmp3lame was a matter of making sure I have the Debian Multimedia source in my sources.list, and then running a quick aptitude install to get it on my disk. If you are using a less privileged distro (or if you don’t have root access to install it system-wide), you’ll have to quite possibly go to the project’s website, and grab a copy there. Audacity’s UI designers haven’t failed you there. A convenient way to go to the website to download the .so is right there.

Overall, seeing this dialog didn’t make me agitated that Audacity wants something I don’t have installed, but instead it made me write this post about something that makes sense, but people fail at doing things like this.

Powered by blahgd