Josef “Jeff” Sipek

FAST 2012

Last week I went to FAST ’12. As always, it’s been fun attending. Overall, the talks were good and gave me a couple of ideas. There was definitely a lot of flash and SSD related work — both of which I don’t find all that exciting. There was however a bunch of work related to dedup and backup.

Anyway, here’s my reading list. I’ve skimmed most of the papers, but I want to take a closer look at these.

  • Characteristics of Backup Workloads in Production Systems
  • WAN Optimized Replication of Backup Datasets Using Stream-Informed Delta Compression
  • Recon: Verifying File System Consistency at Runtime
  • Consistency Without Ordering
  • ZZFS: A Hybrid Device and Cloud File System for Spontaneous Users
  • Serving Large-scale Batch Computed Data with Project Voldemort
  • BlueSky: A Cloud-Backed File System for the Enterprise
  • Extracting Flexible, Replayable Models from Large Block Traces
  • iDedup: Latency-aware, Inline Data Deduplication for Primary Storage

Usenix 2009, Part 2

As promised, here’s more of the day-by-day summary of Usenix ’09.


The last day of the conference. As before, I got to the place at 8:30, and had breakfast.

The first session, System Optimization, was interesting. It started with Reducing Seek Overhead with Application-Directed Prefetching. The idea is pretty obvious. You have a library that takes lists of future accesses from the application, and tries to prefetch the data while monitoring the application’s IO accesses. The first deviation from the prefetch-list causes an invocation of a call-back. This allows the application to specify a new prefetch list.

The second talk of the session, Fido: Fast Inter-Virtual-Machine Communication for Enterprise Appliances, was about a simple and fast way to have multiple VMs communicating. Their target was a collection of virtual machines running in an appliance box. Since the OSes were inherently trustworthy (before virtualization took off and even now, there was one OS that did everything), they achieve zero-copy by mapping all the other OSes into each address space. For example, suppose you have 3 VMs (red, green, blue), their address spaces would be something like:

Fido: address spaces

Each VM gets all the VM’s address spaces read-only, and its own read-write. Then a simple message can be exchanged to specify buffer addresses.

The Web, Internet, Data Center session wasn’t very exciting. The one talk that stuck in my head was RCB: A Simple and Practical Framework for Real-time Collaborative Browsing. What they did was some javascript-fu that synchronized the DOM trees between two (or more?) browsers.

The last session for the day (and the conference) was Bugs and Software Updates. It opened with The Beauty and the Beast: Vulnerabilities in Red Hat’s Packages. The authors did some crazy statistics and found that there was some correlation between packages’ dependencies and the number of vulnerabilities. Several audience members pointed out that publishing these findings may cause a feedback that completely maybe either exaggerate this correlation, or it may cause the developers to make dependency choices to make their software seem less likely to have vulnerabilities.

The second talk, Immediate Multi-Threaded Dynamic Software Updates Using Stack Reconstruction sounded interesting, but I really feel like I need to look at the paper first before drawing any further conclusions.

The last talk of the session, Zephyr: Efficient Incremental Reprogramming of Sensor Nodes using Function Call Indirections and Difference Computation, seemed like 2 talks in one. First, we were told about rsync protocol’s shortcomings, and how they fixed them, and then we were told about their function call indirection scheme to make the deltas smaller. This function call indirection sounded far too much like dynamically linked binaries, with GOT pointer and all that good stuff.

After a short break, the last invited talk began. The speaker was Wikipedia article: David Brin - a science fiction writer. He is one of those people that really knows how to present. He gives off an aura of knowing exactly what he’ll say next. I can’t tell for sure if he does know what he’ll say next, or if he merely has an idea where he wants to get to, and “improvises”; to get there.

I went back to the hotel, but got bored soon after. Not knowing what to do, I went for a walk. I just took the first street away from the hotel. It turned out to be a road that went considerably uphill toward the San Diego Medical Campus (or whatever it was called). After some exploring, I got some food (this was the only place I saw that had fast food joints, the hotel area was about as boring as it can get).


I woke up relatively late - 10am. I blame getting used to the pacific timezone (grr, just in time to fly back!). After packing up, and checking out, I ran into Swami. We went to get breakfast, and talked about all sort of stuff - grad school, conferences, the good ol’ days at FSL. After some time, he and another student from Wisconsin took off. I decided to go for a walk. I ended up in the same fast food place. There, I started typing up the previous blahg post. After about 2 hours of working my laptop, I went back and got a shuttle to the airport.

At the airport, I tried to do some work. Before long, I switched to reading a book. Shortly after, it was time to board. The flight itself was mostly uneventful. Sadly, the 2 hour layover in Charlotte, NC was painful. The free wifi that was there few days earlier disappeared. I survived it. Two-ish hours later, we landed at DTW. Due to some miscommunication, I ended up without a ride back to Ann Arbor. I managed to call up a friend that drove me back.

Usenix 2009, Part 1

Phew! It’s been a fun couple of days. I’m going to provide a day-by-day summary of what’s been going on.


More or less the entire day was take up by travel. The flight was mostly uneventful, with a 1 hour layover in Charlotte, NC. The one hour was long enough to get from one terminal to the other and get food, yet short enough that by the time I was done eating, the first class passengers were about to board. For virtually the entire duration of the flight, I was preoccupied with a book I took with me.


Waking up for a 9am event was never easier! I guess it was the timezone difference. Instead of going to bed at about 3am, I went to bed at around midnight (which happens to be 3am on the other coast!).

I woke up at about 6:30 - not due the alarm. I went back to sleep. At 7:30, I woke up for real. Before long, it was 8:30, and I was already at the conference about to get my proceedings and badge. Free food, namely orange juice and crossionts followed.

Nine o’clock rolled around, and the conference began. The keynote was ok. I wasn’t amazed by it, even though the speaker had valid points.

At 11, the first session began - virtualization. This was one of the sessions I’ve been looking forward to. Well, I was interested in one paper specifically: vNUMA: A Virtual Shared-Memory Multiprocessor. The name summarizes the idea very nicely. Why was I interested in it? Well, a long time ago, when you couldn’t walk down the street and buy a multi-core system, the year was 2003 - I had the same idea. It’s a rather obvious idea, as is the name. At first, I was going to hack the Linux kernel to accomplish it (I still have a couple of patches I started), but other things sapped up my time. (The fact that everyone I mentioned it to told me that the paging over ethernet overheads were going to make it impractical made me want to do it even more!) As it turns out, the folks from University of New South Wales, that got a publication out of it, started working on it in 2002.

Their implementation is for the Itanium architecture. They said that they chose it because at the time they started, Intel was pushing Itanium as the architecture that’ll replace x86. Unfortunately, I didn’t get to talk to any of them at the conference.

The lunchtime invited talk was nice. It was about how faculty at Harvard tried to make the intro to computer science course fun and appealing, yet still retain the same important intro material. The big thing: they used Amazon’s Elastic Compute Cloud (EC2) instead of the on-campus computer network. They liked being in charge of the system (not having to wait for some admin to take care of a request for some change), but at the same time they didn’t like being the admins themselves. Some of the students were also not exactly “thrilled” about hearing about the cloud so often - especially since they didn’t have to use it in a cloud-way. One of the most memorable parts of the talk was when the speaker whipped out a phone book, asked the audience about how to find something in it, and shortly after proceeded to do binary search - ripping the book in half, then tossing a half across the podium, and then ripping the remainder in half.

Afterward, the networking session took place. The one talk that was fun was StrobeLight: Lightweight Availability Mapping and Anomaly Detection. The summary of the idea: ping the subnet, then some time later, ping it again, and count the number of IPs that changed - essentially XOR and then count ones. It’s a simple idea that apparently works rather well.

The next session was about storage. It started with a paper that got a best paper award. One of the authors is Swami - a former FSL member, not at University of Wisconsin-Madison. While he never said it during the talk, they essentially implemented a stackable filesystem. Right after it was a talk by guys from the MIT AI lab, about decentralized deduplication in SAN cluster filesystems.

After a short break, the last invited talk for the day happened. A dude from Sun talked about SunSPOTs. SunSPOT is a small wireless sensor/actuator platform that’s based on Java. It reminded me of several other such platforms, but seemed a lot more polished. Unfortunately, it’s Java-centric way was rather disappointing. (More or less everyone that knows me, knows that I don’t like Java.)

The day concluded with a poster session & food.


Again, waking up wasn’t a problem. The breakfast happened to be bagels.

9am: the first talk of the distributed systems session, was Object Storage on CRAQ: High-Throughput Chain Replication for Read-Mostly Workloads. This is one of the papers I’m intending to look at. The other two I found less interesting.

11am: the kernel development session had 2 interesting talks. The first, Decaf: Moving Device Drivers to a Modern Language talked about taking device drivers, splitting them into two portions (a performance critical section in C, and a non-performance critical section that could be moved to “a modern language” - read: Java). While I strongly disagree with the language choice, the overall idea is interesting. Java (along with other more managed languages) provides stronger static checking than C. They actually took some Linux drivers, and split them up. I want to go over their evaluation again. The next interesting talk was about executing filesystems in a userspace framework (Rump File System: Kernel Code Reborn). Unlike FUSE and other attempts, this one aims to take in-kernel filesystem code, and execute it in userspace without any modifications.

The lunchtime invited talk about about how teaching students how to program is hard. How some error messages the compiler outputs are completely misleading and confuse students.

I zoned out for most of the 2pm session about automated management. There were emails & other things to catch up on.

The short paper session started off nicely with The Restoration of Early UNIX Artifacts. The speaker mentioned that he managed to get his hands on a copy of first edition UNIX kernel source. While it was slightly mangled up, he managed to get it up and running. After a bit of effort, he got his hands on some near-second edition UNIX userspace. Another short paper was about how Linux kernel developers respond to static analysis bug reports.

The last invited talk for the day was unusual. It was about the Wikipedia article: Antikythera mechanism. He used Squeak EToys (a Smalltalk environment) to simulate the mechanism. He put the “software” up on the web:

Afterwards, more food. This time without posters. And after the food, there were some BOFs. The one I went to was about ancient UNIX artifacts. There I got to see first edition UNIX running. Really neat. It felt like UNIX - same but different in some ways. The prompt was ’@’; the ’cd’ command didn’t exist, instead you had ’chdir’… that’s right, the “tersness” of UNIX wasn’t always there!; the ’rwx’ bits you see when you do ls -l were different, you had one ’x’ bit, and 2 rw pairs. On my way out, I got more or less dragged into a mobile cluster BOF (or whatever the title was), the most interesting part was when we got to talk about Plan 9 all the way at the end.

Accepted to EuroSys '08!

My summer internship at IBM resulted in a publication in EuroSys 2008!

So, keep an eye out for my name in EuroSys proceedings near you. :)


As many of you may have noticed, I’ve been really lazy when it comes to updating this blahg of mine…so here’s a short summary of what happened over the past week at SC07. I’m sure I forgot to talk about a ton of things…feel free to leave a comment.

Friday, November 9

Pretty uneventful day…flying from JFK to Reno via LAX, checking into the hotel were the two highlights.

Saturday, November 10

We mis-read the bus schedule, and ended up taking the 6:30 shuttle to the convention center. Waking up that early was quite painful. When we got to the center, we started unpacking the nodes, rack and the TV. Compared to the other teams, we were unfortunate enough to have twelve 8-core nodes, and two 4-core nodes. Yeah, 14 nodes, an infiniband switch, a gigE switch, a TV, and the full-sized rack. That’s 18 things to unpack. Other teams had around 8 nodes and similar interconnect. Either way, we had more to set up.

The organizers of the Cluster Challenge (this is the whole thing about universities, and teams, Stony Brook being one of them - read the link for more info) were nice enough to organize a cruise on lake Tahoe for us…but the only problem with it was, that it was in the evening. So, we got to see a whole lot of big black nothing.

Majority of the Indiana University team, featuring Pikachu:

Majority of the Indiana University team, featuring Pikachu

I must admit, the pikachu hat was a great way to draw attention, I therefore propose that next year, the team looks more like this (photo courtesy of Peter Honeyman):


Sunday, November 11

While most of Saturday was spend setting up hardware, at least half of Sunday was spend setting up software. Somehow, magically, NFS decided to stop working (I’ve been told by the NFS folks that it’s generally not NFS that breaks but something else, but I maintain that NFS is broken :) ). In our case, NFS was a major component - we went the netboot way, and had only 1 disk for the entire cluster. We exported the node root directory image, as well as the home directories over NFS over ethernet, and created a tmpfs (kind of like a ramdisk, but it grows as needed) over NFS over IP over IB. There’s probably a way to remove IP out of the equation, but we just didn’t have enough time to try everything we wanted to - like doing PXE boot over IB, removing the need for ethernet all together. (One of the visitors who stopped by our cluster told me that he does do netboot over IB.)

Stony Brook’s rack

Monday, November 12

The Cluster Challenge started at 20:00. Things got really hectic really quickly, but overall it was all fun. Once everything calmed down, we decided to start the 6-hour shifts. I went back to the hotel. At 4:13 in the morning, I got woken up by a call from the team leader asking me when I’d be back. 4:13 is waaaay too early. I decided to take a pillow and the blanket with me to the conference center.

Colorado team

About 19.5 hours later, still at the conference center, I decided to go to sleep. I didn’t feel like going back to the hotel, so I crashed on one of the couches right by our team’s rack. I hear there is a photo of me sleeping on the couch. Moral of the story: when at a conference, take a pillow and a blanket with you, it might come in handy when you decide to sleep at the conference center.

Tuesday, November 13

Shortly after noon, the entire conference center lost power for a couple of seconds (see The Register). None of the teams were using UPSes (UPSes eat up power, which was quite precious - only 26 Amps per team), all the clusters rebooted. I’ve heard that the team from Taiwan lost more than 10 hours of computation because of that.

Taiwan team

We lost only about 15 minutes wall time of computation (on 96 cores) because we just started a new job.

Taken right after the power outage (notice that the lights are still off):

Right after the power outage, it took them a while to turn back the lights

Wednesday, November 14

The competition ended at 16:00. That’s 44 hours after starting. Everyone was quite tired, but not tired enough to skip what the conference organizers have prepared for us. They rented out an entire arcade in one of the near by hotels. The arcade included a whole lot of games, including laser tag. I wish I had a photo of one of the signs at the laser tag place, because it had quite a number of grammatical mistakes.

Purdue team

Thursday, November 15

Judging/organizer station

The conference ended at 16:00. Everything got promptly torn down, and packed up in boxes. And then…*drumroll* everyone headed to a Blue Man Group show done specifically for SC07 tech badge holders (which included the folks doing the Cluster Challenge - read: us). The show was fantastic, but far too short. Next time I have a pile of PVC pipes, I’m going to have a ton of fun :)

Friday, November 16

After an hour meeting at the center to figure out what could be done better next year, everyone dispersed, and went their own ways. We went to the airport, and headed back to NY - this time via Phoenix. We got to JFK around midnight.

OLS 2007!

So, I submitted a proposal for OLS 2007, and I got in! Stay tuned for the self-inflicted agony associated with actually writing the paper :)

OLS 2006 - Addendum

So, as promised, here are some photos from OLS 2006:

Here’s the only photo of a person I took: John Levon aka. Movement:

John Levon

The Bank of Montreal - in Ottawa!

Bank of Montreal in Ottawa

Crazy turists in Ottawa:

Crazy turists

War & civilization…just 1 km apart!

War & Civilization

Of course there were some nice clouds:


As well as these pesky creatures:

Pesky creature

OLS 2006 - Day 6

I woke up at about 10. Took a shower, packed up and checked out. Just as last year, I decided to take my camera and wonder around Ottawa taking photos of interesting looking things. Well, the center of Ottawa is small. By about 14:00, I was kind of bored. So I decided to go back to the hotel and waste some more time. In front of the hotel, I noticed the shuttle to the airport. I asked the guy how frequently he was going, but he misunderstood me and told me how long it takes to get to the airport instead. He then asked if I was going. Since I didn’t have anything better to do, I told him that I’d just grab my bags.

We got to the airport at 14:36. So it took 15 minutes to get there from the hotel. Not bad. Then, I however realized that there wasn’t much to do at the airport. I sat down on one of the benches. Not even 10 minutes later, Movement poped up out of nowhere. We chatted about how exciting flying is. His flight back to .uk was with a layover in Atlanta - not fun. Around 15:10 he left because he was about to check in. I decided that the terminal couldn’t be any more boring, so I checked in as well. During the chat, I decided to take a photo of him. This year that’s the only OLS photo of a person! All the other photos are of Ottawa. I’ll put it up in few days, when I dig it up.

Good thing I checked in :) When I got the the terminal, I sat down near the first outlet I saw. I was about half way though the boot sequence when I notice about 7 other OLS attendees not far from me. They included Mike Halcrow, Val Henson, and some other people some of which seemed familiar, but who’s names I did not know. We chatted. Mike actually showed me a leak in the FiST templates as well as Unionfs. We chatted, until most of them left because their plane was ready. I and some other dude moved to another cluster of chairs with some more OLS attendees which appeared as we were chatting. There we had an interesting discussion about Australian politics. Some really odd things happen there. :) Some time later, they all headed to their plane (yeah, my flight was at 18:00, and I got to the airport at 14:36…). Few clusters of chairs away, I noticed Dave Jones. I decided to walk up, and ask him if he thinks there were enough changes to the bug tracking issue he was talking about in his keynote last year. He thinks that there were some, but most of them were minor. He was really hoping someone would make some automatic bug-report moving tool which automatically moved kernel bugs from say the Red Hat bugzilla to the bugzilla.

Then they called my flight. It was the same type of a plane as on my way to Ottawa - made by Bombardier - a small jet. The flight itself was uneventful and everything went well and on time. One thing that I found interesting was the fact that the US customs people are in Ottawa and not in New York.

When I got home, I just fell asleep. :)

OLS 2006 - Day 5

The day began with an awesome presentation I gave about Unionfs. :) Shawn was recoding it, but after the presentation, he found out that the video turned out to be crap. He has audio only. I’m sure he’ll share it soon. :) I was pleasantly surprised at the number of people that use Unionfs or were interested in Unionfs.

The keynote was excelent as always. However I must say that Greg K-H made it sound like any piece of code will get into the kernel. Yeah, right :) But he did say few nice things about the status of Linux.

After the keynote, there was the GPG key signing - which I did not attend, although I wanted to. Instead we went to get some food. Food was good, we (I, Dave, Mike Halcrow, and Prof. Zadok) talked about a bunch of things ranging from MythTV and terabyte storage servers, to things like the number of ants in Texas. (Apparently, it is a lot of fun to watch termites and fire ants battle to the death. O_o )

We finished food around 19:45 which was about right to head over to the Black Thorn for the after event party. Just as last year it was quite interesting. Pretty much as soon as I got there, I noticed Peter Baudis aka. pasky - the cogito maintainer. We chatted about how git and Mercurial differ (Matt’s talk the day before came in handy :) ). I mentioned I was slowly working on a generic benchmark script that would test a number of popular SCMs including Mercurial, Subversion, and CVS. He was thrilled about the prospect of knowing exactly where git sucked compared to other SCMs - my guess is that he wants to fix it and make it better, a noble goal, but unnecessary as Mercurial already exists and why reinvent the wheel? ;) Seriously, though, I think a lot of people would benefit from knowing exactly where each SCM excels, and where each sucks. The nice thing about collaborating with the git people would be that it would make it more apparent that this wouldn’t just be yet-another-fake-test. After some time, a bunch of other Czech people poped up right next to us (people like, Pavel Machek, etc.). It was quite interesting. :)

After than I joined a converation with some Intel people. As it turns out, one of the Intel people is working on the e1000 driver — awesome piece of hardware, by the way, don’t ever buy anything other than it. :) Some time later, Jens Axboe joined the group briefly. When he said my name seemed familiar, I mentioned how I tried to implement IO priorities - and failed :) Later on, a guy from University of Toronto joined the group. He approached me earlier in the day about unionfs on clusters. We chatted about things ranging from school (undergraduate program, and grad school) to submitting kernel code to lkml. The e1000 guy said a similar thing that we should split unionfs up into a few patches, and send it off. During the event a few people still asked me about Unionfs, which felt good :)

Then, I decided that it would be fun to talk to some IRC people. I found John Levon and Seth Arnold. We sat down, and had an interesting conversation about a number of things. Since at least some of these were quite interesting, here’s a list:

  1. How can I deal with VFS and not drink vodka or other hard liquer
  2. Everybody hates CDE, even people at Sun
  3. Solaris is dead (well, they didn’t say it, but that’s the feeling I got)
  4. Brittons have some interesting sports or at least some of the expected behavior during the sport is interesting, namely:

  1. darts - you are expected to drink as you play
  2. I can’t recall the name - gigantic pool table
  3. cricket - everyone smokes "reefer" (to quote Movement, I just find this name of the substance mildly amusing) because their games sometimes take several days

After that, they kicked everyone out as it was 2:45 already. We (Seth, John, and I) went back to the hotel. There, Prof. Zadok and Chip (who arrived on Friday) were about to get up and head to the airport. :) I just went to bed.

OLS 2006 - Day 4

Friday was kind of interesting. The talks were little weaker, but there were some interesting ones. For example, Matt Mackall’s Towards a Better SCM: Revlog and Mercurial talk was a nice way to learn how Mercurial stores the history. I got tired so I went back to the hotel, and fell asleep for few hours. I woke up just in time to head over to the conference center for the 20:00 Stackable file systems BOF. That was interesting. A lot of useful people showed up, including Christoph Hellwig, Ted Tso, Val Henson, Steve French (Samba/CIFS guy), Jan Blunck,Eric Van Hensbergen (plan9 fs implementation in Linux) and many more. Topics included limited stackspace (4K on i386), cache consistency, locking, and nameidata structure brain damage.

As we planed before, after the BOF we invited everyone over to the hotel for some snacks and drinks. That’s where things got really interesting. I spent a lot of time with Jan Blunck and Eric Van Hensbergen talking about the proper way to do unions. Three people, three different ways to union files :)

After that we had some fun with the stack space issue and Reiserfs (and Hans’s approach to open source).

stack space
There should be a competition "who can create the largest storage stack without overflowing the stack." For example, unionfs on top of gzipfs on top of versionfs on top of device mapper on top of md on top of scsi over ethernet on top of ppp tunneled thought ssh …. you get the idea
Apparently, Christoph once fixed something trivial in reiserfs code, he sent the patch to Hans, and in return he got a gigantic legal document (I know exactly what he is talking about as I have submitted a patch once as well). Well, he didn’t like it, so he gave Hans a bunch of options one of which included certain sum of money for copyright to the code. Interestingly enough, Hans accepted. Too bad I didn’t know about this story before, I might have made some money of off my one line fix to the reiserfs section in the Kconfig file. :)

I have to say, I really want to look at how plan9 works thanks to the conversation I had with Eric Van Hensbergen. It seems to have a lot of really good ideas. Hrm, now that I think about it, I might install it one of my old boxes. Yeah, I know I am odd.

Powered by blahgd