Josef “Jeff” Sipek

This is a guest post by my wife, Holly.

In the software industry, some problems don't have Google-able solutions. Just like I can't Google where I left my keys, when I last got an oil change, or why I still haven't filed my taxes, I also can't Google what the race condition is in my code. To answer these questions, I must look inside myself. Usually.

"I hope your trip is going awesome and I hate to bother you, but..."

This tale begins in 2013. Jeff and I had recently gotten married and were on our honeymoon. Our team was rather... small... at the time, so we left the storage system we were responsible for in the hands of Josh, our intern of four months or so, with oversight from the Ops team. Halfway through our trip, Josh reluctantly sent us an email with details of an assertion failure in production that he couldn't debug: sem_post() was supposed to return 0, but it was returning -1. It had happened on a couple servers now, and only started after we left. Jeff and I didn't have the resources to debug it remotely, and restarting the service was a viable workaround, so we left it as a problem for our return.

Over the next four months, we made little progress. We discovered that sem_post() was specifically failing with EINVAL, but none of us could find the race condition in the code. Because that's what it means when your calls to concurrency primitives are failing: you have a race condition because you are bad at writing concurrent code. Four months later, Jeff left the company with the bug still unresolved. Josh (no longer an intern) and I continued poring over the code. The failure was rare and we didn't know how to reproduce it -- it happened every ~500,000 server-hours, and given the size of our fleet, that was on the order of once a month -- but when we saw it, we usually spent a few more hours trying to track it down. No dice. It seemed that we were bad at concurrent programming, and even worse at debugging.

Anger makes you do irrational things

One evening, some 16 months after the first occurrence, Josh messaged me. He said he thought he had finally found our elusive race condition!

Inside, I must confess, I did not react maturely. I was not relieved, proud, nor excited. I was incensed to hear he found it, after I'd sunk so much time into this utterly impossible bug. Blinded by my entirely irrational fury, determined to prove him wrong (did I mention "irrational"?), I rage-Googled the problem.

(Again, for the uninitiated, this is equivalent to Googling "where did I leave my keys". Race conditions are a problem you have to solve yourself. Perhaps you could post all the relevant code on a message board and find someone willing to help you debug your specific problem, because the logic is unique to your program and there's no Google-able "general solution" to your errors with semaphores beyond "be better at writing concurrent code.")

One of the search results caught my eye.

Horses, not zebras

There's a saying in medical schools: "When you hear hoofbeats, think of horses, not zebras." When diagnosing a problem, it is more likely that the issue is a common one rather than a rare one. If you have a runny nose, it's probably a cold and not a cerebrospinal fluid leak. If you're getting errors while using concurrency primitives, you probably wrote the code wrong. After all, concurrency is hard and life isn't an episode of House, M.D.

But zebras do exist.

I had clicked the link for a Bugzilla issue titled sem_post/sem_wait race causing sem_post to return EINVAL and started reading. (Meanwhile, Josh told me "Never mind, I didn't find it after all.") The bug report from someone named Don suggested that there is a race in the implementation of glibc semaphore primitives themselves causing sem_post() to access already-freed memory and fail with EINVAL (we were seeing EINVAL!). He said it's hard to reproduce (we saw it once per hundreds of thousands of server hours!). He could reproduce it by adding a sleep() call to the sem_post() function (which was farther than we'd gotten!). This was our problem! It had to be! Maybe we weren't bad at writing concurrent code!

He does not understand your question because he is assuming you are an idiot

One problem with "think of horses, not zebras" is that other people are well aware of the concept.

Following Don's report of the bug was an exchange with Ulrich (one of the main contributors/maintainers for glibc at the time) in which Ulrich asked "Why would this at all be a bug?" and stated "Your code is wrong in assuming what it does." Don patiently outlined his thinking. Ulrich tersely asserted that Don's usage was illegal. Don made one final plea ("Sorry if it seems I am belaboring this"), and Ulrich never responded.

After four months, someone named Pat replied to Don, and opened his response with a sentence that burned itself into my brain. He said, "Ulrich does not understand your question because he is assuming you are an idiot."

It's not an entirely unreasonable assumption. After all, what is more likely? Don is the first person to identify "a fundamental race rendering semaphores useless" in glibc, a library whose semaphores have been used by a great many people for a great many years seemingly without issue, or Don is kind of bad at concurrent programming and the issue with his code is his own? Ulrich decided he was hearing horses and didn't look for zebras. And I almost can't blame him.

When I was a TA for freshman- and sophomore-level programming classes, my students made semi-regular claims that "the compiler is broken." This was their way of expressing that their code did not do what they expected it to do or, in some cases, that their code did not even compile. Of course the compiler isn't broken! These are freshmen who have been writing C++ for two months and g++ had been compiling it since before they were born. Is it more likely that a student who has just begun to learn programming has a syntax error or a bug in their code, or that they have found a heretofore undiscovered problem with software that has been used by millions of people for decades? While I never assumed they were idiots (my students were wonderful; one was the best man at our aforementioned wedding) I admit I gave approximately zero credence to their claims of having discovered a compiler bug, just as Ulrich gave zero credence to Don's bug report.

Over a decade later, I try to channel my inner Pat. (A mere three days after Pat replied in support of Don's claims, a fix for the race was implemented by Rich Felker.) "Think horses" is the type of heuristic that keeps me from wasting too much time on absurd-sounding claims, but at the same time, I try not to get trampled because I was too busy assuming the guy yelling "Zebras!" is an idiot. I'm not perfect, but I'm trying. And I'm pretty sure none of my freshmen ever found a real compiler bug in class...

In the end, we (Josh) worked around the sem_post() race condition, replacing the fundamentally-broken semaphores with locks and a condition variable, because we were already running an outdated version of glibc and knew it might be a while before Ops could update all the servers. After that change, we never saw the failure again.

History doesn't repeat itself, but it rhymes

Why am I writing this post today?

Recently my team had an issue with a cloud service provider (CSP) that we use. It wasn't related to race conditions, but it was the same type of un-Google-able problem. We assumed the issue was a bug in our code and tried to track it down. After a while with no real leads, and a problem scope that continued widening, we opened a case with the CSP.

(Please pardon my deliberate ambiguity in this section, but I am hesitant to divulge as many details about an issue we haven't yet fully resolved internally as I did for a decade-old bug.)

We suspected a hardware issue, they insisted it was a software issue (first ours, but later conceded it might be theirs). Six weeks of back-and-forth later, the glibc semaphore issue crossed my mind, and I Googled my un-Google-able problem once again (out of frustration, but not out of rage). One of the results was a tweet, which led me to a recent-ish hardware research paper, and with every sentence I read, I became more and more convinced that the issue described in this paper was exactly what we were experiencing.

Once more, I had Googled the impossible and found a plausible explanation that lined up with our problem! This time, however, I found myself in Don's shoes. Because, truly, what is more likely? That we (a not-terribly-significant customer of the large CSP) are repeatedly encountering an issue caused by a hardware issue that is both sufficiently rare and sufficiently interesting that research papers are being written about it? Or that we (again, a rather insignificant customer) are idiots who have written some buggy software or are using some buggy libraries, and that's the source of all of our problems?

After another five weeks of meetings in which we provided all the evidence we had been collecting, including sharing the paper with relevant sections highlighted, and imploring them to look into it, I was finally confident that the CSP had brought the right people -- hardware people -- to our sync-up. Two weeks later, they were able to show us the results of the tests they used to identify the specific hardware that was causing our issue. They're pulling it from production, which should mean the end of the problem for us and any other customers who may have encountered it.

Is there a lesson here? I'm not sure. But if I had a nickel for every time Google led me to the right answer for an un-Google-able problem only to encounter challenges getting anyone to believe the diagnosis because it is vanishingly unlikely, I'd have two nickels, which isn't a lot, but it's weird that it happened twice. Perhaps, once a decade or so, you may find yourself on one side or the other of such an issue, and I hope that you can channel your inner Pat to a speedier resolution than these cases had.

P.S. Early on, a coworker tried diagnosing this issue with the help of AI, which told him with "highest probability" it was a bug with the (JIT) compiler. Call me a hypocrite, but I did not try to understand its explanation because I was assuming that it was an idiot.