One of the items on my ever growing TODO list (do these ever shrink?) was to see if inlining Illumos’s atomic_* functions would make any difference. (For the record, these functions atomically manipulate variables. You can read more about them in the various man pages — atomic_add, atomic_and, atomic_bits, atomic_cas, atomic_dec, atomic_inc, atomic_or, atomic_swap.) Of course once I looked at the issue deeply enough, I ended up with five cleanup patches. The gist of it is, inlining them caused not only about 1% kernel performance improvement on the benchmarks, but also reduced the kernel size by a couple of kilobytes. You can read all about it in the associated bugs (5042, 5043, 5044, 5045, 5046, 5047) and the patch 0/6 email I sent to the developer list. In this blahg post, I want to talk about how exactly Illumos presents these atomic functions in a stable ABI but at the same time allows for inlines.
Genesis
It should come as no surprise that the “content” of these functions really needs to be written in assembly. The functions are 100% implemented in assembly in usr/src/common/atomic. There, you will find a directory per architecture. For example, in the amd64 directory, we’ll find the code for a 64-bit atomic increment:
ENTRY(atomic_inc_64)
ALTENTRY(atomic_inc_ulong)
lock
incq (%rdi)
ret
SET_SIZE(atomic_inc_ulong)
SET_SIZE(atomic_inc_64)
The ENTRY, ALTENTRY, and SET_SIZE macros are C preprocessor macros to make writing assembly functions semi-sane. Anyway, this code is used by both the kernel as well as userspace. I am going to ignore the userspace side of the picture and talk about the kernel only.
These assembly functions, get mangled by the C preprocessor, and then are fed into the assembler. The object file is then linked into the rest of the kernel. When a module binary references these functions the krtld (linker-loader) wires up those references to this code.
Inline
Replacing these function with inline functions (using the GNU definition) would be fine as far as all the code in Illumos is concerned. However doing so would remove the actual functions (as well as the symbol table entries) and so the linker would not be able to wire up any references from modules. Since Illumos cares about not breaking existing external modules (both open source and closed source), this simple approach would be a no-go.
Inline v2
Before I go into the next and final approach, I’m going to make a small detour through C land.
extern inline
First off, let’s say that we have a simple function, add, that returns the sum of the two integer arguments, and we keep it in a file called add.c:
#include "add.h"
int add(int x, int y)
{
return x + y;
}
In the associated header file, add.h, we may include a prototype like the following to let the compiler know that add exists elsewhere and what types to expect.
extern int add(int, int);
Then, we attempt to call it from a function in, say, test.c:
#include "add.h"
int test()
{
return add(5, 7);
}
Now, let’s turn these two .c files into a .so. We get the obvious result — test calls add:
test()
test: be 07 00 00 00 movl $0x7,%esi
test+0x5: bf 05 00 00 00 movl $0x5,%edi
test+0xa: e9 b1 fe ff ff jmp -0x14f <0xc90>
And the binary contains both functions:
$ /usr/bin/nm test.so | egrep '(Value|test$|add$)'
[Index] Value Size Type Bind Other Shndx Name
[74] | 3520| 4|FUNC |GLOB |0 |13 |add
[65] | 3536| 15|FUNC |GLOB |0 |13 |test
Now suppose that we modify the header file to include the following (assuming GCC’s inline definition):
extern int add(int, int);
extern inline int add(int a, int b)
{
return a + b;
}
If we compile and link the same .so the same way, that is we feed in the object file with the previously used implementation of add, we’ll get a slightly different binary. The invocation of add will use the inlined version:
test()
test: b8 0c 00 00 00 movl $0xc,%eax
test+0x5: c3 ret
But the binary will still include the symbol:
$ /usr/bin/nm test.so | egrep '(Value|test$|add$)'
[Index] Value Size Type Bind Other Shndx Name
[72] | 3408| 4|FUNC |GLOB |0 |11 |add
[63] | 3424| 6|FUNC |GLOB |0 |11 |test
Neat, eh?
extern inline atomic what?
How does this apply to the atomic functions? Pretty simply. As I pointed out, usr/src/common/atomic contains the pure assembly implementations — these are the functions you’ll always find in the symbol table.
The common header file that defines extern prototypes is usr/src/uts/common/sys/atomic.h.
Now, the trick. If you look carefully at the header file, you’ll spot a check on line 39. If all the conditions are true (kernel code, GCC, inline assembly is allowed, and x86), we include asm/atomic.h — which lives at usr/src/uts/intel/asm/atomic.h. This is where the extern inline versions of the atomic functions get defined.
So, kernel code simply includes <sys/atomic.h>, and if the stars align properly, any atomic function use will get inlined.
Phew! This ended up being longer than I expected. :)