Dumping Memory in MDB
It doesn’t take much reading of documentation, other people’s blogs, and other random web search results to learn how to dump a piece of memory in mdb.
In the following examples, I’ll use the address fffffffffbc30a70. This just so happens to be an avl_tree_t on my system. We can use the ::dump command:
> fffffffffbc30a70::dump \/ 1 2 3 4 5 6 7 8 9 a b c d e f v123456789abcdef fffffffffbc30a70: 801dc3fb ffffffff 0087b1fb ffffffff ................
Or we can use the adb-style /B command:
> fffffffffbc30a70/B kas+0x50: 80
We can even specify the amount of data we want to dump. ::dump takes how many bytes to dump, while /B takes how many 1-byte integers to dump (while for example, /X takes how many 4-byte integers to dump):
> fffffffffbc30a70,20::dump \/ 1 2 3 4 5 6 7 8 9 a b c d e f v123456789abcdef fffffffffbc30a70: 801dc3fb ffffffff 0087b1fb ffffffff ................ fffffffffbc30a80: 20000000 00000000 09000000 00000000 ............... > fffffffffbc30a70,20/B kas+0x50: 80 1d c3 fb ff ff ff ff 0 87 b1 fb ff ff ff ff 20 0 0 0 0 0 0 0 9 0 0 0 0 0 0 0
Things break down if we want to use a walker and pipe the output to ::dump or /B:
> fffffffffbc30a70::walk avl | ::dump \/ 1 2 3 4 5 6 7 8 9 a b c d e f v123456789abcdef fffffffffbc6d2e0: 00000000 00feffff 0000001e 03000000 ................ > fffffffffbc30a70::walk avl | /B kpmseg: kpmseg: 0 0 0 0 0 0 0 0 0
Even though there are 9 entries in the AVL tree, ::dump dumps only the first one. /B does a bit better and it does print what appears to be the first byte of each. What if we want to dump more than just the first byte? Say, the first 32? ::dump is of no use already. Let’s see what we can make /B do:
> fffffffffbc30a70::walk avl | 20/B mdb: syntax error near "20" > fffffffffbc30a70::walk avl | ,20/B mdb: syntax error near ","
No luck.
Solution
Ok, it’s time for the trick that makes it all work. You have to use the ::eval function. For example:
> fffffffffbc30a70::walk avl | ::eval .,20::dump \/ 1 2 3 4 5 6 7 8 9 a b c d e f v123456789abcdef fffffffffbc6d2e0: 00000000 00feffff 0000001e 03000000 ................ fffffffffbc6d2f0: 00000000 00000000 200ac3fb ffffffff ........ ....... \/ 1 2 3 4 5 6 7 8 9 a b c d e f v123456789abcdef fffffffffbc34960: 00000000 00ffffff 00000017 00000000 ................ fffffffffbc34970: 00000000 00000000 200ac3fb ffffffff ........ ....... \/ 1 2 3 4 5 6 7 8 9 a b c d e f v123456789abcdef fffffffffbc31ce0: 00000017 00ffffff 00000080 00000000 ................ fffffffffbc31cf0: 00000000 00000000 200ac3fb ffffffff ........ ....... \/ 1 2 3 4 5 6 7 8 9 a b c d e f v123456789abcdef fffffffffbc35a80: 00000097 00ffffff 0000a0fc 02000000 ................ fffffffffbc35a90: 00000000 00000000 200ac3fb ffffffff ........ ....... \/ 1 2 3 4 5 6 7 8 9 a b c d e f v123456789abcdef fffffffffbc34880: 0000a0d3 03ffffff 00000004 00000000 ................ fffffffffbc34890: 00000000 00000000 200ac3fb ffffffff ........ ....... \/ 1 2 3 4 5 6 7 8 9 a b c d e f v123456789abcdef fffffffffbc31d60: 0000a0d7 03ffffff 000060e8 fb000000 ..........`..... fffffffffbc31d70: 00000000 00000000 200ac3fb ffffffff ........ ....... \/ 1 2 3 4 5 6 7 8 9 a b c d e f v123456789abcdef fffffffffbc7f3a0: 000000c0 ffffffff 00b07f3b 00000000 ...........;.... fffffffffbc7f3b0: 00000000 00000000 200ac3fb ffffffff ........ ....... \/ 1 2 3 4 5 6 7 8 9 a b c d e f v123456789abcdef fffffffffbc7de60: 000080fb ffffffff 00105500 00000000 ..........U..... fffffffffbc7de70: 00000000 00000000 200ac3fb ffffffff ........ ....... \/ 1 2 3 4 5 6 7 8 9 a b c d e f v123456789abcdef fffffffffbc7e000: 000080ff ffffffff 00004000 00000000 ..........@..... fffffffffbc7e010: 00000000 00000000 200ac3fb ffffffff ........ .......
Perfect! ::eval makes repetition with /B work as well:
> fffffffffbc30a70::walk avl | ::eval .,8/B kpmseg: kpmseg: 0 0 0 0 0 fe ff ff kvalloc: kvalloc: 0 0 0 0 0 ff ff ff kpseg: kpseg: 0 0 0 17 0 ff ff ff kzioseg: kzioseg: 0 0 0 97 0 ff ff ff kmapseg: kmapseg: 0 0 a0 d3 3 ff ff ff kvseg: kvseg: 0 0 a0 d7 3 ff ff ff kvseg_core: kvseg_core: 0 0 0 c0 ff ff ff ff ktextseg: ktextseg: 0 0 80 fb ff ff ff ff kdebugseg: kdebugseg: 0 0 80 ff ff ff ff ff
/nap
There is one more trick I want to share in this post. Suppose you have a mostly useless core file, and you want to dump the stack. Not as hex, but rather as a symbol + offset (if possible). The magic command you want is /nap. ‘/’ for printing, ‘n’ for a newline, ‘a’ for symbol + offset (of the value at “dot”), and ‘p’ for symbol (or address) of “dot”. (Formatting differences aside, ‘p’ prints the pointer—“dot”, and ‘a’ prints the value being pointed to—*“dot”.)
For example:
> fd94e3a8,8/nap 0xfd94e3a8: 0xfd94e3a8: 0xfd94f5a8 0xfd94e3ac: libzfs.so.1`namespace_reload+0x394 0xfd94e3b0: 0xfdd6ce28 0xfd94e3b4: 0xfdd6a423 0xfd94e3b8: 0xcc 0xfd94e3bc: libzfs.so.1`__func__.16928 0xfd94e3c0: 0xfdd6ce00 0xfd94e3c4: 0xfdd6ce28
Since the memory happens to be part of the stack, there are no symbols associated with it and therefore the ‘p’ prints a raw hex value.
So, remember: if you have a core file and you think that you need to dump the stack to scavenge for hopefully useful values, you want to…nap. :)