Josef “Jeff” Sipek

setuid/setgid & coreadm

While core files can be a huge nuisance (i.e., “why are there core files all over my file system?!”), it is hard to deny the fact that they often are invaluable when trying to diagnose a bug. Illumos systems allow the system administrator to control the creation of core files in an easy but powerful way—via the coreadm utility. Recently, I had a bit of a run-in with coreadm because I misunderstood one of its features. As I often do, I’m using this space to document my findings for my and others’ benefit.

A Small Introduction

First of all, let me introduce coreadm a little bit. Simply running coreadm without any arguments will print the current settings. For example, these are the default Illumos settings (keep in mind that some distros use different defaults):

# coreadm
     global core file pattern: 
     global core file content: default
       init core file pattern: core
       init core file content: default
            global core dumps: disabled
       per-process core dumps: enabled
      global setid core dumps: disabled
 per-process setid core dumps: disabled
     global core dump logging: disabled

When I first came across coreadm a couple of years ago, I expected a boolean (“should cores be created”) and a pattern for core file names. As you can see above, there are more options than that. While wearing both developer and system administrator hats at the same time, I was very happy to see that it is possible for a misbehaving process to create two cores—a “per-process” core and a “global” one. Since most are familiar with the “per-process” cores that end up in whatever the current working directory was, I’m going to talk only about the global ones. (The term “global” in this context has nothing to do with the global zone. All these settings are per zone.)

The global core settings allow the system administrator to save a copy of every core ever made in a central location for later analysis. So, for example we can make all normal as well as setuid processes dump core in a specified location (e.g., /cores/%f.%p):

# coreadm -e global -e global-setid -g /cores/%f.%p

Running coreadm without any arguments again shows us the new configuration:

# coreadm 
     global core file pattern: /cores/%f.%p
     global core file content: default
       init core file pattern: core
       init core file content: default
            global core dumps: enabled
       per-process core dumps: enabled
      global setid core dumps: enabled
 per-process setid core dumps: disabled
     global core dump logging: disabled

setid

The difference between plain ol’ global core dumps and global setid core dumps is pretty easy to guess: global core dumps are exactly what I described above…except a setid process will not generate a core file unless the setid core dumps are also enabled. The handwavy reasoning here is that setid cores can include sensitive information that could be leaked by sharing the core.

The aspect that bit me recently is that if a process gives up its privileges via setuid(2), it is treated the same way as if it were setuid. This behavior is documented in the setuid syscall handler in uid.c:121:

	/*
	 * A privileged process that gives up its privilege
	 * must be marked to produce no core dump.
	 */

So, when I tried to make the daemon more secure by instructing it to change to a non-privileged user, I was accidentally disabling core dumps for that process. Needless to say, this made debugging the SIGSEGV I was running into significantly harder. Now that I know this, I make sure both global and global setid cores are always enabled.

Performance Co-Pilot: Part 2, Enabling PMDAs

In my previous post, I introduced Performance Co-Pilot (PCP). I know, I promised the next post to be about logging, but I thought I’d make a short detour and show how to install more PMDAs.

After installing PCP on a Linux system, you will have access to somewhere around 850 various metrics from the three basic PMDAs (pmcd, linux, and mmv). There are many more metrics that you can get at if you enable some of the non-default PMDAs.

I pondered what the best way to present a simple howto would be, and then I realized that simply copying & pasting a session where I install a PMDA will do.

First of all, all the PMDAs live in /var/lib/pcp/pmdas/.

# cd /var/lib/pcp/pmdas/
# ls
apache	 gpsd	    lustrecomm	mounts	   news     process   sendmail	systemtap  vmware
bonding  kvm	    mailq	mysql	   pdns     roomtemp  shping	trace	   weblog
cisco	 linux	    memcache	named	   pmcd     samba     simple	trivial    zimbra
dbping	 lmsensors  mmv		netfilter  postfix  sample    summary	txmon

In this post, I will use the PowerDNS PMDA as an example, but the steps are the same for the other PMDAs.

# cd pdns/
# ls
Install  Remove  pmdapdns.pl

As you can see, there are three files in this directory. We are interested in the Install script. Simply run it as root, and when it asks whether you want a collector, a monitor, or both answer appropriately — if you are running the daemon on the same host, answering both is your best bet. (I never had the need to answer anything else.)

# ./Install 
You will need to choose an appropriate configuration for installation of
the "pdns" Performance Metrics Domain Agent (PMDA).

  collector	collect performance statistics on this system
  monitor	allow this system to monitor local and/or remote systems
  both		collector and monitor configuration for this system

Please enter c(ollector) or m(onitor) or b(oth) [b] 
Updating the Performance Metrics Name Space (PMNS) ...
Compiled PMNS contains
	  197 hash table entries
	  847 leaf nodes
	  132 non-leaf nodes
	 8149 bytes of symbol table
Terminate PMDA if already installed ...
Updating the PMCD control file, and notifying PMCD ...
Check pdns metrics have appeared ... 22 warnings, 60 metrics and 42 values

At this point, the PMDA has been installed (take a look at /etc/pmcd/pmcd.conf to see the new config line there enabling the new PMDA). Now, we can see the new metrics using pminfo (there are many more, I just pruned the list for brevity):

# pminfo pdns
pdns.packetcache_hit
pdns.tcp_answers
pdns.packetcache_miss

We are done!

If you decide to uninstall a PMDA, just cd into the directory and run the Remove script.

Formatting Error

Few years ago, I found a formating bug in PowerDNS’s web interface. It took a while for it to get fixed. Since it wasn’t a critical bug, I never tried to make the maintainer see the light — eventually he found it himself :) Now, I can say that PowerDNS is perfect :)

Last night, while roaming around BBC’s website, and I noticed this:

www.bbc.co.uk formating error

I took a screenshot and earlier today I let them know about the problem. I don’t see today’s numbers displayed in engineering format, but today’s numbers are an order of magnitude larger than yesterday’s so I can’t tell if they already fixed it, or if it’s the larger numbers that are hiding the problem. We’ll see if I get a “thank you” note, or something :)

PowerDNS Strikes Again!

I have heard that an unnamed ISP has switched their DNS servers from Bind to something much better — PowerDNS. I, personally, use PowerDNS and I have only positive things to say about it. As for the ISP’s switch, I hear that they reduced they farm of ~190 Bind servers to ~60 PowerDNS servers. Impressive no matter how you look at it! If you are about to set up a DNS server, I suggest you look at PowerDNS.

Powered by blahgd