OpenRC and the hidden segfault

Hi! This time we will take a look at a rabbit hole involving debugging segmentation faults caused at the init(8) boot stage.

I first noticed something was wrong when seeing this in my boot messages:

* Caching service dependencies
segmentation fault
segmentation fault

Investigating

I thought to myself “okay something’s going on with OpenRC caching”. I started to look into how OpenRC caches service dependencies. I ran equery f openrc and thought /lib/rc/bin/rc-depend looked interesting. Running this binary by itself I get nothing, but then I tried /lib/rc/bin/rc-depend -h and see this option:

-u, --update Force an update of the dependency tree.

Running with that option gives me the two segfaults again, and I immediately open it with GDB. I expected to hit a breakpoint but nope! Then I tried set follow-fork-mode child, surely that’d work. But NOPE! (wat). Then I got a suggestion to try ASan which ended up being a huge rabbit hole.

asan madness

First I added a /etc/portage/env/asan with the usual -fsanitize=address on CFLAGS and LDFLAGS, with sys-apps/openrc asan in /etc/portage/package.env/openrc. Apparently you needed a USE-flag for gcc to compile with sanitizers. This USE-flag happened to be masked on musl, but stupid me just unmasked without thinking there could be some issue with sanitizers and gcc.  I emerged again and still got an error about my compiler not being able to produce binaries.

Next time I tried Clang instead to see if it worked, and surprise … no 🙁 I thought it was the same reason as GCC, but after asking for help we found out that the build somewhere turned on ubsan sanitizers too which this Clang build did not support. With some meson arguments turning off ubsan I got it to compile, great. But I still need to figure out the GCC thing …

After searching online we found this: [Feature request] musl support #1080. Ah there’s the issue!

back to openrc

Having successfully compiled OpenRC with Clang I thought that we’d get some answers with ASan, but sadly it gave us nothing, back to GDB, oh wait no that didn’t work uh… Let’s try strace(1). Running strace -ff -o openrc rc-depend -u spammed out a bunch of files (186 to be exact).

Let’s try finding a segfault shall we?

localhost /tmp/openrc # grep -rsin "Segmentation fault"
openrc.19786:147:write(2, "Segmentation fault\n", 19) = 19
openrc.19786:147:write(2, "Segmentation fault\n", 19) = 19

Opening openrc.19786 I saw a bunch of lvm related things… Me having just patched a lvm segfault got a little worried that I had broken something essential in the process… I looked at my lvm version and it turned out that I had rebuilt lvm from git master before my patch was merged because of a revbump, whops. I still learnt a lot though!