Lilblue Linux: release 20140218

I just pushed out a new release of Lilblue Linux 20140218 [1] which you can download from any Gentoo mirror [2].  For those of you who don’t know, Lilblue Linux is a security-enhanced fully featured XFCE4 desktop system for amd64.  It is built with Gentoo’s hardened toolchain [3] and uses Gentoo’s hardened-sources for the kernel which include the Grsec/PaX patches [4] for added security.  Lilblue Linux really is Gentoo, so the name is a bit pretentious, but there is one important and interesting twist: it is built using uClibc [5] as its standard C library rather than glibc, giving it some advantages of an embedded system, such as speed.

Release 20140218 is primarily a maintenance release in which I updated all the packages so as to sync up with maintream Gentoo’s stable amd64 set.  I didn’t touch the toolchain since there was no pressing need, but I did update the kernel to hardened-sources-3.12.6.  There were no known major security issue nor major bugs in the previous release.  But, there was a lot of package flux, with lots of fixes that resolved some annoying issues which plagued the earlier release.  One such annoyance was SMPlayer that used to open up a second window to play a video rather than rendering it in the main window.

If you are already running Lilblue, you can probably just do a `emerge –sync; layman -S; emerge -uvND world` and get caught up [6], but if you are starting a fresh system, the newer image cleans out those annoyances, so you want to start there.  One of the reasons I push out new images every few months is that there are always glitches when updating.  This is true of any Gentoo system but all the moreso of Lilblue because most software is developed under the assumption that we are building against glibc.  These assumptions (GNU-isms) manifest themselves in varioius ways: 1) assumptions about the availability of functions which are GNU extensions, such as secure_getenv() in systemd’s code base which eudev removes [7],  2) assumptions about header stacking, eg. using variadic functions without including stdarg.h (You can sometimes get away with this on a glibc system because it sneaks in via some other included header, but not in uClibc), [8] and 3) missing LDFLAGS like, -liconv -lintl or -largp, which are needed to find these breakout libraries [9].  There are, however, some very deep issue which require serious investigation, such as the removal of poll_waiting in glib (versions above 2.30.3) which lead to a dead lock for all applications linking against it.  It turned out that the issue there was in uClibc’s implementation of eventfd() [10].  Another interesting bug in uClibc-0.9.33.2 was the non-atomic implementation of pread() and pwrite() in terms of open() and lseek() [11].  This caused a race in git-1.8.x which does a multithreaded unpacking of the deltas and requires atomic pread/pwrite.  Mike Frysinger (vapier) had already worked out the implementation in terms of SYS_pread64/pwrite64 but these had not yet been backported.  The latest adventure was on arm architecture (yes I’m thinking of porting Lilblue to arm) where the syscall for pread/pwrite was being done using _syscall5() rather than _syscall6() and not properly aligning the 64-bit value on even register pairs.  This again broke pread/pwrite and git, but only on arm arch! [12]  Mike again had the fix and backported it.

Lilblue is built form a stage3-amd64-uclibc-hardened tarball that can be found on any gentoo mirror under /experimental/uclibc along side the Lilblue image itself [2].  I keep the build scripts on Gentoo’s releng (release engineering) git repo [13] and I run them occassionally to see if any major issues are creeping in as mainstream Gentoo evolves.  If everything goes well, then I don’t push out another release to avoid taxing the mirrors.  But when things get complicated, or a large number of packages need updating, I get the feeling I’d better push out another release.  I hope one day to have a binpkg system going where you can donwload a ~200MB seed image and then install from a binhost but this is more involved than I first suspected.

So give it a try in a virtual machine if you like.  It runs out-of-the-box on VirtualBox.  Installation instructions are on the home page [1].  Or run it as you main desktop as I do on one of my boxes at home!

References:

[1] https://wiki.gentoo.org/wiki/Project:Hardened_uClibc/Lilblue

[2] The image is named desktop-amd64-uclibc-hardened-[release].tar.bz2 and can be found at http://distfiles.gentoo.org/experimental/amd64/uclibc/

[3] https://wiki.gentoo.org/wiki/Hardened/Toolchain

[4] http://grsecurity.net/

[5] http://uclibc.org/

[6] While I’ve tried to get most of the fixes either into the main Gentoo tree, or upstream, some patches still remain and so I maintain a repository for them: http://git.overlays.gentoo.org/gitweb/?p=proj/hardened-dev.git;a=shortlog;h=refs/heads/uclibc

[7] See man 3 getenv.  While getenv conforms to SVr4, POSIX.1-2001, 4.3BSD, C89 and C99, secure_getenv() is a GNU extension.

[8] The header stacking problem works both ways.  In https://bugs.gentoo.org/show_bug.cgi?id=497200, sys-apps/kbd failed to build on uClibc because of a missing stdarg.h when trying to prototype functions with variadic parameters.  Contrast this to https://bugs.gentoo.org/show_bug.cgi?id=486782, where app-cdr/cdrtools fails to build because including stdio.h indirectly includes bits/sched.h which defines clone() (as in man 2 clone).  But this clashes with a definition of clone() in cdrtools’ readcd.c.  Upstream felt that this was a poor implementation on the part of uClibc and the stacking problem there should be fixed.  I can’t disagree, but it is a thorny issue!

BTW, for those interested, you can get a little python script I wrote to analyse header stacking from my dev space: http://dev.gentoo.org/~blueness/misc/header-stack.py

[9] In this way Lilblue is similar to Gentoo on FreeBSD.  Rather than using uClibc’s iconv which has issues, Lilblue pulls in dev-libs/libiconv. The additional LDFLAGS are added on a per package basis using /etc/portage/package.env.

[10] https://bugzilla.gnome.org/show_bug.cgi?id=691168

[11] https://bugs.gentoo.org/show_bug.cgi?id=500382

[12] https://bugs.gentoo.org/show_bug.cgi?id=500382

[13] http://git.overlays.gentoo.org/gitweb/?p=proj/releng.git;a=tree;f=tools-uclibc/desktop

The Gentoo Profile Stacking Problem

I thought I’d write a bit about a long standing problem that the hardened team has been facing with Gentoo’s profile system.  Ever since I joined the team around 2009, we’ve had to deal with the “profile stacking problem”.  Most users and devs just merily go along using `eselect profile` to pick the profile closest to the type of system they want and then tweak the various files under /etc/portage, adding a USE flag here, and keywording or unmasking a package there, until they get the “perfect” system.  What I want to do in this post is expose just what goes into designing the profiles that we publicly export.

I was inpsired to write this because of bug #492312.  There we want to re-introduce the hardened desktop profile for amd64, x86 and arm.  I say “re-introduce” because we had to remove it and its sibling profiles /server and /developer.  So what was going on there?

To start, let me give you a nice pice of python code:

import portage
for p in portage.settings.profiles:
    print("%s" % p)

What this little snippet does is print out the profile stack as the directories inherited from one another via the parent file.  Its a useful tool because profile stacking can get very hard to follow.  When the parent file has something simple like just “..” then the inheritance is easy and that directory just inherits all the package.mask, package.unmask etc of the parent directory, as you would expect from the shell meaning of “..”  But what happens when the parent file looks like this:

../../../base
../../../default/linux
../../../arch/amd64
..

as it does for hardened/linux/amd64?  Well then we get some interesting behavior. The first line says, inherit from base. Easy enough since base inherits from nothing else you get all of base’s settings. The second line says inherit from default/linux, which aslo doesn’t inherit from anything.  These setting just add and override those from base.  Easy enough. Ah! But now we come to arch/amd64, where the parent file says

../base
../../features/multilib/lib32

and the inheritance continues to those directories in order. Finally “..” in hardened/linux/amd64 means inherit hardened/linux which sets most of hardened’s needs via make.defaults, package.mask, use.mask and friends. But alas, hardened/linux has its own parent file which reads

../../releases/13.0

and the trip down the rabbit hole continues!  If you are starting to get a little lost, don’t feel bad. It is hard to wrap your brain around stacking, which is why that little script above is so useful.  But the difficulty in following profiles stacking is not the real problem. If you’re like me, you’re too proud to admit you can’t get your head around any complexity 😉   No, the real problem is that you can’t control the stacking order.

To demonstrate, let me refer again to bug #492312.  There we’d like to have a profile which reads

hardened/linux/amd64/desktop

Okay, but what should we put for its parent file?  We’ll need “..” in there to inherit all of hardened/amd64 settings, but we also would like targets/desktop.  So let’s try a parent file that looks like this

..
../../../../targets/desktop

In that case, our little script tells us that our profile stack as follows:

/usr/portage/profiles/base
/usr/portage/profiles/default/linux
/usr/portage/profiles/arch/base
/usr/portage/profiles/features/multilib
/usr/portage/profiles/features/multilib/lib32
/usr/portage/profiles/arch/amd64
/usr/portage/profiles/releases
/usr/portage/profiles/eapi-5-files
/usr/portage/profiles/releases/13.0
/usr/portage/profiles/hardened/linux
/usr/portage/profiles/hardened/linux/amd64
/usr/portage/profiles/targets/desktop
/usr/portage/profiles/hardened/linux/amd64/destkop

And if you switch the order of .. and targets/desktop, you get

/usr/portage/profiles/targets/desktop
/usr/portage/profiles/base
/usr/portage/profiles/default/linux
/usr/portage/profiles/arch/base
/usr/portage/profiles/features/multilib
/usr/portage/profiles/features/multilib/lib32
/usr/portage/profiles/arch/amd64
/usr/portage/profiles/releases
/usr/portage/profiles/eapi-5-files
/usr/portage/profiles/releases/13.0
/usr/portage/profiles/hardened/linux
/usr/portage/profiles/hardened/linux/amd64
/usr/portage/profiles/hardened/linux/amd64/destkop

The problem with the first ordering is that targets/desktop overrides hardened/linux/amd64 and so any USE flags that we may turn off or on in hardened can get reverse in desktop.  The example here is the jit flag — Just-In-Time compilers write executable code on the fly in areas of memory which must be both writeable and executable.  But a PaX hardened kernel will not allow WX mmap-ings because this is an obvious exploit vector.  Rather, in hardened, we prefer slower and safer methods for compiling/interpreting code on the fly than JIT.

Okay, so what about the second ordering.  It may look strange to have target/desktop before base, but that in itself is not an issue.  Here we have the same problem as above but in an even more subtle way!  (See my comment #9 of bug #492312.)  Consider a fairly important package like dev-libs/libxml2.  In the current state of the tree, `emerge -vp dev-libs/libxml2` would give

 [ebuild   R    ] dev-libs/libxml2-2.9.1-r1:2  USE="ipv6 python* ...

for both stacking choices. But if at some point in the future, someone added the following to profiles/default/linux/package.use

#Python support causes problems on xyz
#Don't pull it in if we don't neeed it
dev-libs/libxml2  -python

The vanilla profile default/linux/amd64/13.0/desktop and our hardened profile with targets/desktop last would not change since they have “dev-libs/libxml2 python” in package.use near the bottom of the stack, but our proposed hardened profile with targets/desktop on top would give

 dev-libs/libxml2-2.9.1-r1:2  USE="ipv6 readline ... -python ...

So, both choices for orderings of “..” and “targets/desktop” in our parent file for hardened/linux/amd64/desktop lead to situations where we can’t control what packages get what use flags. What we would like is a stacking that looks something like this

...
/usr/portage/profiles/targets/desktop
/usr/portage/profiles/hardened/linux/amd64
/usr/portage/profiles/hardened/linux/amd64/destkop

but how do we get that with our current inheritance mechanism? One idea that Magnus (Zorry) had was to gutt out this portion of portage and replace the parsing of the parent file with something along the lines of openrc’s depend() { … } clause. Then we just locally say what has to come before/after what and we let the algorithm figure it out. It sounds like an interesting problem if there were two of me and if there were a good chance that it would actually get implemented. In the mean time, we limp along with what we have and do ad hoc fixes as changes in one part of the profiles means we have to adjust other things. Since we are all responsible for different areas of the tree’s profiles, inevitably we cause one another breakage even with the best of intentions. For example, a few days ago, Mike (vapier) removed a masking on the uclibc USE flag in the base profile.  Doing so makes perfect sense. He didn’t tell me, and why should he have to?, but this lead to a small breakage in hardened/linux/uclibc/amd64 and friends where I had to relax that masking.  I only discovered this upon a catalyst run which is a bit annoying.