Lilblue Linux: release 20140218

I just pushed out a new release of Lilblue Linux 20140218 [1] which you can download from any Gentoo mirror [2].  For those of you who don’t know, Lilblue Linux is a security-enhanced fully featured XFCE4 desktop system for amd64.  It is built with Gentoo’s hardened toolchain [3] and uses Gentoo’s hardened-sources for the kernel which include the Grsec/PaX patches [4] for added security.  Lilblue Linux really is Gentoo, so the name is a bit pretentious, but there is one important and interesting twist: it is built using uClibc [5] as its standard C library rather than glibc, giving it some advantages of an embedded system, such as speed.

Release 20140218 is primarily a maintenance release in which I updated all the packages so as to sync up with maintream Gentoo’s stable amd64 set.  I didn’t touch the toolchain since there was no pressing need, but I did update the kernel to hardened-sources-3.12.6.  There were no known major security issue nor major bugs in the previous release.  But, there was a lot of package flux, with lots of fixes that resolved some annoying issues which plagued the earlier release.  One such annoyance was SMPlayer that used to open up a second window to play a video rather than rendering it in the main window.

If you are already running Lilblue, you can probably just do a `emerge –sync; layman -S; emerge -uvND world` and get caught up [6], but if you are starting a fresh system, the newer image cleans out those annoyances, so you want to start there.  One of the reasons I push out new images every few months is that there are always glitches when updating.  This is true of any Gentoo system but all the moreso of Lilblue because most software is developed under the assumption that we are building against glibc.  These assumptions (GNU-isms) manifest themselves in varioius ways: 1) assumptions about the availability of functions which are GNU extensions, such as secure_getenv() in systemd’s code base which eudev removes [7],  2) assumptions about header stacking, eg. using variadic functions without including stdarg.h (You can sometimes get away with this on a glibc system because it sneaks in via some other included header, but not in uClibc), [8] and 3) missing LDFLAGS like, -liconv -lintl or -largp, which are needed to find these breakout libraries [9].  There are, however, some very deep issue which require serious investigation, such as the removal of poll_waiting in glib (versions above 2.30.3) which lead to a dead lock for all applications linking against it.  It turned out that the issue there was in uClibc’s implementation of eventfd() [10].  Another interesting bug in uClibc-0.9.33.2 was the non-atomic implementation of pread() and pwrite() in terms of open() and lseek() [11].  This caused a race in git-1.8.x which does a multithreaded unpacking of the deltas and requires atomic pread/pwrite.  Mike Frysinger (vapier) had already worked out the implementation in terms of SYS_pread64/pwrite64 but these had not yet been backported.  The latest adventure was on arm architecture (yes I’m thinking of porting Lilblue to arm) where the syscall for pread/pwrite was being done using _syscall5() rather than _syscall6() and not properly aligning the 64-bit value on even register pairs.  This again broke pread/pwrite and git, but only on arm arch! [12]  Mike again had the fix and backported it.

Lilblue is built form a stage3-amd64-uclibc-hardened tarball that can be found on any gentoo mirror under /experimental/uclibc along side the Lilblue image itself [2].  I keep the build scripts on Gentoo’s releng (release engineering) git repo [13] and I run them occassionally to see if any major issues are creeping in as mainstream Gentoo evolves.  If everything goes well, then I don’t push out another release to avoid taxing the mirrors.  But when things get complicated, or a large number of packages need updating, I get the feeling I’d better push out another release.  I hope one day to have a binpkg system going where you can donwload a ~200MB seed image and then install from a binhost but this is more involved than I first suspected.

So give it a try in a virtual machine if you like.  It runs out-of-the-box on VirtualBox.  Installation instructions are on the home page [1].  Or run it as you main desktop as I do on one of my boxes at home!

References:

[1] https://wiki.gentoo.org/wiki/Project:Hardened_uClibc/Lilblue

[2] The image is named desktop-amd64-uclibc-hardened-[release].tar.bz2 and can be found at http://distfiles.gentoo.org/experimental/amd64/uclibc/

[3] https://wiki.gentoo.org/wiki/Hardened/Toolchain

[4] http://grsecurity.net/

[5] http://uclibc.org/

[6] While I’ve tried to get most of the fixes either into the main Gentoo tree, or upstream, some patches still remain and so I maintain a repository for them: http://git.overlays.gentoo.org/gitweb/?p=proj/hardened-dev.git;a=shortlog;h=refs/heads/uclibc

[7] See man 3 getenv.  While getenv conforms to SVr4, POSIX.1-2001, 4.3BSD, C89 and C99, secure_getenv() is a GNU extension.

[8] The header stacking problem works both ways.  In https://bugs.gentoo.org/show_bug.cgi?id=497200, sys-apps/kbd failed to build on uClibc because of a missing stdarg.h when trying to prototype functions with variadic parameters.  Contrast this to https://bugs.gentoo.org/show_bug.cgi?id=486782, where app-cdr/cdrtools fails to build because including stdio.h indirectly includes bits/sched.h which defines clone() (as in man 2 clone).  But this clashes with a definition of clone() in cdrtools’ readcd.c.  Upstream felt that this was a poor implementation on the part of uClibc and the stacking problem there should be fixed.  I can’t disagree, but it is a thorny issue!

BTW, for those interested, you can get a little python script I wrote to analyse header stacking from my dev space: http://dev.gentoo.org/~blueness/misc/header-stack.py

[9] In this way Lilblue is similar to Gentoo on FreeBSD.  Rather than using uClibc’s iconv which has issues, Lilblue pulls in dev-libs/libiconv. The additional LDFLAGS are added on a per package basis using /etc/portage/package.env.

[10] https://bugzilla.gnome.org/show_bug.cgi?id=691168

[11] https://bugs.gentoo.org/show_bug.cgi?id=500382

[12] https://bugs.gentoo.org/show_bug.cgi?id=500382

[13] http://git.overlays.gentoo.org/gitweb/?p=proj/releng.git;a=tree;f=tools-uclibc/desktop

The Gentoo Profile Stacking Problem

I thought I’d write a bit about a long standing problem that the hardened team has been facing with Gentoo’s profile system.  Ever since I joined the team around 2009, we’ve had to deal with the “profile stacking problem”.  Most users and devs just merily go along using `eselect profile` to pick the profile closest to the type of system they want and then tweak the various files under /etc/portage, adding a USE flag here, and keywording or unmasking a package there, until they get the “perfect” system.  What I want to do in this post is expose just what goes into designing the profiles that we publicly export.

I was inpsired to write this because of bug #492312.  There we want to re-introduce the hardened desktop profile for amd64, x86 and arm.  I say “re-introduce” because we had to remove it and its sibling profiles /server and /developer.  So what was going on there?

To start, let me give you a nice pice of python code:

import portage
for p in portage.settings.profiles:
    print("%s" % p)

What this little snippet does is print out the profile stack as the directories inherited from one another via the parent file.  Its a useful tool because profile stacking can get very hard to follow.  When the parent file has something simple like just “..” then the inheritance is easy and that directory just inherits all the package.mask, package.unmask etc of the parent directory, as you would expect from the shell meaning of “..”  But what happens when the parent file looks like this:

../../../base
../../../default/linux
../../../arch/amd64
..

as it does for hardened/linux/amd64?  Well then we get some interesting behavior. The first line says, inherit from base. Easy enough since base inherits from nothing else you get all of base’s settings. The second line says inherit from default/linux, which aslo doesn’t inherit from anything.  These setting just add and override those from base.  Easy enough. Ah! But now we come to arch/amd64, where the parent file says

../base
../../features/multilib/lib32

and the inheritance continues to those directories in order. Finally “..” in hardened/linux/amd64 means inherit hardened/linux which sets most of hardened’s needs via make.defaults, package.mask, use.mask and friends. But alas, hardened/linux has its own parent file which reads

../../releases/13.0

and the trip down the rabbit hole continues!  If you are starting to get a little lost, don’t feel bad. It is hard to wrap your brain around stacking, which is why that little script above is so useful.  But the difficulty in following profiles stacking is not the real problem. If you’re like me, you’re too proud to admit you can’t get your head around any complexity ;)   No, the real problem is that you can’t control the stacking order.

To demonstrate, let me refer again to bug #492312.  There we’d like to have a profile which reads

hardened/linux/amd64/desktop

Okay, but what should we put for its parent file?  We’ll need “..” in there to inherit all of hardened/amd64 settings, but we also would like targets/desktop.  So let’s try a parent file that looks like this

..
../../../../targets/desktop

In that case, our little script tells us that our profile stack as follows:

/usr/portage/profiles/base
/usr/portage/profiles/default/linux
/usr/portage/profiles/arch/base
/usr/portage/profiles/features/multilib
/usr/portage/profiles/features/multilib/lib32
/usr/portage/profiles/arch/amd64
/usr/portage/profiles/releases
/usr/portage/profiles/eapi-5-files
/usr/portage/profiles/releases/13.0
/usr/portage/profiles/hardened/linux
/usr/portage/profiles/hardened/linux/amd64
/usr/portage/profiles/targets/desktop
/usr/portage/profiles/hardened/linux/amd64/destkop

And if you switch the order of .. and targets/desktop, you get

/usr/portage/profiles/targets/desktop
/usr/portage/profiles/base
/usr/portage/profiles/default/linux
/usr/portage/profiles/arch/base
/usr/portage/profiles/features/multilib
/usr/portage/profiles/features/multilib/lib32
/usr/portage/profiles/arch/amd64
/usr/portage/profiles/releases
/usr/portage/profiles/eapi-5-files
/usr/portage/profiles/releases/13.0
/usr/portage/profiles/hardened/linux
/usr/portage/profiles/hardened/linux/amd64
/usr/portage/profiles/hardened/linux/amd64/destkop

The problem with the first ordering is that targets/desktop overrides hardened/linux/amd64 and so any USE flags that we may turn off or on in hardened can get reverse in desktop.  The example here is the jit flag — Just-In-Time compilers write executable code on the fly in areas of memory which must be both writeable and executable.  But a PaX hardened kernel will not allow WX mmap-ings because this is an obvious exploit vector.  Rather, in hardened, we prefer slower and safer methods for compiling/interpreting code on the fly than JIT.

Okay, so what about the second ordering.  It may look strange to have target/desktop before base, but that in itself is not an issue.  Here we have the same problem as above but in an even more subtle way!  (See my comment #9 of bug #492312.)  Consider a fairly important package like dev-libs/libxml2.  In the current state of the tree, `emerge -vp dev-libs/libxml2` would give

 [ebuild   R    ] dev-libs/libxml2-2.9.1-r1:2  USE="ipv6 python* ...

for both stacking choices. But if at some point in the future, someone added the following to profiles/default/linux/package.use

#Python support causes problems on xyz
#Don't pull it in if we don't neeed it
dev-libs/libxml2  -python

The vanilla profile default/linux/amd64/13.0/desktop and our hardened profile with targets/desktop last would not change since they have “dev-libs/libxml2 python” in package.use near the bottom of the stack, but our proposed hardened profile with targets/desktop on top would give

 dev-libs/libxml2-2.9.1-r1:2  USE="ipv6 readline ... -python ...

So, both choices for orderings of “..” and “targets/desktop” in our parent file for hardened/linux/amd64/desktop lead to situations where we can’t control what packages get what use flags. What we would like is a stacking that looks something like this

...
/usr/portage/profiles/targets/desktop
/usr/portage/profiles/hardened/linux/amd64
/usr/portage/profiles/hardened/linux/amd64/destkop

but how do we get that with our current inheritance mechanism? One idea that Magnus (Zorry) had was to gutt out this portion of portage and replace the parsing of the parent file with something along the lines of openrc’s depend() { … } clause. Then we just locally say what has to come before/after what and we let the algorithm figure it out. It sounds like an interesting problem if there were two of me and if there were a good chance that it would actually get implemented. In the mean time, we limp along with what we have and do ad hoc fixes as changes in one part of the profiles means we have to adjust other things. Since we are all responsible for different areas of the tree’s profiles, inevitably we cause one another breakage even with the best of intentions. For example, a few days ago, Mike (vapier) removed a masking on the uclibc USE flag in the base profile.  Doing so makes perfect sense. He didn’t tell me, and why should he have to?, but this lead to a small breakage in hardened/linux/uclibc/amd64 and friends where I had to relax that masking.  I only discovered this upon a catalyst run which is a bit annoying.

Lilblue: a security-enhanced, fully featured XFCE4 desktop for amd64, built on uClibc

Hi everyone,

I’d like to announce a new (fun?) initiative of the hardened uClibc subproject: a security enhanced, fully featured XFCE4 desktop for amd64, built on uClibc, codenamed “Lilblue”, after the little blue penguin of New Zealand [1], a smaller cousin of the Gentoo.

The hardened uClibc subproject aims at producing hardened stage3s for amd64, mips (isa=mips32r2/mipsel3, abi=o32), armv7a (softfloat) and i686 [2].  Recent improvements in uClibc and bugfixes in various Gentoo packages, both downstream and upstream, now make it possible to build an entire desktop system replacing glibc with uClibc.  So, in addition to the stage3s, we are now releasing a fully featured XFCE4 desktop for arch=amd64.  It does *not* depend on busybox to provide its core utilities like most uClibc systems, but coreutils, util-linux and all the usual system packages you find on a generic Gentoo system.  The tarball bundles about 800 packages including ephiphany, claws, hexchat, abiword, gqview, transmission, vinagre, etc.  We have plans to provide binpkgs for up to 7000 packages in all.  The hardening includes all of the usual toolchain and kernel hardening you get in regular hardened glibc-based Gentoo.

The project has been in development for a year but should be considered experimental.  A user base of … uhm … one … does not really qualify it to be labeled as “safe for production” [3]. However, I have had no issues with it (minor bugs of course) and I use it on a daily basis.  For the average user, the main advantage is speed and the system does feel “snappy”.  For developers, its fun to dig into bugs which revolve around what functions are provided by your standard C lib: is this POSIX or a GNU-ism?  should I fix the package or add a new function to uClibc?  what is the best way to implement this fix so it ports across different *libcs?  what do I do about this package whose build system is braindead and doesn’t understand libdir?  If you have too much time on your hands and you’re into that kind of “fun” we have a project for you!  On a serious note, the main reason for this initiative is to explore and expand the usefulness of an alternative standard C library.

The home page is at [4] and a freecode.com announcement at [5].  It can be downloaded from any gentoo mirror [6] at [mirror]/gentoo/experimental/amd64/uclibc/desktop-amd64-uclibc-hardened-[date].tar.bz2. The date of the first release is 20130531.

Ref.
[1] https://en.wikipedia.org/wiki/Little_Penguin
[2] http://www.gentoo.org/proj/en/hardened/uclibc/index.xml
[3] This is not entirely true.  I would like to thank my students for testing, especially Devan Franchini <twitch153@hotmail.com>.
[4] http://www.gentoo.org/proj/en/hardened/uclibc/lilblue.xml
[5] https://freecode.com/projects/lilblue-linux
[6] http://www.gentoo.org/main/en/mirrors2.xml

Some love for mips64el: updates for an XFCE4 Gentoo Desktop on the Lemote Yeeloong

Hi everyone!

Back in September of last year, I pushed out two images for xfce4-based desktop systems for the Yeeloong netbook.  These mips64el machines are manufactured by Lemote in China, and distributed in the US by “Freedom Included“.  Over the summer, I spent time fixing the bugs I hit as I ported over goodies for the desktop: Abiword for word prcessing, GQview for images, Audacity for audio, Claws for email and Dillo for browsing.  Most of these bugs were in build systems that were not multilib aware.  I was building for three ABI’s, o32, n32 and n64 with libraries living in /lib, /lib32 and /lib64, respectively, and similarly under /usr.  Since some build systems would blindly install into /lib and since the default ABI is n32, this caused some issues!

As mentioned, there are two images, one for binaries built with a hardened toolchain (SSP, PIE, FORTIFY_SOURCES=2, bind_now and relro) and the other for binaries built with a vanilla toolchain.  Remarkably, hardening led to no appreciable bugs over the vanilla, but there is about a 5% performance hit when running.  This is most noticeable when compiling which is already slow on these 800-900 MHz machines.  In fact, this is why I decided to distribute entire desktops and not just stage3′s.  It would take a user about two days of no stop building to get to the desktop from a stage3.  Given the occasional failure along the way, this is a recipe for frustration for anyone who wants Gentoo on their Yeeloong.  Hopefully the end user won’t be too unhappy with my choice of software.  In the future, I hope to have a better brower than Dillo and try go get video working.  I’ve had some success with the latter but not enough to push something out.

Anyhow, four months after my initial images and I thought its time to update.  Except for updating the packages, I didn’t change the selection.  My update to the kernel, however, was important: we can now do the usual partitioning suggested in the Gentoo Handbook: boot on sda1 with ext2, swap on sda2 and root on sda3 with ext4.  My first images had to combine boot and root into one ext3 partition because of certain limitations.

I hope if you have the hardware you’ll give the images a test drive and let me know!  They’re available on any Gentoo mirror under <mirror>/experimental/mips/desktop-loongson2f.  The latest version is 20130124 and the instructions for installation are found here.

 

A Gentoo Desktop for the Lemote Yeeloong, a little endian mips64 netbook

Hi everyone,

I’d like to announce a new initiative within the mips arch team. We are now supporting an xfce4-based desktop system for the Lemote Yeeloong netbook.  The images can be found on any gentoo mirrors, under gentoo/experimental/mips/desktop-loongson2f.  The installation instructions can be found here.  The yeeloong netbook is particularly interesting because it only uses “free” hardware, ie. hardware which doesn’t require any proprietary code.  It is manufactured by Lemote in China, and distributed and promoted in the US by “Freedom Included“.  It is how Richard Stallman does his computing.

I’m blogging because I thought it was important for Planet Gentoo to know that mips devices are currently being manufactured and used in netbooks as well as embedded systems.  The gentoo mips team has risen to the challenge of targetting these systems and maintaining natively compiled stage4′s for them.  Why stage4′s?  And why a full desktop for the yeeloong?  These processors are slow, so the time from a stage3 to a desktop is about three days for the yeeloong.  Also, the yeeloong sports a little endian mips64 processor, the loongson2f, and we support three ABIs: o32, n32 and n64, with n32 being the preferred.  This significantly increases the time to build glibc and other core packages.  I provide two images, a vanilla one and a hardened one.  The latter adds full hardening (pie, ssp, _FORTIFY_SOURCES=2, bind now, relro) to the toolchain and userland binaries as we do for amd64 and i686 in hardened gentoo.  I have not ported over the hardened kernel, however.

I allude above to “other” targetted devices.  I am also maintaining some mips uclibc systems (both hardened and vanilla) which are on the gentoo mirrors under experimental/mips/uclibc.  But I will speak more of these later as part of an initiative to maintain hardened uclibc systems on “alternative” architectures such as arm, mips, ppc as well as amd64 and i686.

You can read the full installation instructions, but here’s a quick summary, since it doesn’t follow the usual Gentoo method of starting from a stage3:

  • Prepare either a pen drive or a tftp server with a rescue image: netboot-yeeloong.img
  • Turn on the yeeloong and hit the Del key multiple times until you get the firmware promt: PMON>
  • If netbooting, add an IP address and point to the netboot-yeeloong.img.  If using a pen drive then point to thei image on the drive and boot into the rescue environment.
  • Partition and format the drive.
  • Download the desktop image from a mirror via http or ftp.  Its about 350 MB in size.
  • Unpack the image.  It contains not only the userland, but also a kernel.
  • Reboot to the PMON> prompt.  Aim to the kernel on the drive.  PMON will remember your choice and you will not have to repeat this step.

Once installed, you will log in as an ordinary user with sudo with username and password = “gentoo”.  The root password is also set to “root”.  It is an ordinary Gentoo system, so edit your make.conf, emerge –sync and add whatever packages you like!  File bugs to: blueness@gentoo.org with a CC to mips@gentoo.org.

If you have a Yeeloong or go out and buy one, consider trying out this image.