{"id":207,"date":"2015-08-04T12:58:04","date_gmt":"2015-08-04T12:58:04","guid":{"rendered":"http:\/\/blogs.gentoo.org\/blueness\/?p=207"},"modified":"2015-08-04T12:58:04","modified_gmt":"2015-08-04T12:58:04","slug":"alt-libc-the-state-of-uclibc-and-musl-in-gentoo-part-1","status":"publish","type":"post","link":"https:\/\/blogs.gentoo.org\/blueness\/2015\/08\/04\/alt-libc-the-state-of-uclibc-and-musl-in-gentoo-part-1\/","title":{"rendered":"alt-libc: The state of uClibc and musl in Gentoo (part 1)"},"content":{"rendered":"<p>About five years ago, I became interested in alternative <a href=\"https:\/\/en.wikipedia.org\/wiki\/C_standard_library\" target=\"_blank\">C standard libraries<\/a> like <a href=\"http:\/\/uclibc.org\/\" target=\"_blank\">uClibc<\/a> and <a href=\"http:\/\/www.musl-libc.org\/\" target=\"_blank\">musl<\/a> and their application to more than just embedded systems.\u00a0 Diving into the implementation details of those old familiar C functions can be tedious, but also challenging especially under constraints of size, performance, resource consumption, features and correctness.\u00a0 Yet these details can make a big difference as you can see from<a href=\"http:\/\/www.etalabs.net\/compare_libcs.html\" target=\"_blank\"> this comparison of glibc, uClibc, dietlibc and musl<\/a>.\u00a0 I first encountered libc performance issues when I was doing number crunching work for my ph.d. in physics at Cornell\u00a0 in the late 80&#8217;s.\u00a0 I was using IBM RS6000&#8217;s (yes!) and had to jump into the assembly.\u00a0 It was lots of fun and I&#8217;ve loved this sort of low level stuff ever since.<\/p>\n<p>Over the past four years, I&#8217;ve been working towards producing stage3 tarballs for both <a href=\"https:\/\/wiki.gentoo.org\/wiki\/Project:Hardened_uClibc\" target=\"_blank\">uClibc<\/a> and <a href=\"https:\/\/wiki.gentoo.org\/wiki\/Project:Hardened_musl\" target=\"_blank\">musl<\/a> systems on various arches, and with help from generous contributors (thanks guys!), we now have a pretty good selection on the mirrors.\u00a0 These stages are not strictly speaking embedded in that they do not make use of busybox to provide their base system.\u00a0 Rather, they employ the same packages as our glibc stages and use coreutils, util-linux, net-tools and friends.\u00a0 Except for small details here and there, they only differ from our regular stages in the libc they use.<\/p>\n<p>If you read my last blog posting on <a href=\"https:\/\/blogs.gentoo.org\/blueness\/2015\/07\/31\/the-gentoo-reference-system-suite-a-new-release-engineering-tool\/\" target=\"_blank\">this new release engineering tool I&#8217;m developing called GRS<\/a>, you&#8217;ll know that I recently hit a milestone in this work.\u00a0 I just released three hardened, fully featured XFCE4 desktop systems for amd64.\u00a0 These systems are identical to each other except for their libc, again modulo a few details here and there.\u00a0 I affectionately dubbed these <strong>Bluemoon<\/strong> for glibc, <a href=\"https:\/\/wiki.gentoo.org\/wiki\/Project:Hardened_uClibc\/Lilblue\" target=\"_blank\"><strong>Lilblue<\/strong><\/a> for uClibc, and <strong><a href=\"https:\/\/wiki.gentoo.org\/wiki\/Project:Hardened_musl\/Bluedragon\" target=\"_blank\">Bluedragon<\/a><\/strong> for musl.\u00a0 (If you&#8217;re curious about the names, read their homepages.)\u00a0 You can grab all three off <a href=\"http:\/\/dev.gentoo.org\/~blueness\/theblues\/\" target=\"_blank\">my dev space<\/a> , or off the mirrors under <a href=\"http:\/\/distfiles.gentoo.org\/experimental\/amd64\/\" target=\"_blank\">experimental\/amd64\/{musl,uclibc}<\/a> if you&#8217;re looking for just Lilblue or Bluedragon &#8212; the glibc system is too boring to merit mirror space.\u00a0 I&#8217;ve been maintaining Lilblue for a couple of years now, but with GRS, I can easily maintain all three and its nice to have them for comparison.<\/p>\n<p>If you play with these systems, don&#8217;t expect to be blown away by some amazing differences.\u00a0 They are there and they are noticeable, but they are also subtle.\u00a0 For example, you&#8217;re not going to notice code correctness in, say, pthread_cancel() unless you&#8217;re coding some application and expect certain behavior but don&#8217;t get it because of some bad code in your libc.\u00a0 Rather,\u00a0 the idea here is push the limits of uClibc and musl to see what breaks and then fix it, at least on amd64 for now.\u00a0 Each system includes about 875 packages in the release tarballs, and an extra 5000 or so binpkgs built using GRS.\u00a0 This leads to lots of breakage which I can isolate and address.\u00a0 Often the problem is in the package itself, but occasionally it&#8217;s the libc and that&#8217;s where the fun begins!\u00a0 I&#8217;ve asked Patrick Lauer for some space where I can set up my GRS builds and serve out the binpkgs.\u00a0 Hopefully he&#8217;ll be able to set me up with something.\u00a0 I&#8217;ll also be able to make the build.log&#8217;s available for packages that fail via the web, so that GRS will double as a poor man&#8217;s tinderbox.<\/p>\n<p>In a future article I&#8217;ll discuss musl, but in the remainder of this post, I want to highlight some big ticket items we&#8217;ve hit in uClibc.\u00a0 I&#8217;ve spent a lot of time building up machinery to maintain the stages and desktops, so now I want to focus my attention on fixing the libc problems.\u00a0 The following laundry list is as much a TODO for me as it is for your entertainment.\u00a0 I won&#8217;t blame you if you want to skip it.\u00a0 The selection comes from <a href=\"https:\/\/bugs.gentoo.org\/buglist.cgi?quicksearch=uclibc&amp;list_id=2865798\" target=\"_blank\">Gentoo&#8217;s bugzilla<\/a> and I have yet to compare it to <a href=\"https:\/\/bugs.uclibc.org\/buglist.cgi?bug_status=NEW&amp;bug_status=ASSIGNED&amp;bug_status=REOPENED&amp;product=uClibc&amp;query_format=advanced&amp;order=bug_id%20DESC&amp;query_based_on=\" target=\"_blank\">upstream&#8217;s bugs<\/a> since I&#8217;m sure there&#8217;s some overlap.<\/p>\n<p>Currently there are thirteen uClibc stage3&#8217;s being supported:<\/p>\n<ul>\n<li>stage3-{amd64,i686}-uclibc-{hardened,vanilla}<\/li>\n<li>stage3-armv7a_{softfp,hardfp}-uclibc-{hardened,vanilla}<\/li>\n<li>stage3-mipsel3-uclibc-{hardened,vanilla}<\/li>\n<li>stage3-mips32r2-uclibc-{hardened,vanilla}<\/li>\n<li>stage3-ppc-uclibc-vanilla<\/li>\n<\/ul>\n<p>Here hardened and vanilla refer to the <a href=\"https:\/\/wiki.gentoo.org\/wiki\/Project:Hardened\" target=\"_blank\">toolchain hardening<\/a> as we do for regular glibc systems.\u00a0 Some bugs are arch specific some are common.\u00a0 Let&#8217;s look at each in turn.<\/p>\n<p>* amd64 and i686 are the only stages considered stable and are distributed along side our <a href=\"http:\/\/distfiles.gentoo.org\/releases\/amd64\/autobuilds\/\" target=\"_blank\">regular stage3 releases in the mirrors.<\/a>\u00a0 However, back in May I hit <a href=\"https:\/\/bugs.gentoo.org\/show_bug.cgi?id=548950\" target=\"_blank\">a rather serious bug (#548950)<\/a> in amd64 which is our only 64-bit arch.\u00a0 The problem was in the the implementation of pread64() and pwrite64() and was triggered by <a href=\"http:\/\/git.kernel.org\/cgit\/fs\/ext2\/e2fsprogs.git\/commit\/?id=f00948ad1df100c7d616ef6fbf7609329a2e4001\" target=\"_blank\">a change in the fsck code with e2fsprogs-1.42.12<\/a>.\u00a0 The bug led to data corruption of ext3\/ext4 filesystem which is a very serious issue for a release advertised as stable.\u00a0 The problem was that the wrong _syscall wrapper was being used for amd64.\u00a0 If we don&#8217;t require 64-bit alignment, and you don&#8217;t on a 64-bit arch (see <a href=\"http:\/\/git.uclibc.org\/uClibc\/tree\/libc\/sysdeps\/linux\/x86_64\/bits\/uClibc_arch_features.h#n14\" target=\"_blank\">uClibc\/libc\/sysdeps\/linux\/x86_64\/bits\/uClibc_arch_features.h<\/a>), then you need to use _syscall4, not _syscall6.<\/p>\n<p>The issue was actually fixed by Mike Frysinger (vapier) in uClibc&#8217;s git HEAD but not in the 0.9.33 branch which is the basis of the stages.\u00a0 Unfortunately, there hasn&#8217;t been a new release of uClibc in over three year so <a href=\"http:\/\/git.uclibc.org\/uClibc\/commit\/?h=0.9.33&amp;id=0aaf783f8d0c2748aee458ecd5b846be595c3068\" target=\"_blank\">backporting<\/a> meant disentangling the fix from some new thread cancel stuff and was annoying.<\/p>\n<p>* The armv7a stages are close to being stable, but they are still being distributed in the <a href=\"http:\/\/distfiles.gentoo.org\/experimental\/arm\/\" target=\"_blank\">mirrors under experimental<\/a>.\u00a0 The problem here is not uClibc specific, but due to hardened gcc-4.8 and it affects all our hardened arm stages.\u00a0 With gcc-4.8, we&#8217;re turning on <a href=\"https:\/\/gcc.gnu.org\/onlinedocs\/gccint\/Stack-Checking.html\" target=\"_blank\">-fstack-check=yes<\/a> by default in the specs and this breaks alloca().\u00a0 The workaround for now is to use gcc-4.7, but we should turn off -fstack-check for arm until bug #<a href=\"https:\/\/bugs.gentoo.org\/show_bug.cgi?id=518598\" target=\"_blank\">518598<\/a> &#8211; (<a href=\"https:\/\/gcc.gnu.org\/bugzilla\/show_bug.cgi?id=65958\" target=\"_blank\">PR65958<\/a>) is fixed.<\/p>\n<p>* I hit this one weird bug when building the mips stages, bug #<a href=\"https:\/\/bugs.gentoo.org\/show_bug.cgi?id=544756\" target=\"_blank\">544756<\/a>.\u00a0 An illegal instruction is encountered when building any version of python using gcc-4.9 with -O1 optimization or above, yet it succeeds with -O0.\u00a0 What I suspect happened here is some change in the optimization code for mips between gcc-4.8 and 4.9 introduced the problem.\u00a0 I need to distill out some reduced code before I submit to gcc upstream.\u00a0\u00a0 For now I&#8217;ve p.masked &gt;=gcc-4.9 in default\/linux\/uclibc\/mips.\u00a0 Since mips lacks stable keywords, this actually brings the mips stages in better line with the other stages that use gcc-4.8 (or 4.7 in the case of hardened arm).<\/p>\n<p>* Unfortunately, ppc is plagued with bug #<a href=\"https:\/\/bugs.gentoo.org\/show_bug.cgi?id=517160\" target=\"_blank\">517160<\/a>.\u00a0 <a href=\"http:\/\/www.openbsd.org\/papers\/nycbsdcon08-pie\/\" target=\"_blank\">PIE code<\/a> is broken on ppc and causes a seg fault in plt_pic32.__uClibc_main ().\u00a0 Since PIE is an integral part of how we do hardening in Gentoo, there&#8217; s no hardened ppc-uclibc stage.\u00a0 Luckily, there are no known issues with the vanilla ppc stage3.<\/p>\n<p>Finally, there are five interesting bugs which are common to all arches.\u00a0 These problems lie in the implementation of functions in uClibc and deviate from the expected behavior.\u00a0 I&#8217;m particularly grateful to <span class=\"vcard\"><a class=\"email\" title=\"Ren\u00e9 Rh\u00e9aume &lt;rene.rheaume@gmail.com&gt;\" href=\"mailto:rene.rheaume@gmail.com\" target=\"_blank\"><span class=\"fn\">Ren\u00e9 Rh\u00e9aume<\/span><\/a><span class=\"fn\"> who found them by running the test suite for various packages.<\/span> <\/span><\/p>\n<p>* <a href=\"https:\/\/bugs.gentoo.org\/show_bug.cgi?id=527954\" target=\"_blank\">Bug 527954<\/a> &#8211; One of wget&#8217;s tests makes use of fnmatch(3) which intelligently matches file names or paths.\u00a0 On uClibc, there is an unexpected failure in a test where it should return a non-match when fnmatch&#8217;s options contains FNM_PATHNAME and a matching slash is not present in both strings.\u00a0 Apparently this is a duplicate of a really old bug<a class=\"bz_bug_link&lt;br \/&gt;&lt;br \/&gt;\n          bz_status_CONFIRMED \" title=\"CONFIRMED - uclibc performs fnmatch(..., ..., FNM_PATHNAME) incorrectly when slash only in string and not pattern\" href=\"https:\/\/bugs.gentoo.org\/show_bug.cgi?id=181275\"> (#181275).<\/a><\/p>\n<p>* <a href=\"https:\/\/bugs.gentoo.org\/show_bug.cgi?id=544118\" target=\"_blank\">Bug 544118<\/a> &#8211; <span class=\"vcard\"><span class=\"fn\">Ren\u00e9 noticed this problem in an e2fsprogs test for libss.so.\u00a0 The failure here is due to the fact that <\/span><\/span> setbuf(3) is ineffective at changing the buffer size of stdout if it is run <em>after<\/em> some printf(1).\u00a0 Output to stdout is buffered while output to stderr is not.\u00a0 This particular test tries to preserve the order of output from a sequence of writes to stdout and stderr by setting the buffer size of stdout to zero.\u00a0 But setbuf() only works on uClibc if it is invoked <em>before<\/em> any output to stdout.\u00a0 As soon as there is even one printf(), all invocations to setbuf(stdout, &#8230;) are ineffecitve!<\/p>\n<p>* <a href=\"https:\/\/bugs.gentoo.org\/show_bug.cgi?id=543972\" target=\"_blank\">Bug 543972<\/a> &#8211; This one came up in the gzip test suite.\u00a0 One of the tests there checks to make sure that gzip properly fails if it runs out of disk space.\u00a0 It uses \/dev\/full, which is a pseudo-device provided by the kernel that pretends to always be full.\u00a0 The expectation is that fclose() should set errno = ENOSPC when closing \/dev\/full.\u00a0 It does on glibc but it doesn&#8217;t in uClibc.\u00a0 It actually happens when piping stdout to \/dev\/full, so the problem may even be in dup(2).<\/p>\n<p>* <a href=\"https:\/\/bugs.gentoo.org\/show_bug.cgi?id=543668\" target=\"_blank\">Bug 543668<\/a> &#8211; There is some problem in uClibc&#8217;s dopen()\/dlclose() code.\u00a0 I wrote about this in a <a href=\"https:\/\/blogs.gentoo.org\/blueness\/2014\/12\/16\/lilblue-linux-release-20141212-dlclose-is-a-problem\/\" target=\"_blank\">previous blog post<\/a> and <span class=\"bz_comment_user\"><span class=\"vcard\"><a class=\"email\" title=\"David Flogeras &lt;dflogeras2@gmail.com&gt;\" href=\"mailto:dflogeras2@gmail.com\"> <span class=\"fn\">David Flogeras<\/span><\/a> also hit it with syslog-ng&#8217;s plugin system.\u00a0 A seg fault occurs when unmapping a plugin from the memory space of a process during do_close() in uClibc\/<\/span><\/span>ldso\/libdl\/libdl.c.\u00a0 My guess is that the problem lies in uClibc&#8217;s accounting of the mappings and when it tries to unmap an area of memory which is not mapped or was previously unmapped, it seg faults.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>About five years ago, I became interested in alternative C standard libraries like uClibc and musl and their application to more than just embedded systems.\u00a0 Diving into the implementation details of those old familiar C functions can be tedious, but also challenging especially under constraints of size, performance, resource consumption, features and correctness.\u00a0 Yet these &hellip; <a href=\"https:\/\/blogs.gentoo.org\/blueness\/2015\/08\/04\/alt-libc-the-state-of-uclibc-and-musl-in-gentoo-part-1\/\" class=\"more-link\">Continue reading<span class=\"screen-reader-text\"> &#8220;alt-libc: The state of uClibc and musl in Gentoo (part 1)&#8221;<\/span><\/a><\/p>\n","protected":false},"author":141,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"spay_email":"","jetpack_publicize_message":"","jetpack_is_tweetstorm":false,"jetpack_publicize_feature_enabled":true},"categories":[1,3],"tags":[],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","_links":{"self":[{"href":"https:\/\/blogs.gentoo.org\/blueness\/wp-json\/wp\/v2\/posts\/207"}],"collection":[{"href":"https:\/\/blogs.gentoo.org\/blueness\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blogs.gentoo.org\/blueness\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blogs.gentoo.org\/blueness\/wp-json\/wp\/v2\/users\/141"}],"replies":[{"embeddable":true,"href":"https:\/\/blogs.gentoo.org\/blueness\/wp-json\/wp\/v2\/comments?post=207"}],"version-history":[{"count":11,"href":"https:\/\/blogs.gentoo.org\/blueness\/wp-json\/wp\/v2\/posts\/207\/revisions"}],"predecessor-version":[{"id":218,"href":"https:\/\/blogs.gentoo.org\/blueness\/wp-json\/wp\/v2\/posts\/207\/revisions\/218"}],"wp:attachment":[{"href":"https:\/\/blogs.gentoo.org\/blueness\/wp-json\/wp\/v2\/media?parent=207"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blogs.gentoo.org\/blueness\/wp-json\/wp\/v2\/categories?post=207"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blogs.gentoo.org\/blueness\/wp-json\/wp\/v2\/tags?post=207"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}