Gentoo on Macbookpro 2013 – part2

Go read here for the first part.

What changed

  • The linux 3.13 still doesn’t sport support for the wireless yet. The closed source driver works almost great beside when conman crashes horribly due some bad interaction, luckily doesn’t happen often, sadly I do not have time to debug it.
  • Using grub with the patch pointed in the comments here does make appear the intel gpu and you can enjoy using it for hardware decoding using QSV, patches for Libav availabe in the usual place.
  • vga switcheroo doesn’t let me switch properly, apparently nouveau takes the console framebuffer and does not really wants to release it.
  • the nvidia closed source driver works but you lose the access to the gmux, so you can’t change the brightness, and your console framebuffer is gone as well.
  • bbswitch seems confused enough to give up, I might ask upstream to help me figure out since seems almost everything is there post-grub-setup.
  • My pommed patches are waiting for more testers, then I’ll bake a release for everybody. Here it is working nicely.

What next

  • Probably I’ll try to figure out better how get the intel gpu work fully, since nouveau works mostly fine but Blizzard games on wine do not play at all.
  • Pommed will see more cleanups.
  • Hopefully I’ll play more with the displayports.

So far I’m still quite happy about this model even if the mentioned quirks.

Security & Fuzzing

New year, new bugs and, since apparently lots of people are interested,
new posts about security.

The main topic is obviously libav and the bugs we are fixing here and there thanks to Mateusz and Gynvael kindly providing us fuzzed samples.

Fuzz testing

Many programs expect a certain input and provide a certain output, most of the time you miss a corner case and it leads to unexpected situations.
Fuzzing is one of the most effective black-box testing testing technique and in case of complex input (such as multimedia protocols and codecs) it does wonders spotting unhandled or mishandled conditions.

We are keeping a page about the tools useful to track bugs, since, unluckily, for us most bugs are security issues.

Fuzz testing is tersely explained there and the tools useful for the task are all there. We had a Google Code In mostly devoted in spotting crashes using zuff.

Sadly fuzzing using zuff is time consuming and requires a decent amount of cpu since even AddressSanitizer is relatively slow and in many cases you want to use valgrind: memory leaks are a security issue as well.

Google

Google is using Libavformat and Libavcodec in many projects and last year they started to share with us the results of their huge fuzzing system, what would take to me probably years takes them few days at most.

Before, outside Google developers, just Michael Niedermayer had access to the samples and since we usually do not agree on how to solve problems had been quite a problem figuring out the real problem from his patches and fix it for real.

Now things are quite better and I had the chance to get some feedbacks about new code (such as vp9) before having it landing in the main tree.
We could spot a couple of issues during review and with zuff we could spot some more, Google fuzzing found twice as many. That gives you an idea on how useful this kind of activity is for our code. Thinking about corner cases in complex code is HARD.

Fixing security issues

Initially it was painful, you get a huge amount of samples and you have to run them through avconv instrumented with valgrind or such tools (drmemory, asan, msan), then figure out where the problem is and hopefully fix it. Doing it manually can be tedious.

You need some form of coordination so people can work on different issue and not stomp of each other feet.

Automation

Currently our setup is a bit more organized, we have a central place in which some nice scripts to triage and categorize the samples and provide a sort of nice report with a per-codec and per-format breakout. Me, Martin, Diego, Anton and other interested parties have access to the samples and the scripts so we can work together having 1/2 of the time consuming and boring part done once and for everybody, probably soon I’ll extend it to be even more smarter and have some bug aggregation heuristic.

Integration

Valgrind integrates with gdb quite well, AddressSanitizer more or less on the same level with some few lines of .gdbinit to make the whole experience
smooth. Currently I’m mostly using asan with gcc-4.8* and I’m looking forward to see new drmemory releases since it seem quite promising.

Valgrind is used mostly to make sure memory leaks hadn’t been left around once all the asan-reported issues got fixed.

Fixes and Reviews

One of the annoying problems in fixing security issues is that you first see where it breaks but maybe the reason why is FAR from there.

Usually you might rush and just fix the damn bug where it breaks, it can be as wrong as using duct tape to plug an hole in the ceiling, sure it won’t drip on you from there, but if you don’t go and follow the plumbing or check the roof you never know what will happen next.

You might had spent already an hour sifting through gdb and error logs and you can’t spot a better place where to fix and since it isn’t a job you devote just enough time.

There is where usually reviews shine: having more than a pair of (tired) eyes helps a lot and getting people to take over from where you left and get something better quite good.

Releases

One of the nice perks of the current automated system is that is quite easy to check if the problems are present also in our current supported release branches. Backporting patches is yet another time consuming task and Reinhart, our release manager, couldn’t do that for the past point releases so Sean, Martin and I took the interim for that.

So far

The total amount of samples received is over 1600 of which 240+ are new samples triggering issues in hevc (patches for fixing all of them are already on review luckily).

There are less than 300 samples still waiting for a fix lots of them involve some of the ugliest and oldest lines of our codebase.

Luckily I’m not alone and hopefully in the process we’ll also freshen code untouched since ages and look at how naive we were when we wrote it.

Gentoo on Macbookpro 2013

I eventually decided to buy a newer laptop and ended up pick the latest from apple since looked nice.

Installing

Setting it up is sort of easy with just a couple of caveats

Boot from sysrescuecd

That’s my simplest way to get stuff done, fetched the latest 4.0 beta, installed the minimum needed to replace the kernel with one not sporting the dir entry bug and rebooted. The sysrescuecd standard configuration works decently, you might want to also fetch broadcom-sta to get wireless working.

Disable NCQ

The SSD seems to have problems with the ncq support at least on linux 3.12.2, I patched my 3.12.3 to blacklist APPLE* drives directly,

# echo 1> /sys/block/sda/device/queue_depth

Bootloader

I use refind, it magically finds the kernel and passes the correct root=/dev/foo option, so a no brainer.

X11

I was curious thus installed bumblebee and apparently it works out of box, you need to use mtrack to get decent touchpad support and you are pretty much done.

HiDPI

You might want to set your dpi to at least 110 if you are using GTK applications (not sure if QT has a scaling option, Elementary does and so E17)

Audio

Seems working just fine.

What does not work yet

Webcam

It seems sdio-connected.

b43

The opensource driver does not work, the closed source one works fine.

Pommed

In my github you can find a tentative and not so complete patch, out of box pommed refuses to work and with the patch at least the backlight and keyboard light sort of work.

The Pink Pony

That’s what happens LOADS of time while discussing requirements.

<P O> I want a Pink Pony!

<P M> Ok, what that pony is supposed to do? Does it walk? Does it run? Which kind of user would ride it?

<P O> Oh well, not sure, a pony is cool and cute that’s what important! Ask Marketing for the details.

<P M> According to the information from marketing and design the average use would be a big adult, carrying swords, pikes and shield called Knight. You need a Steed not a pony.

<P O> But but, PINK, my daughter loves pink!

<P M> According to the current poll among the Knights the top colour is pitch black. Your target user wants a Black Steed.

(A longer version will appear sooner or later)

Some ideas regarding / and /usr

What is the / ?

On unix-like system we have the wonderful abstration that let us keep on a single hierarchy all the file systems, it is called mount: we take the logic tree contained in a filesystem and we graft it over another. The initial file system holding all those grafts is the root file system and it lives on the mount point /. Let’s call it rootfs, it usually contains everything needed to start the system.

What (the hell) is /usr ?

Back in the old times where space was constrained the rootfs (/) started to get too big, so, as quick workaround some stuff deemed not fundamental moved on a separate fs mounted as /usr. The quick workaround proved itself useful so started to remain permanent with few exceptions (Hi Hurd, had you been rewritten from scratch this month already again?).

Split-/usr mount, what is the deal?

Linux had since long a quite useful volume-manager, it let you do a number of increasingly complex transformation over storage remaining nearly-agnostic regarding the file system. Being able to extend your storage by adding disks and merging them in a single logical storage seems useful, so is having software-mediated RAID setups.

Gentoo had since long a tutorial how to split all the rootfs mountpoints in different volumes. The idea makes sense or not depending on your tastes. Many people liked it and used it.

The rootfs contained a statically linked set of binaries useful to mount /usr and that was all of it. We can call this set of tools nowadays early-boot tools and the partition holding them early-bootfs. The two usually are the same.

Once /usr is mounted the boot process can start anything else.

Your problem now is figure out what is needed to boot /usr:

  • A volume can be a logical one created by a volume-manager
  • The file system could be a module and the module could be compressed by something the kernel doesn’t understand by itself
  • The file system can be userspace (thanks to fuse) and it could be implemented by an interpreted language such as python
  • Volume and filesystem can be networked, thus you need to bring your network up and the network could require additional components.
  • If you are into crypto, again it could be something mediated by the userspace at volume and file system level.

The possibilities are many and if you want to support them all no matter how unlikely it looks a complex problem.

Everything is broken let’s break it some more!

Some loud mouthed people decided to go against one of the key design item in writing a component such as an init system, that is keeping the single point of failure as simple as possible to reduce the chance it fails.

They kept adding compulsory dependencies to it that use to live on /usr. I think that as a way to get a problem so his arguably half baked solutions can be sell as only future and to make the most annoying situation in the previous paragraph the default.

The rational reaction would had been to just tell them to keep playing with their broken toys in a corner.

Initramfs

So we have this problem, a arbitrary long list of components that would make the rootfs large and some genius actively tries to have the first program started require most of it and shovel the concept down everybody else.

One solution would be just merge rootfs, early-init-bootfs and the whole /usr together somehow, welcome back to the the early 1900! (incidentally you will need also /var mounted but that’s a digression)

Obviously the problems causing the initial split-/usr are still there.

Linux has a neat feature called initramfs, the successor of initrd, it is great to keep modules and all the stuff you might need to mount your rootfs in a place the kernel could always reach no matter what.

So a solution would be keeping all that’s needed to mount the rootfs-now-merged-with-/usr in the initramfs since by definition is always reachable.

It is not exactly the most elegant solution but arguably works as long you get the list of required component right.

The elephant in the /boot

Some rhetoric questions:

  • “The initramfs is somehow related to its kernel, what happens if you keep more than 1 kernel around?”
  • “Which is the sane size limit for it?”
  • “Initramfs can get stale easily, how much time takes to create it and keep it up to date?”

The answers might vary. The short is that you need good tools and lots of space.

Alternatives

You need good tools and good knowledge about what you need for your early boot, you have to put it somewhere and keep it up to date easily. Possibly it shouldn’t depend on your kernel yet be easy to access it.

/boot as early-boot partition

That is one of the simpler ideas, we just keep a separate copy of what is needed /boot, historically most concerned people kept a recovery there so makes sense for them use it as early-boot.

Static and restrict rootfs

If you know what you are doing as long you can keep in your rootfs your tools by linking them statically (so the whole deal about compressed modules is taken care of) and you aren’t using strange stuff (so just lvm and normal fs), you do not care about this whole deal. AS LONG YOUR DISTRIBUTION DOESN’T PLAY GAMES. Nor you drink the kool-aid and use stuff that breaks by design static linking or makes as hard as possible keeping a minimal amount of stuff in the rootfs.

Summary

We always need your help and feedback to make so Gentoo keeps giving you good options and currently working systems keep working in the next future. Thanks for reading.

Early boot fun

Just few notes spurted from a discussion with a friend regarding why he feels we suck badly.

Early boot

Let’s make a quite rough description on how booting could work:

  • Imagine you are the kernel, you just found your rootfs, managed to run your init and you are happy. That’s probably the earliest we care.
  • Init got called and starts running some scripts, maybe checking the rootfs consistency before remounting r/w, maybe checking the other essential mount points before mounting them or maybe start the device manager first and then checking what is going to mount, assuming what is essential still requires some modules loaded and that the device manager will do.
  • Move further and set up the network
  • Maybe now mount the mount points that require the network (nfs?)
  • Now get the other daemons up and running, maybe in parallel, maybe bring up some graphical login.

Now let’s see who are the actors: rootfs, init, device-manager and maybe incidentally volume-manager and networking.

Ideally your rootfs should contain

  • anything required by init to run, easy, init should be as small as possible to make sure this single point of failure really hard to fail.
  • anything required by the device-manager to load modules, should be a no brainer, well, maybe if you want your modules compressed with some new or exotic compressor because it is “faster” that way, you have to fit it in the rootfs.
  • If your essential mountpoints require a volume-manager the same applies, lvm can require something weird depending on the setup so either you link it statically or you have to put it again where it is reachable, same could be said for any kind of advanced crypto at volume leve.
  • We discussed about mount and again we could have fuse-fs using a scripting language or other stuff that make issuing mount a little more complex that we would expect (and again fs-level crypto happening in the userspace)
  • The network would just need some modules loaded right? Wrong, it might need some special daemons doing any kind of bizantine authentication, and if you are really looking for pain you could be willing to netmount those mentioned file systems or even do volume management over bizantine network (ok there is a limit in this kind of perversion and we are just halfway).
  • Once everything is mounted the rest of the system can be brought up w/out much qualms.

So in the end your rootfs can be quite fat contain full copies of python so you can mount that funny file system, have lots of lovely brittle deps because you thought NetworkManager is the only way to get the network up and meanwhile that having some important stuff (e.g. /var) netmounted is all the rage.

Fun (aka pain)

So in short your rootfs could be as big as a compact live distribution and have as many moving parts as one (or more), well it could be just your distribution if you do not keep everything in a separated mount point.

Some years ago that was one of the suggested ways, you keep essential stuff in / and then every other root mountpoint would have its partition, maybe using some advanced stuff just because.

Then you get told that the right place in which you have to fit all discussed above has to be something called initramfs and obviously tell the kernel about it.

Probably nobody would be that crazy to end up with the far corner case, so the initramfs would have to copy just few (20+??) libraries and some (30+?!) binaries in the normal case and you have to keep it synced up properly (joy).

Most people could live happy with just a statically linked lvm and udev living in a small partition easy to mount and that would be the start and the end for them, but certain wise guys will tell you that static linking is harmful, the whole concept is broken anyway since our bluetooth subsystem requires lots of userspace and then you’d be w/out a keyboard in case something goes wrong (so you should shove bluez and happy deps in your initramfs/rootfs?).

Summing up

There are easy, simple and working solution for just some realistic scenarios, but not covering everything that’s possible.

There are more complex, brittle and error prone ones that might cover everything and more (and maybe still fail in some basic situations).

The fact lots of lemmings flocks over the complex/brittle because the guy with the largest mouth is the best speaker is sad, overly sad.

That said if you were happily using since 10 years ago a lvm setup as described by our guides of the time and now you are afraid that your next userspace update your system will break horribly if you don’t go through the hoops of making an initramfs, that won’t work for you out of box and will force you to modify your bootloader or do some other time consuming work:

I’m sorry.

(Luckily somebody prepared a portage hook to prevent some breakages https://gist.github.com/mansr/7289969 not all of them but the most glaring are covered)

Incidentally, you can still pester us, help us getting better programs (e.g. contribute to eudev, kmod, lvm and everything else you use) and take an active part in the community and hopefully protect your simple and working solutions.

Trades

The code must be properly formatted, nice to read. You must have testcases.

  • Always start with simple even dumb code. (e.g. as is written in the spec you write it down no matter how stupid or inefficient)
  • Trade simplicity/clarity for speed, if the gain is big. (e.g. move from the letter-of-the-spec-matching code to something actually faster, sometimes also clarity and simplicity gain from it)
  • Trade space for speed, if the gain is big enough. (e.g. a lookup table usually is a nice solution, sometimes for a different problem)
  • Trade precision for speed if you must, always leave a codepath that isn’t imprecise.
  • Never trade portability, but you might trade slower generic code for faster specialized code for all the platform implementing it. (e.g. implement in asm a function that was inlined before, the plain C code would be slower since you have a call overhead while the asm-optimized version would be 16-fold faster than C)

As Kostya pointed, sometimes you start with a binary specification so point 1 is moot.

Security fun – what’s security?

Since I eventually had access to a batch of broken samples from Google, I spent the past months volunteering time to fix in Libav the issues uncovered (the whole set is over 3000 samples), you probably noticed by the number of releases.

You can consider “security” issues pretty much any kind of bug:

  • A segfault is a security issue.
  • A read/write from not allocated memory is a security issue.
  • An assert triggered IS a security issue and not a way to fix them.
  • A memory leak is a security issue and in most cases the worst kind.

Your security concern is not the same as mine!

Libav has a large surface to attack since you have decoders for every kind of multimedia format, it is a library used in many different situations, what’s a security concern for somebody is a nuisance for somebody else.

If VLC breaks on you when you are trying to decode some incomplete movie you got from bittorrent because one 0 or 1 got misinterpreted is not such an issue. If your transcoding pipeline gets stalled due the same movie being uploaded on Youtube, somebody might be screaming at the idiot that forgot to bound-check that array deep into the code.

If some buffer overflow could lead to code execution, most of the people using avconv to mass transcode won’t care that much, the process is fully sandboxed and they expect it, the people making players are mostly afraid of some buffer overflow being exploitable, their users would feel the pain.

So for us, Libav developers, there isn’t a bug more important or least important. We have to fix all of them and possibly fix them correctly once (so if you move from a buffer overflow to an assert, you just trade a possible code execution to a deny of service). That takes time and resources.

The source of all pain

Most of the bugs are naive assumptions and overlooks piling up over the years, the most common are the following

Off by one
You loop over something and you read one element too many
Corner cases
What happens when your frame has dimension 0? What if it is as large as the maximum representable value?
Faulty assumption
If you think that a malloc cannot fail, think again, if you think realloc won’t ever return NULL so you
can forget about the old pointer and just overwrite it, please DO think again. It can happen, even on Linux
Sloppy coding practices
Some bad practices tend to stick and bad patterns such as not forwarding return values will lead to problems later, usually making the process of tracking back to the root issue HARD.

Even if you are writing something non critical such a fire and forget commandline app you should be a little careful, if you plan to write something more involving such a library that could be used in MANY ways by LOTS of people, you MUST be careful.

Tools of the trade

Tracking bugs is usually annoying and time consuming, if they are crash they are at least apparent, memory leaks and faulty read/write may not trigger an apparent crash, making the whole thing more daunting. Luckily there are good tools help you.

Valgrind

The whole toolset is really valuable, massif and memcheck are the best to figure out where the memory went and who’s the fault.

AddressSanitizer

Asan is a boon since it is much faster than memcheck but also a pain since you have to instrument your code by using a certain compiler (clang or gcc-4.8 and later) and certain flags (-fsanitize=address). You can leverage it in gdb so you can inspect memory while debugging. That had been an huge timesaver most of the time. You can in theory do that also on memcheck adding some lines of code, probably I’ll provide snippets later.

drmemory

If your problem is on non-linux and non-mac you cannot use Asan and Valgrind, the new and coming tool to save you is drmemory. It is the youngest of the set and you can see how green it is by the lack of best practices… So no source releases, naive build system and bad version control system. If you try to build it is better to use the latest svn and hope.

Yet if you have to figure out what’s wrong on windows it is a huge boon already. People with time and will could try to help them on fixing their build system and convince them to move to git.

Automation

Never, ever, ever start hunting this kind of bugs w/out automating the most. Currently I have written a consistent number of lines of bash to automatically triage and check the samples, get the code to build in at least 2-3 flavours (clang and gcc with asan, vanilla gcc for valgrind) and eventually generate additional fate targets so I can run make fate-sec -C .gcc-asan and see if something that was fixed broke when we hadn’t look.

In closing

I still have 200 samples to fix and hopefully I’ll rally more people in helping, if you aren’t running routine tests and make sure your projects are at least valgrind clean (the easiest check to do), you should.

If you are writing code that is a little more critical, better if you use all the tools I briefly described and fix what you overlooked.

The case of defaults (Libav vs FFmpeg)

I tried not to get into this discussion, mostly because it will degenerate to a mud sliding contest.

Alexis did not take well the fact that Tomáš changed the default provider for libavcodec and related libraries.

Before we start, one point:

I am as biased as Alexis, as we are both involved on the projects themselves. The same goes for Diego, but does not apply to Tomáš, he is just a downstream by transition (libreoffice uses gstreamer that uses *only* Libav).

Now the question at hand: which should be the default? FFmpeg or Libav?

How to decide?

– Libav has a strict review policy every patch goes through a review and has to be polished enough before landing the tree.

– FFmpeg merges daily what had been done in Libav and has a more lax approach on what goes in the tree and how.

– Libav has fate running on most architectures, many of those are running Gentoo, usually real hardware.

– FFmpeg has an old fate with less architectures, many qemu emulations.

– Libav defines the API

– FFmpeg follows adding bits here and there to “diversify”

– Libav has a major release per season, minor releases when needed

– FFmpeg releases a lot touting a lot of *Security*Fixes* (usually old code from the ancient times eventually fixed)

– Libav does care about crashes and fixes them, but does not claim every crash is a Security issue.

– FFmpeg goes by leaps to add MORE features, no matter what (including picking wip branches from my personal github and merging them before they are ready…)

– Libav is more careful, thus having less fringe features and focusing more polishing before landing new stuff.

So if you are a downstream you can pick what you want, but if you want something working everywhere you should target Libav.

If you are missing a feature from Libav that is in FFmpeg, feel free to point me to it and I’ll try my best to get it to you.

Again on shoveling stuff in other people mouth

Again we got a fun thread about having to do some extensive change on perfectly working systems because somebody has a *plan* and you must abide to it.

If before the plan was to have systemd as the true and only init system (on why systemd seems to me a bad idea by itself I’ll discuss on a later post, possibly after throughly study its latest iteration and comparing it), now the plan is to force people not to have a separate /usr or use an initramfs with an early boot system because… “because doing otherwise is broken and already had been in ages”.

That doesn’t tell you much and if you have lots of systems running perfectly on a separate /usr setup and you went that way because it was documented as a best practice, you might feel enraged.

Now, let’s make clear that there are operating systems that keep everything in /usr and have next to nothing in / (and system that do not have /usr at all and everything is in /), you can argue a lot about what’s the best and why. FreeBSD or Hurd approaches have both interesting perks.

The fact is that *now* you have lots of people with perfectly working system in a configuration somebody decided that is wrong and *unsupportable*.

If you try to dig down a bit more you’ll discover that the “brokeness” is mainly due:

  • Somebody keen in using a library that traditionally is in /usr for some fringe feature
  • Somebody hell bent to use glib everywhere
  • Somebody wanting to have d-bus running in the early boot phase
  • Some udev rules using some data that currently resides in /usr

All considered forcing people to spend lots of time because somebody might want to use a bluetooth keyboard on early boot (thus requiring bluez, thus requiring d-bus basically because you can’t use bluez without it) or other non widespread use case is not exactly nice.

Surely trying to get a cleaner layout so we have a bare mountpoint directory, a early boot system in initramfs and the rest of the system cleanly split isn’t bad by itself and probably it is something I would consider neat.

But you still need to have a good separation between what is early boot and what is not and you need to make sure the boot process doesn’t get too complex or too tightly coupled with systems that can and will break easily.

I’m quite happy that alternatives are already almost available for simple systems not needing the additional features requiring those extensive changes.

Hopefully somebody will have time to try to add rules marking in udev so complex rules won’t be triggered when the system isn’t ready for them and deploys using special layouts could stay supported in a way or another.

In the other news Gentoo had been accepted to participate to the Google Summer of Code and there are two projects proposed by me, one is about documenting and if needed extending openrc to be a complete viable alternative to systemd, the other about using containers and qemu-user to have better tools to do cross developement.