Security and Tools

Everybody should remember than a 100% secure device is the one unplugged and put in a safe covered in concrete. There is always a trade-off on the impairment we inflict ourselves in order to stay safe.

Antonio Lioy

In the wake of the heartbleed bug. I’d like to return again on what we have to track problems and how they could improve.

The tools of the trade

Memory checkers

I wrote in many places regarding memory checkers, they are usually a boon and they catch a good deal of issues once coupled with good samples. I managed to fix a good number of issues in hevc just by using gcc-asan and running the normal tests and for vp9 took not much time to spot a couple of issues as well (the memory checkers aren’t perfect so they didn’t spot the faulty memcpy I introduced to simplify a loop).

If you maintain some software please do use valgrind, asan (now also available on gcc) and, if you are on windows, drmemory. They help you catch bugs early. Just beware that sometimes certain versions of clang-asan miscompile. Never blindly trust the tools.

Static analyzers

The static analyzers are a mixed bag, sometimes they spot glaring mistakes sometimes they just point at impossible conditions.
Please do not put asserts to make them happy, if they are right you just traded a faulty memory access for a deny of service.

Other checkers

There are plenty other good tools from the *san family one can use, ubsan is maybe the newest available in gcc and it does help. Valgrind has plenty as well and the upcoming drmemory has a good deal of interesting perks, if only upstream hadn’t been so particular with release process and build systems you’d have it in Gentoo since last year…

Regression tests

I guess everybody is getting sick of me talking about fuzzy testing or why I spent weeks to have a fast regression test archive called playground for Libav and I’m sure everybody in Gentoo is missing the tinderbox runs Diego used to run.
Having a good and comprehensive batch of checks to make sure new code and new fixes do not have the uncalled side effect of breaking stuff is nice, coupled with git bisect makes backporting to fix issues in release branches much easier.

Debuggers

We have gdb, that works quite well, and we have lldb that should improve a lot. And many extensions on top of them. When they fail we can always rely on printf, or not

What’s missing

Speed

If security is just an acceptable impairment over performance in order not to crash, using the tools mentioned are an acceptable slow down on the development process in order not to spend much more time later tracking those issues.

The teams behind valgrind and *san are doing their best to just make the execution three-four times as slow when the code is instrumented.

The static analyzers are usually just 5 times as slow as a normal compiler run.

A serial regression test run could take ages and in parallel could make your system not able to do anything else.

Any speed up there is a boon. Bigger hardware and automation mitigates the problem.

Precision

While gdb is already good in getting you information out of gcc-compiled data apparently clang-compiled binaries are a bit harder. Using lldb is a subtle form of masochism right now for many reasons, it getting confused is just the icing of a cake of annoyance.

Integration

So far is a fair fight between valgrind and *san on which integrates better with the debuggers. I started using asan mostly because made introspecting memory as simple as calling a function from gdb. Valgrind has a richer interface but is a pain to use.

Reporting

Some tools are better than other in pointing out the issues. Clang is so far the best with gcc-4.9 coming closer. Most static analyzers are trying their best to deliver the big picture and the detail. gdb so far is incredibly better compared to lldb, but there are already some details in lldb output that gdb should copy.

Thanks

I’m closing this post thanking everybody involved in creating those useful, yet perfectible tools, all the people actually using them and reporting bugs back and everybody actually fixing the mentioned bugs so I don’t have to do myself alone =)

Everything is broken, but we are fixing most of it together.

Fosdem!

About 26h before Fosdem (yes, the beer event is the glorious start of the conference)!

What

I’ll be around bearing chocolate and chocolate for friends and fellow members of the communities I belong to (no beers this time, sorry guys!), hopefully we’ll find some space to discuss anything you’d like to discuss with me.

Topics

  • Libav (We should also have a room to discuss some more Libav10 and Libav11 planned releases)
  • VLC (Probably most discussions during the meeting, where Felix will stab me for not having done hwaccel2)
  • Gentoo/Sabayon (Complaints and rants welcome only during the beer event)
  • Any of my other many projects (contributions welcome btw!)
  • Anything else.

Where

There might be a room to discuss for about 1 hour about Libav10 Sunday, I’ll be around the Gentoo BoF Saturday and obviously I’ll be around attending some of the events.

See you there! (hopefully)

Gentoo on Macbookpro 2013 – part2

Go read here for the first part.

What changed

  • The linux 3.13 still doesn’t sport support for the wireless yet. The closed source driver works almost great beside when conman crashes horribly due some bad interaction, luckily doesn’t happen often, sadly I do not have time to debug it.
  • Using grub with the patch pointed in the comments here does make appear the intel gpu and you can enjoy using it for hardware decoding using QSV, patches for Libav availabe in the usual place.
  • vga switcheroo doesn’t let me switch properly, apparently nouveau takes the console framebuffer and does not really wants to release it.
  • the nvidia closed source driver works but you lose the access to the gmux, so you can’t change the brightness, and your console framebuffer is gone as well.
  • bbswitch seems confused enough to give up, I might ask upstream to help me figure out since seems almost everything is there post-grub-setup.
  • My pommed patches are waiting for more testers, then I’ll bake a release for everybody. Here it is working nicely.

What next

  • Probably I’ll try to figure out better how get the intel gpu work fully, since nouveau works mostly fine but Blizzard games on wine do not play at all.
  • Pommed will see more cleanups.
  • Hopefully I’ll play more with the displayports.

So far I’m still quite happy about this model even if the mentioned quirks.

Gentoo on Macbookpro 2013

I eventually decided to buy a newer laptop and ended up pick the latest from apple since looked nice.

Installing

Setting it up is sort of easy with just a couple of caveats

Boot from sysrescuecd

That’s my simplest way to get stuff done, fetched the latest 4.0 beta, installed the minimum needed to replace the kernel with one not sporting the dir entry bug and rebooted. The sysrescuecd standard configuration works decently, you might want to also fetch broadcom-sta to get wireless working.

Disable NCQ

The SSD seems to have problems with the ncq support at least on linux 3.12.2, I patched my 3.12.3 to blacklist APPLE* drives directly,

# echo 1> /sys/block/sda/device/queue_depth

Bootloader

I use refind, it magically finds the kernel and passes the correct root=/dev/foo option, so a no brainer.

X11

I was curious thus installed bumblebee and apparently it works out of box, you need to use mtrack to get decent touchpad support and you are pretty much done.

HiDPI

You might want to set your dpi to at least 110 if you are using GTK applications (not sure if QT has a scaling option, Elementary does and so E17)

Audio

Seems working just fine.

What does not work yet

Webcam

It seems sdio-connected.

b43

The opensource driver does not work, the closed source one works fine.

Pommed

In my github you can find a tentative and not so complete patch, out of box pommed refuses to work and with the patch at least the backlight and keyboard light sort of work.

Some ideas regarding / and /usr

What is the / ?

On unix-like system we have the wonderful abstration that let us keep on a single hierarchy all the file systems, it is called mount: we take the logic tree contained in a filesystem and we graft it over another. The initial file system holding all those grafts is the root file system and it lives on the mount point /. Let’s call it rootfs, it usually contains everything needed to start the system.

What (the hell) is /usr ?

Back in the old times where space was constrained the rootfs (/) started to get too big, so, as quick workaround some stuff deemed not fundamental moved on a separate fs mounted as /usr. The quick workaround proved itself useful so started to remain permanent with few exceptions (Hi Hurd, had you been rewritten from scratch this month already again?).

Split-/usr mount, what is the deal?

Linux had since long a quite useful volume-manager, it let you do a number of increasingly complex transformation over storage remaining nearly-agnostic regarding the file system. Being able to extend your storage by adding disks and merging them in a single logical storage seems useful, so is having software-mediated RAID setups.

Gentoo had since long a tutorial how to split all the rootfs mountpoints in different volumes. The idea makes sense or not depending on your tastes. Many people liked it and used it.

The rootfs contained a statically linked set of binaries useful to mount /usr and that was all of it. We can call this set of tools nowadays early-boot tools and the partition holding them early-bootfs. The two usually are the same.

Once /usr is mounted the boot process can start anything else.

Your problem now is figure out what is needed to boot /usr:

  • A volume can be a logical one created by a volume-manager
  • The file system could be a module and the module could be compressed by something the kernel doesn’t understand by itself
  • The file system can be userspace (thanks to fuse) and it could be implemented by an interpreted language such as python
  • Volume and filesystem can be networked, thus you need to bring your network up and the network could require additional components.
  • If you are into crypto, again it could be something mediated by the userspace at volume and file system level.

The possibilities are many and if you want to support them all no matter how unlikely it looks a complex problem.

Everything is broken let’s break it some more!

Some loud mouthed people decided to go against one of the key design item in writing a component such as an init system, that is keeping the single point of failure as simple as possible to reduce the chance it fails.

They kept adding compulsory dependencies to it that use to live on /usr. I think that as a way to get a problem so his arguably half baked solutions can be sell as only future and to make the most annoying situation in the previous paragraph the default.

The rational reaction would had been to just tell them to keep playing with their broken toys in a corner.

Initramfs

So we have this problem, a arbitrary long list of components that would make the rootfs large and some genius actively tries to have the first program started require most of it and shovel the concept down everybody else.

One solution would be just merge rootfs, early-init-bootfs and the whole /usr together somehow, welcome back to the the early 1900! (incidentally you will need also /var mounted but that’s a digression)

Obviously the problems causing the initial split-/usr are still there.

Linux has a neat feature called initramfs, the successor of initrd, it is great to keep modules and all the stuff you might need to mount your rootfs in a place the kernel could always reach no matter what.

So a solution would be keeping all that’s needed to mount the rootfs-now-merged-with-/usr in the initramfs since by definition is always reachable.

It is not exactly the most elegant solution but arguably works as long you get the list of required component right.

The elephant in the /boot

Some rhetoric questions:

  • “The initramfs is somehow related to its kernel, what happens if you keep more than 1 kernel around?”
  • “Which is the sane size limit for it?”
  • Initramfs can get stale easily, how much time takes to create it and keep it up to date?”

The answers might vary. The short is that you need good tools and lots of space.

Alternatives

You need good tools and good knowledge about what you need for your early boot, you have to put it somewhere and keep it up to date easily. Possibly it shouldn’t depend on your kernel yet be easy to access it.

/boot as early-boot partition

That is one of the simpler ideas, we just keep a separate copy of what is needed /boot, historically most concerned people kept a recovery there so makes sense for them use it as early-boot.

Static and restrict rootfs

If you know what you are doing as long you can keep in your rootfs your tools by linking them statically (so the whole deal about compressed modules is taken care of) and you aren’t using strange stuff (so just lvm and normal fs), you do not care about this whole deal. AS LONG YOUR DISTRIBUTION DOESN’T PLAY GAMES. Nor you drink the kool-aid and use stuff that breaks by design static linking or makes as hard as possible keeping a minimal amount of stuff in the rootfs.

Summary

We always need your help and feedback to make so Gentoo keeps giving you good options and currently working systems keep working in the next future. Thanks for reading.

Early boot fun

Just few notes spurted from a discussion with a friend regarding why he feels we suck badly.

Early boot

Let’s make a quite rough description on how booting could work:

  • Imagine you are the kernel, you just found your rootfs, managed to run your init and you are happy. That’s probably the earliest we care.
  • Init got called and starts running some scripts, maybe checking the rootfs consistency before remounting r/w, maybe checking the other essential mount points before mounting them or maybe start the device manager first and then checking what is going to mount, assuming what is essential still requires some modules loaded and that the device manager will do.
  • Move further and set up the network
  • Maybe now mount the mount points that require the network (nfs?)
  • Now get the other daemons up and running, maybe in parallel, maybe bring up some graphical login.

Now let’s see who are the actors: rootfs, init, device-manager and maybe incidentally volume-manager and networking.

Ideally your rootfs should contain

  • anything required by init to run, easy, init should be as small as possible to make sure this single point of failure really hard to fail.
  • anything required by the device-manager to load modules, should be a no brainer, well, maybe if you want your modules compressed with some new or exotic compressor because it is “faster” that way, you have to fit it in the rootfs.
  • If your essential mountpoints require a volume-manager the same applies, lvm can require something weird depending on the setup so either you link it statically or you have to put it again where it is reachable, same could be said for any kind of advanced crypto at volume leve.
  • We discussed about mount and again we could have fuse-fs using a scripting language or other stuff that make issuing mount a little more complex that we would expect (and again fs-level crypto happening in the userspace)
  • The network would just need some modules loaded right? Wrong, it might need some special daemons doing any kind of bizantine authentication, and if you are really looking for pain you could be willing to netmount those mentioned file systems or even do volume management over bizantine network (ok there is a limit in this kind of perversion and we are just halfway).
  • Once everything is mounted the rest of the system can be brought up w/out much qualms.

So in the end your rootfs can be quite fat contain full copies of python so you can mount that funny file system, have lots of lovely brittle deps because you thought NetworkManager is the only way to get the network up and meanwhile that having some important stuff (e.g. /var) netmounted is all the rage.

Fun (aka pain)

So in short your rootfs could be as big as a compact live distribution and have as many moving parts as one (or more), well it could be just your distribution if you do not keep everything in a separated mount point.

Some years ago that was one of the suggested ways, you keep essential stuff in / and then every other root mountpoint would have its partition, maybe using some advanced stuff just because.

Then you get told that the right place in which you have to fit all discussed above has to be something called initramfs and obviously tell the kernel about it.

Probably nobody would be that crazy to end up with the far corner case, so the initramfs would have to copy just few (20+??) libraries and some (30+?!) binaries in the normal case and you have to keep it synced up properly (joy).

Most people could live happy with just a statically linked lvm and udev living in a small partition easy to mount and that would be the start and the end for them, but certain wise guys will tell you that static linking is harmful, the whole concept is broken anyway since our bluetooth subsystem requires lots of userspace and then you’d be w/out a keyboard in case something goes wrong (so you should shove bluez and happy deps in your initramfs/rootfs?).

Summing up

There are easy, simple and working solution for just some realistic scenarios, but not covering everything that’s possible.

There are more complex, brittle and error prone ones that might cover everything and more (and maybe still fail in some basic situations).

The fact lots of lemmings flocks over the complex/brittle because the guy with the largest mouth is the best speaker is sad, overly sad.

That said if you were happily using since 10 years ago a lvm setup as described by our guides of the time and now you are afraid that your next userspace update your system will break horribly if you don’t go through the hoops of making an initramfs, that won’t work for you out of box and will force you to modify your bootloader or do some other time consuming work:

I’m sorry.

(Luckily somebody prepared a portage hook to prevent some breakages https://gist.github.com/mansr/7289969 not all of them but the most glaring are covered)

Incidentally, you can still pester us, help us getting better programs (e.g. contribute to eudev, kmod, lvm and everything else you use) and take an active part in the community and hopefully protect your simple and working solutions.

The case of defaults (Libav vs FFmpeg)

I tried not to get into this discussion, mostly because it will degenerate to a mud sliding contest.

Alexis did not take well the fact that Tomáš changed the default provider for libavcodec and related libraries.

Before we start, one point:

I am as biased as Alexis, as we are both involved on the projects themselves. The same goes for Diego, but does not apply to Tomáš, he is just a downstream by transition (libreoffice uses gstreamer that uses *only* Libav).

Now the question at hand: which should be the default? FFmpeg or Libav?

How to decide?

– Libav has a strict review policy every patch goes through a review and has to be polished enough before landing the tree.

– FFmpeg merges daily what had been done in Libav and has a more lax approach on what goes in the tree and how.

– Libav has fate running on most architectures, many of those are running Gentoo, usually real hardware.

– FFmpeg has an old fate with less architectures, many qemu emulations.

– Libav defines the API

– FFmpeg follows adding bits here and there to “diversify”

– Libav has a major release per season, minor releases when needed

– FFmpeg releases a lot touting a lot of *Security*Fixes* (usually old code from the ancient times eventually fixed)

– Libav does care about crashes and fixes them, but does not claim every crash is a Security issue.

– FFmpeg goes by leaps to add MORE features, no matter what (including picking wip branches from my personal github and merging them before they are ready…)

– Libav is more careful, thus having less fringe features and focusing more polishing before landing new stuff.

So if you are a downstream you can pick what you want, but if you want something working everywhere you should target Libav.

If you are missing a feature from Libav that is in FFmpeg, feel free to point me to it and I’ll try my best to get it to you.

Again on shoveling stuff in other people mouth

Again we got a fun thread about having to do some extensive change on perfectly working systems because somebody has a *plan* and you must abide to it.

If before the plan was to have systemd as the true and only init system (on why systemd seems to me a bad idea by itself I’ll discuss on a later post, possibly after throughly study its latest iteration and comparing it), now the plan is to force people not to have a separate /usr or use an initramfs with an early boot system because… “because doing otherwise is broken and already had been in ages”.

That doesn’t tell you much and if you have lots of systems running perfectly on a separate /usr setup and you went that way because it was documented as a best practice, you might feel enraged.

Now, let’s make clear that there are operating systems that keep everything in /usr and have next to nothing in / (and system that do not have /usr at all and everything is in /), you can argue a lot about what’s the best and why. FreeBSD or Hurd approaches have both interesting perks.

The fact is that *now* you have lots of people with perfectly working system in a configuration somebody decided that is wrong and *unsupportable*.

If you try to dig down a bit more you’ll discover that the “brokeness” is mainly due:

  • Somebody keen in using a library that traditionally is in /usr for some fringe feature
  • Somebody hell bent to use glib everywhere
  • Somebody wanting to have d-bus running in the early boot phase
  • Some udev rules using some data that currently resides in /usr

All considered forcing people to spend lots of time because somebody might want to use a bluetooth keyboard on early boot (thus requiring bluez, thus requiring d-bus basically because you can’t use bluez without it) or other non widespread use case is not exactly nice.

Surely trying to get a cleaner layout so we have a bare mountpoint directory, a early boot system in initramfs and the rest of the system cleanly split isn’t bad by itself and probably it is something I would consider neat.

But you still need to have a good separation between what is early boot and what is not and you need to make sure the boot process doesn’t get too complex or too tightly coupled with systems that can and will break easily.

I’m quite happy that alternatives are already almost available for simple systems not needing the additional features requiring those extensive changes.

Hopefully somebody will have time to try to add rules marking in udev so complex rules won’t be triggered when the system isn’t ready for them and deploys using special layouts could stay supported in a way or another.

In the other news Gentoo had been accepted to participate to the Google Summer of Code and there are two projects proposed by me, one is about documenting and if needed extending openrc to be a complete viable alternative to systemd, the other about using containers and qemu-user to have better tools to do cross developement.

Shoveling stuff in other people mouth

Recently I came up to read this thread the initial proposal is to add systemd in some kind of fashion to gnome, with Lennart suggesting new and many features coming from that.

That alone isn’t a problem until people not caring about a broken toy from him (both systemd and pulse had been and are considered as such) can keep playing with gnome with their toys (e.g FreeBSD, any non systemd linux distro). The thread then evolved in something that could be sort of summarized with systemd developer not caring about anything else but Linux and so Gnome should do as well, with extremes from other people suggesting to ditch distributions and have a Gnome OS alone.

In the light of what happened before with HAL and on a minor degree with pulse that makes me wary.

So people willing to use Gnome in Lennart opinion must probably:

– Use Linux

– Use systemd

A quick reality check tell me that:

– Sun contributed a lot to Gnome, there are plenty of BSD users hacking on Gnome and/or using it.

– systemd is still a broken increasingly complex mess and the fact it needs _that_ many linux specific features to try to work tells a lot (Remember the UNIX way: small simple things you can understand and replace quickly)

– Gnome isn’t something that doesn’t have good or better replacements (Unity and Xfce come to mind if we want to consider gtk+)

I already switched from Gnome to e17 long time ago since I want that the WM/visual shell doesn’t get in my way and let me do what I want as I want.

Keeping Posix as baseline is important. Having your software working in different environments help you spot bugs quickly, get a more wider audience and generally improve. Even if I don’t particularly like KDE still I consider their approach to make porting KDE everywhere as easy as possible laudable.

***Update***

Sebastian asked me if there could be something constructive about this post since it looks pretty much a rant. Well, most of the post IS a rant (and categorized as that) and a not so humble request to not rush adopting/forcing upon people random technologies as they are sound and tested when they are not at all.

There isn’t anything constructive to be said, I just hope that we don’t have yet another situation like the one we had with other unproven technologies forced upon people like HAL had been.

On a side note, not completely related I wonder why people wants systemd  and doesn’t ask us to adopt upstart then. It is from Ubuntu as systemd is from Fedora, and Ubuntu at least tries to be present in non-desktop environments while Fedora is quite focused on the Desktop and only that.

Gentoo has OpenRC as init system. It is simpler and smaller. Works well in quite a good number of environments and for a quite large deal of situations. Surely there is space for improvements but at least does not require daemons to be written for it and had been tested in years.

Libav

Probably you already know that my side of FFmpeg got forced to rename itself to Libav. Some people is still wondering why we did that, you might read some short and longer summaries, have a look at our git or our mailing lists to see how we are faring and where we are heading.

So far I’m quite happy with what we are achieving little by little and day by day: a shared and quite defined plan for the future of the library, releases being a first class citizen, long standing issues being tackled and solved.

We were sorely lacking in the communication side and now we are trying to improve there as well. (This blog post and the website work is just part of it)

In the Gentoo land Scarabeus helped me adding libav, now it is pending some migration work to have all the software working with both libav and ffmpeg.

I hope you’ll be pleased by the outcome (people longing for the multithread work being fully merged I think are).