Everybody should remember than a 100% secure device is the one unplugged and put in a safe covered in concrete. There is always a trade-off on the impairment we inflict ourselves in order to stay safe.
In the wake of the heartbleed bug. I’d like to return again on what we have to track problems and how they could improve.
The tools of the trade
Memory checkers
I wrote in many places regarding memory checkers, they are usually a boon and they catch a good deal of issues once coupled with good samples. I managed to fix a good number of issues in hevc just by using gcc-asan and running the normal tests and for vp9 took not much time to spot a couple of issues as well (the memory checkers aren’t perfect so they didn’t spot the faulty memcpy I introduced to simplify a loop).
If you maintain some software please do use valgrind, asan (now also available on gcc) and, if you are on windows, drmemory. They help you catch bugs early. Just beware that sometimes certain versions of clang-asan miscompile. Never blindly trust the tools.
Static analyzers
The static analyzers are a mixed bag, sometimes they spot glaring mistakes sometimes they just point at impossible conditions.
Please do not put asserts to make them happy, if they are right you just traded a faulty memory access for a deny of service.
Other checkers
There are plenty other good tools from the *san family one can use, ubsan is maybe the newest available in gcc and it does help. Valgrind has plenty as well and the upcoming drmemory has a good deal of interesting perks, if only upstream hadn’t been so particular with release process and build systems you’d have it in Gentoo since last year…
Regression tests
I guess everybody is getting sick of me talking about fuzzy testing or why I spent weeks to have a fast regression test archive called playground for Libav and I’m sure everybody in Gentoo is missing the tinderbox runs Diego used to run.
Having a good and comprehensive batch of checks to make sure new code and new fixes do not have the uncalled side effect of breaking stuff is nice, coupled with git bisect makes backporting to fix issues in release branches much easier.
Debuggers
We have gdb, that works quite well, and we have lldb that should improve a lot. And many extensions on top of them. When they fail we can always rely on printf, or not…
What’s missing
Speed
If security is just an acceptable impairment over performance in order not to crash, using the tools mentioned are an acceptable slow down on the development process in order not to spend much more time later tracking those issues.
The teams behind valgrind and *san are doing their best to just make the execution three-four times as slow when the code is instrumented.
The static analyzers are usually just 5 times as slow as a normal compiler run.
A serial regression test run could take ages and in parallel could make your system not able to do anything else.
Any speed up there is a boon. Bigger hardware and automation mitigates the problem.
Precision
While gdb is already good in getting you information out of gcc-compiled data apparently clang-compiled binaries are a bit harder. Using lldb is a subtle form of masochism right now for many reasons, it getting confused is just the icing of a cake of annoyance.
Integration
So far is a fair fight between valgrind and *san on which integrates better with the debuggers. I started using asan mostly because made introspecting memory as simple as calling a function from gdb. Valgrind has a richer interface but is a pain to use.
Reporting
Some tools are better than other in pointing out the issues. Clang is so far the best with gcc-4.9 coming closer. Most static analyzers are trying their best to deliver the big picture and the detail. gdb so far is incredibly better compared to lldb, but there are already some details in lldb output that gdb should copy.
Thanks
I’m closing this post thanking everybody involved in creating those useful, yet perfectible tools, all the people actually using them and reporting bugs back and everybody actually fixing the mentioned bugs so I don’t have to do myself alone =)
Everything is broken, but we are fixing most of it together.