Category: Multimedia

Outreach Program for Women

Libav participated in the summer edition of the OPW. We had three interns Alexandra, Katerina and Nidhi.

Projects

The three interns had different starting skills so the projects picked had a different breadth and scope.

Small tasks

Everybody has to start from a simple task and they did as well. Polishing crufty code is one of the best ways to start learning how it works. In the Libav case we have plenty of spots that require extra care and usually hidden bugs get uncovered that way.

Not so small tasks

Katerina decided to do something radical from the start and she tried to use coccinelle to fix a whole class of issues in a single swoop: I’m still reviewing the patch and splitting it in smaller chunks to single out false positives. The patch itself gave some spotlights to some of the most horrible code still lingering around, hopefully we’ll get to fix those part soon =)

Demuxer rewrite

Alexandra and Katerina showed interest in specific targeted tasks, they honed their skills by reimplementing the ASF and RealMedia demuxer respectively. They even participated in the first Libav Summer Sprint in Torino and worked together with their mentor in person.

They had to dig through the specifications and figure out why some sample files behave in unexpected ways.

They are almost there and hopefully our next release will see brand new demuxers!

Jack of all trades

Libav has plenty of crufty code that requires some love, plenty of overly long files, lots of small quirks that should be ironed out. Libav (as any other big projects) needs some refactoring here and there.

Nidhi’s task was mainly focused on fixing some of those and help others doing the same by testing patches. She had to juggle many different tasks and learn about many different parts of the codebase and the toolset we use.

It might not sound as extreme as replacing ancient code with something completely new (and make it work at least as well as the former), but both kind of tasks are fundamental to keep the project healthy!

In closing

All the projects have been a success and we are looking forward to see further contributions from our new members!

PowerPC is back (and little endian)

Yesterday I fixed a PowerPC issue since ages, it is an endianess issue, and it is (funny enough) on the little endian flavour of it.

PowerPC

I have some ties with this architecture since my interest on the architecture (and Altivec/VMX in particular) is what made me start contributing to MPlayer while fixing issue on Gentoo and from there hack on the FFmpeg of the time, meet the VLC people, decide to part ways with Michael Niedermayer and with the other main contributors of FFmpeg create Libav. Quite a loong way back in the time.

Big endian, Little Endian

It is a bit surprising that IBM decided to use little endian (since big endian is MUCH nicer for I/O processing such as networking) but they might have their reasons.

PowerPC traditionally always had been both-endian with the ability to switch on the fly between the two (this made having foreign-endian simulators lightly less annoying to manage), but the main endianess had always been big.

This brings us to a quite interesting problem: Some if not most of the PowerPC code had been written thinking in big-endian. Luckily since most of the code wrote was using C intrinsics (Bless to whoever made the Altivec intrinsics not as terrible as the other ones around) it won’t be that hard to recycle most of the code.

More will follow.

Libav Release Process

Since the release document is lacking here few notes on how it works, it will be updated soon =).

Versioning

Libav has separate version for each library provided. As usual the major version bump signifies an ABI-incompatible change, a minor version bump marks a specific feature introduction or removal.
It is made this way to let users leverage the pkgconf checks to require features instead of use a compile+link check.
The APIChange document details which version corresponds to which feature.

The Libav global version number e.g. 9.16 provides mainly the following information:

  • If the major number is updated the Libraries have ABI differences.
    • If the major number is Even API-incompatible changes should be expected, downstreams should follow the migration guide to update their code.
    • If the major number is Odd no API-incompatible changes happened and a simple rebuild **must** be enough to use the new library.
  • If the minor number is updated that means that enough bugfixes piled up during the month/2weeks period and a new point release is available.

Major releases

All the major releases start with a major version bump of all the libraries. This automatically enables new ABI incompatible code and disables old deprecated code. Later or within the same patch the preprocessor guards and the deprecated code gets removed.

Alpha

Once the major bump is committed the first alpha is tagged. Alphas live within the master branch, the codebase can still accept features updates (e.g. small new decoders or new demuxers) but the API and ABI cannot have incompatible changes till the next one or two major releases.

Beta

The first beta tag also marks the start of the new release branch.
From this point all the bugfixes that hit the master will be backported, no feature changes are accepted in the branch.

Release

The release is not different from a beta, it is still a tag in the release branch. The level of confidence nothing breaks is much higher though.

Point releases

Point releases are bugfix-only releases and they aim to provide seamless security updates.

Since most bugs in Libav are security concerns users should update as soon the new release is out. We keep our continuous integration system monitoring all the release branches in addition to the master branch to be confident that backported bugfixes do not cause unexpected issues.

Libav 11

The first beta for the release 11 should appear in the next two days, please help us by testing and reporting bugs.

Releases!

Last we made a huge effort to make a release for every supported branch (and even one that is supposed not to be). Lots of patches to fix some old bugs got backported. I hope you appreciate the dedication.

Libav 0.8.15

We made an extra effort, this branch is supposed to be closed and the code is really ancient!
I went the extra mile and I had to run over all the codebase to fix a security issue properly: you might crash if your get_buffer callback doesn’t validate the frame dimension, that code is provided by the library user (e.g. VLC), so the solution is to wrap the get_buffer callback in a function ff_get_buffer and do the check there. For Libav 9 and following we did already for unrelated reasons, for Libav 0.8 I (actually we since the first patch didn’t cover all usage) had sift through the code and replace all the avctx->get_buffer() with ff_get_buffer().

Libav 9.16

This is a standard security release, backporting from Libav 10 might require some manual retouch since code got cleaned up a lot and some internals are different but it is still less painful than backporting from 11 to 0.8

Libav 10.3

This is a quite easy release, backporting fixes is nearly immediate since Libav 11 doesn’t have radical changes in the core internals and the cleanups can apply to release/10.

Libav 11 alpha1

Libav11 is a major release API compatible with Libav10, that makes transitioning as smooth as possible: you enjoy automatically some under-the-hood changes that required an ABI bump (such as the input mime support to speed up AAC webradio startup time) and if you want you can start using the new API features (such as avresample AVFrame API, av_packet_rescale_ts(), AVColor in AVFrame and so on).

You can help!

Libav 11 will be out within the month and help is welcome to polish it and make sure we do not have rough edges.

Update a downstream project you are using

Many downstreams are still using (and sometimes misusing) the old (Libav9) and ancient (Libav0.8) API. We started writing migration guides to help, we contributed many patches already and the Debian packagers did a great job to take care of their side.

Some patches are just waiting to be forwarded to the downstream or, if the package is orphaned, to your favourite distribution packagers.

Triage our bugzilla

Most of the Libav development happens in the mailing-lists and sometimes
bugs reported over bugzilla get not updated timely. Triaging bugs sometimes take a little of time and helps a lot.

Libav 10.1 released (in Gentoo)

I just committed the ebuild in Portage and noticed that Homebrew already updated its Formula.

I spent some time in Berlin at LinuxTAG manning the VideoLAN booth and feeding people with VLC and Libav chocolate (many thanks to Borgodoro for providing me with their fine goods).

During the weekend we held the VideoLAN association meeting in the SoundCloud office, thanks a lot again for the wonderful venue.

Unmask in Gentoo

I’m slowly getting a tinderbox up with the help of Flameeyes so we can make sure nothing unexpected happens, the new refcounted API for Frames and Packets makes quite compelling updating from 9 even if we’ll keep updating both release branches for the next year and half.

You can help

There are a number of packets that depend on old 0.8 ffmpeg application and it won’t really work with the current avconv nor with the ffmpeg provided by the recent versions since the new option parsing code written by Anton ended up there as well. Most of those application are either fully orphaned or they have patches to work with avconv because the Debian and Ubuntu developers took care of it. Nikoli provided me with a list.

HWAccel 1.2

This week-end I eventually merged hwaccel1.2 and I hope to get the AVResample updates finalized by this week.

There had been some discussion regarding backporting the latter to release 10 since they simplify a bit porting to the new resampling library.

Rotation reporting API

Vittorio was working on a mean to export the rotation matrix from MOV and SEI NALs since a while. The devil is usually in the details and I guess it had been an hell of fun for him. Even more since he switched continent meanwhile.

After the Linus rant about not having the automagic support for rotation, we had some decent pressure applied to get it out so people will be able to enjoy it. Hopefully it will appear by the week end as well.

New Libav Releases

We are preparing a new set of releases with more bugfixes (some eventually fixed thanks to ubsan).

Libav 0.8

With the release of Libav 10 the 0.8 is nominally out of the maintainance window, but we’ll try to get a last update (or few more) to ease the transition, volunteers willing to add a section in the migration guides would be welcome.

Libav 9

The Plain Nine serie is actively maintained and being the last one not sporting reference-counted AVFrames, I warmly suggest to consider migrating to Libav 10.
The new release is expected to appear tomorrow, stay tuned!

Libav 10

Soon we’ll provide the first point release, we are discussing if we should backport the simplified avresample API, I’m working on since it should make much easier the life of people used to the old audio decoding API and expecting interleaved PCM as output.

Future

This weekend everybody had been also busy hammering and linting the native Opus codec Anton wrote and while at it I spent some time on finding and fixing few annoying bugs in vp9.

Libav 11 is shaping up nicely and hopefully soon I’ll get back working on libavformat (MXF, segmented formats and some more) and hopefully I’ll have a preview branch for AVScale next month, as stated this release will be source-compatible with Libav 10, Anton is already busy preparing the API cleanse for Libav 12 later this winter, but you’ll be warned by the time this release is out.

This week I’ll in Berlin for the LinuxTAG, if you have questions I’ll be happy to answer, you’ll find me either at the Gentoo or the VLC booth.

Security and Tools

Everybody should remember than a 100% secure device is the one unplugged and put in a safe covered in concrete. There is always a trade-off on the impairment we inflict ourselves in order to stay safe.

Antonio Lioy

In the wake of the heartbleed bug. I’d like to return again on what we have to track problems and how they could improve.

The tools of the trade

Memory checkers

I wrote in many places regarding memory checkers, they are usually a boon and they catch a good deal of issues once coupled with good samples. I managed to fix a good number of issues in hevc just by using gcc-asan and running the normal tests and for vp9 took not much time to spot a couple of issues as well (the memory checkers aren’t perfect so they didn’t spot the faulty memcpy I introduced to simplify a loop).

If you maintain some software please do use valgrind, asan (now also available on gcc) and, if you are on windows, drmemory. They help you catch bugs early. Just beware that sometimes certain versions of clang-asan miscompile. Never blindly trust the tools.

Static analyzers

The static analyzers are a mixed bag, sometimes they spot glaring mistakes sometimes they just point at impossible conditions.
Please do not put asserts to make them happy, if they are right you just traded a faulty memory access for a deny of service.

Other checkers

There are plenty other good tools from the *san family one can use, ubsan is maybe the newest available in gcc and it does help. Valgrind has plenty as well and the upcoming drmemory has a good deal of interesting perks, if only upstream hadn’t been so particular with release process and build systems you’d have it in Gentoo since last year…

Regression tests

I guess everybody is getting sick of me talking about fuzzy testing or why I spent weeks to have a fast regression test archive called playground for Libav and I’m sure everybody in Gentoo is missing the tinderbox runs Diego used to run.
Having a good and comprehensive batch of checks to make sure new code and new fixes do not have the uncalled side effect of breaking stuff is nice, coupled with git bisect makes backporting to fix issues in release branches much easier.

Debuggers

We have gdb, that works quite well, and we have lldb that should improve a lot. And many extensions on top of them. When they fail we can always rely on printf, or not…

What’s missing

Speed

If security is just an acceptable impairment over performance in order not to crash, using the tools mentioned are an acceptable slow down on the development process in order not to spend much more time later tracking those issues.

The teams behind valgrind and *san are doing their best to just make the execution three-four times as slow when the code is instrumented.

The static analyzers are usually just 5 times as slow as a normal compiler run.

A serial regression test run could take ages and in parallel could make your system not able to do anything else.

Any speed up there is a boon. Bigger hardware and automation mitigates the problem.

Precision

While gdb is already good in getting you information out of gcc-compiled data apparently clang-compiled binaries are a bit harder. Using lldb is a subtle form of masochism right now for many reasons, it getting confused is just the icing of a cake of annoyance.

Integration

So far is a fair fight between valgrind and *san on which integrates better with the debuggers. I started using asan mostly because made introspecting memory as simple as calling a function from gdb. Valgrind has a richer interface but is a pain to use.

Reporting

Some tools are better than other in pointing out the issues. Clang is so far the best with gcc-4.9 coming closer. Most static analyzers are trying their best to deliver the big picture and the detail. gdb so far is incredibly better compared to lldb, but there are already some details in lldb output that gdb should copy.

Thanks

I’m closing this post thanking everybody involved in creating those useful, yet perfectible tools, all the people actually using them and reporting bugs back and everybody actually fixing the mentioned bugs so I don’t have to do myself alone =)

Everything is broken, but we are fixing most of it together.

The road to MVC

In the past month or so I started helping Vittorio on adding one of the important missing feature to our h264 decoder. Multi View support.

MVC

The basic idea of this feature is quite simple, you are shooting a movie with multiple angles, something is bound to be sort of common and you’d like to ensure frame precision.

So what about encoding all the simultaneous frames captured in the same elementary stream, share across the different layers as much as you could and then let the decoder output the frames somehow?

Since we know that all the containers have problems might be not completely a bogus idea to have the codec taking care of it. Even better if the resulting aggregated bitstream is more compact than the sum of the single ones.

High level structure

What’s different in h264-mvc than the normal h264?

Random bystander

Not a lot, in fact the main layer is exactly the same and a normal decoder can just skip over the additional bits (3 NALs more or less) and just decode as usual.

Basically there is a NAL unit to signal which layer we are currently working on, a NAL to store the SPS specific per layer and a NAL to keep the actual frame data.

Beside that everything is exactly the same.

Implementation

So why it isn’t already available, you made it look easy?!

Random jb

Sadly it would be easy if the decoder we have isn’t _that_ convoluted with many components entangled in a monolithic entity, with code that grew over the years to adapt to different needs.

Architectural pain points

Per slice multithreaded decoding made the code quite hard to follow since you then have a master context, h that in certain functions is actually h0 and a slice specific copy hx that sometimes becomes h and such.

Per frame multhtreaded decoding luckily doesn’t get in the way too much for now.

Having to touch a large file of about 4k lines of code in itself isn’t _so_ nice, split view as you like for editing, you end up waiting a single core of you cpu doing the work.

Community constraints

The h264-mvc is a fringe feature for many and if you care about speed you want to not have all the cruft around slowing down. What’s is for you a feature, for many is just cruft.

  • MVC support must be completely optional or not slow down the normal decoding at all.
  • MVC support must not make the code harder to follow than it is now, so hacking your way is not an option.
  • MVC should give me a pony, purple

The plan

First take the low hanging fruits while you think what’s the best route to achieve your goal.

Random wise person

Refactor

The first step is always refactor and cleanup. As you, hopefully, do not cook on a dirty kitchen, people shouldn’t
write code on top of crufty one.

Split the monster

In Libav everything compiles quite fast beside for vc1(vc1dec.c is 6k loc) and h264(h264.c was around 6k loc).
New codecs such as vp9 or hevc landed already split in smaller chunks.

Shuffling the code should be simple enough, so we had h264.c split in h264_slice.c, h264_mb.c and such. That helps having shorter (re)build time and makes you easier to focus.

Untangle it

Vittorio tried to remove the dependency over the mpeg12 context in order to make easier to follow the code, it was one of the pending issues since years. Now h264 doesn’t require mpeg12 in order to build, that will make probably happier our friends working on Chrome and everybody else needing to have _just_ few selected features in their build.

Pave the road

Once you divided the problem in smaller sub problems (parsing the new nals, store the information in an appropriate data structure, do the actual decoding and store the results somewhere accessible) you can start working on adapting the code to fit. That means reordering some code, splitting functions that would be shared and maybe slay some bugs hidden in the code weed while at it.

So far

We are halfway!

Random optimist

Done

We got the frame splitting, nal parsing pretty much in working shape and is not sent for review just because in itself is not
useful.

Doing

The frame data decoding is pending some patches from me that try to simplify the slice header parsing so enough of it could be shared w/out adding more branches. I hacked it once and I know the approach used works.

The code to store multiple views in a single frame has a whole blueprint being evaluated.

To Do

Test the actual decoding and hopefully make so the frame reference code behaves as expected, this will be probably the most annoying and time consuming task if we are unlucky. That code bites.

Libav 10 – release

New release

After several months spent finalizing, we are now pleased to announce the release of Libav 10.

One of the main features of this release is the addition of reference-counted data buffers to Libav and their use in various structures. Specifically, the data buffers used by AVPacket and AVFrame can now be reference counted, which should allow to significantly simplify many use cases. In addition, reference-counted AVFrames> can now be used in libavfilter, avoiding the need for a separate libavfilter-specific frame structure. Frames can now be passed straight from the decoders into filters or from filters to encoders.

These additions made it necessary to bump the major versions of libavcodec, libavformat, libavdevice, libavfilter, and libavutil, which was accompanied by dropping some old deprecated APIs. These libraries are thus not ABI- or API- compatible with the previous release. All the other libraries (libavresample and libswscale) remain ABI- and API-compatible.

Another major point is the inclusion of the HEVC (AKA H.265, the successor of H.264) decoder in the main codebase. It was started in 2012 as a Libav Google Summer of Code project by Guillaume Martres and subsequently completed with the assistance of the OpenHEVC project and several Libav developers.

As usual, this release also contains support for other new formats, many smaller new features and countless bug fixes. We can highlight a native VP9 decoder, with encoding provided through libvpx, native decoders for WebP, JPEG 2000, and AIC, as well as improved WavPack support with encoding through libwavpack, support for more AAC flavors (LD – low delay, ELD – enhanced low delay), slice multithreading in libavfilter, or muxing chapters in ASF. Furthermore a few new filters have been introduced, namely compand, to change audio dynamics, framepack, to create stereoscopic videos, asetpts, to set audio pts, and interlace, to convert progressive video to interlaced. Finally there is more fine-grained detection of host and target libc, which should allow better portability to various cross compilation scenarios.

See the Changelog file for a fuller list of significant changes.

You can download the new release, as usual, from our download page.

Release 10 took a lot of time to get out, mostly due the fact we spent lots of time to help downstream to adapt to the new API and we tried our best to provide patches to most of the projects we were aware of.

Now we are settled to have migration guides so the next API breaking releases won’t require that much effort.

Thanks

I want to thank everybody in the Libav team for spending that much time on the annoying, depressing and unrewarding task of coping with the release process, fixing fringe bugs, bake patches for projects not really used and help cleaning up the documentation.

Special thanks also go to the people from VideoLan and mpv since they helped us a lot in many different ways (testing, giving feedback on the new APIs and also provide patches) and to the Google security team that provided me on short notice a large batch of samples for HEVC and VP9 that I used to validate the new decoders.

Future releases

Release process update

This is the plan for the next 4 releases (spanning more or less from spring till winter), it is the result of all the feedback regarding our release process and requests.

Enough people, mostly mpv, vlc and other downstreams tracking us by git commit, would like to have quicker major releases. The API changes
introduced are mostly caused by us trying to satisfy their needs after all.

On the other hand, a good amount of people, distribution managers/packagers and the people tending to orphaned packages used but not really developed further, have quite a problem keeping up with the changes if the API gets incompatible too often.

In order to help them we already opened a dedicated section to our bugzilla and started writing migration guides, but they would really like not having to patch old packages that often anyway.

Trying to satisfy those two, apparently conflicting, requirements that’s what we aim for:

  • Every odd major releases should not break the API, must happen quickly once enough features are available and just augment the API. ABI breaks still possible, thus the version bumps.
  • Major releases removing old APIs, thus normally source incompatible with downstream not tracking git, should happen at most once per season
    or twice per year.
  • All the API changes will get an entry in the migration guide when it is committed.
  • We do remain committed to backport security-impacting bugfixes through a window of API-breaking releases, thus not leaving in the cold who couldn’t or didn’t update often enough.

I hope ~8 feature improvements and ~4 api cleanups per year would make most people happy.

Next releases

Libav 11

It would just provide new features, more optimizations for the usual platforms and the new ones, support for a good number of fringe codecs, such as the elusive vp7, will be added.

As stated above no API breakages are to be expected.

Libav 12

This release will contain major changes, including possibly a new scaling library.
The wiki has a Blueprint section tracking the most prominent ones, you are welcome to discuss them with us.

What I’ll be working on

I’m personally involved in the following items:

  • Extend MXF support: The format is quite bizantine and had been extended even further over time.[libav11]
  • Hwaccel2: because the current situation is far from being easy to use.[libav11]
  • mime-type support in Input Formats: since we support as output I don’t see why we should not leverage it on input to speed-up probing formats.[libav11]
  • AVScale: a replacement to swscale, trying to be more rational and not pointlessly lose information doing all pointless intermediate conversions to YUV. Incidentally also support hardware scalers when available [libav12]
  • libmfx: Intel tried its best to give an uniform interface that spans Linux, Windows and possibly MacOSX, I have working decoders and encoders wrappers, soon also hwaccel1.2 support. [libav11]
  • MVC support: multiview support is nice to have if you want to watch your blu-ray disks. [libav11]
  • Apple VDA and VT hwaccel: Since the introduction of hwaccel1.2 supporting them properly should be easier.[libav11]

If some of them are important to you actual help or even sponsorship is welcome.

Libav10 Release Progress

We are working on getting some few remaining bits in the tree before we could eventually branch release/10, lots of low hanging fruits are being reaped right now. Soon we’ll get the first beta out, thus freezing the featureset. If something you need doesn’t make it don’t be afraid the release/11 will be much quicker and appear in spring.

Missing from the next beta

There are a number of speed improvement people would like to land before we open the branch and that are polished this week-end.

The results of the preliminary work Vittorio and I are doing on MVC are getting merged already since they just make the h264 decoder nicer to read.

Some less famous codecs are getting some slices of attention, even if some fringe ones requiring additional data structures might not make 10.

Missing from the actual release

Downstream updates cross-distribution

We are doing our best to track which downstream projects need an API update, help is more than welcome. Soon we’ll have a tinderbox run to check what’s the status in Gentoo, Debian has already some reports and hopefully we’ll get some feedback from our friends at OpenSuse as well.

Fuzz cleansing some pending bugs

I still have some in my list and I’m fixing them while not merging the pending patchsets deemed useful for this release. Luckily that part must be finished by the last beta and not the first one.

Documentation

We are getting more data in the wiki and our doxy is getting polished and the examples section is getting richer and more structured (eventually). By the time we hit this release all the pending work on this side will be available for consumption.

Sprint in March

I decided to postpone the first Libav sprint/meeting by one month and probably the focus will change a bit, the wiki will be updated accordingly.