{"id":577,"date":"2016-03-21T20:45:20","date_gmt":"2016-03-21T20:45:20","guid":{"rendered":"http:\/\/blogs.gentoo.org\/lu_zero\/?p=577"},"modified":"2016-03-21T20:45:21","modified_gmt":"2016-03-21T20:45:21","slug":"bitstream-filtering","status":"publish","type":"post","link":"https:\/\/blogs.gentoo.org\/lu_zero\/2016\/03\/21\/bitstream-filtering\/","title":{"rendered":"Bitstream Filtering"},"content":{"rendered":"<p>Last weekend, after few months of work, the new bitstream filter API eventually landed.<\/p>\n<h2>Bitstream filters<\/h2>\n<p>In Libav is possible to manipulate raw and encoded data in many ways, the most common being<\/p>\n<ul>\n<li>Demuxing: extracting single data packets and their timing information<\/li>\n<li>Decoding: converting the compressed data packets in raw video or audio frames<\/li>\n<li>Encoding: converting the raw multimedia information in a compressed form<\/li>\n<li>Muxing:   store the compressed information along timing information and additional information.<\/li>\n<\/ul>\n<p>Bitstream filtering is somehow less considered even if the are widely used under the hood to demux and mux many widely used formats.<\/p>\n<p>It could be consider an optional final demuxing or muxing step since it works on <strong>encoded data<\/strong> and its main purpose is to reformat the data so it can be accepted by decoders consuming only a specific serialization of the many supported (e.g. the <strong>HEVC<\/strong> QSV decoder) or it can be correctly muxed in a <strong>container<\/strong> format that stores only a specific kind.<\/p>\n<p>In Libav this kind of reformatting happens normally automatically with the annoying exception of <strong>MPEGTS<\/strong> muxing.<\/p>\n<h2>New API<\/h2>\n<p>The new API is modeled against the <a href=\"https:\/\/blogs.gentoo.org\/lu_zero\/2015\/10\/02\/decoupling-an-api-part-ii\/\">pull\/push paradigm<\/a> I described for AVCodec <a href=\"https:\/\/blogs.gentoo.org\/lu_zero\/2015\/10\/02\/decoupling-an-api-part-ii\/\">before<\/a>, it works on <code>AVPacket<\/code>s and has the following concrete implementation:<\/p>\n<div class=\"codehilite\">\n<pre><span class=\"c1\">\/\/ Query<\/span>\n<span class=\"k\">const<\/span> <span class=\"n\">AVBitStreamFilter<\/span> <span class=\"o\">*<\/span><span class=\"nf\">av_bsf_next<\/span><span class=\"p\">(<\/span><span class=\"kt\">void<\/span> <span class=\"o\">**<\/span><span class=\"n\">opaque<\/span><span class=\"p\">);<\/span>\n<span class=\"k\">const<\/span> <span class=\"n\">AVBitStreamFilter<\/span> <span class=\"o\">*<\/span><span class=\"nf\">av_bsf_get_by_name<\/span><span class=\"p\">(<\/span><span class=\"k\">const<\/span> <span class=\"kt\">char<\/span> <span class=\"o\">*<\/span><span class=\"n\">name<\/span><span class=\"p\">);<\/span>\n\n<span class=\"c1\">\/\/ Setup<\/span>\n<span class=\"kt\">int<\/span> <span class=\"nf\">av_bsf_alloc<\/span><span class=\"p\">(<\/span><span class=\"k\">const<\/span> <span class=\"n\">AVBitStreamFilter<\/span> <span class=\"o\">*<\/span><span class=\"n\">filter<\/span><span class=\"p\">,<\/span> <span class=\"n\">AVBSFContext<\/span> <span class=\"o\">**<\/span><span class=\"n\">ctx<\/span><span class=\"p\">);<\/span>\n<span class=\"kt\">int<\/span> <span class=\"nf\">av_bsf_init<\/span><span class=\"p\">(<\/span><span class=\"n\">AVBSFContext<\/span> <span class=\"o\">*<\/span><span class=\"n\">ctx<\/span><span class=\"p\">);<\/span>\n\n<span class=\"c1\">\/\/ Usage<\/span>\n<span class=\"kt\">int<\/span> <span class=\"nf\">av_bsf_send_packet<\/span><span class=\"p\">(<\/span><span class=\"n\">AVBSFContext<\/span> <span class=\"o\">*<\/span><span class=\"n\">ctx<\/span><span class=\"p\">,<\/span> <span class=\"n\">AVPacket<\/span> <span class=\"o\">*<\/span><span class=\"n\">pkt<\/span><span class=\"p\">);<\/span>\n<span class=\"kt\">int<\/span> <span class=\"nf\">av_bsf_receive_packet<\/span><span class=\"p\">(<\/span><span class=\"n\">AVBSFContext<\/span> <span class=\"o\">*<\/span><span class=\"n\">ctx<\/span><span class=\"p\">,<\/span> <span class=\"n\">AVPacket<\/span> <span class=\"o\">*<\/span><span class=\"n\">pkt<\/span><span class=\"p\">);<\/span>\n\n<span class=\"c1\">\/\/ Cleanup<\/span>\n<span class=\"kt\">void<\/span> <span class=\"nf\">av_bsf_free<\/span><span class=\"p\">(<\/span><span class=\"n\">AVBSFContext<\/span> <span class=\"o\">**<\/span><span class=\"n\">ctx<\/span><span class=\"p\">);<\/span>\n<\/pre>\n<\/div>\n<p>In order to use a <strong>bsf<\/strong> you need to:<\/p>\n<ul>\n<li>Look up its definition <code>AVBitStreamFilter<\/code> using a <strong>query<\/strong> function.<\/li>\n<li>Set up a specific context <code>AVBSFContext<\/code>, by allocating, configuring and then initializing it.<\/li>\n<li>Feed the input using <code>av_bsf_send_packet<\/code> function and get the processed output once it is ready using <code>av_bsf_receive_packet<\/code>.<\/li>\n<li>Once you are done <code>av_bsf_free<\/code> cleans up the memory used for the context and the internal buffers.<\/li>\n<\/ul>\n<h3>Query<\/h3>\n<p>You can <strong>enumerate<\/strong> the available filters<\/p>\n<div class=\"codehilite\">\n<pre><span class=\"kt\">void<\/span> <span class=\"o\">*<\/span><span class=\"n\">state<\/span> <span class=\"o\">=<\/span> <span class=\"nb\">NULL<\/span><span class=\"p\">;<\/span>\n\n<span class=\"k\">const<\/span> <span class=\"n\">AVBitStreamFilter<\/span> <span class=\"o\">*<\/span><span class=\"n\">bsf<\/span><span class=\"p\">;<\/span>\n\n<span class=\"k\">while<\/span> <span class=\"p\">((<\/span><span class=\"n\">bsf<\/span> <span class=\"o\">=<\/span> <span class=\"n\">av_bsf_next<\/span><span class=\"p\">(<\/span><span class=\"o\">&amp;<\/span><span class=\"n\">state<\/span><span class=\"p\">))<\/span> <span class=\"p\">{<\/span>\n    <span class=\"n\">av_log<\/span><span class=\"p\">(<\/span><span class=\"nb\">NULL<\/span><span class=\"p\">,<\/span> <span class=\"n\">AV_LOG_INFO<\/span><span class=\"p\">,<\/span> <span class=\"s\">&quot;%s<\/span><span class=\"se\">\\n<\/span><span class=\"s\">&quot;<\/span><span class=\"p\">,<\/span> <span class=\"n\">bsf<\/span><span class=\"o\">-&gt;<\/span><span class=\"n\">name<\/span><span class=\"p\">);<\/span>\n<span class=\"p\">}<\/span>\n<\/pre>\n<\/div>\n<p>or directly pick the one you need by <strong>name<\/strong>:<\/p>\n<div class=\"codehilite\">\n<pre><span class=\"k\">const<\/span> <span class=\"n\">AVBitStreamFilter<\/span> <span class=\"o\">*<\/span><span class=\"n\">bsf<\/span> <span class=\"o\">=<\/span> <span class=\"n\">av_bsf_get_by_name<\/span><span class=\"p\">(<\/span><span class=\"s\">&quot;hevc_mp4toannexb&quot;<\/span><span class=\"p\">);<\/span>\n<\/pre>\n<\/div>\n<h3>Setup<\/h3>\n<p>A <strong>bsf<\/strong> may use some <strong>codec parameters<\/strong> and <strong>time_base<\/strong> and provide updated ones.<\/p>\n<div class=\"codehilite\">\n<pre><span class=\"n\">AVBSFContext<\/span> <span class=\"o\">*<\/span><span class=\"n\">ctx<\/span><span class=\"p\">;<\/span>\n\n<span class=\"n\">ret<\/span> <span class=\"o\">=<\/span> <span class=\"n\">av_bsf_alloc<\/span><span class=\"p\">(<\/span><span class=\"n\">bsf<\/span><span class=\"p\">,<\/span> <span class=\"o\">&amp;<\/span><span class=\"n\">ctx<\/span><span class=\"p\">);<\/span>\n<span class=\"k\">if<\/span> <span class=\"p\">(<\/span><span class=\"n\">ret<\/span> <span class=\"o\">&lt;<\/span> <span class=\"mi\">0<\/span><span class=\"p\">)<\/span>\n    <span class=\"k\">return<\/span> <span class=\"n\">ret<\/span><span class=\"p\">;<\/span>\n\n<span class=\"n\">ret<\/span> <span class=\"o\">=<\/span> <span class=\"n\">avcodec_parameters_copy<\/span><span class=\"p\">(<\/span><span class=\"n\">ctx<\/span><span class=\"o\">-&gt;<\/span><span class=\"n\">par_in<\/span><span class=\"p\">,<\/span> <span class=\"n\">in<\/span><span class=\"o\">-&gt;<\/span><span class=\"n\">codecpar<\/span><span class=\"p\">);<\/span>\n<span class=\"k\">if<\/span> <span class=\"p\">(<\/span><span class=\"n\">ret<\/span> <span class=\"o\">&lt;<\/span> <span class=\"mi\">0<\/span><span class=\"p\">)<\/span>\n    <span class=\"k\">goto<\/span> <span class=\"n\">fail<\/span><span class=\"p\">;<\/span>\n\n<span class=\"n\">ctx<\/span><span class=\"o\">-&gt;<\/span><span class=\"n\">time_base_in<\/span> <span class=\"o\">=<\/span> <span class=\"n\">in<\/span><span class=\"o\">-&gt;<\/span><span class=\"n\">time_base<\/span><span class=\"p\">;<\/span>\n\n<span class=\"n\">ret<\/span> <span class=\"o\">=<\/span> <span class=\"n\">av_bsf_init<\/span><span class=\"p\">(<\/span><span class=\"n\">ctx<\/span><span class=\"p\">);<\/span>\n<span class=\"k\">if<\/span> <span class=\"p\">(<\/span><span class=\"n\">ret<\/span> <span class=\"o\">&lt;<\/span> <span class=\"mi\">0<\/span><span class=\"p\">)<\/span>\n    <span class=\"k\">goto<\/span> <span class=\"n\">fail<\/span><span class=\"p\">;<\/span>\n\n<span class=\"n\">ret<\/span> <span class=\"o\">=<\/span> <span class=\"n\">avcodec_parameters_copy<\/span><span class=\"p\">(<\/span><span class=\"n\">out<\/span><span class=\"o\">-&gt;<\/span><span class=\"n\">codecpar<\/span><span class=\"p\">,<\/span> <span class=\"n\">ctx<\/span><span class=\"o\">-&gt;<\/span><span class=\"n\">par_out<\/span><span class=\"p\">);<\/span>\n<span class=\"k\">if<\/span> <span class=\"p\">(<\/span><span class=\"n\">ret<\/span> <span class=\"o\">&lt;<\/span> <span class=\"mi\">0<\/span><span class=\"p\">)<\/span>\n    <span class=\"k\">goto<\/span> <span class=\"n\">fail<\/span><span class=\"p\">;<\/span>\n\n<span class=\"n\">out<\/span><span class=\"o\">-&gt;<\/span><span class=\"n\">time_base<\/span> <span class=\"o\">=<\/span> <span class=\"n\">ctx<\/span><span class=\"o\">-&gt;<\/span><span class=\"n\">time_base_out<\/span><span class=\"p\">;<\/span>\n<\/pre>\n<\/div>\n<h3>Usage<\/h3>\n<p>Multiple <code>AVPackets<\/code> may be consumed before an <code>AVPacket<\/code> is emitted or multiple <code>AVPackets<\/code> may be produced out of a single input one.<\/p>\n<div class=\"codehilite\">\n<pre><span class=\"n\">AVPacket<\/span> <span class=\"o\">*<\/span><span class=\"n\">pkt<\/span><span class=\"p\">;<\/span>\n\n<span class=\"k\">while<\/span> <span class=\"p\">(<\/span><span class=\"n\">got_new_packet<\/span><span class=\"p\">(<\/span><span class=\"o\">&amp;<\/span><span class=\"n\">pkt<\/span><span class=\"p\">))<\/span> <span class=\"p\">{<\/span>\n    <span class=\"n\">ret<\/span> <span class=\"o\">=<\/span> <span class=\"n\">av_bsf_send_packet<\/span><span class=\"p\">(<\/span><span class=\"n\">ctx<\/span><span class=\"p\">,<\/span> <span class=\"n\">pkt<\/span><span class=\"p\">);<\/span>\n    <span class=\"k\">if<\/span> <span class=\"p\">(<\/span><span class=\"n\">ret<\/span> <span class=\"o\">&lt;<\/span> <span class=\"mi\">0<\/span><span class=\"p\">)<\/span>\n        <span class=\"k\">goto<\/span> <span class=\"n\">fail<\/span><span class=\"p\">;<\/span>\n\n    <span class=\"k\">while<\/span> <span class=\"p\">((<\/span><span class=\"n\">ret<\/span> <span class=\"o\">=<\/span> <span class=\"n\">av_bsf_receive_packet<\/span><span class=\"p\">(<\/span><span class=\"n\">ctx<\/span><span class=\"p\">,<\/span> <span class=\"n\">pkt<\/span><span class=\"p\">))<\/span> <span class=\"o\">==<\/span> <span class=\"mi\">0<\/span><span class=\"p\">)<\/span> <span class=\"p\">{<\/span>\n        <span class=\"n\">yield_packet<\/span><span class=\"p\">(<\/span><span class=\"n\">pkt<\/span><span class=\"p\">);<\/span>\n    <span class=\"p\">}<\/span>\n\n    <span class=\"k\">if<\/span> <span class=\"p\">(<\/span><span class=\"n\">ret<\/span> <span class=\"o\">==<\/span> <span class=\"n\">AVERROR<\/span><span class=\"p\">(<\/span><span class=\"n\">EAGAIN<\/span><span class=\"p\">)<\/span>\n        <span class=\"k\">continue<\/span><span class=\"p\">;<\/span>\n    <span class=\"n\">IF<\/span> <span class=\"p\">(<\/span><span class=\"n\">ret<\/span> <span class=\"o\">==<\/span> <span class=\"n\">AVERROR_EOF<\/span><span class=\"p\">)<\/span>\n        <span class=\"k\">goto<\/span> <span class=\"n\">end<\/span><span class=\"p\">;<\/span>\n    <span class=\"k\">if<\/span> <span class=\"p\">(<\/span><span class=\"n\">ret<\/span> <span class=\"o\">&lt;<\/span> <span class=\"mi\">0<\/span><span class=\"p\">)<\/span>\n        <span class=\"k\">goto<\/span> <span class=\"n\">fail<\/span><span class=\"p\">;<\/span>\n<span class=\"p\">}<\/span>\n\n<span class=\"c1\">\/\/ Flush<\/span>\n<span class=\"n\">ret<\/span> <span class=\"o\">=<\/span> <span class=\"n\">av_bsf_send_packet<\/span><span class=\"p\">(<\/span><span class=\"n\">ctx<\/span><span class=\"p\">,<\/span> <span class=\"nb\">NULL<\/span><span class=\"p\">);<\/span>\n<span class=\"k\">if<\/span> <span class=\"p\">(<\/span><span class=\"n\">ret<\/span> <span class=\"o\">&lt;<\/span> <span class=\"mi\">0<\/span><span class=\"p\">)<\/span>\n    <span class=\"k\">goto<\/span> <span class=\"n\">fail<\/span><span class=\"p\">;<\/span>\n\n<span class=\"k\">while<\/span> <span class=\"p\">((<\/span><span class=\"n\">ret<\/span> <span class=\"o\">=<\/span> <span class=\"n\">av_bsf_receive_packet<\/span><span class=\"p\">(<\/span><span class=\"n\">ctx<\/span><span class=\"p\">,<\/span> <span class=\"n\">pkt<\/span><span class=\"p\">))<\/span> <span class=\"o\">==<\/span> <span class=\"mi\">0<\/span><span class=\"p\">)<\/span> <span class=\"p\">{<\/span>\n    <span class=\"n\">yield_packet<\/span><span class=\"p\">(<\/span><span class=\"n\">pkt<\/span><span class=\"p\">);<\/span>\n<span class=\"p\">}<\/span>\n\n<span class=\"k\">if<\/span> <span class=\"p\">(<\/span><span class=\"n\">ret<\/span> <span class=\"o\">!=<\/span> <span class=\"n\">AVERROR_EOF<\/span><span class=\"p\">)<\/span>\n    <span class=\"k\">goto<\/span> <span class=\"n\">fail<\/span><span class=\"p\">;<\/span>\n<\/pre>\n<\/div>\n<p>In order to signal the end of stream a NULL pkt should be fed to <code>send_packet<\/code>.<\/p>\n<h3>Cleanup<\/h3>\n<p>The cleanup function matches the <code>av_freep<\/code> signature so it takes the address of the AVBSFContext pointer.<\/p>\n<div class=\"codehilite\">\n<pre>    <span class=\"n\">av_bsf_free<\/span><span class=\"p\">(<\/span><span class=\"o\">&amp;<\/span><span class=\"n\">ctx<\/span><span class=\"p\">);<\/span>\n<\/pre>\n<\/div>\n<p>All the memory is freed and the <code>ctx<\/code> pointer is set to <strong>NULL<\/strong>.<\/p>\n<h2>Coming Soon<\/h2>\n<p>Hopefully next I&#8217;ll document the new HWAccel layer that already landed and some other API that I discussed with <a href=\"http:\/\/codecs.multimedia.cx\/?p=1057\">Kostya<\/a> before.<br \/>\nSadly my blog-time (and spare time in general) shrunk a lot in the past months so he rightfully <strong>blamed<\/strong> me a lot.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Last weekend, after few months of work, the new bitstream filter API eventually landed. Bitstream filters In Libav is possible to manipulate raw and encoded data in many ways, the most common being Demuxing: extracting single data packets and their timing information Decoding: converting the compressed data packets in raw video or audio frames Encoding: &hellip; <a href=\"https:\/\/blogs.gentoo.org\/lu_zero\/2016\/03\/21\/bitstream-filtering\/\" class=\"more-link\">Continue reading <span class=\"screen-reader-text\">Bitstream Filtering<\/span><\/a><\/p>\n","protected":false},"author":10,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"spay_email":"","jetpack_publicize_message":"","jetpack_is_tweetstorm":false,"jetpack_publicize_feature_enabled":true},"categories":[14],"tags":[19],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/p1aGWH-9j","_links":{"self":[{"href":"https:\/\/blogs.gentoo.org\/lu_zero\/wp-json\/wp\/v2\/posts\/577"}],"collection":[{"href":"https:\/\/blogs.gentoo.org\/lu_zero\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blogs.gentoo.org\/lu_zero\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blogs.gentoo.org\/lu_zero\/wp-json\/wp\/v2\/users\/10"}],"replies":[{"embeddable":true,"href":"https:\/\/blogs.gentoo.org\/lu_zero\/wp-json\/wp\/v2\/comments?post=577"}],"version-history":[{"count":1,"href":"https:\/\/blogs.gentoo.org\/lu_zero\/wp-json\/wp\/v2\/posts\/577\/revisions"}],"predecessor-version":[{"id":578,"href":"https:\/\/blogs.gentoo.org\/lu_zero\/wp-json\/wp\/v2\/posts\/577\/revisions\/578"}],"wp:attachment":[{"href":"https:\/\/blogs.gentoo.org\/lu_zero\/wp-json\/wp\/v2\/media?parent=577"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blogs.gentoo.org\/lu_zero\/wp-json\/wp\/v2\/categories?post=577"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blogs.gentoo.org\/lu_zero\/wp-json\/wp\/v2\/tags?post=577"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}