{"id":549,"date":"2015-11-08T21:57:30","date_gmt":"2015-11-08T21:57:30","guid":{"rendered":"http:\/\/blogs.gentoo.org\/lu_zero\/?p=549"},"modified":"2015-11-13T16:10:18","modified_gmt":"2015-11-13T16:10:18","slug":"trusting-the-context","status":"publish","type":"post","link":"https:\/\/blogs.gentoo.org\/lu_zero\/2015\/11\/08\/trusting-the-context\/","title":{"rendered":"Trusting the context"},"content":{"rendered":"<p>This mini-post spurred from this <a href=\"https:\/\/bugzilla.libav.org\/show_bug.cgi?id=911\">bug<\/a>.<\/p>\n<h2>AVFrame and AVCodecContext<\/h2>\n<p>In <a href=\"http:\/\/libav.org\">Libav<\/a> there are a number of patterns shared across most of the components.<br \/>\nDoes not matter if it models a codec, a demuxer or a resampler: You interact with it using a <code>Context<\/code> and you get data in or out of the module using some kind of <code>Abstraction<\/code> that wraps data and useful information such as the timestamp. Today&#8217;s post is about AVFrames and AVCodecContext.<\/p>\n<h3>AVFrame<\/h3>\n<p>The most used abstraction in Libav by far is the <code>AVFrame<\/code>. It wraps some kind of <code>raw data<\/code> that can be produced by <strong>decoders<\/strong> and fed to <strong>encoders<\/strong>, passed through <strong>filters<\/strong>, <strong>scalers<\/strong> and <strong>resamplers<\/strong>.<\/p>\n<p>It is quite flexible and contains the <em>data<\/em> and all the information to understand it e.g.:<\/p>\n<ul>\n<li><code>format<\/code>: Used to describe either the pixel format for video and the sample format for audio.<\/li>\n<li><code>width<\/code> and <code>height<\/code>: The dimension of a video frame.<\/li>\n<li><code>channel_layout<\/code>, <code>nb_samples<\/code> and <code>sample_rate<\/code> for audio frames.<\/li>\n<\/ul>\n<h3>AVCodecContext<\/h3>\n<p>This context contains all the information useful to <em>describe<\/em> a codec and to <em>configure<\/em> an encoder or a decoder (the generic, common features, there are <strong>private<\/strong> options for specific features).<\/p>\n<p>Being shared with encoder, decoder and (until Anton&#8217;s plan to avoid it is deployed) container streams this context is fairly large and a good deal of its fields are a little confusing since they seem to replicate what is present in the <strong>AVFrame<\/strong> or because they aren&#8217;t marked as <strong>write-only<\/strong> since they might be read in few situation.<\/p>\n<p>In the <a href=\"https:\/\/bugzilla.libav.org\/show_bug.cgi?id=911\">bug<\/a> mentioned <code>channel_layout<\/code> was the confusing one but also <code>width<\/code> and <code>height<\/code> caused problems to people thinking the value of those fields in the <code>AVCodecContext<\/code> would represent what is in the <code>AVFrame<\/code> (then you&#8217;d wonder why you should have them in two different places&#8230;).<\/p>\n<p>As a rule of thumb everything that is set in a context is either the starting configuration and bound to change in the future.<\/p>\n<p>Video decoders can reconfigure themselves and output video frames with completely different geometries, audio decoders can report a completely different number of channels or variations in their layout and so on.<\/p>\n<p>Some encoders are able to reconfigure on the fly as well, but usually with more strict constraints.<\/p>\n<h2>Why their information is not the same<\/h2>\n<p>The fields in the AVCodecContext are used internally and updated as needed by the decoder. The decoder can be multithreaded so the AVFrame you are getting from one of the <code>avcodec_decode_something()<\/code> functions is not the last frame decoded.<\/p>\n<blockquote><p>\n  Do not expect any of the fields with names similar to the ones provided by AVFrame to stay immutable or to match the values provided by the AVFrame.\n<\/p><\/blockquote>\n<h2>Common pitfalls<\/h2>\n<h3>Allocating video surfaces<\/h3>\n<p>Some quite common mistake is to use the <code>AVCodecContext<\/code> <code>coded_width<\/code> and <code>coded_height<\/code> to allocate the surfaces to present the decoded frames.<\/p>\n<p>As said the frame geometry can <strong>change<\/strong> mid-stream, so if you do that best case you have some lovely green surrounding your picture, worst case you have a bad crash.<\/p>\n<p>I suggest to always check that the <code>AVFrame<\/code> dimensions fit and be ready to reconfigure your video out when that happens.<\/p>\n<h3>Resampling audio<\/h3>\n<p>If you are using a current version of Libav you have <code>avresample_convert_frame()<\/code> doing most of the work for you, if you are not you need to check that <code>format<\/code> <code>channel_layout<\/code> and <code>sample_rate<\/code> do not change and manually reconfigure.<\/p>\n<h3>Rescaling video<\/h3>\n<p>Similarly you can misconfigure <strong>swscale<\/strong> and you should check manually that <code>format<\/code>, <code>width<\/code> and <code>height<\/code> and reconfigure as well. The <strong>AVScale<\/strong> draft API on provides an <code>avscale_process_frame()<\/code>.<\/p>\n<h2>In closing<\/h2>\n<p>Be extra careful, think twice and beware of the examples you might find on internet, they might work until they wont.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>This mini-post spurred from this bug. AVFrame and AVCodecContext In Libav there are a number of patterns shared across most of the components. Does not matter if it models a codec, a demuxer or a resampler: You interact with it using a Context and you get data in or out of the module using some &hellip; <a href=\"https:\/\/blogs.gentoo.org\/lu_zero\/2015\/11\/08\/trusting-the-context\/\" class=\"more-link\">Continue reading <span class=\"screen-reader-text\">Trusting the context<\/span><\/a><\/p>\n","protected":false},"author":10,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"spay_email":"","jetpack_publicize_message":"","jetpack_is_tweetstorm":false,"jetpack_publicize_feature_enabled":true},"categories":[14],"tags":[],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/p1aGWH-8R","_links":{"self":[{"href":"https:\/\/blogs.gentoo.org\/lu_zero\/wp-json\/wp\/v2\/posts\/549"}],"collection":[{"href":"https:\/\/blogs.gentoo.org\/lu_zero\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blogs.gentoo.org\/lu_zero\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blogs.gentoo.org\/lu_zero\/wp-json\/wp\/v2\/users\/10"}],"replies":[{"embeddable":true,"href":"https:\/\/blogs.gentoo.org\/lu_zero\/wp-json\/wp\/v2\/comments?post=549"}],"version-history":[{"count":5,"href":"https:\/\/blogs.gentoo.org\/lu_zero\/wp-json\/wp\/v2\/posts\/549\/revisions"}],"predecessor-version":[{"id":557,"href":"https:\/\/blogs.gentoo.org\/lu_zero\/wp-json\/wp\/v2\/posts\/549\/revisions\/557"}],"wp:attachment":[{"href":"https:\/\/blogs.gentoo.org\/lu_zero\/wp-json\/wp\/v2\/media?parent=549"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blogs.gentoo.org\/lu_zero\/wp-json\/wp\/v2\/categories?post=549"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blogs.gentoo.org\/lu_zero\/wp-json\/wp\/v2\/tags?post=549"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}