{"id":415,"date":"2015-03-23T00:54:50","date_gmt":"2015-03-23T00:54:50","guid":{"rendered":"http:\/\/blogs.gentoo.org\/lu_zero\/?p=415"},"modified":"2015-06-05T11:02:33","modified_gmt":"2015-06-05T11:02:33","slug":"decoupling-an-api","status":"publish","type":"post","link":"https:\/\/blogs.gentoo.org\/lu_zero\/2015\/03\/23\/decoupling-an-api\/","title":{"rendered":"Decoupling an API"},"content":{"rendered":"

This weekend on #libav-devel we discussed again a bit about the problems with the current core avcodec<\/a> api.<\/p>\n

Current situation<\/h2>\n

Decoding<\/h3>\n

We have 3 decoding functions<\/a> for each of the supported kind of media types: Audio, Video and Subtitles.<\/p>\n

Subtitles are already a sore thumb since they are not using AVFrame but a specialized structure, let’s ignore it for now. Audio and Video share pretty much the same signature:<\/p>\n

\n
int<\/span> avcodec_decode_something<\/span>(<\/span>AVCodecContext<\/span> *<\/span>avctx<\/span>,<\/span> AVFrame<\/span> *<\/span>f<\/span>,<\/span> int<\/span> *<\/span>got_frame<\/span>,<\/span> AVPacket<\/span> *<\/span>p<\/span>)<\/span>\n<\/pre>\n<\/div>\n
It takes a context<\/strong> pointer containing the decoder state, consumes a demuxed packet<\/strong> and optionally outputs a decoded frame<\/em> containing raw<\/em> data in a certain format (audio samples, a video frame).<\/p>\n
The usage model is quite simple it takes packets<\/strong> and whenever it has enough encoded data to emit a frame it emits one, the got_frame<\/strong> pointer signals if a frame is ready or more data is needed.<\/p>\n
Problem:<\/strong><\/p>\n
\nWhat if 1 AVPacket is near always enough to output 2 or more frames<\/strong> of raw data?\n<\/p><\/blockquote>\n
This happens with MVC<\/a> and other real-world scenarios.<\/p>\n
In general our current API cannot cope with it cleanly.<\/p>\n
While working with the MediaSDK interface from Intel and now with MMAL for the Rasberry Pi, similar problems arisen due the natural parallelism the underlying hardware has.<\/p>\n
Encoding<\/h3>\n
We have again 3 functions<\/a> again Subtitles are somehow different, while Audio and Video are sort of nicely uniform.<\/p>\n
\n
int<\/span> avcodec_encode_something<\/span>(<\/span>AVCodecContext<\/span> *<\/span>avctx<\/span>,<\/span> AVPacket<\/span> *<\/span>p<\/span>,<\/span> const<\/span> AVFrame<\/span> *<\/span>f<\/span>,<\/span> int<\/span> *<\/span>got_packet<\/span>)<\/span>\n<\/pre>\n<\/div>\n
It is pretty much the dual of the decoding<\/strong> function: the context pointer is the same, a frame<\/strong> of raw data enters and a packet<\/strong> of encoded data. Again we have a pointer to signal if we had enough data and an encoded packet<\/strong> had been outputted.<\/p>\n
Problem:<\/strong><\/p>\n
\nAgain we might get multiple AVPacket produced out of a single AVFrame data fed.\n<\/p><\/blockquote>\n
This happens when the HEVC “workaround”<\/em> to encode interlaced content makes the encoder to output the two separate fields as separate encoded frames.<\/p>\n
Again, the API cannot cope with it cleanly and threaded or otherwise parallel encoding fit the model just barely.<\/p>\n
Decoupling the process<\/h2>\n
To fix this issue (and make our users life simpler) the idea<\/a> is to split the feeding<\/strong> data function from the one actually providing the processed data.<\/p>\n
\n
int<\/span> avcodec_decode_push<\/span>(<\/span>AVCodecContext<\/span> *<\/span>avctx<\/span>,<\/span> AVPacket<\/span> *<\/span>packet<\/span>);<\/span>\nint<\/span> avcodec_decode_pull<\/span>(<\/span>AVCodecContext<\/span> *<\/span>avctx<\/span>,<\/span> AVFrame<\/span> *<\/span>frame<\/span>);<\/span>\nint<\/span> avcodec_decode_need_data<\/span>(<\/span>AVCodecContext<\/span> *<\/span>avctx<\/span>);<\/span>\nint<\/span> avcodec_decode_have_data<\/span>(<\/span>AVCodecContext<\/span> *<\/span>avctx<\/span>);<\/span>\n<\/pre>\n<\/div>\n
\n
int<\/span> avcodec_encode_push<\/span>(<\/span>AVCodecContext<\/span> *<\/span>avctx<\/span>,<\/span> AVFrame<\/span> *<\/span>frame<\/span>);<\/span>\nint<\/span> avcodec_encode_pull<\/span>(<\/span>AVCodecContext<\/span> *<\/span>avctx<\/span>,<\/span> AVPacket<\/span> *<\/span>packet<\/span>);<\/span>\nint<\/span> avcodec_encode_need_data<\/span>(<\/span>AVCodecContext<\/span> *<\/span>avctx<\/span>);<\/span>\nint<\/span> avcodec_encode_have_data<\/span>(<\/span>AVCodecContext<\/span> *<\/span>avctx<\/span>);<\/span>\n<\/pre>\n<\/div>\n
From a single function 4 are provided, why it is simple?<\/p>\n
The current workflow is more or less like<\/p>\n
\n
while<\/span> (<\/span>get_packet_from_demuxer<\/span>(<\/span>&<\/span>pkt<\/span>))<\/span> {<\/span>\n    ret<\/span> =<\/span> avcodec_decode_something<\/span>(<\/span>avctx<\/span>,<\/span> frame<\/span>,<\/span> &<\/span>got_frame<\/span>,<\/span> pkt<\/span>);<\/span>\n    if<\/span> (<\/span>got_frame<\/span>)<\/span> {<\/span>\n        render_frame<\/span>(<\/span>frame<\/span>);<\/span>\n    }<\/span>\n    if<\/span> (<\/span>ret<\/span> <<\/span> 0<\/span>)<\/span> {<\/span>\n        manage_error<\/span>(<\/span>ret<\/span>);<\/span>\n    }<\/span>\n}<\/span>\n<\/pre>\n<\/div>\n
The get_packet_from_demuxer()<\/code> is a function that dequeues from some queue<\/strong> the encoded data or directly call the demuxer (beware: having your I\/O-intensive demuxer function blocking<\/strong> your CPU-intensive decoding function isn’t nice), render_frame()<\/code> is as well either something directly talking to some kind of I\/O-subsystem or enqueuing the data to have the actual rendering<\/strong> (including format conversion, overlaying and scaling) in another thread.<\/p>\n
The new API makes much easier to keep the multiple area of concern separated, so they won’t trip each other while the casual user would have something like<\/p>\n
\n
while<\/span> (<\/span>ret<\/span> >=<\/span> 0<\/span>)<\/span> {<\/span>\n    while<\/span> ((<\/span>ret<\/span> =<\/span> avcodec_decode_need_data<\/span>(<\/span>avctx<\/span>))<\/span> ><\/span> 0<\/span>)<\/span> {<\/span>\n        ret<\/span> =<\/span> get_packet_from_demuxer<\/span>(<\/span>&<\/span>pkt<\/span>);<\/span>\n        if<\/span> (<\/span>ret<\/span> <<\/span> 0<\/span>)<\/span>\n           ...<\/span>\n        ret<\/span> =<\/span> avcodec_decode_push<\/span>(<\/span>avctx<\/span>,<\/span> &<\/span>pkt<\/span>);<\/span>\n        if<\/span> (<\/span>ret<\/span> <<\/span> 0<\/span>)<\/span>\n           ...<\/span>\n    }<\/span>\n    while<\/span> ((<\/span>ret<\/span> =<\/span> avcodec_decode_have_data<\/span>(<\/span>avctx<\/span>))<\/span> ><\/span> 0<\/span>)<\/span> {<\/span>\n        ret<\/span> =<\/span> avcodec_decode_pull<\/span>(<\/span>avctx<\/span>,<\/span> frame<\/span>);<\/span>\n        if<\/span> (<\/span>ret<\/span> <<\/span> 0<\/span>)<\/span>\n           ...<\/span>\n        render_frame<\/span>(<\/span>frame<\/span>);<\/span>\n    }<\/span>\n}<\/span>\n<\/pre>\n<\/div>\n
That has probably few more lines.<\/p>\n
Asyncronous API<\/h2>\n
Since the decoupled API is that simple, is possible to craft something more immediate for the casual user.<\/p>\n
\n
typedef<\/span> struct<\/span> AVCodecDecodeCallback<\/span> {<\/span>\n    int<\/span> (<\/span>*<\/span>pull_packet<\/span>)(<\/span>void<\/span> *<\/span>priv<\/span>,<\/span> AVPacket<\/span> *<\/span>pkt<\/span>);<\/span>\n    int<\/span> (<\/span>*<\/span>push_frame<\/span>)(<\/span>void<\/span> *<\/span>priv<\/span>,<\/span> AVFrame<\/span> *<\/span>frame<\/span>);<\/span>\n    void<\/span> *<\/span>priv_data<\/span>;<\/span>\n}<\/span> AVCodecDecodeCallback<\/span>;<\/span>\n\nint<\/span> avcodec_register_decode_callbacks<\/span>(<\/span>AVCodecContext<\/span> *<\/span>avctx<\/span>,<\/span> AVCodecDecodeCallback<\/span> *<\/span>cb<\/span>);<\/span>\n\nint<\/span> avcodec_decode_loop<\/span>(<\/span>AVCodecContext<\/span> *<\/span>avctx<\/span>)<\/span>\n{<\/span>\n    AVCodecDecodeCallback<\/span> *<\/span>cb<\/span> =<\/span> avctx<\/span>-><\/span>cb<\/span>;<\/span>\n    int<\/span> ret<\/span>;<\/span>\n    while<\/span> ((<\/span>ret<\/span> =<\/span> avcodec_decode_need_data<\/span>(<\/span>avctx<\/span>))<\/span> ><\/span> 0<\/span>)<\/span> {<\/span>\n        ret<\/span> =<\/span> cb<\/span>-><\/span>pull_packet<\/span>(<\/span>cb<\/span>-><\/span>priv_data<\/span>,<\/span> &<\/span>pkt<\/span>);<\/span>\n        if<\/span> (<\/span>ret<\/span> <<\/span> 0<\/span>)<\/span>\n            return<\/span> ret<\/span>;<\/span>\n        ret<\/span> =<\/span> avcodec_decode_push<\/span>(<\/span>avctx<\/span>,<\/span> &<\/span>pkt<\/span>);<\/span>\n        if<\/span> (<\/span>ret<\/span> <<\/span> 0<\/span>)<\/span>\n            return<\/span> ret<\/span>;<\/span>\n    }<\/span>\n    while<\/span> ((<\/span>ret<\/span> =<\/span> avcodec_decode_have_data<\/span>(<\/span>avctx<\/span>))<\/span> ><\/span> 0<\/span>)<\/span> {<\/span>\n        ret<\/span> =<\/span> avcodec_decode_pull<\/span>(<\/span>avctx<\/span>,<\/span> frame<\/span>);<\/span>\n        if<\/span> (<\/span>ret<\/span> <<\/span> 0<\/span>)<\/span>\n            return<\/span> ret<\/span>;<\/span>\n        ret<\/span> =<\/span> cb<\/span>-><\/span>push_frame<\/span>(<\/span>cb<\/span>-><\/span>priv_data<\/span>,<\/span> frame<\/span>);<\/span>\n    }<\/span>\n    return<\/span> ret<\/span>;<\/span>\n}<\/span>\n<\/pre>\n<\/div>\n
So the actual minimum decoding loop can be just 2 calls:<\/p>\n
\n
ret<\/span> =<\/span> avcodec_register_decode_callbacks<\/span>(<\/span>avctx<\/span>,<\/span> cb<\/span>);<\/span>\nif<\/span> (<\/span>ret<\/span> <<\/span> 0<\/span>)<\/span>\n   ...<\/span>\nwhile<\/span> ((<\/span>ret<\/span> =<\/span> avcodec_decode_loop<\/span>(<\/span>avctx<\/span>))<\/span> >=<\/span> 0<\/span>);<\/span>\n<\/pre>\n<\/div>\n
Cute, isn’t it?<\/p>\n
Theory is simple …<\/h2>\n
… the practice not so much:
\n– there are plenty of implementation issues to take in account.
\n– LOTS<\/strong> of tedious work converting all the codecs to the new API.
\n– lots of details to iron out (e.g. have_data()<\/code> and need_data()<\/code> should block or not?)<\/p>\n
We did radical overhauls before, such as introducing reference-counted<\/strong> AVFrames thanks to Anton, so we aren’t much scared of reshaping and cleaning the codebase once more.<\/p>\n
If you like the ideas posted above or you want to discuss them more, you can join the Libav irc channel<\/a> or mailing list<\/a> to discuss and help.<\/p>\n","protected":false},"excerpt":{"rendered":"
This weekend on #libav-devel we discussed again a bit about the problems with the current core avcodec api. Current situation Decoding We have 3 decoding functions for each of the supported kind of media types: Audio, Video and Subtitles. Subtitles are already a sore thumb since they are not using AVFrame but a specialized structure, … Continue reading Decoupling an API<\/span><\/a><\/p>\n","protected":false},"author":10,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"spay_email":"","jetpack_publicize_message":"","jetpack_is_tweetstorm":false,"jetpack_publicize_feature_enabled":true},"categories":[14,6],"tags":[19],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/p1aGWH-6H","_links":{"self":[{"href":"https:\/\/blogs.gentoo.org\/lu_zero\/wp-json\/wp\/v2\/posts\/415"}],"collection":[{"href":"https:\/\/blogs.gentoo.org\/lu_zero\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blogs.gentoo.org\/lu_zero\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blogs.gentoo.org\/lu_zero\/wp-json\/wp\/v2\/users\/10"}],"replies":[{"embeddable":true,"href":"https:\/\/blogs.gentoo.org\/lu_zero\/wp-json\/wp\/v2\/comments?post=415"}],"version-history":[{"count":6,"href":"https:\/\/blogs.gentoo.org\/lu_zero\/wp-json\/wp\/v2\/posts\/415\/revisions"}],"predecessor-version":[{"id":467,"href":"https:\/\/blogs.gentoo.org\/lu_zero\/wp-json\/wp\/v2\/posts\/415\/revisions\/467"}],"wp:attachment":[{"href":"https:\/\/blogs.gentoo.org\/lu_zero\/wp-json\/wp\/v2\/media?parent=415"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blogs.gentoo.org\/lu_zero\/wp-json\/wp\/v2\/categories?post=415"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blogs.gentoo.org\/lu_zero\/wp-json\/wp\/v2\/tags?post=415"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}