{"id":454,"date":"2015-05-02T20:54:30","date_gmt":"2015-05-02T20:54:30","guid":{"rendered":"http:\/\/blogs.gentoo.org\/lu_zero\/?p=454"},"modified":"2015-06-05T11:02:15","modified_gmt":"2015-06-05T11:02:15","slug":"splitting-a-library-hashes","status":"publish","type":"post","link":"https:\/\/blogs.gentoo.org\/lu_zero\/2015\/05\/02\/splitting-a-library-hashes\/","title":{"rendered":"Splitting a library – hashes"},"content":{"rendered":"

libavutil<\/a> contains lots that is common with the other libraries that compose Libav<\/a>. It grown a lot over the years and it’s time to consider splitting it.<\/p>\n

Monolithic vs Modular<\/h2>\n

There will always be some discussion on which approach is globally better.
\n– Jumbling everything together so you have everything there and doesn’t matter what, you have your super hammer<\/em> supporting screws, bolts, nuts and nails.
\n– Keeping the tools in separate boxes so you carry only the set of spanners you need when you need it.<\/p>\n

For software libraries you have this kind of problem all the time and at multiple levels:
\n– Do you want to have a single huge header file<\/code> with every function your library provides or a set of them organized to keep all the function related together?
\n– Do you want to link a single library or have the concerns split in multiple so you do not have to carry lots of stuff you do not use (storage and memory are still important in some applications).<\/p>\n

Usually modularity comes with the price of additional initial effort (you have to think<\/strong> about what you are going to use a little harder) and maintenance (which library should I update<\/strong>?).<\/p>\n

This blogpost is about trying to group and split bunch of unrelated functions present in a library and try to get a better API for some of them.<\/p>\n

Libavutil<\/h2>\n

The Libav libraries are written mainly in languages (C, asm) and they focus a lot on being portable. Libavutil is the foundation.<\/p>\n

It contains all the code that is common<\/strong> across libraries from the basics such as memory management<\/a> to higher level data structures<\/a>, to video and audio-specific<\/a> basic manipulation and hashes, cryptographic primitives and lossless compressors<\/a>.<\/p>\n

A lot indeed<\/strong>.<\/p>\n

Problems<\/h2>\n

Irregular Mushroom-API<\/h3>\n

Some of the highest level part of the library appeared little by little, first you need md5<\/code> and you add it, then is aes<\/code>, then you want lzo<\/code>. All the crypto expose direct functions to that specific hash, making those components non-optional even if you do not need them.<\/p>\n

\n
# libavutil\/aes.h<\/span>\nstruct<\/span> AVAES<\/span>;<\/span>\n\nstruct<\/span> AVAES<\/span> *<\/span>av_aes_alloc<\/span>(<\/span>void<\/span>);<\/span>\n\nint<\/span> av_aes_init<\/span>(<\/span>struct<\/span> AVAES<\/span> *<\/span>a<\/span>,<\/span> const<\/span> uint8_t<\/span> *<\/span>key<\/span>,<\/span> int<\/span> key_bits<\/span>,<\/span> int<\/span> decrypt<\/span>);<\/span>\nvoid<\/span> av_aes_crypt<\/span>(<\/span>struct<\/span> AVAES<\/span> *<\/span>a<\/span>,<\/span> uint8_t<\/span> *<\/span>dst<\/span>,<\/span> const<\/span> uint8_t<\/span> *<\/span>src<\/span>,<\/span> int<\/span> count<\/span>,<\/span> uint8_t<\/span> *<\/span>iv<\/span>,<\/span> int<\/span> decrypt<\/span>);<\/span>\n\n# libavutil\/xtea.h<\/span>\ntypedef<\/span> struct<\/span> AVXTEA<\/span> {<\/span>\n    uint32_t<\/span> key<\/span>[<\/span>16<\/span>];<\/span>\n}<\/span> AVXTEA<\/span>;<\/span>\n\nvoid<\/span> av_xtea_init<\/span>(<\/span>struct<\/span> AVXTEA<\/span> *<\/span>ctx<\/span>,<\/span> const<\/span> uint8_t<\/span> key<\/span>[<\/span>16<\/span>]);<\/span>\nvoid<\/span> av_xtea_crypt<\/span>(<\/span>struct<\/span> AVXTEA<\/span> *<\/span>ctx<\/span>,<\/span> uint8_t<\/span> *<\/span>dst<\/span>,<\/span> const<\/span> uint8_t<\/span> *<\/span>src<\/span>,<\/span>\n                   int<\/span> count<\/span>,<\/span> uint8_t<\/span> *<\/span>iv<\/span>,<\/span> int<\/span> decrypt<\/span>);<\/span>\n\n# libavutil\/sha.h<\/span>\nstruct<\/span> AVSHA<\/span>;<\/span>\n\nstruct<\/span> AVSHA<\/span> *<\/span>av_sha_alloc<\/span>(<\/span>void<\/span>);<\/span>\nint<\/span> av_sha_init<\/span>(<\/span>struct<\/span> AVSHA<\/span>*<\/span> context<\/span>,<\/span> int<\/span> bits<\/span>);<\/span>\nvoid<\/span> av_sha_update<\/span>(<\/span>struct<\/span> AVSHA<\/span>*<\/span> context<\/span>,<\/span> const<\/span> uint8_t<\/span>*<\/span> data<\/span>,<\/span> unsigned<\/span> int<\/span> len<\/span>);<\/span>\nvoid<\/span> av_sha_final<\/span>(<\/span>struct<\/span> AVSHA<\/span>*<\/span> context<\/span>,<\/span> uint8_t<\/span> *<\/span>digest<\/span>);<\/span>\n\n# libavutil\/md5.h<\/span>\nstruct<\/span> AVMD5<\/span>;<\/span>\n\nstruct<\/span> AVMD5<\/span> *<\/span>av_md5_alloc<\/span>(<\/span>void<\/span>);<\/span>\nvoid<\/span> av_md5_init<\/span>(<\/span>struct<\/span> AVMD5<\/span> *<\/span>ctx<\/span>);<\/span>\nvoid<\/span> av_md5_update<\/span>(<\/span>struct<\/span> AVMD5<\/span> *<\/span>ctx<\/span>,<\/span> const<\/span> uint8_t<\/span> *<\/span>src<\/span>,<\/span> const<\/span> int<\/span> len<\/span>);<\/span>\nvoid<\/span> av_md5_final<\/span>(<\/span>struct<\/span> AVMD5<\/span> *<\/span>ctx<\/span>,<\/span> uint8_t<\/span> *<\/span>dst<\/span>);<\/span>\nvoid<\/span> av_md5_sum<\/span>(<\/span>uint8_t<\/span> *<\/span>dst<\/span>,<\/span> const<\/span> uint8_t<\/span> *<\/span>src<\/span>,<\/span> const<\/span> int<\/span> len<\/span>);<\/span>\n<\/pre>\n<\/div>\n
As you might notice it got to have lots and lots of expose, similar-but-non-uniform API popping out.<\/p>\n
And if it was acceptable having a couple of hashes always around it gets not so nice if you have more to add.<\/p>\n
Right now libavutil<\/code> exposes 50 separate headers.<\/p>\n
Extending it is painful now<\/h3>\n
Since we already have that many different components inside it you think twice about adding more<\/em> stuff (if you are careful and caring), Libav is fairly modular and people do appreciate that.<\/p>\n
In my wishlist I have few items such as getting more decompressors natively implemented.<\/p>\n
Every new API is a burden to maintain (if you care about legacy and you keep maintaining releasing your older software) so adding or exposing more is always something you should consider.<\/p>\n
\nAbstracting<\/b> some details always helps, think what would be the API if each of the supported codecs has an exposed, non-uniform<\/b> set of functions to decode each?\n<\/div>\n
Ideal structure<\/h2>\n
Ideally I’d have the following layout:
\n– libavutil:  basic memory abstraction, error, logs and not much else
\n– libavdata: basic data structures, including refcounted buffers, dictionaries, trees and such
\n– libavmedia: audio samples, pixel formats, metadata, frames, packets, side data types.
\n– libavhash: hashes such md5, sha and such
\n– libavcomp: compressors such as lzo
\n– libavcrypto: aes, blowfish and such<\/p>\n
API<\/h3>\n
I already described<\/a> my ideal api for the codecs<\/strong>, today I’d detail the hashes<\/strong><\/p>\n
As seen above it is common to have init<\/code>, update<\/code>
\nfinal<\/code> and an optional utility function sum<\/code> (or calc<\/code>) that takes whole buffer buffer and returns the hash.<\/p>\n
\n
typedef<\/span> struct<\/span> AVHashLibrary<\/span>;<\/span>\ntypedef<\/span> struct<\/span> AVHash<\/span>;<\/span>\ntypedef<\/span> struct<\/span> AVHashContext<\/span>;<\/span>\n\nint<\/span> av_hash_register_all<\/span>(<\/span>AVHashLibrary<\/span> *<\/span>hashes<\/span>)<\/span>\n\nconst<\/span> AVHash<\/span> *<\/span>av_hash_get<\/span>(<\/span>AVHashLibrary<\/span> *<\/span>hashes<\/span>,<\/span> const<\/span> char<\/span> *<\/span>name<\/span>);<\/span>\n\nAVHashContext<\/span> *<\/span>avhash_open<\/span>(<\/span>AVHash<\/span> *<\/span>hash<\/span>,<\/span> AVDictionary<\/span> *<\/span>opts<\/span>);<\/span>\n\nint<\/span> av_hash_update<\/span>(<\/span>AVHashContext<\/span> *<\/span>ctx<\/span>,<\/span> const<\/span> uint8_t<\/span> *<\/span>src<\/span>,<\/span> const<\/span> int<\/span> len<\/span>);<\/span>\n\nuint8_t<\/span> *<\/span>av_hash_final<\/span>(<\/span>AVHashContext<\/span> *<\/span>ctx<\/span>,<\/span> int<\/span> *<\/span>len<\/span>);<\/span>\n\nuint8_t<\/span> *<\/span>av_hash_sum<\/span>(<\/span>AVHashContext<\/span> *<\/span>ctx<\/span>,<\/span> const<\/span> uint8_t<\/span> *<\/span>src<\/span>,<\/span> const<\/span> uint64_t<\/span> src_len<\/span>,<\/span> int<\/span> *<\/span>out_len<\/span>);<\/span>\n\nvoid<\/span> avhash_close<\/span>(<\/span>AVHashContext<\/span> *<\/span>hash<\/span>);<\/span>\n<\/pre>\n<\/div>\n
The structures are fully opaque, the AVHashLibrary<\/code> contains the list of available hashes and possibly some additional hidden state. In Libav we are trying to remove all the global variables so the list of hashes is explicit.<\/p>\n
The register_all<\/code> function just populates the list of hashes and possibly creates accessory lookup tables when needed.<\/p>\n
The get<\/code> call let you look up the hash by name, additional can be made to look it up by id.<\/p>\n
The open<\/code> function takes a dictionary for hash-specific configuration.<\/p>\n
The update<\/code> and final<\/code> function let you calculate the hash incrementally, the sum<\/code> function is a simple utility that takes a full buffer (assumed to fit an uint64_t<\/code>) and produces the hash.<\/p>\n
The NihAV<\/a> from Kostya hopefully will have a similar API with TypeLibrary<\/code>, Type<\/code> and TypeContext<\/code> structs.<\/p>\n","protected":false},"excerpt":{"rendered":"
libavutil contains lots that is common with the other libraries that compose Libav. It grown a lot over the years and it’s time to consider splitting it. Monolithic vs Modular There will always be some discussion on which approach is globally better. – Jumbling everything together so you have everything there and doesn’t matter what, … Continue reading Splitting a library – hashes<\/span><\/a><\/p>\n","protected":false},"author":10,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"spay_email":"","jetpack_publicize_message":"","jetpack_is_tweetstorm":false,"jetpack_publicize_feature_enabled":true},"categories":[14,6],"tags":[19],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/p1aGWH-7k","_links":{"self":[{"href":"https:\/\/blogs.gentoo.org\/lu_zero\/wp-json\/wp\/v2\/posts\/454"}],"collection":[{"href":"https:\/\/blogs.gentoo.org\/lu_zero\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blogs.gentoo.org\/lu_zero\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blogs.gentoo.org\/lu_zero\/wp-json\/wp\/v2\/users\/10"}],"replies":[{"embeddable":true,"href":"https:\/\/blogs.gentoo.org\/lu_zero\/wp-json\/wp\/v2\/comments?post=454"}],"version-history":[{"count":5,"href":"https:\/\/blogs.gentoo.org\/lu_zero\/wp-json\/wp\/v2\/posts\/454\/revisions"}],"predecessor-version":[{"id":466,"href":"https:\/\/blogs.gentoo.org\/lu_zero\/wp-json\/wp\/v2\/posts\/454\/revisions\/466"}],"wp:attachment":[{"href":"https:\/\/blogs.gentoo.org\/lu_zero\/wp-json\/wp\/v2\/media?parent=454"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blogs.gentoo.org\/lu_zero\/wp-json\/wp\/v2\/categories?post=454"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blogs.gentoo.org\/lu_zero\/wp-json\/wp\/v2\/tags?post=454"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}