{"id":713,"date":"2018-12-30T09:25:47","date_gmt":"2018-12-30T09:25:47","guid":{"rendered":"http:\/\/blogs.gentoo.org\/lu_zero\/?p=713"},"modified":"2018-12-30T09:25:47","modified_gmt":"2018-12-30T09:25:47","slug":"making-and-using-c-compatible-libraries-in-rust-present-and-future","status":"publish","type":"post","link":"https:\/\/blogs.gentoo.org\/lu_zero\/2018\/12\/30\/making-and-using-c-compatible-libraries-in-rust-present-and-future\/","title":{"rendered":"Making and using C-compatible libraries in rust: present and future"},"content":{"rendered":"<p>Since there are plenty of blogposts about what people would like to have or will implement in rust in 2019 here is mine.<\/p>\n<p>I spent the last few weeks of my spare time making a C-api for <a href=\"https:\/\/github.com\/xiph\/rav1e\/\">rav1e<\/a> called <a href=\"https:\/\/github.com\/lu-zero\/crav1e\">crav1e<\/a>, overall the experience had been a mixed bag and there is large space for improvement.<\/p>\n<p>Ideally I&#8217;d like to have by the end of the year something along the lines of:<\/p>\n<pre><code>$ cargo install-library --prefix=\/usr --libdir=\/usr\/lib64 --destdir=\/staging\/place\n<\/code><\/pre>\n<p>So that it would:<br \/>\n&#8211; build a valid <code>cdylib<\/code>+<code>staticlib<\/code><br \/>\n&#8211; produce a correct header<br \/>\n&#8211; produce a correct pkg-config file<br \/>\n&#8211; install all of it in the right paths<\/p>\n<p>All of this requiring a quite basic <code>build.rs<\/code> and, probably, an applet.<\/p>\n<h2>What is it all about?<\/h2>\n<p>Building and installing properly shared libraries is quite hard, even more on multiple platforms.<\/p>\n<p>Right now <a href=\"https:\/\/github.com\/rust-lang\/cargo\">cargo<\/a> has quite limited install capabilities with <a href=\"https:\/\/github.com\/rust-lang\/rfcs\/pull\/2376\">some work<\/a> pending on extending it and has an <a href=\"https:\/\/github.com\/rust-lang\/cargo\/issues\/5045\">open issue<\/a> and a <a href=\"https:\/\/github.com\/rust-lang\/cargo\/pull\/6298\">patch<\/a>.<\/p>\n<p>Distributions that, probably way too early since the <code>rust-ABI<\/code> is not stable nor being stabilized yet, are experimenting in building everything as shared library also have those problems.<\/p>\n<h2>Why it is important<\/h2>\n<p><a href=\"https:\/\/rust-lang.org\">rust<\/a> is a pretty good language and has a fairly simple way to interact in both direction with any other language that can produce or consume C-ABI-compatible object code.<\/p>\n<p>This is already quite useful if you want to build a small static archive and link it in your larger application and\/or library.<\/p>\n<p>An example of this use-case is <a href=\"https:\/\/gitlab.gnome.org\/GNOME\/librsvg\/blob\/master\/Makefile.am\">librsvg<\/a>.<\/p>\n<p>Such heterogeneous environment warrant for a modicum of additional machinery and complication.<\/p>\n<p>But if your whole library is written in rust, it is a fairly annoying amount of boilerplate that you would rather avoid.<\/p>\n<h2>Current status<\/h2>\n<p>If you want to provide C-bindings to your crates you do not have a single perfect solution right now.<\/p>\n<h3>What works well already<\/h3>\n<p>Currently building the library itself works fine and it is really straightforward:<\/p>\n<ul>\n<li>It is quite easy to mark data types and functions to be C-compatible:<\/li>\n<\/ul>\n<pre><code class=\"language-rust\">#[repr(C)]\npub struct Foo {\n    a: Bar,\n    ...\n}\n\n#[no_mangle]\npub unsafe extern \"C\" fn make_foo() -&gt; *mut Foo {\n   ...\n}\n<\/code><\/pre>\n<ul>\n<li><a href=\"https:\/\/doc.rust-lang.org\/reference\/linkage.html\">rustc<\/a> and <a href=\"https:\/\/doc.rust-lang.org\/cargo\/reference\/manifest.html#building-dynamic-or-static-libraries\">cargo<\/a> are aware of different <code>crate-types<\/code>, selecting <code>staticlib<\/code> produces a valid library<\/li>\n<\/ul>\n<pre><code class=\"language-toml\">[lib]\nname = \"rav1e\"\ncrate-type = [\"staticlib\"]\n<\/code><\/pre>\n<ul>\n<li><a href=\"https:\/\/github.com\/eqrion\/cbindgen\/\">cbindgen<\/a> can produce a usable <code>C-header<\/code> from a crate using few lines of <code>build.rs<\/code> or a stand-alone applet and a <code>toml<\/code> configuration file.<\/li>\n<\/ul>\n<pre><code class=\"language-rust\">extern crate cbindgen;\n\nfn main() {\n    let crate_dir = std::env::var(\"CARGO_MANIFEST_DIR\").unwrap();\n    let header_path: std::path::PathBuf = [\"include\", \"rav1e.h\"].iter().collect();\n\n    cbindgen::generate(crate_dir).unwrap().write_to_file(header_path);\n\n    println!(\"cargo:rerun-if-changed=src\/lib.rs\");\n    println!(\"cargo:rerun-if-changed=cbindgen.toml\");\n}\n<\/code><\/pre>\n<pre><code class=\"language-toml\">header = \"\/\/ SPDX-License-Identifier: MIT\"\nsys_includes = [\"stddef.h\"]\ninclude_guard = \"RAV1E_H\"\ntab_width = 4\nstyle = \"Type\"\nlanguage = \"C\"\n\n[parse]\nparse_deps = true\ninclude = ['rav1e']\nexpand = ['rav1e']\n\n[export]\nprefix = \"Ra\"\nitem_types = [\"enums\", \"structs\", \"unions\", \"typedefs\", \"opaque\", \"functions\"]\n\n[enum]\nrename_variants = \"ScreamingSnakeCase\"\nprefix_with_name = true\n<\/code><\/pre>\n<p>Now issuing <code>cargo build --release<\/code> will get you a <code>.h<\/code> in the <code>include\/<\/code> dir and a <code>.a<\/code> library in <code>target\/release<\/code>, so far it is simple enough.<\/p>\n<h3>What sort of works<\/h3>\n<p>Once have a static library, you need an external mean to track what are its dependencies.<\/p>\n<p>Back in the old ages there were <code>libtool archives<\/code> (.la), now we have <code>pkg-config<\/code> files providing more information and in a format that is way easier to parse and use.<\/p>\n<p><code>rustc<\/code> has <code>--print native-static-libs<\/code> to produce the additional libraries to link, <strong>BUT<\/strong> prints it to stderr and only as a side-effect of the actual build process.<\/p>\n<p>My, fairly ugly, hack had been adding a dummy empty subcrate just to produce the link-line using<\/p>\n<pre><code>cargo rustc -- --print native-static-libs 2&gt;&amp;1| grep native-static-libs | cut -d ':' -f 3\n<\/code><\/pre>\n<p>And then generate the <code>.pc<\/code> file from a template.<\/p>\n<p>This is anything but straightforward and because how <code>cargo rustc<\/code> works, you may end up adding an empty subcrate just to extract this information quickly.<\/p>\n<h3>What is missing<\/h3>\n<p>Once you have your library, your header and your pkg-config file, you probably would like to install the library somehow and\/or make a package out of it.<\/p>\n<p><code>cargo install<\/code> does not currently cover it. It works only for binaries and just binaries alone. It <a href=\"https:\/\/github.com\/rust-lang\/rfcs\/pull\/2376\">will<\/a> hopefully change, but right now you just have to pick the external build system you are happy with and hack your way to integrate the steps mentioned above.<\/p>\n<p>For <code>crav1e<\/code> I ended up hacking a quite crude <a href=\"https:\/\/github.com\/lu-zero\/crav1e\/blob\/master\/Makefile\">Makefile<\/a>.<\/p>\n<p>And with that at least a pure-rust static library can be built and installed with the common:<\/p>\n<pre><code>make DESTDIR=\/staging\/place prefix=\/usr libdir=\/usr\/lib64\n<\/code><\/pre>\n<h3>Dynamic libraries<\/h3>\n<p>Given <code>rustc<\/code> and <code>cargo<\/code> have the <code>cdylib<\/code> crate type, one would assume we could just add the type, modify our build-system contraption a little and go our merry way.<\/p>\n<p>Sadly not. A dynamic library (or shared object) requires in most of the common platform some additional metadata to guide the runtime linker.<\/p>\n<p>The current widespread practice is to use tools such as <a href=\"https:\/\/nixos.org\/patchelf.html\">patchelf<\/a> or <code>install_name_tool<\/code>, but it is quite brittle and might require tools.<\/p>\n<h2>My plans for the 2019<\/h2>\n<p><code>rustc<\/code> has a mean to pass the information to the compile-time linker but there is no proper way to pass it in <code>cargo<\/code>, I already <a href=\"https:\/\/github.com\/rust-lang\/cargo\/pull\/6298\">tried to provide a solution<\/a>, but I&#8217;ll have to go through the <em>rfc<\/em> route to make sure there is community consensus and the feature is properly documented.<\/p>\n<p>Since kind of metadata is platform-specific so would be better to have this information produced and passed on by something external to the main cargo. Having it as applet or a <code>build.rs<\/code> dependency makes easier to support more platforms little by little and have overrides without having to go through a main cargo update.<\/p>\n<p>The applet could also take care of properly create the <code>.pc<\/code> file and installing since it would have access to all the required information.<\/p>\n<p>Some efforts could be also put on streamlining the process of extracting the library link line for the static data and spare some roundtrips.<\/p>\n<p>I guess that&#8217;s all for what I&#8217;d really like to have next year in rust and I&#8217;m confident I&#8217;ll have time to deliver myself \ud83d\ude42<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Since there are plenty of blogposts about what people would like to have or will implement in rust in 2019 here is mine. I spent the last few weeks of my spare time making a C-api for rav1e called crav1e, overall the experience had been a mixed bag and there is large space for improvement. &hellip; <a href=\"https:\/\/blogs.gentoo.org\/lu_zero\/2018\/12\/30\/making-and-using-c-compatible-libraries-in-rust-present-and-future\/\" class=\"more-link\">Continue reading <span class=\"screen-reader-text\">Making and using C-compatible libraries in rust: present and future<\/span><\/a><\/p>\n","protected":false},"author":10,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"spay_email":"","jetpack_publicize_message":"","jetpack_is_tweetstorm":false,"jetpack_publicize_feature_enabled":true},"categories":[24],"tags":[],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/p1aGWH-bv","_links":{"self":[{"href":"https:\/\/blogs.gentoo.org\/lu_zero\/wp-json\/wp\/v2\/posts\/713"}],"collection":[{"href":"https:\/\/blogs.gentoo.org\/lu_zero\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blogs.gentoo.org\/lu_zero\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blogs.gentoo.org\/lu_zero\/wp-json\/wp\/v2\/users\/10"}],"replies":[{"embeddable":true,"href":"https:\/\/blogs.gentoo.org\/lu_zero\/wp-json\/wp\/v2\/comments?post=713"}],"version-history":[{"count":5,"href":"https:\/\/blogs.gentoo.org\/lu_zero\/wp-json\/wp\/v2\/posts\/713\/revisions"}],"predecessor-version":[{"id":718,"href":"https:\/\/blogs.gentoo.org\/lu_zero\/wp-json\/wp\/v2\/posts\/713\/revisions\/718"}],"wp:attachment":[{"href":"https:\/\/blogs.gentoo.org\/lu_zero\/wp-json\/wp\/v2\/media?parent=713"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blogs.gentoo.org\/lu_zero\/wp-json\/wp\/v2\/categories?post=713"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blogs.gentoo.org\/lu_zero\/wp-json\/wp\/v2\/tags?post=713"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}