Michał Górny

A few words on the topic of optimization

Michał Górny — Thu, 23 Aug 2012 08:27:22 +0000

Optimization is a very broad topic when referring to compiled languages like C or C++. There are many good guides on that topic. However, I see that people nevertheless forget about a few basic principles; thus I’d like to write a few short words myself, explaining how to avoid common pitfalls when optimizing or at least trying to.

First of all, I’d like to notice that I will be considering optimization as targeted towards making the program execution time shorter through use of faster code (algorithms, methods). I will not cover the area in detail but just give a few tips or remainders which should be taken into account when optimizing.

Know your assembly!

Measurement is a very important tool when it comes to optimize. Measure before, measure after and see whether there was improvement. Developers like measures because they give numbers, and numbers simply say faster or slower, or sometimes not sure but whatever. But remember that numbers can be mis-interpreted easily, and then they can lead to wrong (or even ridiculous) conclusions.

To be honest, measurement is a very broad topic itself, and it is really hard to perform good and trustworthy measures. And I believe that in order to do so, you first need to know deeply what you are measuring. And in fact you are measuring code; but more than the code you have just written, you are measuring the assembly generated by the compiler.

Thus, I believe that before starting to measure anything, you should obtain the resulting assembly and analyze it. Or at least compare with the previous result. Otherwise, you can end up trying to measure a difference between two variants of code resulting in the same assembly! And trying to improve your measurement methods to get a consistent, meaningful difference.

Of course, you shouldn’t trust the assembly by itself either. Sometimes seemingly insignificant code changes can cause noticeable differences in execution time. Sometimes four instructions (with longer execution time) actually execute faster than two instructions because the former were done in parallel while the latter couldn’t; and this is when measurement can come in handy.

Optimize the compiler, do not perform its job

There are people who believe that the only possible way of getting good code is by writing the assembly themselves. Shortly saying — don’t do it! Most importantly, you are making your program less portable; if the assembly is optional, the replacement part becomes not well-tested. And anyway, you are writing the assembly for your CPU. If you are writing for Intel, then AMD can work better with another code; if you are writing for AMD, then VIA may like yet another solution. And I’m just considering x86 here.

If you are writing in high-level programming language like C or C++, write in it. If the compiler doesn’t generate desired assembly, try to change the code to give it a hint. Sometimes you just need to reorder the variables a little, or introduce a helper (mid-result) variable.

Often it is actually beneficial to replace your complex, optimized construct with a simpler one. The latter will benefice both the users (which will find it easier to understand it, and fiddle with it) and the compiler (which will find it easier to understand it, and replace it with a much better optimized code).

And if you still can’t get your desired assembly, and you are sure that it will be actually better and beneficial, report a bug. Get your compiler fixed so that it could do its job better, and benefit all the projects using it, not only yours.

The impact of C++ templates on library ABI

Michał Górny — Mon, 20 Aug 2012 10:38:03 +0000

Author:	Michał Górny
Date:	20 Aug 2012
Copyright:	http://creativecommons.org/licenses/by/3.0/
Source:	http://dev.gentoo.org/~mgorny/articles/the-impact-of-cxx-templates-on-library-abi.rst

Preamble

The general aspect of maintaining binary compatibility of C++ library interfaces has been already covered thoroughly multiple times. A good reference of articles on the topic can be found on wiki page of ABI compliance checker tool [1]. Sadly, those articles usually consider the topic of C++ templates only briefly, if at all.

While in fact the topic is fairly complex, and I believe that considering the overall usefulness and popularity of the templates, it should be considered more thoroughly. Thus, in this article I will try to address the issues arising from use of templates, methods of dealing with them and trying to prevent them.

Both the overall topic of templates in respect to the programming techniques, and the wide topic of ABI are already explained in detail in many other articles and guides. Moreover, I believe that myself I am not fluent enough to be able to cover those topics in detail here. Thus, I will assume that a reader of this article is already familiar with both the general topic of templates in C++, and the basic aspects of an ABI and its compatibility.

Moreover, in the solutions and problems listed here I will assume that a particular toolchain in question does conform to the C++98 standard, and is able to properly support templates with regard to multiple instantiations.

[1]	http://ispras.linuxbase.org/index.php/ABI_compliance_checker#Articles

ABI consistency in regular functions and class methods

Before proceeding further, I wish to quickly summarize the aspects of ABI compliance in regard to regular (non-template and non-inline) functions and class methods.

In shared libraries, the interface of library functions is defined within the header files which are installed along with the library. The implementation code, however, is defined in the source files which are compiled into the shared library itself. Effectively, the implementation is kept opaque to the library consumers; and those consumers (either programs or other shared libraries) dynamically link to the library [2].

Nevertheless, if the function ABI is to change, such a change can be signaled through changing the SONAME of the library. As executables and other shared libraries are linked to a specific SONAME, they have to be rebuilt in order to use the new version of the library. That way, it is ensured that the consumers will be rebuilt to use the new ABI before using the library.

Effectively, it is quite straightforward to maintain the ABI of a shared library using regular functions and classes. The number of ABI changes is minimized through keeping the implementation opaque to the consumers; and the actual ABI changes are propagated through use of SONAME and other symbol versioning techniques.

[2]	The topic of static linking is not covered here as of no relevance.

The problem of handling template functions and classes

The requirements put on the compiler for template support made the solution presented above no longer possible. In order to achieve generic programming, the compiler must be able to adjust both the interface and the implementation of template functions and classes according to the template arguments. Considering that those arguments can practically hold any consumer-provided type or value (including types defined by consumers themselves), it is no longer possible to precompile the implementation code and provide it in a opaque shared library.

A completely different solutions need to be used in order to support templates. Firstly, both the interface and the implementation code must be public to the consumers; thus both are provided in the headers installed by the library. Effectively, a number of C++ libraries, Eigen [3] for instance, actually consist of header files only.

Secondly, compiler must be able to instantiate and compile the templates at will. Unlike with regular function which are compiled within a single source unit, templates have to be compiled on use, and can be used in multiple independent files. Effectively, the compiler must be able to handle multiple definitions of the same template sanely.

To sum up, templates basically no longer form a uniform shared library. They are propagated through header files, and can be instantiated practically anywhere — either in the library providing them (through use in other functions, or through forced instantiation), in a shared library using them or in the final executable. Moreover, the same template can be instantiated in multiple locations at the same time. The implementation part can no longer be opaque, and is spread along with the instantiations.

The lack of ability to provide opaque implementation makes it harder to change the code without breaking ABI compatibility. Moreover, the decentralization makes it even harder both to propagate implementation changes (including bug fixes) and to handle ABI changes.

[3]	http://eigen.tuxfamily.org/

Avoiding multiple, incompatible instantiations

One of the most common issues being result of a library ABI change is the occurrence of multiple, incompatible instantiations of the same template. I will explain this on a recent example.

With the recent improvement of C++11 standard support in gcc 4.7.1, the std::list template class has been changed to hold the list length. The change has been made conditional to the C++11 standard being enabled; effectively, to the -std=c++11 option [4].

Now, if I have a particular program which uses std::list, and I compile that program in the C++11 mode, the extended template will be used. To that moment, everything is fine.

However, if I would like to use a shared library using std::list as well, and that library is compiled for another C++ standard (-std=c++98, for example), the program may no longer work correctly.

This is because both the program executable and the shared library would try to instantiate an incompatible versions of std::list. Only one of those versions will be actually used, and the code written for the other one may break randomly.

The main issue solving that incompatibility is that if the library providing the template is based only on header files, it is no longer able to enforce propagation of the ABI change through rebuilding all of the libraries and executables using it.

[4]	As a side note, the change has been reverted and postponed to be introduced unconditionally in the next standard C++ library version.

Using templates in final executables only

A one particular way in which the problem could be avoided is to limit the template instantiation to a single executable. For practical reasons, that executable would supposedly be the final program executable, allowing the programmer to use templates freely and all the libraries to use them equally.

This can be achieved through making all the implementation using templates… a template as well. For example, consider the following code:

#include 

class C
{
public:
        int f();

        // ...
};

int C::f()
{
        std::vector value_vector;

        // ...
}

Here function f() from class C uses a std::vector internally. In order to avoid local instantiation of the vector, we need to make the class C a template class. In this particular case, we can achieve an additional benefit from that — allowing the user to specify the allocator for value_vector:

#include 
#include 

template < class allocator = std::allocator >
class C
{
public:
        int f()
        {
                typedef typename allocator::template rebind
                        ::other alloc_type;
                std::vector value_vector;

                // ...
        }
};

This way, as long as all classes using C are templates as well, the only template instantiation will take place in the final executable where the last template class is used. Of course, this has many implications for quite a little benefit. Most importantly:

all template classes have to be used along with template parameters; instead of C, one has to use C<>;
all template classes have to make the implementation public, and start to suffer from all template-related issues explained here;
the code generation is deferred to the final executable. Code duplication occurs, shared libraries no longer serve a purpose and compiling the program becomes time- and memory-consuming.

Just as a note, the fore-mentioned approach can be used when there is no logical use for a template as well. An unused template parameter can be introduced then:

template 
class C
{
        // ...
};

Using custom types in shared library templates

An alternate and possibly better approach is to prevent non-local instantiations of a particular template specializations through scoping of the types used in them.

Please consider the following example:

#include 

// class C is a public class provided by the library

void f()
{
        std::list l;

        // ...
}

Here, the std::list template is instantiated. However, the type C is part of the public library API and an API consumer may want to create a list of that type as well. In order to avoid the list being instantiated twice, the library may replace C with a local-scope derived class:

#include 

namespace
{
        class C_ : public C
        {
                // ...
        };
};

void f()
{
        std::list l;

        // ...
}

Since C and C_ are distinct types, std::list will instantiate a different specialization than std::list. And as C_ is limited to the local scope, no consumer should be able to
create a list of that type.

Using inline methods and functions

There is one more solution which I am ought to mention, yet its use is limited and it is not guaranteed to work at all. The one remaining way of preventing multiple instantiations is to enforce inline instantiations of template methods.

This approach has a few limitations. Firstly, it requires modifying the template itself, so it won’t work with external templates, like fore-mentioned std::list. Additionally, since the compiler is free to ignore the inline keyword at will, this solution may only work partially or not work at all.

For completeness, I will present an example:

template 
class C
{
        T sth;

public:
        inline T& get()
        {
                return sth;
        }

        inline void set(T& new_val)
        {
                sth = new_val;
        }
};

The above code uses the inline keyword extensively, hoping that the compiler will inline all the method calls rather than instantiating them as separate methods. If it succeeds for all of the methods, external instantiations will not have anything to collide with.

Achieving a (partially) opaque implementation with templates

While the solutions listed above address a real issue, none of them could be considered a really good solution. They mostly try to circumvent a specific issue rather than addressing the deeper one which I would like to point out.

Those solutions are good as long as we’re only interested in keeping the executables mostly working. However, the template library works alike a static library there, suffering from the same limitations that a static library has.

Most importantly, it is impossible to change the implementation of a particular function or method, and propagate the change without rebuilding all the libraries and programs using the template. And this becomes important if the particular code suffers from security vulnerabilities or bugs which could result in data loss.

In order to solve that problem, we need to take a step back and see what can we do to make template implementation more opaque once again.

Explicitly instantiating opaque templates

One particular solution which solves the issue completely is to keep the implementation completely opaque, and force the compiler to explicitly instantiate necessary templates. However, a very important disadvantage of this solution is that the possible uses of the template are limited to the ones provided by the library.

For an example, please consider we’re having a MyList container which works similarly to std::list. In order to make the list implementation opaque, the header files contain only the interface of MyList:

template 
class MyList
{
        struct priv_type;
        priv_type* priv;

public:
        // ...
};

As you can see in the above example, the PImpl (private implementation) idiom is used there to hide the private class members. That kind of PImpl implementation does not make sense with regular templates, since the private members would have to be explicitly listed along with the implementation. However, in this case we are keeping the implementation private.

The implementation along with the supported instantiations is provided in the source file of the library:

#include "mylist"

template class MyList;
template class MyList;
template class MyList;

// ...

With the above declarations, followed by the method and struct priv_type implementations will result in compiler outputting the template code for int, double and void* types. That code will be placed in the shared library, and made available for the consumers of the shared library.

Now, if a consumer uses MyList and links to the shared library, it will use the opaque implementation from the library. Alike with regular functions, any ABI-safe changes will be propagated to the programs; and ABI-unsafe will result in necessity of rebuilding them, propagated through SONAME change.

However, if the consumer decides to use MyList, the compiler will no longer be able to satisfy that requirement. The linker will refuse to link the program because of no implementation of MyList methods.

Using common base classes and functions

Sadly, the fore-mentioned approach limits the possible uses of the template to a predefined parameter sets. This may be acceptable in very specific cases but usually it just invalidates the whole point of using templates.

An alternative approach is to use a partially opaque implementation, with the opaque being subject to easy implementation changes, and the public behaving like a regular template. This could be accomplished in a number of ways.

One of them is to develop a common, opaque base class which performs the most important tasks in a type-agnostic way and a derived template class which wraps the former in a nice API. For example, consider a pointer list like the following:

template 
class PointerList;

template <>
class PointerList
{
        struct priv_data;
        priv_data* priv;

public:
        // ...
};

template 
class PointerList
        : private PointerList
{
public:
        // ...
};

The pointer list template is first specialized for the void* type. That specialization uses a PImpl idiom, private code and explicit instantiation (in the source file) to provide an opaque implementation of void* pointer list. On top of that, a public generic pointer list template is built.

With such a solution, the most important parts of implementation (in this case that would be the list handling code) are kept opaque, and thus changes in that code are propagated the usual way. The remaining public parts (in this case, that part would probably just cast types) suffer the regular issues with template classes.

It should be noted that the example presented above was very optimistic. Usually, providing a common opaque implementation is a more demanding task. Sometimes even it is not possible to split out a common amount of code for the base class, and only small pieces of code could be considered common. In these cases, the presented approach may not be beneficial at all.

Enforcing ABI versioning with template libraries

Up to the moment, I have mostly considered avoiding issues resulting from ABI incompatibilities, and avoiding introducing those incompatibilities. But even if the author protects executables from the fore-mentioned issues and provides a mostly-opaque implementation, he should still prepare the library for future ABI (or implementation) changes.

As mentioned before, the most common method used in POSIX-compliant systems is based on interface versioning (or even a more fine-grained symbol versioning in GNU systems). The concept is mostly straightforward — a shared library declares a range of supported interface versions, and the programs linking against it copy the current version into themselves. When the library is upgraded, and does not support the old interface anymore, the interface version in programs does not match the one in the library, and they refuse to use it without being rebuilt.

Although this technique is used in regular C and C++ libraries to handle ABI changes (effectively propagating the ABI change to consumers), it can be used to enforce implementation changes for templates as well. This can become particularly useful if a particular template code suffers from serious vulnerabilities or bugs.

The –as-needed problem

The fore-mentioned interface versioning technique requires the program using templates to link against a shared library where the version is defined. This becomes problematic if the template class does not use opaque implementation in the used code (i.e. all methods used by a particular consumer are defined in the template) and the –as-needed GNU linker option is used.

The intent of that option is to link the program executable only with those libraries which it uses directly. Although this often is advantageous and thus the option gained popularity, it thwarts our plans. Because the code using the templates does not call any function from the library we are forcing it to link, the linker will simply skip that library. Effectively, the ABI link will be broken, and interface changes from the library won’t be propagated to template consumers.

Although this specific problem could supposedly be worked around through disabling that option, that is not really feasible in this case. Most importantly, the task of disabling it would either need to be done by the library consumers directly (effectively, making it unlikely to happen) or handled through an external -config application which will hurt cross-compilation.

To cover the topic completely, I’ll note that such a solution would first have to check whether –as-needed is actually enabled. And due to the specific synopsis of -Wl compiler option, finding that would be a semi-difficult task (and different compilers will have different options). Then, it would have to add -Wl,–no-as-needed, the relevant libraries and re-enable -Wl,–as-needed.

Since that thing is neither easy to implement, nor portable, and we can’t forget that other (future) linkers can inhibit similar behavior using different options or even unconditionally, I believe we should look for another solution. One which works around the actual issue rather than trying to disable the feature exposing it.

Using dummy symbols to enforce linking

As mentioned earlier, the reason why linker silently ignores our library is that the program doesn’t actually use any symbol from it. So, in order to work around that, we simply need to define a dummy symbol and use it. Although that may look simple at first, we should give it some thought.

First of all, considering that the symbol has no real meaning, we should try hard to avoid unnecessary impact on the code, both in terms of performance and its size. Preferably, it should be as short as possible, and it should appear on unused (or likely-unused) code paths. However, we should also make sure that the code will actually be reachable — and thus the compiler won’t optimize it out.

For example, an obvious solution seems to add a dummy() method to our template class, and access the symbol there. Sadly, since the method is never actually used, the compiler doesn’t even instantiate it. After some thinking, you may reach a consensus that the only safe place for the call is in the constructor or destructor — but if the template class is used as a static class, these may not be used as well.

I described a similar problem on stackoverflow [5], and was given a pretty good tip there. Thanks to that, I assembled the following universal solution:

namespace mylib
{
        extern int dummy;

        namespace
        {
                class Dummy
                {
                public:
                        inline Dummy()
                        {
                                dummy = 0;
                        }
                };

                static const Dummy dummy;
        }
}

With the int dummy variable being declared in the library code. The main (and possibly the only one) advantage of that solution is that it works with any code. The variable setting code may have side effects, thus the compiler is not allowed to optimize it out. It can however practically optimize it to a single mov instruction.

The disadvantage of that solution is that the assignment operation will be performed once for every single file including the header file. Thus, I would consider the fore-mentioned solution as a last resort, while preferring placing similar assignments in rarely-used (but always compiled) code, whenever possible.

For example, such a code may be within a rare error or exception handler. Just make sure that the compiler will not be allowed to assume that a particular branch will never be executed. For example, the following may be a bad idea:

template 
void C::f(int always_one = 1)
{
        if (always_one != 1)
        {
                dummy = 0;
                throw MyException();
        }

        // ...
}

Although most likely this code will work as expected, if a compiler decides to inline the function it may notice that it is always called with 1, effectively removing the whole branch. Similarly, the code should not be placed in unreachable code (because compiler may notice that) or assertions (because with NDEBUG they will be removed). And don’t forget that if f() will not be used directly at all, the whole function can be optimized out.

To sum up: if you have a safe, unlikely-called branch in your code, then it’s the best place for the assignment. If you don’t have one, then universal solution is probably the best you can get. Although it will cause unnecessary repetitions of the assignments, every other place is likely to cause even larger number of them.

[5]	http://stackoverflow.com/questions/11917014/enforce-linking-with-a-shared-library-with-wl-as-needed-when-only-templates

A short summary

In this article I tried to point out a few common problems related to the ABI in shared libraries using templates. Firstly, I’d like to note I’ve focused only on issues arising from their use in the implementation code of a library, without exposing them in the public API.

If templates are exposed in the public API, i.e. used as function arguments or return types, their impact increases rapidly. In that case they become part of the library ABI, and changes in them effectively change the ABI of the library itself. There are solutions which try to avoid that, for example the bridge pattern [6] but they’re out of scope of this article.

Still, all the fore-mentioned issues apply. When designing a library of templates, you should be aware of them and know how to handle them. Most importantly, I believe that you should always be aware that your ABI may change at some point, even if it is very simple and dedicated to a very specific solution, and even if you are going to make it opaque, or take other precautions to avoid it changing in the future.

Thus, the important point here is to design your library in such a way that its consumers will be able to handle these ABI changes gracefully. The second but nevertheless important problem is actually avoiding ABI changes, either by good design (if possible) or through introducing new classes or methods rather than modifying the existing ones.

However, note that the latter method mentioned is going to require you to either permanently duplicate code (and effectively allowing the consumers to still use the old one), or just delay the ABI change until the compatibility code is removed in favor of requiring your consumers to update their code, thus effectively changing both the API and the ABI.

[6]	http://en.wikipedia.org/wiki/Bridge_pattern

Research question: integral type sizes on various platforms

Michał Górny — Wed, 15 Aug 2012 09:23:25 +0000

I’m a bit curious about sizes of various integral types on different platforms, and I’d really appreciate a little help from people running various non-common architectures/toolchains. I’ve prepared a little package which just tries to get various type sizes using the C++ compiler, and I’d really appreciate if you could run it and paste the results in a comment.

To run it:

wget http://dev.gentoo.org/~mgorny/cxx-type-sizes-0.tar.bz2
tar -xf cxx-type-sizes-0.tar.bz2
cd cxx-type-sizes-0/
./configure
make
cat output/_all

It will try to compile a few programs, and then run them. Then it concatenates the results into output/_all and that’s the file I’d like to get, along with your platform, toolchain, CHOST and ARCH, ABI and everything else you consider relevant.

I’d really like to get a single output for each architecture, and possibly additional outputs if some toolchain/other magic resulted in different results than the previous one. I’ll put the results then into a nice table. Thanks in advance.

Current results.

GitHub — or how to re-centralize a DVCS

Michał Górny — Tue, 03 Jul 2012 09:44:29 +0000

What are the most important advantages of git? I think one which should come out pretty early is that it is «distributed», or «decentralized». This simply means that the actual, complete repository moves along with the project. At some point, you could think: where it is hosted shouldn’t matter that much.

Although git doesn’t facilitate working completely without centralized server (because you need to find updates somewhere), it should be pretty clear that the repository content should be independent of the hosting service. In other words, hosting service should serve the repository, not enforce its contents.

I think all the madness started on Google Code project hosting. There, the project wikis were hosted as a subdirectory to svnroot (e.g. in gecko-mediaplayer sources). I’m not sure if I can say «it is wrong». On one hand, it’s bad to keep completely separate codebases in the same repository. On the other, the design of subversion is simply pure madness, and so everything in the repository follows it…

On the other hand, Google got git correctly. When a particular project decides to use git there, it gets three separate repositories (look at pkgcore sources for an example).

GitHub got wikis right as well. But GitHub Pages… They actually misuse branches in a horrible, messy way. Just look how to create project pages manually — they tell you to create an «orphan» branch!

In other words, they tell you to create two repositories in a single repository. Two independent histories. Complete madness! And whether you want it or not, you pull them with every single clone you do. Yes, that could be some kind of advantage but nevertheless it has nothing to do with the source code.

There’s also this old, ignored issue that they encourage you to rename your README file to their invented suffix just to have it rendered correctly. Once again, hosting services enforces the layout of your repository. And I get really angry getting all those README.md.bz2 in my docdir. This is all against the purpose of markup…

Shortly saying, the sole purpose of markup formats like Markdown, reStructuredText, asciidoc is to provide a complete markup on top of plain text. The text which sould be still completely usable for any regular text viewer. And this means that their naming should also follow the common text file naming rules, which means either uppercase names in *nix or .txt suffix in Windows. No custom .md, and certainly not .asciidoc!

It’s really sad that the very common git hosting sites, instead of encouraging people to use git correctly, force them to hack it around to achieve some minor madness.

vim: smart C/C++ boilerplate templates

Michał Górny — Wed, 13 Jun 2012 11:33:59 +0000

A simple vim scriptie for those who are interested. It is triggered when new C/C++ files are created (e.g. via vim new-file.cxx), and fills it in with boilerplate unit code. What’s special about it is that it tries to find tips about that code in other files in that or parent directory.

The templates

The template used for .c/.cxx files is:

/* header-comment
 */

#ifdef HAVE_CONFIG_H
#	include "config.h"
#endif

While for .h/.hxx files the following is used:

/* header-comment
 */

#pragma once

#ifndef PREFIX_FN_H
#define PREFIX_FN_H 1

#endif /*PREFIX_FN_H*/

In order to simplify the work flow, the following magic is done:

header-comment is actually copied from any other file with the same suffix,
FN_H is substituted for actual filename,
PREFIX is copied from any other file with the same suffix,
if no file is found in the same directory as the new file, the scripts falls back to parent directory, and appends the subdirectory name to PREFIX.

The script

Just a dirty, quick vimrc. Feel free to improve:

function CTemplate()
  let fext = expand('%:e')
  let ucext = toupper(fext)
  let myprefix = substitute(toupper(expand('%')), '[^A-Z0-9]', '_', 'g')

  let head_done = 0
  let head_line = 0
  let prefix_done = 0

  if fext =~ '^h'
    call append(1, '#pragma once')
    call append(2, '')
    call append(3, '#ifndef @fn@')
    call append(4, '#define @fn@ 1')
    call append(5, '')
    call append(6, '#endif /*@fn@*/')
  else
    call append(1, '#ifdef HAVE_CONFIG_H')
    call append(2, '#  include "config.h"')
    call append(3, '#endif')
    let prefix_done = 1
  endif

  let mywd = expand("%:p:h")
  echo mywd
  let myfiles = glob(mywd . '/*.' . fext, 0, 1)
  let myfiles += glob(mywd . '/../*.' . fext, 0, 1)

  for myfile in myfiles
    if filereadable(myfile)
      for myline in readfile(myfile)
        if !head_done
          if head_line != 0
            call append(head_line, myline)
            if myline =~ '[*][/]'
              let head_done = 1
            else
              let head_line += 1
            endif
          endif

          if head_line == 0 && myline =~ '^[/][*]'
            call append(0, myline)
            let head_line += 1

            if myline =~ '[*][/]'
              let head_done = 1
            endif
          endif
        endif

        if !prefix_done && myline =~ '^#ifndef .*_'.ucext
          if myfile =~ '^../'
            let mydir = toupper(expand('%:p:h:t'))

            let myprefix = mydir . '_' . myprefix
          endif

          let myprefix = substitute(myline[8:],
            \ '[^_]*_[^_]*$', '', '') . myprefix
          let prefix_done = 1
        endif
      endfor
    endif

    if head_done && prefix_done
      break
    endif
  endfor

  if fext =~ '^h'
    exec '%s/@fn@/'.myprefix.'/'
  endif
endfunction

A C API for C++ and Python ones — or making of libh2o

Michał Górny — Sun, 10 Jun 2012 16:47:02 +0000

Lately I spent a lot of time working on a small project of mine called libh2o. Its goal is to provide a library of routines implementing IAPWS IF97 equations for water and steam properties. With the core written in C, and providing a nice-to-use API for C++ and Python.

At first, I thought about not providing a «high level» C API at all. It was like: if you want to use plain C, you’ve gotta glue all the low-level equations yourself. However, after some thinking I decided to provide one, and built the two remaining APIs (C++ and Python) on top of it.

The main reason for doing this was that Python (well, CPython) is written in C. Although I’ve seen people writing Python extensions in C++, and even using some of C++ features to make them a little nicer, that’s still a bunch of ugly C hacks and pointer casts. I don’t see a really good reason to write a Python extension in C++, nor to make it depend on a C++ compiler when it’s all limited to C-based CPython API anyway.

And that means that I have either to duplicate all the high-level logic in the Python extension, or just create a C API first and reuse that. Since the whole logic was simple enough to be covered completely and clearly in C, I have chosen this way.

As it happens when people choose C, I had to implement some kind of poor man’s objectivity. Not something as wide (and ugly) as GObject (someone, please kill it!) but a few bits necessary to keep the state. In other words, a structure keeping the «object» and a bunch of nicely named functions taking it as their first argument.

Before I learnt C++, I would assume that the object structure should be a private (and obscured) blob, and the object type should be an incomplete pointer to it. User should just grab that pointer from a «constructor», pass it around and finally free it through a «destructor». Advantage: the exact struct contents are not the part of ABI.

But now I’ve decided to go the other way; way similar to how C++ classes work. I’ve created a structure with explicitly listed private fields (and a very simple /*private:*/ comment), and used that as the public type. It doesn’t need to keep any memory allocated, and is simple enough to be allocated on stack. Advantages: no need for a destructor, and an ability to pack that struct in the C++ class which will wrap it.

Then the usual stuff: a bunch of functions with common prefixes. One prefix for the «namespace», another one for the function (new, get…). All in nice and clear fashion, either to be used directly or wrapped in the C++ or Python APIs.

A five commandments for XML format designers

Michał Górny — Wed, 25 Apr 2012 18:32:00 +0000

If you’re designing an XML-based data format, then I beg you, please read the few following rules and obey them. XML may look easy, and even is easy but that doesn’t mean that writing a good one is. And if you’re going to invent second HTML, then please, just use JSON or any other random container. That will be easier for you, and easier for us.

1. Thou shalt always write a schema

Every XML format should be well described. And no, your ten-stanza poem is not enough. Complete, dedicated Wiki neither. These usually describe nicely (or less nicely) how to write your XML. That could be great if that’s all you’re interested in. But if that’s supposed to be some public format, there is one more important thing…

It’s called reading. Or parsing. Or just transforming. If you need to handle random XML files, coming from various sources, written by random people, you have to know what you can expect and what can you assume. It’s not enough to say what does — I need to know where it can appear and what I can find inside.

There are already well-deployed XML description formats such as DTD, Relax-NG or XML Schema. Please use one of them, I will be grateful. Not only they describe the format strictly and accurately but they also provide a very simple means to validate XML files. It’s helpful both to us, who parse it, and to people who actually write such XML.

An XML without spec is an XML where every element can appear anywhere in the document. In other words, it’s not even XML but an ugly tag soup.

2. Thy XML shalt be structured, not flat

XML provides means to create neat, hierarchical structures. Use them. If your documents consists of logical parts like sections or chapters, put their complete content in a single

or , or any other thing that may come into your head. That’s the correct way of doing that in XML.

Random headings and separators are not enough. Even if your spec says they always and definitely start a new section, that’s not enough. If you don’t believe us, try splitting that thing into parts yourself. Especially when you have sub-headings, sub-sub-headings and so on.

A flat-structured XML is no real XML. It’s just a text file with a few unnecessary elements.

3. Thou shalt split text into blocks using XML, not text delimeters

Even if you think that’ll make writing much easier, do not ever try to use simple character delimiters to split text into blocks. If you need a list, create a list of XML elements. Like the following:

elem1
elem2
elem3

And yes, I know elem1,elem2,elem3 is shorter and easier to type. But guess what — it’s hell to parse. It isn’t even XML — you either have to handle it externally or create a complex recursive template which will split it and handle each token separately. That’s very bad.

An XML which uses random delimeters to create lists is no XML. It’s called CSV.

4. Thou shalt not allow insane structures

Even if you think noone will create an insane structure in your document, it’s not enough. Saying it’s disallowed on your awesome Wiki is not enough either. Forbid it if it’s supposed to be forbidden.

Otherwise, someone finally will use it. He or she will deliberately ignore your warning because it works. And even if they don’t, we will have to support it anyway in a compliant parser.

If you expect your data to be interchangeable with widely used formats, take a look at them. Don’t allow insane things which none of these formats do — or we’ll have to either refuse to convert some files, convert them incorrectly or waste our time writing complex blocks converting them to sane ones.

Simply, don’t do it. Even HTML doesn’t do that… well, that much.

5. Thou shalt write readable XML, not bytecode

The major point of using XML is that the data is both readable to machines and humans. Leave it that way. You have the whole human language at your disposal, so don’t write zeros, ones and other random numbers which are explained on your great Wiki.

Say, an attribute called type should actually name some type. Say, article can be some type. 1 usually ain’t. And if that type only describes width of indent, then name it so! Calling it a type is as useful as calling it a thing. Or some-other-thing and a-third-thing.

XML without human-readable text is no XML. Hell, even byte-compiled XML should have readable element names! That’s the whole point with it. Otherwise, you just end up developing another custom, useless format.

The suggested dependencies problem

Michał Górny — Fri, 13 Apr 2012 16:46:22 +0000

Optional runtime dependencies (or «suggested dependencies») are one of the late problems we’re facing in Gentoo. There’s definitely a need for some standard solution, and it’d be great to put it in the next EAPI. Sadly, there’s no consensus how to solve it.

The optional dependencies problem

Gentoo has a very neat solution for handling optional dependencies and optional features, probably ever since the beginning. It is called «USE flags» and they work very well with the «traditional» optional dependencies. By that, I mean optional dependencies which are both build- and run-time.

Such a dependencies have to be pulled in before the build process starts, and usually require passing specific options in the configure phase. What’s important, both enabling and disabling features requires rebuilding the program in question because of code branches being switched. Thus, it’s perfectly fine if changing USE flags implies rebuilding the package.

Sadly, when it comes to optional runtime dependencies, USE flags are not a perfect solution. «Switching» such a dependency doesn’t require rebuilding the program anymore. It’s usually not even switching — the program can determine in runtime whether a particular dependency is available, and either enable or disable respective features. Simple like that.

If one decides to use USE flags for that, they become partially meaningless. Unless flags start stripping off the code (which is a bad idea), feature availability is dependency- rather than flag-based. So, USE=-ssl is irrelevant if, say, pyopenssl is installed. What’s even worse, flag imply needless rebuilding of such packages just to pull in an additional dependency.

The simple hack — pkg_postinst() messages

The simplest solution right now is just listing the suggested dependencies in pkg_postinst() messages. Combined with has_version helper, those messages can give a pretty nice output, pointing out already installed packages — just take a look at sys-apps/systemd ebuild.

Of course, it’s not a real solution, rather relying on user doing the hard work. The biggest disadvantage is that the dependencies are often going to end up in @world. And then, if user decides to unmerge our package, portage is unable to find and unmerge them as well.

The SDEPEND solution

A pretty common idea is to establish a new variable called SDEPEND (for «suggested»). Such a variable would simply list relevant dependencies, and let portage handle the UI part somehow. It is a minimalistic solution, quite consistent with other parts of PMS. Sadly, it has a few big shortcomings.

First, using our current dependency syntax, you can’t specify that a particular feature requires more than one package; in other words, that two or more suggested dependencies are supposed to be pulled in together. Of course, solving this one would be pretty easy — e.g. by allowing grouping them with parantheses.

A much more important issue is describing what particular dependencies do. Although sometimes this could be guessed by package descriptions pretty well, usually a more friendly text would be great. So, we end up having to implement that somehow.

And that’s usually when Ciaran comes in with ugly exherbism DEPENDENCIES. Sure, it solves most of the issues pointed out here but, hell, do we really want such a thing? Isn’t dependency syntax obscure enough already?

And it’s all rather dependency-oriented. In other words, package comes first, then goes the feature description. «Pass dev-python/pyopenssl or dev-python/python-gnutls to enable secure connections support». I don’t think that’s the most user friendly solution.

The USE flag solution

Another solution is brining a new category of USE flags. It’s not important whether they would be specified using a special variable, common USE_EXPAND or another magical features. In fact, that could be a thing totally separate from USE flags. The point is that some of the package flags would be runtime-switchable.

Unlike traditional USE flags, such flags wouldn’t be stored in vdb. They would be evaluated in place instead, using package.use or similar files, and the dependency tree would use current state of such flags. Of course, they would be allowed for RDEPEND (PDEPEND) use only.

Why reuse USE flags for that? Because it’s the most user-friendly solution. User doesn’t have to learn anything new. He/she enables a flag, does emerge -vDtN @world and notices that new dependencies are pulled in but the package doesn’t have to be rebuilt for that.

If we just add some additional magic for regular USE flags, enabling run-time dependant SSL support could be exactly the same as enabling build-time one — even using the same USE=ssl.

And we could basically even give «backwards» support for older EAPIs. Package managers not supporting the new feature would simply treat runtime-switchable USE flags as regular USE flags, requiring rebuilds of the package.

Moving systemd into /usr — the technical side

Michał Górny — Wed, 04 Jan 2012 22:51:46 +0000

Now that I think of it, I really regret I didn’t make systemd ebuild install it to /usr from the very beginning. But the harm has been done already, and I’d like to move it ASAP and that’s why I’d like to sum up problems with that and possible ways of proceeding with it.

The idea

The idea is simple as it is: move systemd install to /usr prefix completely. Right now, there are no technical benefits from keeping it in rootfs. It already depends on libdbus, which is installed in /usr, and I expect more dependencies over time. There’s no reason to move all those packages into rootfs.

Most importantly, the above information allows me to assume that such move won’t hurt our split-/usr users — because they already had to have /usr mounted for systemd to run.

The trouble

The main problem with the move is that unit files were installed into /lib location by a number of packages. The files can be moved into the new location cleanly only through rebuilding the packages which provide it. They need to stay in search path for systemd to work.

These unit files are symlinked from within /etc/systemd as well. Whenever we move a single unit, we need to update the symlink as well. I’d really like to avoid forcing users to manually fix that, and the eclass doesn’t export pkg_postinst() which could help doing it automatically.

The last problem is that people have the systemd location hardcoded into the kernel command-line. This one should be relatively easy to avoid as we can keep compatibility symlink for some time.

Solution 1: one big move

The simplest solution for the migration is to move all the relevant units in the systemd ebuild. This way, the unit integrity can be preserved, and symlinks could be updated at once as well. The risk of system breakage should be reduced to minimum.

However, there is an important disadvantage of that method. All those files would be moved out-of-scope of the Package Manager. Thus, after rebuild of every single package providing systemd units, all of our users will have to fight file collisions.

The same will likely apply to our new users, because they will have at least some units installed by random packages already. Users not ever intending to use systemd won’t be hurt because the move of unit files will be transparent to them as any other package file move.

Solution 2: temporary support for two locations

Right now, systemd already supports multiple locations for unit files. As a temporary hack, we could just add /lib/systemd/system to that list. This way, all unit files still installed in /lib will still work as expected when systemd is installed into /usr.

Sadly, this won’t handle updating /etc symlinks. I could, however, fix that easily by adding a simple .path unit or another solution updating symlinks as soon as files are removed from /lib.

Why there’s no IUSE=systemd, logrotate, bash-completion…

Michał Górny — Fri, 16 Dec 2011 17:09:46 +0000

Next tides of users slowly notice that a number of unneeded files is installed on their systems. They enumerate systemd unit files, logrotate files, take their pitchforks and start their cruciates against Gentoo developers wasting their precious disk space.

Let me tell you a story. The story starts when Uncle Scarabeus wants to add bash-completion support into libreoffice ebuild. He considers this a minor addon, not worth the half a day necessary to rebuild libreoffice, so he doesn’t revbump it. He simply assumes the change will be propagated nicely when users upgrade to the next version.

Of course, some users will already come shouting here: that’s against the policy! Yes, indeed it is. But is it worth the hassle? Should all libreoffice users be forced to rebuild that huge package just to get a single tiny file installed? He could wait and add that along with the next version. Well, if he wouldn’t forget about it.

But that’s not really important part here. Because, to his surprise, many users have actually noticed the change. That’s because the use of bash-completion.eclass has caused the ebuild to have IUSE=bash-completion; and many of the --newuse Portage option users have rebuilt the package. A few others, like me, just stopped using that option.

That’s when the discussion started. We — the few devs actually caring about discussing — decided that it is quite pointless to control installing tiny files through USEflags. Of course, the libreoffice is an elephant-case here but so-called regular packages aren’t much better here. Is there really a reason to rebuild even 10 C files when the only thing going to change is a single, tiny text file being installed or not?

Another solution is to split those files into separate ebuilds. But that’s usually inconvenient both for users and devs. Users have to notice that they need to emerge an additional package to get the particular file installed, and devs need to maintain that additional package. That starts to become really ridiculous with files like systemd units which are often generated during build-time and store installation paths.

So what to use? INSTALL_MASK, obviously. It’s an ancient Portage magic which allows you to control which files will be punted from installed files. You can use app-portage/install-mask to quickly set it for the files you don’t want. It’s as simple as:

# install-mask -a systemd logrotate