Emerge tmpfs wrapper

The following is a wrapper script for emerge, written by Maciej Mrozowski (reavertm). What it does is mounts the temporary emerge dir (usually /var/tmp/portage) into tmpfs, where the amount of memory that emerge will use is defined in line 3:

TMPDIR=`portageq envvar PORTAGE_TMPDIR`/portage
. /etc/init.d/functions.sh
if [[ -z $(mount | grep "$TMPDIR") ]]; then
	ebegin "Mounting $MEMSIZE of memory to $TMPDIR"
	mount -t tmpfs tmpfs -o size=$MEMSIZE "$TMPDIR"
	eend $?
touch "$STAMP"
einfo "Emerging $*"
emerge $*
rm -f "$STAMP"
if [[ -z `find $TMPDIR -name '.keep-*'` ]]; then
	ebegin "Unmounting ramdisk"
	umount -l "$TMPDIR"
	eend $?
exit $retval

This is supposed to speed up your build times, since it uses the RAM instead of the hard drive, which is significally faster in IO tasks. To use it, make it executable and place it either at /usr/local/bin (which will substitute the emerge command) or anywhere else you want so you can call it only when you want to, like this:

./emerge-wrapper -av1 kdelibs

Powered by Blogilo

17 Responses to Emerge tmpfs wrapper

  1. RealNC says:

    What’s this good for? All you need is mount it in your /etc/fstab.

    If a script was going to be useful, it would be one that *disables* tmpfs, not one that enables it. You should always have tmpfs active, but sometimes you need to disable it (for emerging openoffice). The other way around doesn’t make any sense 😛

  2. tampakrap says:

    It’s useful to quickly mount/unmount /var/tmp/portage only when needed (which means during an emerge), use it only when you want it and easily set the amount of the memory that will be used in each emerge

  3. RealNC says:

    Yes, but why? There’s no reason to unmount it. I suppose you missed one small detail: tmpfs will not use *any* memory if it’s empty. If you don’t have anything in it, it uses 0MB. If you have a file in there that’s 50MB big, it will use 50MB RAM, and so on :)

    There’s no reason to ever unmount it.

  4. tampakrap says:

    There is a reason and your answer is on your own first comment: in case I want to emerge something like openoffice or in case something else is eating my memory, I can choose between mounting tmpfs (through the wrapper script) or use the usual /usr/bin/emerge command (and in fact that’s what I do already)

  5. RealNC says:

    My first comment also says that it doesn’t make sense. There shouldn’t be a script for enabling tmpfs; there should be one that disables it instead for the cases where you don’t want to use tmpfs. Otherwise, an emerge -u world will be tmpfs-less :)

    The way proposed with this script has got it totally backwards.

  6. tampakrap says:

    I see your point, but it actually fits my needs, that’s why I wanted to share (after the approval of the author of course).
    Anyway, in case in the future you or somebody else implements what you have in mind, feel free to contact me, I’d be happy to promote it to gentooland.

  7. sean says:

    My own /var/tmp/portage is a tmpfs partition defined in /etc/fstab and mounted normally, and when I want to compile a package that wouldn’t fit in memory such as chromium I simply change the temp dir via the environment variable, e.g.

    PORTAGE_TMPDIR=/var/tmp/portage2 emerge www-client/chromium

  8. non7top says:

    I wonder if it’s possible to set PORTAGE_TMPDIR=/var/tmp/portage2 in env override files from /etc/portage/env

  9. BRULE Herman says:

    Same here. But for me, the right behavior is the kernel use buffer/cache for do like ram disk when manipulate file size is greatly < to the system ram.

  10. Jabbas says:

    I’m using RealNC method and it works well for me except for openoffice.
    I’ve found a solution for ooo, use aufs/unionfs which will start writing to the harddisk when space on tmpfs runs out, but don’t have a time (and knowledge of aufs) to test it.

  11. crazyfish says:

    I’m using the fstab method, but for some packages that uses to much space, I make sure i have enough swap (for openoffice I create an extra swapfile that I activate only when building it) and increase the size of the tmpfs mount

    mount -o remount,size=10G /var/tmp/portage

    This work very well and the kernel handles very good that tmpfs size is greater than physical memory. (! 32bit machines with 4G memory size limit applies to tmpfs, too, including swap. I’m using this only on 64bit machines)

  12. dilfridge says:

    I can also confirm that swap works fine… :)

    huettel@pinacolada ~ $ df /var/tmp
    Filesystem 1K-blocks Used Available Use% Mounted on
    none 15728640 31232 15697408 1% /var/tmp
    huettel@pinacolada ~ $

  13. Mart Raudsepp says:

    I also use a permanently mounted tmpfs – in fact my /tmp is that and I don’t bother with a separate mount by setting PORTAGE_TMPDIR=/tmp in make.conf. Compiling openoffice goes with extra swap files (as then kernel LRU caching and swapping rules hopefully still make the most benefit of it), but the normal swap is also 2GB (4GB RAM). Webkit based things typically go with CFLAGS override that doesn’t do -ggdb.

    The benefits I could see from such a wrapper script:

    * Memory seems to be guaranteed to be freed up on failed merges (I prefer to have the build.log until a restart or manual cleanup);
    * If one wants a shorter alias to emerge anyway, could wrap it with this for opt-in tmpfs instead of proposed above opt-out approach.

  14. greg says:

    Could anyone explain me why when I used this wrapper compilation time was longer then in normal way?

    lenovo greg # time /usr/local/sbin/emerge-wrapper -qv digikam
    * Mounting 700M of memory to /var/tmp/portage … [ ok ]
    * Emerging -qv digikam
    >>> Verifying ebuild manifests
    >>> Emerging (1 of 1) media-gfx/digikam-1.4.0-r1
    >>> Installing (1 of 1) media-gfx/digikam-1.4.0-r1
    >>> Jobs: 1 of 1 complete Load avg: 2.28, 2.27, 1.63
    * GNU info directory index is up-to-date.
    * Unmounting ramdisk … [ ok ]

    real 10m3.084s
    user 7m44.535s
    sys 7m43.668s
    lenovo greg # time emerge -qv digikam
    >>> Verifying ebuild manifests
    >>> Emerging (1 of 1) media-gfx/digikam-1.4.0-r1
    >>> Installing (1 of 1) media-gfx/digikam-1.4.0-r1
    >>> Jobs: 1 of 1 complete Load avg: 2.01, 1.98, 1.57
    * GNU info directory index is up-to-date.

    real 8m5.238s
    user 6m51.500s
    sys 6m16.625s

  15. katabami says:

    I’ve posted a forum topic now:

    tmpfs emerging is not so much beneficial.

    > greg
    Are you sure cache is cleaned? And tmpfs usage is less than your physical ram (see the above forum topic).

  16. Ryan Hill says:

    I would also disable ccache and set the log dir to also be in the tmpfs.

  17. Markos Chandras says:

    Thank you! It really speeds up compilation process on my laptop!


Leave a Reply