Emerge tmpfs wrapper

The following is a wrapper script for emerge, written by Maciej Mrozowski (reavertm). What it does is mounts the temporary emerge dir (usually /var/tmp/portage) into tmpfs, where the amount of memory that emerge will use is defined in line 3:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
#!/bin/bash
 
MEMSIZE=2G
TMPDIR=`portageq envvar PORTAGE_TMPDIR`/portage
 
. /etc/init.d/functions.sh
if [[ -z $(mount | grep "$TMPDIR") ]]; then
	ebegin "Mounting $MEMSIZE of memory to $TMPDIR"
	mount -t tmpfs tmpfs -o size=$MEMSIZE "$TMPDIR"
	eend $?
	mounted=1
fi
 
STAMP=$TMPDIR/.keep-`uuidgen`
 
touch "$STAMP"
einfo "Emerging $*"
emerge $*
retval=$?
rm -f "$STAMP"
 
if [[ -z `find $TMPDIR -name '.keep-*'` ]]; then
	ebegin "Unmounting ramdisk"
	umount -l "$TMPDIR"
	eend $?
fi
 
exit $retval

This is supposed to speed up your build times, since it uses the RAM instead of the hard drive, which is significally faster in IO tasks. To use it, make it executable and place it either at /usr/local/bin (which will substitute the emerge command) or anywhere else you want so you can call it only when you want to, like this:

./emerge-wrapper -av1 kdelibs

=-=-=-=-=
Powered by Blogilo

17 Responses to Emerge tmpfs wrapper

  1. RealNC says:

    What’s this good for? All you need is mount it in your /etc/fstab.

    If a script was going to be useful, it would be one that *disables* tmpfs, not one that enables it. You should always have tmpfs active, but sometimes you need to disable it (for emerging openoffice). The other way around doesn’t make any sense :P

    Reply
  2. tampakrap says:

    It’s useful to quickly mount/unmount /var/tmp/portage only when needed (which means during an emerge), use it only when you want it and easily set the amount of the memory that will be used in each emerge

    Reply
  3. RealNC says:

    Yes, but why? There’s no reason to unmount it. I suppose you missed one small detail: tmpfs will not use *any* memory if it’s empty. If you don’t have anything in it, it uses 0MB. If you have a file in there that’s 50MB big, it will use 50MB RAM, and so on :)

    There’s no reason to ever unmount it.

    Reply
  4. tampakrap says:

    There is a reason and your answer is on your own first comment: in case I want to emerge something like openoffice or in case something else is eating my memory, I can choose between mounting tmpfs (through the wrapper script) or use the usual /usr/bin/emerge command (and in fact that’s what I do already)

    Reply
  5. RealNC says:

    My first comment also says that it doesn’t make sense. There shouldn’t be a script for enabling tmpfs; there should be one that disables it instead for the cases where you don’t want to use tmpfs. Otherwise, an emerge -u world will be tmpfs-less :)

    The way proposed with this script has got it totally backwards.

    Reply
  6. tampakrap says:

    I see your point, but it actually fits my needs, that’s why I wanted to share (after the approval of the author of course).
    Anyway, in case in the future you or somebody else implements what you have in mind, feel free to contact me, I’d be happy to promote it to gentooland.

    Reply
  7. sean says:

    My own /var/tmp/portage is a tmpfs partition defined in /etc/fstab and mounted normally, and when I want to compile a package that wouldn’t fit in memory such as chromium I simply change the temp dir via the environment variable, e.g.

    PORTAGE_TMPDIR=/var/tmp/portage2 emerge www-client/chromium

    Reply
  8. non7top says:

    I wonder if it’s possible to set PORTAGE_TMPDIR=/var/tmp/portage2 in env override files from /etc/portage/env

    Reply
  9. BRULE Herman says:

    Same here. But for me, the right behavior is the kernel use buffer/cache for do like ram disk when manipulate file size is greatly < to the system ram.

    Reply
  10. Jabbas says:

    I’m using RealNC method and it works well for me except for openoffice.
    I’ve found a solution for ooo, use aufs/unionfs which will start writing to the harddisk when space on tmpfs runs out, but don’t have a time (and knowledge of aufs) to test it.

    Reply
  11. crazyfish says:

    I’m using the fstab method, but for some packages that uses to much space, I make sure i have enough swap (for openoffice I create an extra swapfile that I activate only when building it) and increase the size of the tmpfs mount

    mount -o remount,size=10G /var/tmp/portage

    This work very well and the kernel handles very good that tmpfs size is greater than physical memory. (! 32bit machines with 4G memory size limit applies to tmpfs, too, including swap. I’m using this only on 64bit machines)

    Reply
  12. dilfridge says:

    I can also confirm that swap works fine… :)

    huettel@pinacolada ~ $ df /var/tmp
    Filesystem 1K-blocks Used Available Use% Mounted on
    none 15728640 31232 15697408 1% /var/tmp
    huettel@pinacolada ~ $

    Reply
  13. Mart Raudsepp says:

    I also use a permanently mounted tmpfs – in fact my /tmp is that and I don’t bother with a separate mount by setting PORTAGE_TMPDIR=/tmp in make.conf. Compiling openoffice goes with extra swap files (as then kernel LRU caching and swapping rules hopefully still make the most benefit of it), but the normal swap is also 2GB (4GB RAM). Webkit based things typically go with CFLAGS override that doesn’t do -ggdb.

    The benefits I could see from such a wrapper script:

    * Memory seems to be guaranteed to be freed up on failed merges (I prefer to have the build.log until a restart or manual cleanup);
    * If one wants a shorter alias to emerge anyway, could wrap it with this for opt-in tmpfs instead of proposed above opt-out approach.

    Reply
  14. greg says:

    Could anyone explain me why when I used this wrapper compilation time was longer then in normal way?

    lenovo greg # time /usr/local/sbin/emerge-wrapper -qv digikam
    * Mounting 700M of memory to /var/tmp/portage … [ ok ]
    * Emerging -qv digikam
    >>> Verifying ebuild manifests
    >>> Emerging (1 of 1) media-gfx/digikam-1.4.0-r1
    >>> Installing (1 of 1) media-gfx/digikam-1.4.0-r1
    >>> Jobs: 1 of 1 complete Load avg: 2.28, 2.27, 1.63
    * GNU info directory index is up-to-date.
    * Unmounting ramdisk … [ ok ]

    real 10m3.084s
    user 7m44.535s
    sys 7m43.668s
    lenovo greg # time emerge -qv digikam
    >>> Verifying ebuild manifests
    >>> Emerging (1 of 1) media-gfx/digikam-1.4.0-r1
    >>> Installing (1 of 1) media-gfx/digikam-1.4.0-r1
    >>> Jobs: 1 of 1 complete Load avg: 2.01, 1.98, 1.57
    * GNU info directory index is up-to-date.

    real 8m5.238s
    user 6m51.500s
    sys 6m16.625s

    Reply
  15. katabami says:

    I’ve posted a forum topic now:
    http://forums.gentoo.org/viewtopic-t-847293.html

    tmpfs emerging is not so much beneficial.

    > greg
    Are you sure cache is cleaned? And tmpfs usage is less than your physical ram (see the above forum topic).

    Reply
  16. Ryan Hill says:

    I would also disable ccache and set the log dir to also be in the tmpfs.

    Reply
  17. Markos Chandras says:

    Thank you! It really speeds up compilation process on my laptop!

    Reply

Leave a Reply