{"id":597,"date":"2017-07-23T20:03:02","date_gmt":"2017-07-23T18:03:02","guid":{"rendered":"https:\/\/blogs.gentoo.org\/mgorny\/?p=597"},"modified":"2017-07-23T20:58:41","modified_gmt":"2017-07-23T18:58:41","slug":"optimizing-ccache-using-per-package-caches","status":"publish","type":"post","link":"https:\/\/blogs.gentoo.org\/mgorny\/2017\/07\/23\/optimizing-ccache-using-per-package-caches\/","title":{"rendered":"Optimizing ccache using per-package caches"},"content":{"rendered":"

ccache<\/a> can be of great assistance to Gentoo developers and\u00a0users who frequently end up rebuilding similar versions of packages. By providing a\u00a0caching compiler frontend, it can speed up builds by removing the\u00a0need to build files that have not changed again. However, it uses a\u00a0single common cache directory by\u00a0default which can be suboptimal even if you are explicitly enabling ccache only for a\u00a0subset of packages needing that.<\/p>\n

The\u00a0likeliness of cross-package ccache hits is pretty low \u2014 majority of the\u00a0hits occurs within a\u00a0single package. If you use a\u00a0single cache directory for all affected packages, it grows pretty quick. Besides a\u00a0possible performance hit from having a\u00a0lot of files in\u00a0every directory, this means that packages built later can shift earlier packages out of the\u00a0cache, resulting in\u00a0meaninglessly lost cache hits. A\u00a0simple way to avoid both of the\u00a0problems is to use separate ccache directories.<\/p>\n

<\/p>\n

In\u00a0my solution, a\u00a0separate subdirectory of \/var\/cache\/ccache<\/kbd> is used for every package, named after the\u00a0category, package name and\u00a0slot. While the\u00a0last one is not strictly necessary, it can be useful for slotted packages such as\u00a0LLVM where I do not want frequently changing live package sources to shift the\u00a0release versions out of the\u00a0cache.<\/p>\n

To use it, put a\u00a0code similar to the\u00a0following in\u00a0your \/etc\/portage\/bashrc<\/kbd>:<\/p>\n

if [[ ${FEATURES} == *ccache* && ${EBUILD_PHASE_FUNC} == src_* ]]; then\r\n\tif [[ ${CCACHE_DIR} == \/var\/cache\/ccache ]]; then\r\n\t\texport CCACHE_DIR=\/var\/cache\/ccache\/${CATEGORY}\/${PN}:${SLOT}\r\n\t\tmkdir -p \"${CCACHE_DIR}\" || die\r\n\tfi\r\nfi<\/code><\/pre>\n
The\u00a0first condition makes sure the\u00a0code is only run when ccache is enabled, and\u00a0only for src_*<\/kbd> phases where we can rely on userpriv being used consistently. The second one makes sure the\u00a0code only applies to a\u00a0specific (my initial) value of\u00a0CCACHE_DIR<\/kbd> and\u00a0therefore avoids both nesting the\u00a0cache indefinitely when Portage calls subsequent phase functions, and\u00a0applying the\u00a0replacement if user overrides CCACHE_DIR<\/kbd>.<\/p>\n
You need to either adjust the\u00a0value used here to the\u00a0directory used on your system, or\u00a0change it in your \/etc\/portage\/make.conf<\/kbd>:<\/p>\n
CCACHE_DIR=\"\/var\/cache\/ccache\"<\/code><\/pre>\n
Once this is done, Portage should start creating separate cache directories for every package where you enable ccache. This should improve the\u00a0cache hit ratio, especially if you are using ccache for large packages (why else would you need it?). However, note that you will no longer have a\u00a0single cache size limit \u2014 every package will have its own limit. Therefore, you may want to reduce the\u00a0limits per-package, or manually look after the\u00a0cache periodically.<\/p>\n","protected":false},"excerpt":{"rendered":"
ccache can be of great assistance to Gentoo developers and\u00a0users who frequently end up rebuilding similar versions of packages. By providing a\u00a0caching compiler frontend, it can speed up builds by removing the\u00a0need to build files that have not changed again. However, it uses a\u00a0single common cache directory by\u00a0default which can be suboptimal even if you … Continue reading “Optimizing ccache using per-package caches”<\/span><\/a><\/p>\n","protected":false},"author":137,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"jetpack_publicize_message":"","jetpack_is_tweetstorm":false,"jetpack_publicize_feature_enabled":true},"categories":[3],"tags":[],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","_links":{"self":[{"href":"https:\/\/blogs.gentoo.org\/mgorny\/wp-json\/wp\/v2\/posts\/597"}],"collection":[{"href":"https:\/\/blogs.gentoo.org\/mgorny\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blogs.gentoo.org\/mgorny\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blogs.gentoo.org\/mgorny\/wp-json\/wp\/v2\/users\/137"}],"replies":[{"embeddable":true,"href":"https:\/\/blogs.gentoo.org\/mgorny\/wp-json\/wp\/v2\/comments?post=597"}],"version-history":[{"count":5,"href":"https:\/\/blogs.gentoo.org\/mgorny\/wp-json\/wp\/v2\/posts\/597\/revisions"}],"predecessor-version":[{"id":602,"href":"https:\/\/blogs.gentoo.org\/mgorny\/wp-json\/wp\/v2\/posts\/597\/revisions\/602"}],"wp:attachment":[{"href":"https:\/\/blogs.gentoo.org\/mgorny\/wp-json\/wp\/v2\/media?parent=597"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blogs.gentoo.org\/mgorny\/wp-json\/wp\/v2\/categories?post=597"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blogs.gentoo.org\/mgorny\/wp-json\/wp\/v2\/tags?post=597"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}