{"id":1172,"date":"2020-09-02T09:12:28","date_gmt":"2020-09-02T07:12:28","guid":{"rendered":"http:\/\/blogs.gentoo.org\/mgorny\/?p=1172"},"modified":"2020-09-02T09:12:28","modified_gmt":"2020-09-02T07:12:28","slug":"new-tools-to-help-with-package-cleanups","status":"publish","type":"post","link":"https:\/\/blogs.gentoo.org\/mgorny\/2020\/09\/02\/new-tools-to-help-with-package-cleanups\/","title":{"rendered":"New tools to help with package cleanups"},"content":{"rendered":"

Did you ever have had Croaker shout at you because you removed an old version that just happened to be still required by some other package? Did you have to run your cleanups past (slow-ish) CI just to avoid that? If you did, I have just released app-portage\/mgorny-dev-scripts<\/a>, version 6 that has a tool just for that!
\n<\/p>\n

check-revdep to check depgraph of reverse dependencies<\/h2>\n

If you have used mgorny-dev-tools before, then you may know already about the rdep<\/kbd> tool that prints reverse dependency information collected from qa-reports.gentoo.org. Now I’ve put it into a trivial pipeline with pkgcheck<\/kbd>, and made check-revdep<\/kbd>. The idea is really trivial: it fetches list of reverse dependencies from the server, filters it through qatom<\/kbd> (from app-portage\/portage-utils<\/kbd>) and passes to pkgcheck scan -c VisibilityCheck<\/kbd>.<\/p>\n

So you do something like:<\/p>\n

\r\n$ cd dev-python\/unidecode\r\n$ git rm unidecode-0.04.21.ebuild \r\nrm 'dev-python\/unidecode\/unidecode-0.04.21.ebuild'\r\n$ check-revdep \r\n== rdep of dev-python\/unidecode ==\r\n== ddep of dev-python\/unidecode ==\r\n== bdep of dev-python\/unidecode ==\r\n== pdep of dev-python\/unidecode ==\r\ncat: \/tmp\/pindex\/dev-python\/unidecode: No such file or directory\r\ndev-python\/awesome-slugify\r\n  NonexistentDeps: version 1.6.5: RDEPEND: nonexistent package: <dev-python\/unidecode-0.05\r\n  NonsolvableDepsInDev: version 1.6.5: nonsolvable depset(rdepend) keyword(~amd64) dev profile (default\/linux\/amd64\/17.0\/no-multilib\/prefix\/kernel-3.2+) (33 total): solutions: [ <dev-python\/unidecode-0.05 ]\r\n  NonsolvableDepsInStable: version 1.6.5: nonsolvable depset(rdepend) keyword(~amd64) stable profile (default\/linux\/amd64\/17.0) (38 total): solutions: [ <dev-python\/unidecode-0.05 ]\r\n[...]\r\n$ git restore --staged --worktree .\r\n<\/code><\/pre>\n
…and you know you can’t clean it up.<\/p>\n
Warning: the tooling uses data from qa-reports that is updated periodically.  If the data is not up-to-date (read: someone just added a dependency on your package), check-revdep may miss something.<\/em><\/p>\n
Enable cache to speed things up<\/h2>\n
The rdep<\/kbd> also supports using a local cache to avoid fetching everything from the server (= 4 requests per package).  To populate the cache using the current data from server, just run:<\/p>\n
$ rdep-fetch-cache \r\n--2020-09-02 09:00:05--  https:\/\/qa-reports.gentoo.org\/output\/genrdeps\/rdeps.tar.xz\r\nResolving qa-reports.gentoo.org (qa-reports.gentoo.org)... 140.211.166.190, 2001:470:ea4a:1:230:48ff:fef8:9fdc\r\nConnecting to qa-reports.gentoo.org (qa-reports.gentoo.org)|140.211.166.190|:443... connected.\r\nHTTP request sent, awaiting response... 200 OK\r\nLength: 1197176 (1,1M) [application\/x-xz]\r\nSaving to: \u2018STDOUT\u2019\r\n\r\n-                                    100%[=====================================================================>]   1,14M   782KB\/s    in 1,5s    \r\n\r\n2020-09-02 09:00:08 (782 KB\/s) - written to stdout [1197176\/1197176]<\/code><\/pre>\n
The script will fetch the data as a tarball from server, and unpack to \/tmp\/*index<\/kbd>.<\/p>\n
pkgcheck also has its own caching, so successive checks will run faster if packages don’t change.<\/p>\n
Combining rdep with other tools<\/h2>\n
You can also pass rdep<\/kbd> output through to other tools, such as eshowkw<\/kbd> (app-portage\/gentoolkit<\/kbd>) or gpy-showimpls<\/kbd> (app-portage\/gpyutils<\/kbd>).  The recommended pipeline is:<\/p>\n
$ rdep $(pkg) | grep -v '^\\[B' | xargs qatom -C -F '%{CATEGORY}\/%{PN}' | sort -u | xargs gpy-showimpls | less\r\n== rdep of dev-python\/unidecode ==\r\n== ddep of dev-python\/unidecode ==\r\n== bdep of dev-python\/unidecode ==\r\n== pdep of dev-python\/unidecode ==\r\ncat: \/tmp\/pindex\/dev-python\/unidecode: No such file or directory\r\napp-misc\/khard:0\r\n          0.13.0: S             3.7 3.8\r\n          0.17.0: ~             3.7 3.8 3.9\r\napp-text\/pelican:0\r\n           3.7.1: S   #     3.6\r\n           4.0.0: ~   #     3.6\r\n           4.0.1: ~   #     3.6\r\n           4.1.2: ~   #     3.6\r\n           4.2.0: S         3.6 3.7\r\n            9999:           3.6 3.7\r\ndev-python\/awesome-slugify:0\r\n           1.6.5: ~         3.6 3.7 3.8\r\ndev-python\/pretty-yaml:0\r\n          20.4.0: S         3.6 3.7 3.8 3.9\r\ndev-python\/python-slugify:0\r\n           1.2.6: S   #     3.6 3.7 3.8 3.9\r\n           4.0.1: S         3.6 3.7 3.8 3.9\r\nmedia-sound\/beets:0\r\n        1.4.9-r2: ~ s       3.6 3.7 3.8\r\n            9999:   s       3.6 3.7 3.8\r\nwww-apps\/nikola:0\r\n          7.8.15: S         3.6\r\n       7.8.15-r1: ~   #     3.6\r\n           8.0.4: ~   #     3.6 3.7 3.8\r\n           8.1.0: ~   #     3.6 3.7 3.8\r\n           8.1.1: ~   #     3.6 3.7 3.8\r\n        8.1.1-r1: ~         3.6 3.7 3.8\r\n<\/code><\/pre>\n
Getting redundant versions from pkgcheck<\/h2>\n
Another nice trick is to have pkgcheck scan for redundant versions, and output it in format convenient for machine use.<\/p>\n
For example, I often use:<\/p>\n
git grep -l mgorny@ '**\/metadata.xml' | cut -d\/ -f1-2 | uniq | xargs pkgcheck scan -c RedundantVersionCheck -R FormatReporter --format '( cd {category}\/{package} && eshowkw -C )'| sort -u | bash - |& less<\/code><\/pre>\n
that gives eshowkw<\/kbd> output for all packages maintained by me that have potentially redundant versions.  Plus, in another terminal:<\/p>\n
$ git grep -l mgorny@ '**\/metadata.xml' | cut -d\/ -f1-2 | uniq | xargs pkgcheck scan -c RedundantVersionCheck -R FormatReporter --format 'git rm {category}\/{package}\/${package}-{version}.ebuild' | less<\/code><\/pre>\n
that gives convenient commands to copy-paste-execute.<\/p>\n","protected":false},"excerpt":{"rendered":"
Did you ever have had Croaker shout at you because you removed an old version that just happened to be still required by some other package? Did you have to run your cleanups past (slow-ish) CI just to avoid that? If you did, I have just released app-portage\/mgorny-dev-scripts, version 6 that has a tool just … Continue reading “New tools to help with package cleanups”<\/span><\/a><\/p>\n","protected":false},"author":137,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"jetpack_publicize_message":"","jetpack_is_tweetstorm":false,"jetpack_publicize_feature_enabled":true},"categories":[5],"tags":[],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","_links":{"self":[{"href":"https:\/\/blogs.gentoo.org\/mgorny\/wp-json\/wp\/v2\/posts\/1172"}],"collection":[{"href":"https:\/\/blogs.gentoo.org\/mgorny\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blogs.gentoo.org\/mgorny\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blogs.gentoo.org\/mgorny\/wp-json\/wp\/v2\/users\/137"}],"replies":[{"embeddable":true,"href":"https:\/\/blogs.gentoo.org\/mgorny\/wp-json\/wp\/v2\/comments?post=1172"}],"version-history":[{"count":12,"href":"https:\/\/blogs.gentoo.org\/mgorny\/wp-json\/wp\/v2\/posts\/1172\/revisions"}],"predecessor-version":[{"id":1184,"href":"https:\/\/blogs.gentoo.org\/mgorny\/wp-json\/wp\/v2\/posts\/1172\/revisions\/1184"}],"wp:attachment":[{"href":"https:\/\/blogs.gentoo.org\/mgorny\/wp-json\/wp\/v2\/media?parent=1172"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blogs.gentoo.org\/mgorny\/wp-json\/wp\/v2\/categories?post=1172"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blogs.gentoo.org\/mgorny\/wp-json\/wp\/v2\/tags?post=1172"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}