{"id":572,"date":"2017-07-16T10:40:49","date_gmt":"2017-07-16T08:40:49","guid":{"rendered":"https:\/\/blogs.gentoo.org\/mgorny\/?p=572"},"modified":"2018-08-13T18:16:37","modified_gmt":"2018-08-13T16:16:37","slug":"glep-73-check-results-explained","status":"publish","type":"post","link":"https:\/\/blogs.gentoo.org\/mgorny\/2017\/07\/16\/glep-73-check-results-explained\/","title":{"rendered":"GLEP 73 check results explained"},"content":{"rendered":"<p>The\u00a0pkgcheck instance run for the\u00a0<a rel='external' href='https:\/\/wiki.gentoo.org\/wiki\/Project:Repository_mirror_and_CI'>Repo mirror&amp;CI project<\/a> has finished gaining a\u00a0full support for <a rel='external' href='https:\/\/wiki.gentoo.org\/wiki\/User:MGorny\/GLEP:ReqUse'>GLEP 73<\/a> REQUIRED_USE validation and\u00a0verification today. As a\u00a0result, it can report 5 new issues defined by that GLEP. In\u00a0this article, I&#8217;d like to shortly summarize them and\u00a0explain how to interpret and\u00a0solve the\u00a0reports.<\/p>\n<p><!--more--><\/p>\n<p><small>Technical note: the\u00a0GLEP number has not been formally assigned yet. However, since there is no other GLEP request open at the\u00a0moment, I have taken the\u00a0liberty of using the\u00a0next free number in\u00a0the\u00a0implementation.<\/small><\/p>\n<h2>GLEP73Syntax: syntax violates GLEP 73<\/h2>\n<p>GLEP 73 specifies <a rel='external' href='https:\/\/wiki.gentoo.org\/wiki\/User:MGorny\/GLEP:ReqUse#Restrictions_on_REQUIRED_USE_format'>a few syntax restrictions<\/a> as compared to the\u00a0pretty much free-form syntax allowed by the\u00a0PMS. The\u00a0restrictions could be shortly summarized as:<\/p>\n<ul>\n<li><code>||<\/code>, <code>^^<\/code> and <code>??<\/code> can not not be empty,<\/li>\n<li><code>||<\/code>, <code>^^<\/code> and <code>??<\/code> can not not be nested,<\/li>\n<li>USE-conditional groups can not be used inside <code>||<\/code>, <code>^^<\/code> and <code>??<\/code>,<\/li>\n<li>All-of groups (expressed using parentheses without a\u00a0prefix) are banned completely.<\/li>\n<\/ul>\n<p><a rel='external' href='https:\/\/wiki.gentoo.org\/wiki\/User:MGorny\/GLEP:ReqUse#Restrictions_for_allowed_REQUIRED_USE_syntax'>The\u00a0full rationale for the\u00a0restrictions<\/a>, along with examples and\u00a0proposed fixes is provided in\u00a0the\u00a0GLEP. For the\u00a0purpose of this article, it is enough to say that in\u00a0all the\u00a0cases found, there was a\u00a0simpler (more obvious) way of expressing the\u00a0same constraint.<\/p>\n<p>Violation of this syntax prevents pkgcheck from performing any of the\u00a0remaining checks. But more importantly, the\u00a0report indicates that the\u00a0constraint is unnecessarily complex and\u00a0could result in\u00a0REQUIRED_USE mismatch messages that are unnecessarily confusing to the\u00a0user. Taking a\u00a0real example, compare:<\/p>\n<pre><code>  The following REQUIRED_USE flag constraints are unsatisfied:\r\n    exactly-one-of ( ( !32bit 64bit ) ( 32bit !64bit ) ( 32bit 64bit ) )<\/code><\/pre>\n<p>and\u00a0the\u00a0effect of a\u00a0valid replacement:<\/p>\n<pre><code>  The following REQUIRED_USE flag constraints are unsatisfied:\r\n\tany-of ( 64bit 32bit )<\/code><\/pre>\n<p>While we could debate about usefulness of the\u00a0Portage output, I think it is clear that the\u00a0second output is simpler to comprehend. And\u00a0the\u00a0best proof is that you actually need to think a\u00a0bit before confirming that they&#8217;re equivalent.<\/p>\n<h2>GLEP73Immutability: REQUIRED_USE violates immutability rules<\/h2>\n<p>This one is rather simple: it means <q>this constraint may tell user to enable (disable) a\u00a0flag that is use.masked\/forced<\/q>. Taking a\u00a0trivial example:<\/p>\n<pre><code>a? ( b )<\/code><\/pre>\n<p>GLEP73Immutability report will trigger if a\u00a0profile masks the\u00a0<kbd>b<\/kbd> flag. This means that if the\u00a0user has <kbd>a<\/kbd> enabled, the\u00a0PM would normally tell him to enable <kbd>b<\/kbd> as well. However, since <kbd>b<\/kbd> is masked, it can not be enabled using normal methods (we assume that altering use.mask is not normally expected).<\/p>\n<p>The\u00a0alternative is to disable <kbd>a<\/kbd> then. But what&#8217;s the\u00a0point of letting user enable it if we afterwards tell him to disable it anyway? It is more friendly to disable both flags together, and\u00a0this is pretty much what the\u00a0check is about. So in\u00a0this case, the\u00a0solution is to mask <kbd>a<\/kbd> as well.<\/p>\n<p>How to read it? Given the\u00a0generic message of:<\/p>\n<blockquote>\n<p>REQUIRED_USE violates immutability rules: [<em>C<\/em>] requires [<em>E<\/em>] while the opposite value is enforced by use.force\/mask (in profiles: [<em>P<\/em>])<\/p>\n<\/blockquote>\n<p>It indicates that in\u00a0profiles <em>P<\/em> (a\u00a0lot of profiles usually indicates you&#8217;re looking for base or top-level arch profile), <em>E<\/em> is forced or masked, and\u00a0that you probably need to force\/mask <em>C<\/em> appropriately as well.<\/p>\n<h2>GLEP73SelfConflicting: impossible self-conflicting condition<\/h2>\n<p>This one is going to be extremely rare. It indicates that somehow the\u00a0REQUIRED_USE nested a\u00a0condition and\u00a0its negation, causing it to never evaluate to true. It is best explained using the\u00a0following trivial example:<\/p>\n<pre><code>a? ( !a? ( b ) )<\/code><\/pre>\n<p>This constraint will never be enforced since <kbd>a<\/kbd> and\u00a0<kbd>!a<\/kbd> can not be true simultaneously.<\/p>\n<p>Is there a\u00a0point in\u00a0having such a\u00a0report at all? Well, such a\u00a0thing is extremely unlikely to happen. However, it would break the\u00a0verification algorithms and\u00a0so we need to account for it explicitly. Since we account for it anyway and\u00a0it is a\u00a0clear mistake, why not report it?<\/p>\n<h2>GLEP73Conflict: request for conflicting states<\/h2>\n<p>This warning indicates that there are at least two constraints that can apply simultaneously and\u00a0request the\u00a0opposite states for the\u00a0same USE flag. Again, best explained on a\u00a0generic example:<\/p>\n<pre><code>a? ( c ) b? ( !c )<\/code><\/pre>\n<p>In this example, any USE flag set with both <kbd>a<\/kbd> and\u00a0<kbd>b<\/kbd> enabled could not satisfy the constraint. However, Portage will happily led us astray:<\/p>\n<pre><code>  The following REQUIRED_USE flag constraints are unsatisfied:\r\n\ta? ( c )<\/code><\/pre>\n<p>If\u00a0we follow the\u00a0advice and\u00a0enable <kbd>c<\/kbd>, we get:<\/p>\n<pre><code>  The following REQUIRED_USE flag constraints are unsatisfied:\r\n\tb? ( !c )<\/code><\/pre>\n<p>The\u00a0goal of this check is to avoid such a\u00a0bad advices, and\u00a0to\u00a0require constraints to clearly indicate a\u00a0suggested way forward. For example, the\u00a0above case could be modified to:<\/p>\n<pre><code>a? ( !b c ) b? ( !c )<\/code><\/pre>\n<p>to indicate that <kbd>a<\/kbd> takes precedence over <kbd>b<\/kbd>, and\u00a0that <kbd>b<\/kbd> should be disabled to avoid the\u00a0impossible constraint. The opposite can be stated similarly \u2014 however, note that you need to reorder the\u00a0constraints to make sure that the\u00a0PM will get it right:<\/p>\n<pre><code>b? ( !a !c ) a? ( c )<\/code><\/pre>\n<p>How to read it? Given the\u00a0generic message of:<\/p>\n<blockquote>\n<p>REQUIRED_USE can request conflicting states: [<em>C<sub>i<\/sub><\/em>] requires [<em>E<sub>i<\/sub><\/em>] while [<em>C<sub>j<\/sub><\/em>] requires [<em>E<sub>j<\/sub><\/em>]<\/p>\n<\/blockquote>\n<p>It means that if the\u00a0user enables <em>C<sub>i<\/sub><\/em> and\u00a0<em>C<sub>j<\/sub><\/em> simultaneously, the\u00a0PM will request conflicting <em>E<sub>i<\/sub><\/em> and\u00a0<em>E<sub>j<\/sub><\/em>. Depending on the\u00a0intent, the\u00a0solution might involve negating one of\u00a0the\u00a0conditions in\u00a0the\u00a0other constraint, or reworking the\u00a0REQUIRED_USE towards another solution.<\/p>\n<h2>GLEP73BackAlteration: previous condition starts applying<\/h2>\n<p>This warning is the\u00a0most specific and\u00a0the\u00a0least important from all the\u00a0additions at the\u00a0moment. It indicates that the\u00a0specific constraint may cause a\u00a0preceding condition to start to apply, enforcing additional requirements. Consider the\u00a0following example:<\/p>\n<pre><code>b? ( c ) a? ( b )<\/code><\/pre>\n<p>If the\u00a0user has only <kbd>a<\/kbd> enabled, the\u00a0second rule will enforce <kbd>b<\/kbd>. Then the\u00a0condition for the\u00a0first rule will start matching, and\u00a0additionally enforce <kbd>c<\/kbd>. Is this a\u00a0problem? Usually not. However, for the\u00a0purpose of GLEP\u00a073 we prefer that the\u00a0REQUIRED_USE can be enforced while processing left-to-right, in a\u00a0single iteration. If a\u00a0previous rule starts applying, we may need to do another iteration.<\/p>\n<p>The\u00a0solution is usually trivial: to reorder (swap) the\u00a0constraints. However, in\u00a0some cases developers seem to prefer copying the\u00a0enforcements into the\u00a0subsequent rule, e.g.:<\/p>\n<pre><code>b? ( c ) a? ( b c )<\/code><\/pre>\n<p>Either way works for the\u00a0purposes of GLEP 73, though the\u00a0latter increases complexity.<\/p>\n<p>How to read it? Given the\u00a0generic message of:<\/p>\n<blockquote>\n<p>REQUIRED_USE causes a preceding condition to start applying: [<em>C<sub>j<\/sub><\/em>] enforces [<em>E<sub>j<\/sub><\/em>] which may cause preceding [<em>C<sub>i<\/sub><\/em>] enforcing [<em>E<sub>i<\/sub><\/em>] to evaluate to true<\/p>\n<\/blockquote>\n<p>This indicates that if\u00a0<em>C<sub>j<\/sub><\/em> is true, <em>E<sub>j<\/sub><\/em> needs to be true as well. Once it is true, a\u00a0preceding condition of\u00a0<em>C<sub>i<\/sub><\/em> may also become true, adding another requirement for <em>E<sub>i<\/sub><\/em>. To fix the\u00a0issue, you need to either move the\u00a0latter constraint before the\u00a0former, or include the\u00a0enforcement of\u00a0<em>E<sub>i<\/sub><\/em> in\u00a0the\u00a0rule for <em>C<sub>j<\/sub><\/em>, rendering the\u00a0application of the\u00a0first rule unnecessary.<\/p>\n<h2>Constructs using ||, ^^ and ?? operators<\/h2>\n<p>GLEP 73 specifies a\u00a0leftmost-preferred behavior for the\u00a0<code>||<\/code>, <code>^^<\/code> and <code>??<\/code> operators. It is expressed in a\u00a0simple <a rel='external' href='https:\/\/wiki.gentoo.org\/wiki\/User:MGorny\/GLEP:ReqUse#Concept_for_transforming_REQUIRED_USE_into_implications'>transformation into implications (USE-conditional groups)<\/a>. Long story short:<\/p>\n<ul>\n<li><code>||<\/code> and <code>^^<\/code> groups force the leftmost unmasked flag if\u00a0none of the\u00a0flags are enabled already, and<\/li>\n<li><code>??<\/code> and <code>^^<\/code> groups disable all but the leftmost enabled flag if more than one flag is enabled.<\/li>\n<\/ul>\n<p>All the\u00a0verification algorithms work on the transformed form, and\u00a0so their output may list conditions resulting from it. For example, the\u00a0following construct:<\/p>\n<pre><code>|| ( a b c ) static? ( !a )<\/code><\/pre>\n<p>will report a\u00a0conflict between <em>!b !c \u21d2 a<\/em> and\u00a0<em>static \u21d2 !a<\/em>. This indicates the\u00a0fact that per the\u00a0forementioned rule, <code>||<\/code> group is transformed into <code>!b? ( !c? ( a ) )<\/code> which explains that if\u00a0none of the\u00a0flags are enabled, the\u00a0first one is preferred, causing a\u00a0conflict with the\u00a0<code>static<\/code> flag.<\/p>\n<p>In\u00a0this particular case you could debate that the\u00a0algorithm could choose <code>b<\/code> or\u00a0<code>c<\/code> instead in\u00a0order to avoid the\u00a0problem. However, we determined that this kind of heuristic is not a\u00a0goal for GLEP 73, and\u00a0instead we always obide the\u00a0developer&#8217;s preference expressed in\u00a0the\u00a0ordering. The\u00a0only exception to this rule is when the\u00a0leftmost flag can not match due to a\u00a0mask, in\u00a0which case the\u00a0first unmasked flag is used.<\/p>\n<p>For completeness, I should add that <code>??<\/code> and\u00a0<code>^^<\/code> blocks create implications in\u00a0the\u00a0form of: <em>a \u21d2 !b !c\u2026<\/em>, <em>b \u21d2 !c\u2026<\/em> and\u00a0so on.<\/p>\n<p>At\u00a0some point I might work on making the\u00a0reports include the\u00a0original form to avoid ambiguity.<\/p>\n<h2>The future<\/h2>\n<p>The\u00a0most important goal for GLEP 73 is to make it possible for users to install packages out-of-the-box without having to fight through mazes of REQUIRED_USE, and\u00a0for developers to use REQUIRED_USE not only sparingly but whenever possible to improve the\u00a0visibility of resulting package configuration. However, there is still a\u00a0lot of\u00a0testing, some fixing and\u00a0many bikesheds before that could happen.<\/p>\n<p>Nevertheless, I think we can all agree that most of the\u00a0reports produced so far (with the\u00a0exception of the\u00a0back-alteration case) are meaningful even without automatic enforcing of\u00a0REQUIRED_USE, and\u00a0fixing them would benefit our users already. I\u00a0would like to ask you to look for the\u00a0reports on your packages and\u00a0fix them whenever possible. Feel free to ping me if you need any help with that.<\/p>\n<p>Once the\u00a0number of non-conforming packages goes down, I will convert the\u00a0reports successively into warning levels, making the\u00a0CI report new issues and\u00a0the\u00a0pull request scans proactively complain about them.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>The\u00a0pkgcheck instance run for the\u00a0Repo mirror&amp;CI project has finished gaining a\u00a0full support for GLEP 73 REQUIRED_USE validation and\u00a0verification today. As a\u00a0result, it can report 5 new issues defined by that GLEP. In\u00a0this article, I&#8217;d like to shortly summarize them and\u00a0explain how to interpret and\u00a0solve the\u00a0reports.<\/p>\n","protected":false},"author":137,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"jetpack_publicize_message":"","jetpack_is_tweetstorm":false,"jetpack_publicize_feature_enabled":true},"categories":[11],"tags":[],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","_links":{"self":[{"href":"https:\/\/blogs.gentoo.org\/mgorny\/wp-json\/wp\/v2\/posts\/572"}],"collection":[{"href":"https:\/\/blogs.gentoo.org\/mgorny\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blogs.gentoo.org\/mgorny\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blogs.gentoo.org\/mgorny\/wp-json\/wp\/v2\/users\/137"}],"replies":[{"embeddable":true,"href":"https:\/\/blogs.gentoo.org\/mgorny\/wp-json\/wp\/v2\/comments?post=572"}],"version-history":[{"count":23,"href":"https:\/\/blogs.gentoo.org\/mgorny\/wp-json\/wp\/v2\/posts\/572\/revisions"}],"predecessor-version":[{"id":596,"href":"https:\/\/blogs.gentoo.org\/mgorny\/wp-json\/wp\/v2\/posts\/572\/revisions\/596"}],"wp:attachment":[{"href":"https:\/\/blogs.gentoo.org\/mgorny\/wp-json\/wp\/v2\/media?parent=572"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blogs.gentoo.org\/mgorny\/wp-json\/wp\/v2\/categories?post=572"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blogs.gentoo.org\/mgorny\/wp-json\/wp\/v2\/tags?post=572"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}