MBF: Packages which break with nocheck

[Santiago Vila]

Hello.

After building all the archive (trixie/sid) with nocheck, I only found 33 new
packages which fail to build with nocheck that were not reported before. Admittedly
a little bit more than I expected, but certainly not "hundreds" as some people feared.

(The main reason there are not so many is that Helmut Grohne has been reporting those
every now and then).

My current plan for now would be to report them as "important" (using some
usertag) with the following disclaimer:

--------------------------------------------------------------------------------
Disclaimer: This was going to be a release goal for trixie, but I'm reporting it
as "important" because it's not clear how many bugs of this type are acceptable
as RC at this point of the release cycle.
--------------------------------------------------------------------------------

but of course I'm open to suggestions regarding the above text, particularly
from Release Managers.


On a personal note, I consider those bugs interesting to fix because I think there
should be a safe procedure to build all packages in the archive in a way which minimizes
build failures as much as possible.

Currently such procedure would be to build all the packages in the regular way and then retry
with nocheck those which fail to build. It would be more simple and straightforward if we were
able to build everything with nocheck from the beginning. It would be a sign of quality
and a milestone if one day we could get zero build failures doing that.

Thanks.

[Simon McVittie]

Post by Santiago Vila
After building all the archive (trixie/sid) with nocheck, I only found 33 new
packages which fail to build with nocheck that were not reported before. Admittedly
a little bit more than I expected, but certainly not "hundreds" as some people feared.
(The main reason there are not so many is that Helmut Grohne has been reporting those
every now and then).

I think there are two subtly different things that you could mean by
"with nocheck":

1. DEB_BUILD_OPTIONS=nocheck, but no special build profiles
- therefore <!nocheck> build-dependencies are still installed
2. DEB_BUILD_OPTIONS=nocheck DEB_BUILD_PROFILES=nocheck
- therefore <!nocheck> build-dependencies are likely to be missing

(DEB_BUILD_PROFILES=nocheck without DEB_BUILD_OPTIONS=nocheck is not
allowed by https://wiki.debian.org/BuildProfileSpec, and for convenience
some build tools automatically convert it into (2.), with a warning.)

Failing to build in either of those configurations is a bug, and both
could be argued to be either RC or non-RC depending on opinions and
priorities, but in practice I think (1.) is going to succeed more often
than (2.).

For example #1102605 is an example of a package that FTBFS when we do
(2.) but would have succeeded if we do (1.). This is a fairly common
failure mode, but I would expect packages that FTBFS in scenario (1.),
but do build successfully if their tests had been run, to be very rare.

Post by Santiago Vila
My current plan for now would be to report them as "important" (using some
usertag)

I think that seems reasonable, but in the template email please be clear
about which scenario it was that you tried. Helmut's wording "fails to
build from source in unstable when enabling the nocheck build profile"
seems good - that unambiguously identifies scenario (2.).

Post by Santiago Vila
On a personal note, I consider those bugs interesting to fix because I think there
should be a safe procedure to build all packages in the archive in a way which minimizes
build failures as much as possible.

If that's what you want, I think scenario (1.) is the one that will
maximize your number of successful builds, although possibly at the cost
of shipping software that compiles but does not work (in ways that
build-time tests would have detected). Running build-time tests is a
trade-off: it makes us less likely to ship broken software, at the cost
of sometimes treating unimportant bugs (either in the software under
test, or in the tests themselves) as more serious than they necessarily
need to be.

Helmut has been testing scenario (2.) and reporting bugs when it
fails, because it's interesting as a way to reduce build-dependencies
for bootstrappability and cross-compiling, but the price he pays for
that is that he'll sometimes see build failures like #1102605.

smcv

[Santiago Vila]

Post by Santiago Vila
On a personal note, I consider those bugs interesting to fix because I think there
should be a safe procedure to build all packages in the archive in a way which minimizes
build failures as much as possible.

If that's what you want, I think scenario (1.) is the one that will maximize your number of successful builds, although possibly at the cost of shipping software that compiles but does not work (in ways that build-time tests would have detected).

Yes, that's what I tested and what I was going to report. I'll make sure that the bug text is
not ambiguous, and also will use some usertag like ftbfs-deb-build-options-nocheck to
make it even more clear.

Thanks a lot!

[Santiago Vila]

Post by Simon McVittie
1. DEB_BUILD_OPTIONS=nocheck, but no special build profiles
    - therefore <!nocheck> build-dependencies are still installed
2. DEB_BUILD_OPTIONS=nocheck DEB_BUILD_PROFILES=nocheck
    - therefore <!nocheck> build-dependencies are likely to be missing

Ok, in the end there was a lot of overlap between those two, so I decided
to report in a single bug the subset of (1) which also fails (2).

In some future mass rebuild maybe I will try the nocheck build profile
in full (not as a subset of DEB_BUILD_OPTIONS=nocheck).

For reference, I've used this usertag for all the bugs (26 new and 3 old):

https://bugs.debian.org/cgi-bin/pkgreport.cgi?users=debian-***@lists.debian.org;tag=ftbfs-nocheck-profile

Thanks a lot.

[Paul Gevers]

Hi Santiago,

Post by Santiago Vila
https://bugs.debian.org/cgi-bin/pkgreport.cgi?users=debian-

In one of the reports I read this:
"""
* When a package is built with the nocheck profile, it means:

- DEB_BUILD_OPTIONS=nocheck
(the tests should be skipped during the build)

- DEB_BUILD_PROFILES=nocheck
(Build-Depends marked <!nocheck> are not installed)

- The contents of the package is allowed to be different than normal
"""

I don't think that's true. I was very much under the impression that
with the build profile nocheck all binaries have to be build and the
content has to be the same.

Paul

[Simon McVittie]

Post by Paul Gevers
"""
- DEB_BUILD_OPTIONS=nocheck
(the tests should be skipped during the build)
- DEB_BUILD_PROFILES=nocheck
(Build-Depends marked <!nocheck> are not installed)
- The contents of the package is allowed to be different than normal
"""
I don't think that's true. I was very much under the impression that
with the build profile nocheck all binaries have to be build and the
content has to be the same.

The definition of the nocheck build profile on
https://wiki.debian.org/BuildProfileSpec agrees with Paul's impression.

I think it's reasonable for Santiago's mass-rebuild not to verify whether
those desired properties were actually true (to verify them, he would have
had to build the package twice, with and without nocheck, defeating the
purpose of wanting to skip potentially flaky or slow build-time tests)
but it would be better if future bug reporting for this scenario didn't
encourage maintainers to implement nocheck incorrectly.

So perhaps instead of

The contents of the package is allowed to be different than normal

the bug template should be more like

The contents of the resulting package are meant to be identical to
the package produced by a normal build, but this was not checked
during this particular mass-rebuild

or something along those lines?

smcv

[Santiago Vila]

Post by Simon McVittie
the bug template should be more like
    The contents of the resulting package are meant to be identical to
    the package produced by a normal build, but this was not checked
    during this particular mass-rebuild
or something along those lines?

Yes, sorry, I'll change the template and will add extra information
to the already submitted bugs.

For some reason I believed having a different package contents was allowed
in one case but not in the other, when it's never allowed. My fault.

I wish reproducible-builds people would activate DEB_BUILD_OPTIONS=nocheck
for the second build, I think it would help. I don't think anybody will
use that as an excuse to file more RC bugs.

Thanks.

[Simon McVittie]

Post by Santiago Vila
I wish reproducible-builds people would activate DEB_BUILD_OPTIONS=nocheck
for the second build

https://bugs.debian.org/786644

smcv

[Paul Gevers]

Hi,

Post by Santiago Vila
I wish reproducible-builds people would activate DEB_BUILD_OPTIONS=nocheck
for the second build, I think it would help. I don't think anybody will
use that as an excuse to file more RC bugs.

They mentioned earlier on IRC that they'll do just that *after* trixie
release.

Paul

[Santiago Vila]

Hi,

Post by Santiago Vila
I wish reproducible-builds people would activate DEB_BUILD_OPTIONS=nocheck
for the second build, I think it would help. I don't think anybody will
use that as an excuse to file more RC bugs.

They mentioned earlier on IRC that they'll do just that *after* trixie release.

Yes, I know that they plan to do that after trixie, but they also said this

the release team doesnt want hundreds of RC bugs filed based on these failures
right now

which seems a strange rationale to me given that they are neither hundreds nor RC.

Thanks.

[Holger Levsen]

Post by Santiago Vila

Post by Santiago Vila
I wish reproducible-builds people would activate DEB_BUILD_OPTIONS=nocheck
for the second build, I think it would help. I don't think anybody will
use that as an excuse to file more RC bugs.

They mentioned earlier on IRC that they'll do just that *after* trixie release.

Yes, I know that they plan to do that after trixie, but they also said this

the release team doesnt want hundreds of RC bugs filed based on these failures
right now

which seems a strange rationale to me given that they are neither hundreds nor RC.

the reproducible builds folks have also 100s of other issues on their plates
and only this many spoons.

enabling nocheck now, takes time and attention, and frankly is not really
that much related to our core mission, it's just one of the many corners
where we already do a lot of qa work and where we could do even more.

time and attention are our most scare ressources.

HTH.
--
cheers,
Holger

⢀⣎⠟⠻⢶⣊⠀
⣟⠁⢠⠒⠀⣿⡁ holger@(debian|reproducible-builds|layer-acht).org
⢿⡄⠘⠷⠚⠋⠀ OpenPGP: B8BF54137B09D35CF026FE9D 091AB856069AAA1C
⠈⠳⣄

Menschen, die sich um die Klimakatastrophe sorgen, sind keine Klimaaktivisten
und auch nicht woke. Sie haben schlicht die FÀhigkeit einen Wissenschaftlichen
Befund zu lesen und zu verstehen. (@DGlatzkopp)

[Santiago Vila]

Post by Simon McVittie
it would be better if future bug reporting for this scenario didn't
encourage maintainers to implement nocheck incorrectly.

You are absolutely right and I apologize for my mistake. Not just future
but also *present*, so I've now added a clarification note to all the
bugs I reported before.

Thanks.

[Helmut Grohne]

Hi Santiago,

Post by Santiago Vila
After building all the archive (trixie/sid) with nocheck, I only found 33 new
packages which fail to build with nocheck that were not reported before. Admittedly
a little bit more than I expected, but certainly not "hundreds" as some people feared.

Thanks for performing these builds!

For the rest of this mail, I am going to assume that "with nocheck"
means "when enabling the nocheck build profile" (see Simon's mail for
context). Please correct me if I'm wrong.

Post by Santiago Vila
My current plan for now would be to report them as "important" (using some

I argue that the correct severity is serious. The release team agreed to
treat them as rc bugs about two years ago and I have reported more than
33 at rc severity. If we were not treating them as rc, the autoremover
could break trixie in the sense that it would no longer be
self-contained. So we should either have them rc, or change the behavior
of the autoremover to disregard <!nocheck> restrictions.

The other side of this is that erroneous <!nocheck> restrictions (and
that's what it is most of the time) are trivial to "fix". With rare
exceptions, you simply drop them. So while 33 may be a worrisome number
of additional rc bugs, the effort spent on fixing them is rather low in
practice.

That said, Emilio explicitly asked them not to be filed as rc on irc.
That feels like RT is not internally consistent here. How about filing
them as rc now and tagging them trixie-ignore later if we deem the
effort too big?

When filing them, please ensure that you file them with the source
package ("Source: ..." or "Package: src:...") and add the ftbfs tag such
that other automatic ftbfs reporters don't file duplicates.

Helmut

[Paul Gevers]

Hi,

Post by Helmut Grohne
That said, Emilio explicitly asked them not to be filed as rc on irc.
That feels like RT is not internally consistent here. How about filing
them as rc now and tagging them trixie-ignore later if we deem the
effort too big?

What I think he means, and I agree with him if I'm right, we don't want
a _new_ systematic QA effort like this _at this stage_ of the release to
add lots of RC bugs for a case where the RC-ness only comes from the way
we run our autoremoval. I'm monitoring the "self-contained" state of
trixie [1] for more then a year now and I've been filing (RC) bugs to
make packages aware of problems (not only for this class of issues).
Currently trixie is nearly fully self-contained in this respect. So at
this moment this RC problem (running with nocheck build profile changes
the content of the package) is in practice not a critical issue for trixie.

What I really want to avoid is that people get afraid to add the
!nocheck profile. It's valuable to have, let's not scare people so late
in the release cycle.

Paul

[1] https://qa.debian.org/dose/debcheck/src_testing_main/