Salsa CI job 'missing-breaks' to be enabled by default starting March 1st

[Otto Kekäläinen]

Hi all package maintainers who use Salsa CI,

Salsa CI has had for many years the job 'missing-breaks' that
complements piuparts by checking that the files a package introduce
don't clash with files shipped by any other package in the
distribution without having proper Breaks/Replaces in the
`debian/control` file. This job works well, being quick to run and has
had zero false positives in our experience.

We plan to enable this job next weekend, but allow the job to fail. In
a month from now we plan to switch it to enforcing mode so that this
job must pass for the pipeline to pass.

If you have any comments, feel free to reply to this email thread or
take part in the discussion at
https://salsa.debian.org/salsa-ci-team/pipeline/-/merge_requests/578

## Schedule

1. March 1st: Enable this job by default, but in allow_failure mode,
making Salsa CI yellow on packages that fail on this job
2. March 31st: Remove the allow_failure mode, potentially making the
Salsa CI red for packages that fail on this job

## Error examples

Example output when end users run into this category of issues:

$ apt upgrade
...
Unpacking example (from .../foo_1.0-1_all.deb) ...
dpkg: error processing /var/cache/apt/archives/foo_1.0-1_all.deb (--unpack):
trying to overwrite '/usr/share/man/man8/example.8.gz', which is also
in package bar 2.2

Example output from the job on failure:

$ check_for_missing_breaks_replaces.py -o
${WORKING_DIR}/missing_breaks.xml --changes-file
${WORKING_DIR}/*.changes
[ERROR] Missing Breaks/Replaces found
[ERROR] foo conflicts with bar files: {'/usr/share/man/man8/example.8.gz'}


PS. If you are interested in many smaller improvements soon landing in
Salsa CI, you are welcome to read and comment (and review) the 32
currently open MRs at
https://salsa.debian.org/salsa-ci-team/pipeline/-/merge_requests

[Guillem Jover]

Hi!

Post by Otto Kekäläinen
Salsa CI has had for many years the job 'missing-breaks' that
complements piuparts by checking that the files a package introduce
don't clash with files shipped by any other package in the
distribution without having proper Breaks/Replaces in the
`debian/control` file. This job works well, being quick to run and has
had zero false positives in our experience.

I think that while this check seems worthwhile to me to avoid file
conflicts, the check name itself…

Post by Otto Kekäläinen
## Error examples
$ apt upgrade
...
Unpacking example (from .../foo_1.0-1_all.deb) ...
trying to overwrite '/usr/share/man/man8/example.8.gz', which is also
in package bar 2.2
$ check_for_missing_breaks_replaces.py -o
${WORKING_DIR}/missing_breaks.xml --changes-file
${WORKING_DIR}/*.changes
[ERROR] Missing Breaks/Replaces found
[ERROR] foo conflicts with bar files: {'/usr/share/man/man8/example.8.gz'}

…and this advice, are not ideal, and can lead to potentially worse
problems.

I think the check should be renamed to something like
check-file-conflicts (probably with compat names for existing repos
using the old name). And the advice should probably point to a Wiki
page or similar, where the various cases can be explained, and how to
solve them. For example for conflicting programs with different
interfaces the advice would incur in a policy violation, for conflicts
between a current shared library SOVERSION and the one in the archive
with a lower SOVERSION that makes upgrades and transitions harder,
etc.

Thanks,
Guillem

[Arnaud Rebillout]

Post by Otto Kekäläinen
$ check_for_missing_breaks_replaces.py -o
${WORKING_DIR}/missing_breaks.xml --changes-file
${WORKING_DIR}/*.changes
[ERROR] Missing Breaks/Replaces found
[ERROR] foo conflicts with bar files: {'/usr/share/man/man8/example.8.gz'}


and this advice, are not ideal, and can lead to potentially worse
problems.
I think the check should be renamed to something like
check-file-conflicts (probably with compat names for existing repos
using the old name). And the advice should probably point to a Wiki
page or similar, where the various cases can be explained, and how to
solve them. For example for conflicting programs with different
interfaces the advice would incur in a policy violation, for conflicts
between a current shared library SOVERSION and the one in the archive
with a lower SOVERSION that makes upgrades and transitions harder,
etc.

I suppose there are two main situations:

1) package includes the offending file(s) by mistake, in that case the
fix is to stop distributing the file (and NOT to use Breaks/Replaces)

2) package includes the file(s) for good reason, in that case it's as
Guillem said above

[ERROR] You might want to rename/remove the offending files if that's

suitable

For situation 2, maybe point to Debian Policy:
https://www.debian.org/doc/debian-policy/ch-relationships.html, in
particular sections 7.3, 7.4 and 7.6? Unless there's a better piece of
doc on this topic?

Best,
Arnaud

[Otto Kekäläinen]

Thanks for the feedback!

Ideally the script
https://salsa.debian.org/salsa-ci-team/pipeline/-/blob/master/images/scripts/check_for_missing_breaks_replaces.py
would live in the devscripts package and be extended to have the
features suggested in this thread. However, even in its current form
it has worked well for many years, and I having it enabled by default
in Salsa CI now will catch some regressions from entering the archive
(and Trixie), and is a better choice than postponing the change with
expectations that would have more features in near future.

[Simon Richter]

Hi,

can this be fixed to suggest Conflicts instead of Breaks/Replaces if the file in the other package is still shipped in the current version?

Breaks/Replaces implies that ownership should be transferred from the other package to this one. This is most often correct, but care needs to be taken to not build a Replaces loop, something that Conflicts nicely avoids.

Also, unversioned Breaks, Conflicts and Replaces should give a warning at least.

   Simon

[Lorenzo]

Hello Otto,

[please keep me in CC, I'm not subscribed]

Post by Otto Kekäläinen
Salsa CI has had for many years the job 'missing-breaks' that
complements piuparts by checking that the files a package introduce
don't clash with files shipped by any other package in the
distribution without having proper Breaks/Replaces in the
`debian/control` file. This job works well, being quick to run and has
had zero false positives in our experience.

In salsa CI now I see:

$ check_for_missing_breaks_replaces.py -o ${WORKING_DIR}/missing_breaks.xml --changes-file ${WORKING_DIR}/*.changes
[ERROR] Missing Breaks/Replaces found
[ERROR] runit-init conflicts with init-system-helpers files: {'/usr/share/man/man8/invoke-rc.d.8.gz', '/usr/sbin/service', '/usr/sbin/invoke-rc.d', '/usr/share/man/man8/service.8.gz'}
Uploading artifacts for failed job

this looks like false positive, file are in fact diverted. Does the test
check for for diversions?

Post by Otto Kekäläinen
## Schedule
1. March 1st: Enable this job by default, but in allow_failure mode,
making Salsa CI yellow on packages that fail on this job
2. March 31st: Remove the allow_failure mode, potentially making the
Salsa CI red for packages that fail on this job

Could you please consider delaying 2. until diversion are properly
detected?

Best Regards,
Lorenzo

[NoisyCoil]

Post by Lorenzo
Hello Otto,
[please keep me in CC, I'm not subscribed]

Post by Otto Kekäläinen
Salsa CI has had for many years the job 'missing-breaks' that
complements piuparts by checking that the files a package introduce
don't clash with files shipped by any other package in the
distribution without having proper Breaks/Replaces in the
`debian/control` file. This job works well, being quick to run and has
had zero false positives in our experience.

$ check_for_missing_breaks_replaces.py -o ${WORKING_DIR}/missing_breaks.xml --changes-file ${WORKING_DIR}/*.changes
[ERROR] Missing Breaks/Replaces found
[ERROR] runit-init conflicts with init-system-helpers files: {'/usr/share/man/man8/invoke-rc.d.8.gz', '/usr/sbin/service', '/usr/sbin/invoke-rc.d', '/usr/share/man/man8/service.8.gz'}
Uploading artifacts for failed job
this looks like false positive, file are in fact diverted. Does the test
check for for diversions?

Post by Otto Kekäläinen
## Schedule
1. March 1st: Enable this job by default, but in allow_failure mode,
making Salsa CI yellow on packages that fail on this job
2. March 31st: Remove the allow_failure mode, potentially making the
Salsa CI red for packages that fail on this job

Could you please consider delaying 2. until diversion are properly
detected?

Another instance of diversions not being detected is in linux's pipeline
[1,2]: linux-libc-dev and oss4-dev both install
/usr/include/linux/soundcard.h, oss4-dev diverts it, missing-break
fails. If my understanding is correct, this will make all unstable/exp
(oss4-dev is in unstable only) src:linux pipelines break starting March
31st.

I agree that diversions should be detected.


[1] https://salsa.debian.org/kernel-team/linux/-/jobs/7205419
[2] https://salsa.debian.org/kernel-team/linux/-/jobs/7182906

Post by Lorenzo
Best Regards,
Lorenzo

[Bastian Blank]

Post by NoisyCoil
Another instance of diversions not being detected is in linux's pipeline
[1,2]: linux-libc-dev and oss4-dev both install
/usr/include/linux/soundcard.h, oss4-dev diverts it, missing-break fails. If
my understanding is correct, this will make all unstable/exp (oss4-dev is in
unstable only) src:linux pipelines break starting March 31st.

Open an serious bug report against oss4-dev. No need to wait, it needs
to go.

Bastian

[NoisyCoil]

Post by Bastian Blank
Open an serious bug report against oss4-dev. No need to wait, it needs
to go.

oss4-dev is fine (unless diversions of files in linux-libc-dev are
forbidden): oss4-dev is correctly diverting the header, as a consequence
it needs not Break or Conflict with linux-libc-dev.

The issue here is that the new missing-breaks pipeline job has no clue
that packages are correctly diverting files, and it flags as missing
Breaks packages which, in fact, do not miss Breaks because they aren't
supposed to have any.

[Bastian Blank]

Post by NoisyCoil
oss4-dev is fine (unless diversions of files in linux-libc-dev are
forbidden): oss4-dev is correctly diverting the header, as a consequence it
needs not Break or Conflict with linux-libc-dev.

linux-libc-dev defines the interface the kernel provides. Random
packages overriding that makes for nasty surprises.

So there are multiple solutions:
- Rename the header and move out of the linux dir.
- Move the header outside of /usr/include and explicitely use this
directory in the include path.

Post by NoisyCoil
The issue here is that the new missing-breaks pipeline job has no clue that
packages are correctly diverting files, and it flags as missing Breaks
packages which, in fact, do not miss Breaks because they aren't supposed to
have any.

Because diverts are kind of sledgehammers. Without coordination they
break stuff.

Bastian

[Matthias Urlichs]

Post by Bastian Blank
Because diverts are kind of sledgehammers. Without coordination they
break stuff.

OK, so in *this* case the diversion is in error (in your opinion anyway;
not having looked at the contents of these headers, I have no opinion here).

However, I'm reasonably certain that there are other cases where
diversions are intentional and perfectly valid. Unless 'missing-breaks'
learns to handle them it will report CI failures for *all* of them, at
which point we might as well change Policy to state that diversions are
a legacy feature that cannot be used in new uploads.

I kindof doubt that we'd get a majority to sign off on that.

--
-- regards
--
-- Matthias Urlichs

[Bastian Blank]

However, I'm reasonably certain that there are other cases where diversions
are intentional and perfectly valid. Unless 'missing-breaks' learns to
handle them it will report CI failures for *all* of them, at which point we
might as well change Policy to state that diversions are a legacy feature
that cannot be used in new uploads.

Do you have an example?

I kindof doubt that we'd get a majority to sign off on that.

You don't need to use it, so don't? All of that is opt-in.

Bastian

[Matthias Urlichs]

Post by Bastian Blank

However, I'm reasonably certain that there are other cases where diversions
are intentional and perfectly valid.

Do you have an example?

Does perldoc diverting the stub perldoc so that it can install the real
one count?

(That's the only one on my desktop that's not glx-diversions or usrmerge.)

Post by Bastian Blank

I kindof doubt that we'd get a majority to sign off on that.

You don't need to use it, so don't? All of that is opt-in.

Fair enough.

--
-- regards
--
-- Matthias Urlichs

[Otto Kekäläinen]

Hi!

Post by Lorenzo

Post by Otto Kekäläinen
Salsa CI has had for many years the job 'missing-breaks' that
complements piuparts by checking that the files a package introduce
don't clash with files shipped by any other package in the
distribution without having proper Breaks/Replaces in the
`debian/control` file. This job works well, being quick to run and has
had zero false positives in our experience.

$ check_for_missing_breaks_replaces.py -o ${WORKING_DIR}/missing_breaks.xml --changes-file ${WORKING_DIR}/*.changes
[ERROR] Missing Breaks/Replaces found
[ERROR] runit-init conflicts with init-system-helpers files: {'/usr/share/man/man8/invoke-rc.d.8.gz', '/usr/sbin/service', '/usr/sbin/invoke-rc.d', '/usr/share/man/man8/service.8.gz'}
Uploading artifacts for failed job
this looks like false positive, file are in fact diverted. Does the test
check for for diversions?

No, it does not. This is now tracked at
https://salsa.debian.org/salsa-ci-team/pipeline/-/issues/418

The challenge is that most of the debian/ contents is declarative,
such as the file lists in debian-*.install files, while the diversions
are procedural code inside maintainer scripts and challenging to
automatically analyze.

If I get more people interested in collaborating on getting the script
first included in devscripts
(https://salsa.debian.org/debian/devscripts/-/merge_requests/478), we
could later take a stab at adding somekind of diversion detection.

Post by Lorenzo

Post by Otto Kekäläinen
## Schedule
1. March 1st: Enable this job by default, but in allow_failure mode,
making Salsa CI yellow on packages that fail on this job
2. March 31st: Remove the allow_failure mode, potentially making the
Salsa CI red for packages that fail on this job

Could you please consider delaying 2. until diversion are properly
detected?

If this CI job is too incorrect for your package, you can also opt out
but having this in your salsa-ci.yml file:

variables:
SALSA_CI_DISABLE_MISSING_BREAKS: 1


The point with having this job enabled for a month in allow_failure
mode is precisely to collect this feedback. If there is enough
packages that use diversions or have other issues, we might indeed
postpone this change, or roll it back completely. Thanks for sharing
your feedback!

[Chris Hofstaedtler]

Post by Otto Kekäläinen

Post by Lorenzo

Post by Otto Kekäläinen
Salsa CI has had for many years the job 'missing-breaks' that
complements piuparts by checking that the files a package introduce
don't clash with files shipped by any other package in the
distribution without having proper Breaks/Replaces in the
`debian/control` file. This job works well, being quick to run and has
had zero false positives in our experience.

^^^^

https://binarycontrol.debian.net/?q=dpkg-divert&path=unstable%2F

This doesn't give a huge list of packages using diversions, but very
high profile packages are in there. I wonder what your testing
strategy was :-)

Post by Otto Kekäläinen

Post by Lorenzo
$ check_for_missing_breaks_replaces.py -o ${WORKING_DIR}/missing_breaks.xml --changes-file ${WORKING_DIR}/*.changes
[ERROR] Missing Breaks/Replaces found
[ERROR] runit-init conflicts with init-system-helpers files: {'/usr/share/man/man8/invoke-rc.d.8.gz', '/usr/sbin/service', '/usr/sbin/invoke-rc.d', '/usr/share/man/man8/service.8.gz'}
Uploading artifacts for failed job
this looks like false positive, file are in fact diverted. Does the test
check for for diversions?

No, it does not. This is now tracked at
https://salsa.debian.org/salsa-ci-team/pipeline/-/issues/418
The challenge is that most of the debian/ contents is declarative,
such as the file lists in debian-*.install files, while the diversions
are procedural code inside maintainer scripts and challenging to
automatically analyze.

Yup. dumat has a lot of code to deal with diversions. But overall it
seems totally doable/reusable.

live-tools also seems to divert files from initramfs-tools, which is
how I found out about this mail thread.

Best,
Chris

[Matthias Urlichs]

Post by Chris Hofstaedtler
This doesn't give a huge list of packages using diversions, but very
high profile packages are in there.

Keep in mind that many of those are there to support usrmerge; these
probably should be filtered out.

On the other hand, on my system lots of diversions are tagged
"glx-diversions"; the packages that installed these don't show up in
https://binarycontrol.debian.net/?q=dpkg-divert&path=unstable.
Apparently "dpkg-maintscript-helper symlink_to_dir" does (did?) some
interesting behind-the-scenes magic there.

Bottom line: It's not *that* easy.

--
-- regards
--
-- Matthias Urlichs

[Otto Kekäläinen]

Hi,

Would Matthias or anyone else active in this discussion be willing to
be the reviewer for
https://salsa.debian.org/debian/devscripts/-/merge_requests/478?
It has been pending for almost two weeks with zero reviews. I don't
feel it would be prudent of me to merge it with no additional eyeballs
checking it at least superficially.

- Otto