Discussion:
DEP17 /usr-move: most mitigations (M18) for aliased diversions (P3) are broken
(too old to reply)
Helmut Grohne
2025-01-08 15:00:02 UTC
Permalink
write some tests. As a result, I'm attaching a shell script. It
A kind soul reminded me of actually attaching something.

Helmut
G. Branden Robinson
2025-01-09 19:50:02 UTC
Permalink
Hi Helmut,

I have a general remark and a technical one.
I have bad news about /usr-move. The primary mitigation (M18) for
aliased diversions (P3) as implemented by me in quite a few packages
is broken.
[...]
If your have a lot of time and are interested in the underlying
technical details, I invite you to continue reading. Otherwise,
consider doing something more pleasant.
[...]

I'd like to call out this response as salutary example of how to cope
with a setback in project management. Your message is candid, and if
it's not too effusive an utterance, honorable.

There is a saying in management circles, "fix the problem, not the
blame".[1] However, there is another saying, "the officers of the firm
have a legal duty to maximize returns to the shareholders".

The latter is a myth,[2] but an extremely useful fiction for those who
either want to contrive constructive discharge of an otherwise
productive and unobjectionable employee,[3] believe they can derive
greater revenue for the firm, just their department, or simply for
themselves by leaving the problem unfixed--or if someone manages to fix
it anyway on their own damnable initiative, do one's level best to keep
the fix from being widely deployed.[4]

I'll climb off the soapboax and down to technical matters now.
Fundamentally, I assumed that dpkg-divert --rename would only move a
file if it was considered installed according to the dpkg database.
What really happens is that --rename moves a file as long as it
exists.
Is that within dpkg's specification? If it is, in fact, a bug, should
we consider tackling it for the next development cycle?

If it's valuable behavior nevertheless, should dpkg-divert support an
additional option to force it to operate even on files not installed per
the database? (I assume no one thinks the default shouldn't be to
restrict itself to installed files. Everything I see in the
dpkg-divert(1) man page suggests to me that its only contemplated scope
of application is the set of files actually in a package's payload.)
Since I was failing to get this right earlier, I figured I could as
well write some tests.
<applause>

A long time ago I dreaded writing tests. Now, I often get a perverse
satisfaction out of it.

But about as often I get frustrated with the code under test, because I
find out just how under-specified it was. Fantasies of violence
often follow.[5]
write some tests. As a result, I'm attaching a shell script. It
A kind soul reminded me of actually attaching something.
Happens to me (almost) every time...

Regards,
Branden

[1] I tried to trace this quote; the best I can do is to observe that it
was popularized by Michael Crichton in a 1993 novel.

https://www.goodreads.com/work/quotes/2045220-rising-sun

For Crichton personally, a corollary would appear to be "and if
fixing the problem would inconvenience the wealthy people with whom
you rub elbows, deny that the problem exists."

https://www.brookings.edu/articles/michael-crichton-and-global-warming/

Thus can we take a lesson in distinguishing wisdom from the identity
of the oracle uttering it.

[2] https://www.investopedia.com/terms/s/shareholder-value.asp#toc-what-is-the-shareholder-value-maximization-myth
[3] https://webapps.dol.gov/elaws/eta/warn/glossary.asp?p=constructive%20discharge

[4] Here's an American computer business story from the 1980s with a
Unix connection.

https://paste.c-net.org/TruthsBottoms

[5] https://en.wikipedia.org/wiki/Rampage_(1986_video_game)
Helmut Grohne
2025-01-10 14:50:02 UTC
Permalink
Hi Branden,
Post by G. Branden Robinson
Fundamentally, I assumed that dpkg-divert --rename would only move a
file if it was considered installed according to the dpkg database.
What really happens is that --rename moves a file as long as it
exists.
Is that within dpkg's specification? If it is, in fact, a bug, should
we consider tackling it for the next development cycle?
Really? We are very deep in the land of implementation-defined
behaviour. From a specification point of view, I'd call doing diversions
on aliased files undefined behaviour. It really does not matter at all
whether we call this a bug or anything. At the time our maintainer
scripts run, we must expect them to run with bookworm's or trixie's dpkg
and we cannot rely on any change in behaviour to be applied to the
former one. The only thing that we care about is that as little as
possible implementation-defined behaviour in this area changes between
bookworm and trixie. Luckily, Guillem put his effort into other valuable
areas of dpkg that are less of a concern for the /usr-move.

Then when it comes to the next development cycle, the idea is that we
have left aliasing behind. In the common case, if there is an untracked
file in a location that is being diverted, the present --rename
semantics arguably are useful as the untracked file will be renamed (on
installation) and back (on removal). I asked Guillem off list to
consider documenting this aspect.

Helmut
Helmut Grohne
2025-01-15 13:10:01 UTC
Permalink
Since I was failing to get this right earlier, I figured I could as well
write some tests. As a result, I'm attaching a shell script. It
constructs a few demo packages and attempts a hopefully exhaustive
combination of using them and dpkg-divert by writing its experiments to
the working directory. Running it should be fairly boring. If you happen
to review it, I appreciate a response.
Despite multiple requests to actually attach my tests, nobody seems to
have read them and spot the missing test cases. And of course they were
failing.

The case of upgrading from a single, aliased diversion to a duplicated
diversion was not covered. In this case, we want the canonicalized
diversion target to be used by the canonicalized diversion and
therefore, the aliased diversion must change its diversion target.
Failure to do so results in file loss and/or left-over files.

I'm attaching an updated version that tests the transition from aliased
diversion to duplicated diversion both in the case of the divertee being
absent and present. It's a lot of branches now, but I don't quite see
how we could do it with less.

Again, I appreciate someone actually reading the test cases and telling
me where I am wrong such that I don't have to redo too many redone
patches. At least the one I submitted for bfh-container fails an
additional test case.

Helmut
Helmut Grohne
2025-01-15 13:50:01 UTC
Permalink
On Wed, Jan 08, 2025 at 03:21:43PM +0100, Helmut Grohne wrote:
codesearching DEP17.*M18 (and a few other patterns) suggest the
* bfh-metapackages
Open #1092955. I first got it wrong and then sent another update.
* clonezilla
Open #1093135. Though this one was broken in more ways and I guess it is
dead code.
* cryptsetup-nuke-password (and cryptsetup)
Not affected in a non-obvious way. cryptsetup-nuke-password performs the
duplicate dpkg-divert --add --rename. It first renames the canonicalized
path wrongly renames an aliased askpass. However, it also depends on the
canonicalized cryptsetup so by the time its postinst runs, the wrongly
moved file is overwritten by the cryptsetup update and things are fine
again.
* debian-installer-utils
Open
https://salsa.debian.org/installer-team/debian-installer-utils/-/merge_requests/11
* isc-dhcp
Not affected in a non-obvious way. It does the duplicate dpkg-divert
--add --rename and therefore moves an aliased file to the wrong location
in its preinst. However, the diverting package also depends on a version
of the divertee that is canonicalized. The upgrade of the divertee will
clobber the wrongly moved file and after postinst the effects are ok.
This kinda is the simplified case in m18_diversions.sh.
* molly-guard
Open. #1093132. Avoiding --rename as much as possible now.
* live-build
Open
https://salsa.debian.org/live-team/live-build/-/merge_requests/394#note_569976
* open-infrastructure-system-tools
Open. #1093131. Similar to debian-installer-utils.
* opensysusers
The duplicate --rename is there, but in the mean time systemd-sysusers
implementations started conflicting with one another, so the use of
diversions is no longer required. I propose dropping the diversion mess
entirely: #1092745.
* piuparts
Fixed. Aparently, I understood this back in March 2024 and then failed
to transfer what I learned to other packages. :-(
https://salsa.debian.org/debian/piuparts/-/commit/1b9a3843964b42b983921a1b011f14d61e430944
* progress-linux-metapackages
Open #1092956. Similar to bfh-container.
* systemd
Not affected. It is being diverted, but does no --rename.
* zutils (and gzip)
Installing zutils/trixie on bookworm, looses gzip files. #1092737 fixed
in zutils/1.14-1.

* runit

Not affected. It is being diverted, but does no --rename.

* sysvinit

Not affected. It is being diverted and it diverts manual pages with
--rename.

So this hopefully concludes the mess as I am not aware of any other
affected cases. What this does not handle is the breakage left behind
trixie or sid systemd that have taken all intermediate steps. If you
find that you have any files named *.usr-is-merged, that's a bad sign
and you may reach out to me. My focus has been fixing the bookworm ->
trixie upgrade, but we may also handle unstable upgrades if that happens
to affect non-trivial amounts of users.

Helmut

Loading...