Proposal: drop libcrypt-dev dependency from libc6-dev

[Helmut Grohne]

Hello fellow developers,

how about libc6-dev stops depending on libcrypt-dev?

I mean for real. We cannot do it right away. The proposal is forky
material of course. Also libc6-dev would still "Recommends:
libcrypt-dev", but libcrypt-dev would no longer be build-essential.

The obvious "why" question needs an answer. libxcrypt and glibc pose a
2-cycle during architecture cross bootstrap and dropping the dependency
solves that cycle. It also is not that many packages that actually use
libcrypt-dev as we shall see.

Let's move into the "how" part. The easy part is preparing a glibc
package with the dependency dropped. You may find one at
https://people.debian.org/~helmutg/glibc-no-crypt/build/. Santiago Vila
kindly performed an archive rebuild using this modified glibc (and with
libcrypt-dev removed) and handed me some logs. I sorted those logs and
you may now find them at
https://people.debian.org/~helmutg/glibc-no-crypt/logs/.

Now we need to talk about numbers. It's 803 FTBFS. Of those, 576 issue a
dependency on libperl-dev and since perl #includes <crypt.h> somewhere,
libperl-dev should be depending on libcrypt-dev. There are also 95
packages that build a perl extension module without depending on
perl-xs-dev. Such a practice causes them to FTCBFS, but with
libcrypt-dev that FTCBFS is turned into a FTBFS. Beyond that, 11
packages build a perl extension module for testing purposes and
therefore need "Build-Depends: perl-xs-dev <!nocheck>". One package
actually embeds a perl interpreter without depending on libperl-dev.

So far so good. That's 1 + 95 + 11 + 1 = 108 source packages from the
perl ecosystem in need of changes. What about others?

I classified 110 of the packages as needing a direct libcrypt-dev
dependency as they use it in their source directly. And then we have a
few packages that indirectly use libcrypt. The following packages ship a
pkgconfig file containing -lcrypt and therefore need "Depends:
libcrypt-dev".
* guile-2.2-dev
* guile-3.0-dev
* heimdal-multidev
* libapr1-dev
* libidzebra-2.0-dev
* librep-dev
* libuser1-dev
* ruby3.1-dev
* ruby3.3-dev
In addition, gauche-dev has gauche-config that also emits -lcrypt. Now
matching this up with the build failures yields four that will be fixed
be one of these being modified.

There is one further category. 6 packages link -lcrypt without using it
in any way. I went right ahead and filed patches:
https://udd.debian.org/cgi-bin/bts-usertags.cgi?user=***@debian.org&tag=libcrypt-unused

So that's modifying another 110 + 9 + 1 + 6 = 126 source packages
outside the perl ecosystem. You'll find all of the mentioned categories
in the published logs as subdirectories. Please bear in mind that among
the packages that FTBFS in unstable, a small fraction would additionally
FTBFS without libcrypt and I've missed those. Expect a few more.

If we assume all of these changes, we've addressed all FTBFS, but we're
not done yet. What about packages that check for libcrypt and silently
opt out of it when it is unavailable? Typically those packages would
emit a runtime dependency on libcrypt1, right? So I also investigated
all apt-cache rdepends libcrypt1. That results in 151 source packages.
Of those, a large fraction will already be covered by investigating
build failures. Additionally, we had to add runtime dependencies to a
few packages above. And then there are 11 source packags that already
build depend on libcrypt-dev (mostly to support bootstrapping). So if
you disregard all of those duplicates, what remains is 28 packages
missed in the FTBFS-based analysis:
* alpine
* asterisk
* burp
* ccrypt
* courier-authlib
* cpu
* cvm
* freewnn
* frr
* guestfs-tools
* ircii
* lighttpd
* lsh-utils
* mariadb
* netatalk
* pike8.0
* ppp
* python-oslo.utils
* python3.12
* regina-rexx
* rserve
* scrollz
* steam-installer
* tinymux
* ukui-control-center
* virt-v2v
* x11vnc
* xemacs21

I also considered running autopkgtests of glibc's reverse dependencies,
but that proved to be harder than expected.

In the end, we also need to modify glibc to actually demote the
dependency. So now we've reached my final number. 108 + 126 + 28 + 1 =
263 source packages need modifications. Some issues may remain here, but
at this time I don't expect large numbers anymore. On the perl side, a
practical side effect will be fixing couple of cross builds.

Question 1: Do you see important aspects missed in this analysis?

Question 2: Do you agree that this change is worth the effort?

Assuming "no" and "yes" as answers, I'd like to proceed with mass bug
filing. I propose severity normal for all bugs. The additional
dependencies can be piggy-backed onto uploads aimed for trixie as their
risk of causing breakage is really low. For the perl-related changes we
might skip filing the bugs and consider mass-committing the changes for
a later upload. The glibc change must not be done during the trixie
cycle, but can happen early in the forky one. At that point, the
remaining bugs would become RC.

Question 3: Can I move forward with the MBF?

I'd appreciate explicit replies from:
* libxcrypt maintainer Marco
* glibc maintainers (e.g. Aurelien)
* Debian Perl Group (e.g. Gregor) (high number of affected packages)

Thanks for considering

Helmut

[Simon McVittie]

Post by Helmut Grohne
how about libc6-dev stops depending on libcrypt-dev?

I think this is a good idea for early in the forky cycle.

Post by Helmut Grohne
I also investigated
all apt-cache rdepends libcrypt1. That results in 151 source packages.

...

Post by Helmut Grohne
* steam-installer

This particular one is a false positive, it hard-codes a dependency on
libcrypt1 (or more precisely libcrypt1 | libc6 (<< 2.29-4)) as one of
the SONAMEs that Steam relies on the host system to provide (because it
was historically part of glibc). It shouldn't need a -dev build-dependency
for this purpose.

smcv

[Marco d'Itri]

Post by Helmut Grohne
how about libc6-dev stops depending on libcrypt-dev?

Sure.

Post by Helmut Grohne
libcrypt-dev", but libcrypt-dev would no longer be build-essential.

What purpose would this Recommends solve?

Post by Helmut Grohne
Assuming "no" and "yes" as answers, I'd like to proceed with mass bug
filing. I propose severity normal for all bugs. The additional

Go for it, great job!
Removing the package from the build-essential set is a worthwhile goal.

But please mention prominently that there is no need to work on this
before the release.

[Helmut Grohne]

Hi Marco,

Post by Marco d'Itri

Post by Helmut Grohne
how about libc6-dev stops depending on libcrypt-dev?

Sure.

Thanks for the feedback.

Post by Marco d'Itri

Post by Helmut Grohne
libcrypt-dev", but libcrypt-dev would no longer be build-essential.

What purpose would this Recommends solve?

My thinking here was to reduce the annoyance for users by degrading the
dependency softly. I was told that some users expect to be able to build
perl extensions without installing libperl-dev (which is why libperlVER
contains the headers that #include <crypt.h>). If we were right dropping
the dependency, libcrypt-dev could be automatically be removed from
those systems and we'd get angry bug reports. Going to Recommends is a
means to limit the annoyance while still meeting the original goal.

This is not to be read as an endorsement of libc6-dev permanently
recommending libcrypt-dev. For instance downgrading it to Suggests or
dropping it entirely could be an option for later. To me, the key
feature here is making it removable in the first place.

In any case, I'll leave this particular aspect to the glibc and
libxcrypt maintainers. If you feel, Recommends is not warranted, so be
it.

Helmut

[Santiago Vila]

[ Thanks for this. Awesome work! ]

Post by Helmut Grohne
My thinking here was to reduce the annoyance for users by degrading the
dependency softly. I was told that some users expect to be able to build
perl extensions without installing libperl-dev (which is why libperlVER
contains the headers that #include <crypt.h>). If we were right dropping
the dependency, libcrypt-dev could be automatically be removed from
those systems and we'd get angry bug reports. Going to Recommends is a
means to limit the annoyance while still meeting the original goal.

On the other hand:

- A recommends will only have some effect for people not having
libcrypt-dev installed yet, so people upgrading from trixie to forky
who have both libc6-dev and libcrypt-dev installed will not see the
libcrypt-dev package mysteriously removed from their systems.

- In general, Debian users are already aware that all sorts of breakage in
unstable is expected to happen shortly after a stable release, so I would
not really expect angry bugs about this from users.

So, I agree with Marco that Recommends is not necessary.

For the purposes of being nice to people, I would maybe extend
the period where the bugs are not RC as necessary so that we don't
introduce a lots of RC bugs at once (i.e. raise to serious not
in a date-based scheme but when the number of bugs is low enough).

Thanks.

[gregor herrmann]

Post by Helmut Grohne
I sorted those logs and
you may now find them at
https://people.debian.org/~helmutg/glibc-no-crypt/logs/.

Thanks.

Post by Helmut Grohne
Beyond that, 11
packages build a perl extension module for testing purposes and
therefore need "Build-Depends: perl-xs-dev <!nocheck>".

I learned something new :)

Post by Helmut Grohne
For the perl-related changes we
might skip filing the bugs and consider mass-committing the changes for
a later upload.

At first glance that sounds reasonable (and save us from the work of
correlating and closing bugs in packages). At second glance, i.e.
after looking at
<https://people.debian.org/~helmutg/glibc-no-crypt/logs/needs-perl-xs-dev/>,
I note that many packages there don't match the 'lib.+-perl' pattern,
i.e. are not maintained by the Debian Perl Team, and from the
remaining ones, (without checking them) I also don't recognize quite
a few package names. ¹

I think the "mass" for "mass-commiting" would not be high enough to
rely on this mechanism, so I guess including them in the MBF makes
more sense.


Cheers,
gregor


¹ I'm not surprised that this problem mainly affects
non-team-maintained packages, as we've been using perl-xs-dev since
quite some time 


[Aurelien Jarno]

Hi,

Post by Helmut Grohne
Hello fellow developers,

[ Snip ]

Post by Helmut Grohne
Question 1: Do you see important aspects missed in this analysis?

No

Post by Helmut Grohne
Question 2: Do you agree that this change is worth the effort?

I don't know. I do not see a huge benefit from the glibc point of view,
but I understand that it could be a benefit for other parts of Debian.
As long as these other part leads the effort that sounds good.

Post by Helmut Grohne
Assuming "no" and "yes" as answers, I'd like to proceed with mass bug
filing. I propose severity normal for all bugs. The additional
dependencies can be piggy-backed onto uploads aimed for trixie as their
risk of causing breakage is really low. For the perl-related changes we
might skip filing the bugs and consider mass-committing the changes for
a later upload. The glibc change must not be done during the trixie
cycle, but can happen early in the forky one. At that point, the
remaining bugs would become RC.

It seems to me that this way of doing might block glibc development, ie
prevent glibc from migrating to testing for many weeks, as I guess it
will causes autopkgtest failures in addition to FTBFS.

I think we should proceed with the change on the glibc side once we know
that the bugs can be be solved *in testing* in a reasonable time frame.
This could mean massive NMUs to fix the remaining issues (but this could
be seen as aggressive). An other option is to upgrade the severity to
serious before the change on the glibc side, which should remove most
unfixed packages from testing within 1 month, and the remaining packages
can be fixed through NMU.

Post by Helmut Grohne
Question 3: Can I move forward with the MBF?

Sounds good to me. Whatever is decided, having an explicit dependency is
always better, than relying on de facto build-essential.

Regards
Aurelien

[Helmut Grohne]

Hello fellow developers,

Post by Helmut Grohne
how about libc6-dev stops depending on libcrypt-dev?

with minor disagreement in details, I have received much positive
feedback and therefore moved forward.

Post by Helmut Grohne
So far so good. That's 1 + 95 + 11 + 1 = 108 source packages from the
perl ecosystem in need of changes. What about others?

All of the perl ones have been filed and gregor already fixed (thanks!)
a significant fraction including all 11 perl-xs-dev <!nocheck> ones. I
guess that on third of these is fixed in git or unstable.

Post by Helmut Grohne
few packages that indirectly use libcrypt. The following packages ship a
libcrypt-dev".
* guile-2.2-dev
* guile-3.0-dev
* heimdal-multidev
* libapr1-dev
* libidzebra-2.0-dev
* librep-dev
* libuser1-dev
* ruby3.1-dev
* ruby3.3-dev
In addition, gauche-dev has gauche-config that also emits -lcrypt. Now
matching this up with the build failures yields four that will be fixed
be one of these being modified.

Bugs with patches are filed for these.
https://udd.debian.org/cgi-bin/bts-usertags.cgi?user=***@debian.org&tag=libcrypt-pkgconf
Notably, libuser is missing as I did a QA upload instead.

Post by Helmut Grohne
So that's modifying another 110 + 9 + 1 + 6 = 126 source packages
outside the perl ecosystem. You'll find all of the mentioned categories
in the published logs as subdirectories. Please bear in mind that among
the packages that FTBFS in unstable, a small fraction would additionally
FTBFS without libcrypt and I've missed those. Expect a few more.

...

Post by Helmut Grohne
build depend on libcrypt-dev (mostly to support bootstrapping). So if
you disregard all of those duplicates, what remains is 28 packages

I have not yet filed bugs for packages lacking "Build-Depends:
libcrypt-dev". That's a next step. I consider the perl-xs-dev
dependencies and the runtime dependencies more important as both of them
also affect other use cases (such as cross building).
Thanks for the quick feedback!

Regarding the timing of the glibc upload, I also am in favor of not
upgrading lots of these bugs to rc severity. Given the usertags, we may
monitor how the situation evolves in forky. I suggest once the remaining
unfixed bug count (across all categories) is 30 or less, we may proceed
and upgrade the remaining ones to rc.

Helmut