Supporting alternative zlib implementations

[Mark Brown]

A recurrning question with the zlib package in Debian is interest in the
various alternative zlib implementations that are out there. There was
a long period where upstream zlib development seemed very stalled,
during that period people who wanted improvements started forking their
own projects. The main ones I'm aware of are:

zlib-ng: https://github.com/zlib-ng/zlib-ng
chromium: https://chromium.googlesource.com/chromium/src/third_party/zlib

zlib-ng seems pretty healthy, the chromium fork is less generally active
but is used by Chrome/ChromeOS which is a big userbase.

The main thing people seem excited about is performance work for modern
platforms though both projects have been doing other work on the code.
Unfortunately it looks like there is little interest in bringing these
forks together in spite of zlib's upstream development having picked up
a bit again.

Fedora did a transition to zlib-ng relatively recently, in version 40:

https://fedoraproject.org/wiki/Changes/ZlibNGTransition
https://packages.fedoraproject.org/pkgs/zlib/zlib/
https://packages.fedoraproject.org/pkgs/zlib-ng/zlib-ng/

In the past I've pushed back on doing anything here since zlib is
essential and it seemed better to be consistent over the ecosystem than
to use a more niche implementation, and some of the early optimisation
efforts had not worked well on CPUs other than their immediate targets.
However given the user feedback and looking at the Fedora experience I
think it might be time to reevaluate that.

Obviously it's far too late to do anything with the default for trixie,
we might want to evaluate doing something after the release but for now
it's too late.

There's been some ongoing discussion (which sadly I wasn't looped into
most of) of zlib-ng in WNPP:

https://bugs.debian.org/1002056

with some packaging done, but not AIUI building the zlib compatible ABI
for zlib-ng yet which would allow it to be used as a replacement.
Adding support for the compatible ABI allowing it to be an alternative
for standard zlib seems to me like an obvious step we could take, it
would need a lot of care given that zlib is essentially but would let
people get zlib-ng if they wanted, and if there are problems it can be
held in unstable (or experimental) to avoid impact on trixie. This
would allow people to kick the tires.

Does anyone have thoughts on this?

[Guillem Jover]

Hi!

Post by Mark Brown
In the past I've pushed back on doing anything here since zlib is
essential and it seemed better to be consistent over the ecosystem than
to use a more niche implementation, and some of the early optimisation
efforts had not worked well on CPUs other than their immediate targets.
However given the user feedback and looking at the Fedora experience I
think it might be time to reevaluate that.

Great! I'm happy to hear that.

Post by Mark Brown
Obviously it's far too late to do anything with the default for trixie,
we might want to evaluate doing something after the release but for now
it's too late.

Personally I don't think it's too late, there should be several months
until the freeze, and I think if we wanted to switch we could perhaps
do a staged transition and see how it goes and only do the final
replacement if everything seems fine.

Post by Mark Brown
There's been some ongoing discussion (which sadly I wasn't looped into
https://bugs.debian.org/1002056
with some packaging done, but not AIUI building the zlib compatible ABI
for zlib-ng yet which would allow it to be used as a replacement.
Adding support for the compatible ABI allowing it to be an alternative
for standard zlib seems to me like an obvious step we could take, it
would need a lot of care given that zlib is essentially but would let
people get zlib-ng if they wanted, and if there are problems it can be
held in unstable (or experimental) to avoid impact on trixie. This
would allow people to kick the tires.

Sorry, I've been meaning to bring this up again to your attention,
given that as you mention zlib-ng has seen steady development and
buy in from the community at large. But at the same time, I've been
both a bit reluctant to upload anything to avoid the impression of
some kind of attempt to a hostile takeover, and to bring this up to
you as from your earlier push back I thought that would require some
(perhaps) exceeding changed circumstances. But given your mail, I'm
happy to work on this again and start with say uploading some initial
stuff into experimental for example, after this thread settles a bit?

(I'll start by refreshing the packaging first though.)

Post by Mark Brown
Does anyone have thoughts on this?

Personally, I think fully migrating from zlib to zlib-ng would sound
great (even for trixie), but I guess we can take it slow if you do not
feel confident or have concerns over this.

Also if you'd prefer to take over the zlib-ng ITP, as a continuation
of zlib, that'd seem fine with me too.

Thanks,
Guillem

[Fay Stegerman]

* Guillem Jover <***@debian.org> [2024-09-24 17:45]:
[...]

Post by Guillem Jover
Personally, I think fully migrating from zlib to zlib-ng would sound
great (even for trixie), but I guess we can take it slow if you do not
feel confident or have concerns over this.

As using an alternative zlib implementation could impact Reproducible Builds
[1], I would recommend taking that into consideration before deciding on this
kind of change.

- Fay

[1] https://lists.reproducible-builds.org/pipermail/rb-general/2024-September/003526.html

[Guillem Jover]

Hi!

Post by Fay Stegerman

Post by Guillem Jover
Personally, I think fully migrating from zlib to zlib-ng would sound
great (even for trixie), but I guess we can take it slow if you do not
feel confident or have concerns over this.

As using an alternative zlib implementation could impact Reproducible Builds
[1], I would recommend taking that into consideration before deciding on this
kind of change.

Ah, this is related to something I wanted to mention too and forgot.

I don't think the specific case you mention is in itself a concern for
Debian, because we only guarantee reproducibility given the same inputs,
which includes the set of packages and their versions that were used
when building the binaries. So if there was a switch, those would end up
being recorded as well, and used when reproducing the outputs. And this
could also happen with a newer version of zlib itself.

The problem though is, that because the compressed stream is going to
change, that can make certain test suites fail if we perform this
switch, which I think would be the main fallout that we'd see from
this and would need manual fixing, although I assume Fedora has probably
handled most of these already. For example when I added explicit
zlib-ng support to dpkg, I had to fix its test suite to parametrize
sizes for test artifacts.

I think it would be pretty easy to at least see the extent of this
fallout by performing a mass rebuild for packages build-depending
on zlib1g-dev with a zlib-ng version.

Thanks,
Guillem

[Mike Hommey]

Post by Guillem Jover
Hi!

Post by Fay Stegerman

Post by Guillem Jover
Personally, I think fully migrating from zlib to zlib-ng would sound
great (even for trixie), but I guess we can take it slow if you do not
feel confident or have concerns over this.

As using an alternative zlib implementation could impact Reproducible Builds
[1], I would recommend taking that into consideration before deciding on this
kind of change.

Ah, this is related to something I wanted to mention too and forgot.
I don't think the specific case you mention is in itself a concern for
Debian, because we only guarantee reproducibility given the same inputs,
which includes the set of packages and their versions that were used
when building the binaries. So if there was a switch, those would end up
being recorded as well, and used when reproducing the outputs. And this
could also happen with a newer version of zlib itself.
The problem though is, that because the compressed stream is going to
change, that can make certain test suites fail if we perform this
switch, which I think would be the main fallout that we'd see from
this and would need manual fixing, although I assume Fedora has probably
handled most of these already. For example when I added explicit
zlib-ng support to dpkg, I had to fix its test suite to parametrize
sizes for test artifacts.

As someone who recently tested a rust port of zlib-ng, another factor to
take into account is that while zlib-ng is faster, it also looks like it
compresses less at the same compression level.
Using higher compression levels with the same compression rate is also
usually faster too, but without touching the compression levels, you
end up with something that compresses in less time, but also with a
bigger output.

Mike

[Simon Richter]

Hi,

Post by Guillem Jover
So if there was a switch, those would end up
being recorded as well, and used when reproducing the outputs. And this
could also happen with a newer version of zlib itself.

I have a POWER9 box, which includes a NX-GZIP coprocessor, which is
currently not used for anything, but it makes even -9 compression very
cheap (9 GB/s). If I were to use it (probably through the kernel
subsystem), I should probably record it somewhere.

Thinking about it, I'd expect POWER to generate different output than
x86 when building an arch:all package with the result of some floating
point computation, at least when -ffast-math is active (x86 being wrong).

Simon

[Mark Brown]

Post by Guillem Jover
The problem though is, that because the compressed stream is going to
change, that can make certain test suites fail if we perform this
switch, which I think would be the main fallout that we'd see from
this and would need manual fixing, although I assume Fedora has probably
handled most of these already. For example when I added explicit
zlib-ng support to dpkg, I had to fix its test suite to parametrize
sizes for test artifacts.

I guess this is also a risk for zlib upgrades, seems a bit fragile.

[Fay Stegerman]

Post by Guillem Jover
Hi!

Post by Fay Stegerman

Post by Guillem Jover
Personally, I think fully migrating from zlib to zlib-ng would sound
great (even for trixie), but I guess we can take it slow if you do not
feel confident or have concerns over this.

As using an alternative zlib implementation could impact Reproducible Builds
[1], I would recommend taking that into consideration before deciding on this
kind of change.

Ah, this is related to something I wanted to mention too and forgot.
I don't think the specific case you mention is in itself a concern for
Debian, because we only guarantee reproducibility given the same inputs,
which includes the set of packages and their versions that were used
when building the binaries. So if there was a switch, those would end up
being recorded as well, and used when reproducing the outputs. And this
could also happen with a newer version of zlib itself.
The problem though is, that because the compressed stream is going to
change, that can make certain test suites fail if we perform this
switch, which I think would be the main fallout that we'd see from
this and would need manual fixing, although I assume Fedora has probably
handled most of these already. For example when I added explicit
zlib-ng support to dpkg, I had to fix its test suite to parametrize
sizes for test artifacts.

Whilst it indeed may not affect the reproducibility guarantees for Debian
packages themselves, it does affect being able to use a Debian system for
Reproducible Builds of other software for which the reference artefacts were
built with regular zlib and thus can no longer be reproduced on Debian if that
uses a different zlib implementation (so far I've only encountered the reverse,
which seems relatively rare -- for now).

For example, ZIP files or Android APKs built on a Debian system will have a
different compressed stream, like the test files you mention. Which will likely
break Reproducible Builds tooling like apksigcopier [1] and
reproducible-apk-tools [2].

AFAIK all rebuilders (including my own [3]) for Android APKs use Debian base
systems, so this could cause quite a bit of breakage for Reproducible Builds
within that ecosystem, which is something I would like to avoid (or at least
have a decent workaround for -- e.g. being able to easily choose between
multiple zlib implementations during runtime in my Python tooling would be
great).

As you point out, we've been lucky that zlib has remained backwards-compatible
for a long time (even though it doesn't provide any guarantees of that AFAIK).
Which also makes me wonder how much more likely zlib-ng might be to produce
different compressed streams between different versions or using different
hardware (configurations).

There might also be issues with reproducibility of Debian packages themselves if
e.g. zlib-ng output can differ on different hardware (e.g. number of cores) even
with an otherwise identical build environment. At the very least I think it
would be good to know how all this could be affected (and how likely things are
to remain as stable as zlib has been so far) before making a decision to switch.

Post by Guillem Jover
I think it would be pretty easy to at least see the extent of this
fallout by performing a mass rebuild for packages build-depending
on zlib1g-dev with a zlib-ng version.

- Fay

[1] https://tracker.debian.org/pkg/apksigcopier
[2] https://github.com/obfusk/reproducible-apk-tools
[3] https://github.com/obfusk/rbtlog

[Sebastian Andrzej Siewior]

Post by Fay Stegerman
For example, ZIP files or Android APKs built on a Debian system will have a
different compressed stream, like the test files you mention. Which will likely
break Reproducible Builds tooling like apksigcopier [1] and
reproducible-apk-tools [2].

wouldn't it work to compare the decompressed stream? Is an identical ZIP
file a requirement?

Post by Fay Stegerman
There might also be issues with reproducibility of Debian packages themselves if
e.g. zlib-ng output can differ on different hardware (e.g. number of cores) even
with an otherwise identical build environment. At the very least I think it
would be good to know how all this could be affected (and how likely things are
to remain as stable as zlib has been so far) before making a decision to switch.

I don't know at this time. Maybe we could throw it into exp first and
evaluate the situtation.

Post by Fay Stegerman
- Fay

Sebastian

[Fay Stegerman]

Post by Sebastian Andrzej Siewior

Post by Fay Stegerman
For example, ZIP files or Android APKs built on a Debian system will have a
different compressed stream, like the test files you mention. Which will likely
break Reproducible Builds tooling like apksigcopier [1] and
reproducible-apk-tools [2].

wouldn't it work to compare the decompressed stream? Is an identical ZIP
file a requirement?

By definition a Reproducible Build means a bit-by-bit identical APK, including
the signature (which is why I built a tool to extract an existing signature and
use it as a build input instead of the private key). Which means you need
identical compressed data for Reproducible Builds.

Having identical uncompressed data gets you pretty close to the goals of RB, but
unpacking and/or skipping over signatures is very very hard to get right and
simply cannot provide the same guarantees as having two bitwise identical files.

And it's impossible to create an APK you can actually install if it's not
bit-by-bit identical as the signature would not be valid otherwise. So yes,
unfortunately an identical ZIP file is a requirement and comparing the
decompressed stream not an option, which is why this kind of change is not
something we can just consider an implementation detail or work around.

I wrote more about the very messy situation Fedora's switch to zlib-ng already
created for Android Reproducible Builds [1]. Which likely would have broken a
lot more reproducible Android apps already if Fedora's OpenJDK packages linked
against the system zlib like Debian's OpenJDK packages do (instead of using an
embedded copy of regular zlib).

- Fay

[1] https://lists.reproducible-builds.org/pipermail/rb-general/2024-September/003547.html

[Konstantin Demin]

One minor moment: zlib-ng doesn't seem to be fully backward compatible.
E.g. Angie (nginx's fork with enhancements) is unable to perform gzip
compression [1] if built against zlib-ng.
It's highly likely that nginx is affected too.

[1] https://t.me/angie_support/4205

Post by Fay Stegerman

Post by Sebastian Andrzej Siewior

Post by Fay Stegerman
For example, ZIP files or Android APKs built on a Debian system will have a
different compressed stream, like the test files you mention. Which will likely
break Reproducible Builds tooling like apksigcopier [1] and
reproducible-apk-tools [2].

wouldn't it work to compare the decompressed stream? Is an identical ZIP
file a requirement?

By definition a Reproducible Build means a bit-by-bit identical APK, including
the signature (which is why I built a tool to extract an existing signature and
use it as a build input instead of the private key). Which means you need
identical compressed data for Reproducible Builds.
Having identical uncompressed data gets you pretty close to the goals of RB, but
unpacking and/or skipping over signatures is very very hard to get right and
simply cannot provide the same guarantees as having two bitwise identical files.
And it's impossible to create an APK you can actually install if it's not
bit-by-bit identical as the signature would not be valid otherwise. So yes,
unfortunately an identical ZIP file is a requirement and comparing the
decompressed stream not an option, which is why this kind of change is not
something we can just consider an implementation detail or work around.
I wrote more about the very messy situation Fedora's switch to zlib-ng already
created for Android Reproducible Builds [1]. Which likely would have broken a
lot more reproducible Android apps already if Fedora's OpenJDK packages linked
against the system zlib like Debian's OpenJDK packages do (instead of using an
embedded copy of regular zlib).
- Fay
[1] https://lists.reproducible-builds.org/pipermail/rb-general/2024-September/003547.html

[Mark Brown]

Post by Guillem Jover

Post by Mark Brown
Obviously it's far too late to do anything with the default for trixie,
we might want to evaluate doing something after the release but for now
it's too late.

Personally I don't think it's too late, there should be several months
until the freeze, and I think if we wanted to switch we could perhaps
do a staged transition and see how it goes and only do the final
replacement if everything seems fine.

We do OTOH package more software than most distros on more architectures
so we got a lot more exposure for testing coverage, and the revert would
involve switching the entire implementation which complicates things a
bit compared to a risky patch within a package. I'm not totally
opposed, and if everything goes smoothly we could definitely implement
it within the timeframe, but it feels like an impactful change to
introduce now not having considered it sooner.

Post by Guillem Jover
(perhaps) exceeding changed circumstances. But given your mail, I'm
happy to work on this again and start with say uploading some initial
stuff into experimental for example, after this thread settles a bit?
(I'll start by refreshing the packaging first though.)

Sure.

Post by Guillem Jover

Post by Mark Brown
Does anyone have thoughts on this?

Personally, I think fully migrating from zlib to zlib-ng would sound
great (even for trixie), but I guess we can take it slow if you do not
feel confident or have concerns over this.
Also if you'd prefer to take over the zlib-ng ITP, as a continuation
of zlib, that'd seem fine with me too.

I'm fine with you carrying on with it (actually there is some slight
non-technical complication for me with doing it myself), or we could
also consider a packaging team. I think there was some other interest
in helping out but ICBW. If you're packaging it I'm also more confident
in letting you worry about how risky it is to transition and deal with
any fallout! :P

[Guillem Jover]

Hi!

[ I'll try to summarize the current discussion and status, what might
be blockers, and a potential incremental way forward. ]

Post by Mark Brown

Post by Guillem Jover

Post by Mark Brown
Obviously it's far too late to do anything with the default for trixie,
we might want to evaluate doing something after the release but for now
it's too late.

Personally I don't think it's too late, there should be several months
until the freeze, and I think if we wanted to switch we could perhaps
do a staged transition and see how it goes and only do the final
replacement if everything seems fine.

We do OTOH package more software than most distros on more architectures
so we got a lot more exposure for testing coverage, and the revert would
involve switching the entire implementation which complicates things a
bit compared to a risky patch within a package. I'm not totally
opposed, and if everything goes smoothly we could definitely implement
it within the timeframe, but it feels like an impactful change to
introduce now not having considered it sooner.

True, also two months have passed since (that's on me!). At this time,
I'm now not sure whether it is feasible to consider such a switch, even
if there was agreement to do it. As it is, I think there are too many
unknowns!

Post by Mark Brown

Post by Guillem Jover
(perhaps) exceeding changed circumstances. But given your mail, I'm
happy to work on this again and start with say uploading some initial
stuff into experimental for example, after this thread settles a bit?
(I'll start by refreshing the packaging first though.)

Sure.

I did that, and the current WIP zlib-ng packaging provides now two
builds, one with the new native zng_* API and another (tentatively)
with the compat API/ABI one in libz-dev and libz1 binary packages.

I've tentatively chosen those package names for the compat libraries
to avoid having to go through NEW multiple times (with the assumption
that we'd either go ahead with the switch or the packages could then
simply be dropped). I think this should initially only be uploaded to
experimental, to avoid getting packages built with either zlib or
zlib-ng. But depending on the outcome of this discussion, I think other
(probably better) options would be to perhaps name the compat packages
something like libz-ng-compat*, or drop them completely?

WIP package at <https://git.hadrons.org/cgit/wip/debian/pkgs/zlib-ng.git/>.

Post by Mark Brown

Post by Guillem Jover

Post by Mark Brown
Does anyone have thoughts on this?

Personally, I think fully migrating from zlib to zlib-ng would sound
great (even for trixie), but I guess we can take it slow if you do not
feel confident or have concerns over this.
Also if you'd prefer to take over the zlib-ng ITP, as a continuation
of zlib, that'd seem fine with me too.

I'm fine with you carrying on with it (actually there is some slight
non-technical complication for me with doing it myself), or we could
also consider a packaging team. I think there was some other interest
in helping out but ICBW. If you're packaging it I'm also more confident
in letting you worry about how risky it is to transition and deal with
any fallout! :P

Ok, so after the feedback on this thread, and Sebastian asking how we
can proceed, here are the concerns brought on this thread, along my
own and things I think we need to check or consider:

* There were concerns (from Fay) about whether given same input the
output changes per arch or hw setup, we'd need to check this; I'd
expect this not to be the case for different arches, but it might
be an issue with number of cores for example, but if either is true
this would be a serious blocker.
* I've had concerns both about providing the zlib compat API and the
native zlib-ng API in sid, and then getting a mess of packages
linking against (true) zlib and against (native) zlib-ng, or
packages relying on specific behaviors from either and breaking
when switching from (true) zlib to zlib-ng-compat or vice versa,
for example.
* There were concerns (from Fay) about the output stream changing due
to a potential implementation switch and that affecting external
reproducibility. Personally I think while I can see how this is
annoying for the involved parties, it's part of the "you need
the same tools to generate the same output" premise that we also
assume in Debian. I guess keeping both implementations around
indefinitely, I think, would make this less of an issue, with the
potential drawbacks mentioned in the previous point.
* There was a concern (from Konstantin) about at least one known
upstream (Angie) misbehaving with zlib-ng generated streams.
* There were concerns (from Mark) that even though projects like
Fedora have done such switch, we have way more packages and
architectures, so we might see more fallout that has not already
been handled.
* There was a concern (from Mike) about whether the performance gain
at the cost of stream size makes sense, given that the compression
level could be reduced instead to similar effect (?). I'm not sure
how these compare, so it would be interesting to analyze this,
because perhaps that's a less traumatic way to look at it (but that
might require redefining compression level semantics globally in
zlib, or patching users, with neither look very enticing options).
My perception from when I tested it is that the speed up was
significant enough and the size increase not so much, but… In any
case switching to zlib-ng upstream would also imply other benefits,
like (supposedly) a more responsive upstream with more frequent
releases, the new native API, and an implementation other
distributions are switching to.
* Some upstreams have started to use the zlib-ng native API, so
regardless of whether we plan a switch or not, I guess packaging
zlib-ng (w/ or w/ the compat API) might still make sense.
* To consider a switch we'd need to do a mass rebuild of the
archive. Ideally running autopkgtests and similar to exercise the
packages?


After having written the above, and if Mark agrees, I think I'd opt for
uploading zlib-ng to experimental, with the compat packages renamed to
libz-ng-compat* or similar (even if that implies later on another trip
through NEW if we want to perform a full switch), because that might
make it easier to move them to sid as a way less disruptive change,
even if we decide not to switch the default zlib implementation.

OTOH and unfortunately I don't think I'm currently prepared to drive any
of what I think might be required mass archive rebuilds and testing or
the analysis mentioned above.

Thanks,
Guillem

[Simon McVittie]

Post by Guillem Jover
depending on the outcome of this discussion, I think other
(probably better) options would be to perhaps name the compat packages
something like libz-ng-compat*, or drop them completely?

You might find the history of the sdl12-compat source package interesting.
It's a reimplementation of the SDL 1.2 API/ABI in terms of SDL 2, and it
went through several stages:

- only SDL 2 available (as of bullseye);
- sdl12-compat available as non-default and outside the normal library
search path, but can take over from "classic" SDL 1.2 if explicitly
requested via LD_LIBRARY_PATH or by installing a extra "shim" package
(this state is what we shipped in bookworm);
- sdl12-compat takes over binary packages from "classic" SDL 1.2,
"classic" SDL 1.2 disappears, sdl12-compat's old binary packages become
transitional (this is the current state in trixie);
- in forky, we'll presumably drop the transitional packages

Having the second step made it a lot easier to check for regressions.

Switching from glibc's libcrypt to the external libxcrypt, and from IJG
libjpeg to libjpeg-turbo, both went through similar processes.

sdl12-compat is a bit different because it's *only* the
backwards-compatible API/ABI, and not the "native" API/ABI (but you could
think of libsdl2, in its own source package, as being the corresponding
"native" API/ABI).

libxcrypt has a "native" legacy-free API/ABI available, but in Debian we don't
seem to be compiling it at the moment, and we only have the glibc- and
musl-compatible backwards-compatible ABIs (but I think some other distros
like Arch have moved to the legacy-free ABI).

libjpeg-turbo has a "native" legacy-free API/ABI libturbojpeg* in parallel
with a backwards-compatible API/ABI resembling IJG libjpeg 6b, so in that
sense it seems the closest to what you're proposing for zlib-ng.

Post by Guillem Jover
* I've had concerns [...] about providing the zlib compat API and the
native zlib-ng API in sid, and then getting a mess of packages
linking against (true) zlib and against (native) zlib-ng

Is there any problem with that, other than duplicate functionality?

Post by Guillem Jover
From the other things you've said about the native API, it sound as

though the symbols all have different names, so this is no worse than
the duplication you would get if you have indirect dependencies on both
zlib and libdeflate, or on both OpenSSL and GNUTLS; it isn't like trying
to link to both libreadline and libedit (which export essentially the
same symbols and so will collide badly), or more than one of GTK 2/3/4
(likewise).

smcv

[Bill Allombert]

Switching .... from IJG
libjpeg to libjpeg-turbo, both went through similar processes.

Independently of the merit of your proposal, this is not historically
correct.

Cheers,

[Sebastian Andrzej Siewior]

Post by Guillem Jover
Hi!

Hi,

…

Post by Guillem Jover
WIP package at <https://git.hadrons.org/cgit/wip/debian/pkgs/zlib-ng.git/>.

just built that.
…

Post by Guillem Jover
* To consider a switch we'd need to do a mass rebuild of the
archive. Ideally running autopkgtests and similar to exercise the
packages?

So you upload it to experimental then autopkgtests should pick it up and
run against unstable. You should the results in
https://release.debian.org/britney/pseudo-excuses-experimental.html

Post by Guillem Jover
After having written the above, and if Mark agrees, I think I'd opt for
uploading zlib-ng to experimental, with the compat packages renamed to
libz-ng-compat* or similar (even if that implies later on another trip
through NEW if we want to perform a full switch), because that might
make it easier to move them to sid as a way less disruptive change,
even if we decide not to switch the default zlib implementation.
OTOH and unfortunately I don't think I'm currently prepared to drive any
of what I think might be required mass archive rebuilds and testing or
the analysis mentioned above.

I just kicked a rebuild locally with libz1 in the chroot. The proper
-dev gets installed if asked so that should work. Dose computed 38659
packages so this could a while.

Post by Guillem Jover
Thanks,
Guillem

Sebastian

[Sebastian Andrzej Siewior]

Post by Sebastian Andrzej Siewior

Post by Guillem Jover
Hi!

Hi,

Hi,

Post by Sebastian Andrzej Siewior

Post by Guillem Jover
WIP package at <https://git.hadrons.org/cgit/wip/debian/pkgs/zlib-ng.git/>.

just built that.

…

One thing I didn't debug and it might be expected: If I install
libz1_2.2.2-1_amd64.deb and then try to install build dependencies of
0ad via
apt build-dep 0ad

then it removes libz1 while doing so. sbuild aborts complains and
aborts. I worked around it by installing the lib + header files in the
chroot before building.

Post by Sebastian Andrzej Siewior
I just kicked a rebuild locally with libz1 in the chroot. The proper
-dev gets installed if asked so that should work. Dose computed 38659
packages so this could a while.

ongoing… 59% left. In the meantime I uploaded what looks like possible
fallout to
https://people.debian.org/~bigeasy/zlib-ng/todo/

I moved cpp-httplib to done after filling #1088193 and
https://udd.debian.org/cgi-bin/bts-usertags.cgi?user=zlib-***@tracker.debian.org&tag=zlib-ng-ftbfs

lists it, too.

Post by Sebastian Andrzej Siewior

Post by Guillem Jover
Thanks,
Guillem

Sebastian

[Sebastian Andrzej Siewior]

On 2024-11-24 21:36:25 [+0100], To debian-***@lists.debian.org wrote:
…

I've been looking at cdebootstrap. It is one of the failing. This
happens in the rules files:
| ( echo -n "misc:Built-Using="; dpkg-query -f='${source:Package} (=
| ${source:Version}), ' -W libc6-dev libdebian-installer4-dev
| libdebconfclient0-dev libbz2-dev liblzma-dev zlib1g-dev; echo ) >>
| debian/cdebootstrap-static.substvars

and then
| dpkg-gencontrol: warning: Depends field of package cdebootstrap-static: substitution variable ${shlibs:Depends} used, but is not defined
| dpkg-gencontrol: warning: can't parse dependency zlib1g-dev (= )
| dpkg-gencontrol: error: parsing package 'cdebootstrap-static' Built-Using field: bzip2 (= 1.0.8-6), glibc (= 2.40-4), cdebconf (= 0.273), libdebian-installer (= 0.125), xz-utils (= 5.6.3-1), zlib1g-dev (= ),

Since we don't have zlib1g-dev anymore, this goes boom. If we exclude
zlib1g-dev then it is not recorded. Can't we smart about this?

Sebastian

[Sebastian Andrzej Siewior]

Post by Sebastian Andrzej Siewior
…
I've been looking at cdebootstrap. It is one of the failing. This
| ( echo -n "misc:Built-Using="; dpkg-query -f='${source:Package} (=
| ${source:Version}), ' -W libc6-dev libdebian-installer4-dev
| libdebconfclient0-dev libbz2-dev liblzma-dev zlib1g-dev; echo ) >>
| debian/cdebootstrap-static.substvars
and then
| dpkg-gencontrol: warning: Depends field of package cdebootstrap-static: substitution variable ${shlibs:Depends} used, but is not defined
| dpkg-gencontrol: warning: can't parse dependency zlib1g-dev (= )
| dpkg-gencontrol: error: parsing package 'cdebootstrap-static' Built-Using field: bzip2 (= 1.0.8-6), glibc (= 2.40-4), cdebconf (= 0.273), libdebian-installer (= 0.125), xz-utils (= 5.6.3-1), zlib1g-dev (= ),
Since we don't have zlib1g-dev anymore, this goes boom. If we exclude
zlib1g-dev then it is not recorded. Can't we smart about this?

Rebuild done. Rebuild all packages as suggested by ratt for zlib.

Total : 38655
Passed : 37188
Failed with zlib-ng: 64
Failed always : 1401

The total amount of build logs is ~20GiB. I uploaded compressed logs of
the which failed with zlib-ng and overview to
https://people.debian.org/~bigeasy/zlib-ng/

Three packages failed due to the way Build-Using is generated. I moved
those to another folder (cdebootstrap, dar, sash).

Sebastian

[Fay Stegerman]

* Guillem Jover <***@debian.org> [2024-11-22 12:29]:
[...]

Post by Guillem Jover
* There were concerns (from Fay) about the output stream changing due
to a potential implementation switch and that affecting external
reproducibility. Personally I think while I can see how this is
annoying for the involved parties, it's part of the "you need
the same tools to generate the same output" premise that we also
assume in Debian. I guess keeping both implementations around
indefinitely, I think, would make this less of an issue, with the
potential drawbacks mentioned in the previous point.

[...]

As I reported to the Reproducible Builds mailing list in September, Fedora
switching to zlib-ng has created a very messy situation [1] for reproducibility
of Android APKs, which would have been far worse if not for the fact that
Fedora's own OpenJDK packages use a bundled zlib instead of linking against the
system zlib-ng.

Unfortunately, Reproducible Builds for Android APKs involve reproducing APKs
built by individual upstreams, not Debian buildds. Most upstreams don't use
Debian, many even use Windows. It's simply not feasible to use the exact same
build environment as upstream in a lot of cases, nor can we expect them to
switch to building on Debian to match the rebuilders.

So far, we have still been able get identical results 99% of the time because
the Android toolchain generates identical results on different platforms -- with
some exceptions, like newlines, that we have created workarounds for, though
those workarounds rely on being able to recreate a bitwise identical zlib output
stream.

I agree that it *should not* be Debian's responsibility to ensure compatibility
with Fedora/Windows/etc., but the reality is that if "you need the same tools to
generate the same output" -- which right now means using the same JDK and
Android toolchain but in 99% of cases doesn't require using the same OS since
everyone, including Google [2], standardised on zlib -- becomes "you cannot
reproduce APKs built on a OS other than Debian on Debian", that's not just
"annoying for the involved parties": it will effectively break the ability to
verify reproducibility of many Android apps.

Having both zlib implementations available and being able to choose between them
at runtime (perhaps using LD_LIBRARY_PATH in the tooling when needed) would help
a lot to at least allow creating workarounds, but that would still require
significant changes that someone needs to implement in order to unbreak
something that currently works fine.

- Fay

[1] https://lists.reproducible-builds.org/pipermail/rb-general/2024-September/003547.html
[2] https://android-developers.googleblog.com/2016/12/saving-data-reducing-the-size-of-app-updates-by-65-percent.html

[Chris Hofstaedtler]

Post by Fay Stegerman
I agree that it *should not* be Debian's responsibility to ensure compatibility
with Fedora/Windows/etc., but the reality is that if "you need the same tools to
generate the same output" -- which right now means using the same JDK and
Android toolchain but in 99% of cases doesn't require using the same OS since
everyone, including Google [2], standardised on zlib -- becomes "you cannot
reproduce APKs built on a OS other than Debian on Debian", that's not just
"annoying for the involved parties": it will effectively break the ability to
verify reproducibility of many Android apps.

Sorry, but what exactly are you saying here? That Debian should be
bound by the decision of a BigTech Corporate and by thousands of
individual Android developers, neither of which might be interested
in Debian?

To maybe make the argument the other way 'round: if Google switches
to zlib-ng tomorrow, should Debian be required to switch to zlib-ng?

I just don't see that's how it works.

Chris

[Fay Stegerman]

Post by Chris Hofstaedtler

Post by Fay Stegerman
I agree that it *should not* be Debian's responsibility to ensure compatibility
with Fedora/Windows/etc., but the reality is that if "you need the same tools to
generate the same output" -- which right now means using the same JDK and
Android toolchain but in 99% of cases doesn't require using the same OS since
everyone, including Google [2], standardised on zlib -- becomes "you cannot
reproduce APKs built on a OS other than Debian on Debian", that's not just
"annoying for the involved parties": it will effectively break the ability to
verify reproducibility of many Android apps.

Sorry, but what exactly are you saying here? That Debian should be
bound by the decision of a BigTech Corporate and by thousands of
individual Android developers, neither of which might be interested
in Debian?
To maybe make the argument the other way 'round: if Google switches
to zlib-ng tomorrow, should Debian be required to switch to zlib-ng?

What I'm saying is that this change will have consequences for downstreams using
Debian for Reproducible Builds. That includes e.g. hundreds of F-Droid apps,
which would no longer be able to get updates if Reproducible Builds break.

That's clearly not Debian's fault. And of course Debian isn't *required* to do
anything. I agree Debian should not be bound by the decisions of a "BigTech
Corporate" and thousands of individual Android developers. And I very much
dislike the fact that this matters.

But Debian's choices here do have consequences for downstreams, and I think
that's something we should take into account when reasonably possible. Similar
to how we didn't switch i386 to t64 to avoid breaking running existing legacy
binaries.

For example, if it can be made easy to install both and choose between zlib and
zlib-ng at runtime, so it's easy to build APKs using either zlib or zlib-ng as
needed, downstream breakage can be avoided. Considering whether that can
reasonably be done doesn't seem like an unreasonable request to me.

What I would like is to be able to continue to use Debian for Reproducible
Builds regardless of what Google does or doesn't do. Right now that means being
able to choose to keep using the original zlib for backwards compatibility. If
Google switched to zlib-ng I would be asking if Debian could provide a way to
opt in to using that instead.

- Fay

[Mark Brown]

Post by Fay Stegerman
For example, if it can be made easy to install both and choose between zlib and
zlib-ng at runtime, so it's easy to build APKs using either zlib or zlib-ng as
needed, downstream breakage can be avoided. Considering whether that can
reasonably be done doesn't seem like an unreasonable request to me.
What I would like is to be able to continue to use Debian for Reproducible
Builds regardless of what Google does or doesn't do. Right now that means being
able to choose to keep using the original zlib for backwards compatibility. If
Google switched to zlib-ng I would be asking if Debian could provide a way to
opt in to using that instead.

TBH the whole situation here seems incredibly fragile - you're also
vulnerable to someone making an improvement in the zlib compression
code and changing the generated output that way. It feels like the
wrong question is being asked here.

[Mark Brown]

Post by Guillem Jover
[ I'll try to summarize the current discussion and status, what might
be blockers, and a potential incremental way forward. ]

You CCed some people but not me so I only saw this today...

Post by Guillem Jover

Post by Mark Brown
We do OTOH package more software than most distros on more architectures
so we got a lot more exposure for testing coverage, and the revert would
involve switching the entire implementation which complicates things a
bit compared to a risky patch within a package. I'm not totally
opposed, and if everything goes smoothly we could definitely implement
it within the timeframe, but it feels like an impactful change to
introduce now not having considered it sooner.

True, also two months have passed since (that's on me!). At this time,
I'm now not sure whether it is feasible to consider such a switch, even
if there was agreement to do it. As it is, I think there are too many
unknowns!

It does seem more safe to offer zlib-ng as an alternative at this
point...

Post by Guillem Jover
I did that, and the current WIP zlib-ng packaging provides now two
builds, one with the new native zng_* API and another (tentatively)
with the compat API/ABI one in libz-dev and libz1 binary packages.
I've tentatively chosen those package names for the compat libraries
to avoid having to go through NEW multiple times (with the assumption
that we'd either go ahead with the switch or the packages could then
simply be dropped). I think this should initially only be uploaded to
experimental, to avoid getting packages built with either zlib or
zlib-ng. But depending on the outcome of this discussion, I think other
(probably better) options would be to perhaps name the compat packages
something like libz-ng-compat*, or drop them completely?

The packages could have a different name and support diversion or
replacement of the zlib library packages? That'd let people use them
if they wanted to.

Post by Guillem Jover
* I've had concerns both about providing the zlib compat API and the
native zlib-ng API in sid, and then getting a mess of packages
linking against (true) zlib and against (native) zlib-ng, or
packages relying on specific behaviors from either and breaking
when switching from (true) zlib to zlib-ng-compat or vice versa,
for example.

What are those concerns? Like Simon says the zlib and zlib-ng APIs are
just two separate APIs, they happen to have overlapping functionality
but while that might be a bit wasteful it's not obvious that there's any
actual problem.

Post by Guillem Jover
* There were concerns (from Fay) about the output stream changing due
to a potential implementation switch and that affecting external
reproducibility. Personally I think while I can see how this is
annoying for the involved parties, it's part of the "you need
the same tools to generate the same output" premise that we also
assume in Debian. I guess keeping both implementations around
indefinitely, I think, would make this less of an issue, with the
potential drawbacks mentioned in the previous point.

Yes, I don't think this should be a blocker - this seems like it's on a
similar level to needing to use the same compilers.

Post by Guillem Jover
* There was a concern (from Mike) about whether the performance gain
at the cost of stream size makes sense, given that the compression
level could be reduced instead to similar effect (?). I'm not sure
how these compare, so it would be interesting to analyze this,
because perhaps that's a less traumatic way to look at it (but that
might require redefining compression level semantics globally in
zlib, or patching users, with neither look very enticing options).
My perception from when I tested it is that the speed up was
significant enough and the size increase not so much, but
 In any
case switching to zlib-ng upstream would also imply other benefits,
like (supposedly) a more responsive upstream with more frequent
releases, the new native API, and an implementation other
distributions are switching to.
* Some upstreams have started to use the zlib-ng native API, so
regardless of whether we plan a switch or not, I guess packaging
zlib-ng (w/ or w/ the compat API) might still make sense.

Yes, it seems clear that we should package zlib-ng and there's just a
question about handling of the compat interface it provides.

Post by Guillem Jover
After having written the above, and if Mark agrees, I think I'd opt for
uploading zlib-ng to experimental, with the compat packages renamed to
libz-ng-compat* or similar (even if that implies later on another trip
through NEW if we want to perform a full switch), because that might
make it easier to move them to sid as a way less disruptive change,
even if we decide not to switch the default zlib implementation.

Yes, that sounds sensible to me.

[Fay Stegerman]

* Guillem Jover <***@debian.org> [2024-11-22 12:29]:
[...]

Post by Guillem Jover
* There were concerns (from Fay) about whether given same input the
output changes per arch or hw setup, we'd need to check this; I'd
expect this not to be the case for different arches, but it might
be an issue with number of cores for example, but if either is true
this would be a serious blocker.
* There were concerns (from Fay) about the output stream changing due
to a potential implementation switch and that affecting external
reproducibility. Personally I think while I can see how this is
annoying for the involved parties, it's part of the "you need
the same tools to generate the same output" premise that we also
assume in Debian. I guess keeping both implementations around
indefinitely, I think, would make this less of an issue, with the
potential drawbacks mentioned in the previous point.

[...]

I did some more testing with zlib-ng. With the original zlib, you will always
get an identical output stream given the same input stream and compressor
parameters (compression level being the only one that's commonly varied in ZIP
files). I expected that zlib-ng would often produce a different output steam
than the original, but what I found was a lot more non-deterministic than just
that.

With zlib-ng, feeding the data into the compressor in e.g. 1024-byte chunks
always gave me a different output stream than using 4096-byte chunks (at
compression level 6). In fact, every chunk size I tried gave a different
output. And that's with fixed size chunks, which is not a given if you're
handling e.g. a stream of input.

Even using the same buffer size, I cannot get an identical compressed output
stream with Python and Java any more, presumably because of subtle
implementation differences (in the stdlib code that ends up calling zlib to do
the compression) that do not affect zlib but clearly do affect zlib-ng.

Which makes zlib-ng unsuitable for use cases where you need to be able to create
an identical output stream without knowing exactly how the bytes were fed into
the zlib compressor (or simply have no way to control this). This fundamentally
breaks my tooling in ways I can't fix by using the same build environment.
Because programs that used to produce identical and deterministic output with
zlib no longer do with zlib-ng, despite using the exact same zlib-ng .so.

- Fay

[Charles Plessy]

Post by Mark Brown
zlib-ng: https://github.com/zlib-ng/zlib-ng

Hi Mark, just out of curiosity, would the carbon footprint of Debian be
lower or higher after replacing zlib with zlib-ng?

Have a nice day,

Charles

[Mark Brown]

Post by Charles Plessy

Post by Mark Brown
zlib-ng: https://github.com/zlib-ng/zlib-ng

Hi Mark, just out of curiosity, would the carbon footprint of Debian be
lower or higher after replacing zlib with zlib-ng?

You could probably calculate it either way depending on how you want to
make up the numbers; running faster will take less time but larger
outputs might take more storage.