Moving apt (and hence bootstraps) from GnuPG to Sequioa (via gpgv-sq)

[Julian Andres Klode]

I've just finished more or less, adjusting the APT test suite
to test gpgv-sq. I plan to upload APT that tests gpgv-sq
tomorrow. This ensures full compatibility between apt and
gpgv-sq going forward.

After that migrates to testing next week, I want to make
the switch: APT by default should use gpgv-sq. Previous
discussions with the security team did not reveal any
blockers for that, despite the strenuous nature of
security updates for Rust packages.

My plan here is to use

Depends: gpgv-from-sq | gpgv-sq | gpgv
Recommends: gpgv-sq

To do 2 things:

1) The Depends will install gpgv-from-sq on new systems. The
gpgv-from-sq package diverts /usr/bin/gpgv to gpgv-g10code
and Provides gpgv, installing a symlink to gpgv-sq.

This ensures that while switching to gpgv-sq, the default
bootstrap does not change in exposed functionality: There
is a /usr/bin/gpgv available.

This does not affect existing systems, as the dependency
is already satisfied by the gpgv alternative.

An optimal mechanism would instea

2) The Recommends ensures that the signature verification
as used by APT is changed, as APT also will prefer gpgv-sq
over gpgv if installed.

This means existing systems will have two gpgv implementations:

- /usr/bin/gpgv from GnuPG
- /usr/bin/gpgv-sq from Sequioa

This is not necessarily the best outcome. We could change
it to gpgv-from-sq as well, such that systems remain with
a single /usr/bin/gpgv. I don't particularly like the idea
of forcing a software change on users, but in the sense of
two systems behaving the same way that of course would be
preferable.

As for ports without gpgv-sq, this does not affect them,
they can be served by the gpgv alternative. Once a gpgv-sq
is available, it's important to note that no migration happens
for them: APT does not install Recommends that were previously
unsatisfied.

[Gioele Barabucci]

Post by Julian Andres Klode
1) The Depends will install gpgv-from-sq on new systems. The
gpgv-from-sq package diverts /usr/bin/gpgv to gpgv-g10code
and Provides gpgv, installing a symlink to gpgv-sq.

Unrelated to the apt proposal, how come dpkg-divert is being used here
by gpgv-from-sq instead of using the what-is-python/python-is-pythonX
approach of just shipping a symlink?

The python-is-pythonX approach is much simpler, fully declarative (as
in: the future state of the system is statically described by the
content of the package, not at runtime), and removes the need for the
maintscripts to know about the other implementations.

Regards,

[Simon Richter]

Hi,

Post by Gioele Barabucci
Unrelated to the apt proposal, how come dpkg-divert is being used here
by gpgv-from-sq instead of using the what-is-python/python-is-pythonX
approach of just shipping a symlink?

Because there is no coordination between gpgv and gpgv-sq packages, and
dependent packages should have no reason to care. gpgv-sq unilaterally
claims compatibility, and if something breaks as a result, that is a bug
in gpgv-sq, because the interface of gpgv is defined by the g10code
implementation.

Post by Gioele Barabucci
The python-is-pythonX approach is much simpler, fully declarative (as
in: the future state of the system is statically described by the
content of the package, not at runtime), and removes the need for the
maintscripts to know about the other implementations.

The python-is-pythonX approach is for *incompatible* changes, where
depending packages have a need to specify which version they expect to
use, and introducing indirect conflicts is acceptable because it cannot
be avoided.

Simon

[Holger Levsen]

Post by Simon Richter
Because there is no coordination between gpgv and gpgv-sq packages,

that's not entirely true.

Post by Simon Richter
and
dependent packages should have no reason to care. gpgv-sq unilaterally
claims compatibility, and if something breaks as a result, that is a bug in
gpgv-sq, because the interface of gpgv is defined by the g10code
implementation.

this is true.
--
cheers,
Holger

⢀⣎⠟⠻⢶⣊⠀
⣟⠁⢠⠒⠀⣿⡁ holger@(debian|reproducible-builds|layer-acht).org
⢿⡄⠘⠷⠚⠋⠀ OpenPGP: B8BF54137B09D35CF026FE9D 091AB856069AAA1C
⠈⠳⣄

It's not climate change nor climate crisis, it's climate disaster.

[Iustin Pop]

Post by Julian Andres Klode
I've just finished more or less, adjusting the APT test suite
to test gpgv-sq. I plan to upload APT that tests gpgv-sq
tomorrow. This ensures full compatibility between apt and
gpgv-sq going forward.

Excuse my ignorance, but what is gpgv-sq/Sequoia? I just saw very
recently the package name (like, two days ago), and while the
description says "it is an alternative implementation", that doesn't say
what's the context here.

I'm not happy with the heavyness that one gets via gpg just for
verifying signatures, so I'd be all for a lighter-weight solution.

I found https://lwn.net/Articles/830902/ which explains what Sequoia is,
but in Debian, is there a bigger plan for GnuPG -> Sequoia, or just
piecemeal adoption?

thanks!
iustin

[Antonio Russo]

Post by Iustin Pop
I'm not happy with the heavyness that one gets via gpg just for
verifying signatures, so I'd be all for a lighter-weight solution.

gpgv is lighter weight than gpgv-sq. Surprisingly, it's not just
the rustc-static-links-everything problem, since gpgv-static is
also smaller than gpgv-sq.

Antonio

[Julian Andres Klode]

Post by Antonio Russo

Post by Iustin Pop
I'm not happy with the heavyness that one gets via gpg just for
verifying signatures, so I'd be all for a lighter-weight solution.

gpgv is lighter weight than gpgv-sq. Surprisingly, it's not just
the rustc-static-links-everything problem, since gpgv-static is
also smaller than gpgv-sq.

That's correct due to some overlinking in the Rust toolchain,
there needs to be some more crate splitting to make it smaller.

[Marco d'Itri]

Post by Julian Andres Klode
That's correct due to some overlinking in the Rust toolchain,
there needs to be some more crate splitting to make it smaller.

Do you have reasons to believe that this is going to happen in time for
the next release?
The size increase is not trivial. :-(

[Marco d'Itri]

Post by Julian Andres Klode
I've just finished more or less, adjusting the APT test suite
to test gpgv-sq. I plan to upload APT that tests gpgv-sq
tomorrow. This ensures full compatibility between apt and
gpgv-sq going forward.

OK, but why?

Did you make an analysis of how much the size of a minimal system would
change?

[Julian Andres Klode]

Post by Marco d'Itri

Post by Julian Andres Klode
I've just finished more or less, adjusting the APT test suite
to test gpgv-sq. I plan to upload APT that tests gpgv-sq
tomorrow. This ensures full compatibility between apt and
gpgv-sq going forward.

OK, but why?
Did you make an analysis of how much the size of a minimal system would
change?

We currently see a size increase of 8% (9MB uncompressed, 4MB gzipped) in an
essential + apt bootstrap:

$ mmdebstrap --variant=essential --include=apt unstable unstable.tar
$ mmdebstrap --variant=essential --include=gpgv-from-sq,apt,gpgv- unstable unstable-with-sq.tar
$ $ ls -lh unstable*.tar
-rw-r--r-- 1 jak jak 114M Nov 22 13:39 unstable.tar
-rw-r--r-- 1 jak jak 123M Nov 22 13:39 unstable-with-sq.tar
$ gzip unstable*.tar
$ ls -lh unstable*.tar*
-rw-r--r-- 1 jak jak 46M Nov 22 13:39 unstable.tar.gz
-rw-r--r-- 1 jak jak 50M Nov 22 13:39 unstable-with-sq.tar.gz
$ diff <(tar xOf unstable.tar.gz ./var/lib/dpkg/status | grep ^Package\\\|Installed-Size) <(tar xOf unstable-with-sq.tar.gz ./var/lib/dpkg/status | grep ^Package\\\|Installed-Size) -U0
diff --git dev/fd/63 dev/fd/62
--- dev/fd/63
+++ dev/fd/62
@@ -29,2 +29,4 @@ Installed-Size: 109
-Package: gpgv
-Installed-Size: 509
+Package: gpgv-from-sq
+Installed-Size: 14
+Package: gpgv-sq
+Installed-Size: 8167
@@ -110,0 +113,2 @@ Installed-Size: 368
+Package: libsqlite3-0
+Installed-Size: 1833

There are a bunch of toolchain/crate-splitting issues here: Rust
overlinks, neither do we use all 8MB of the code that's linked
into gpgv-sq, nor does it actually need libsqlite3-0. This happens
because gpgv-sq uses gpg-sq crate, if they were split up, things
should go down quite a bit is my understanding.

[Julian Andres Klode]

Post by Julian Andres Klode

Post by Marco d'Itri

Post by Julian Andres Klode
I've just finished more or less, adjusting the APT test suite
to test gpgv-sq. I plan to upload APT that tests gpgv-sq
tomorrow. This ensures full compatibility between apt and
gpgv-sq going forward.

OK, but why?
Did you make an analysis of how much the size of a minimal system would
change?

We currently see a size increase of 8% (9MB uncompressed, 4MB gzipped) in an
$ mmdebstrap --variant=essential --include=apt unstable unstable.tar
$ mmdebstrap --variant=essential --include=gpgv-from-sq,apt,gpgv- unstable unstable-with-sq.tar
$ $ ls -lh unstable*.tar
-rw-r--r-- 1 jak jak 114M Nov 22 13:39 unstable.tar
-rw-r--r-- 1 jak jak 123M Nov 22 13:39 unstable-with-sq.tar
$ gzip unstable*.tar
$ ls -lh unstable*.tar*
-rw-r--r-- 1 jak jak 46M Nov 22 13:39 unstable.tar.gz
-rw-r--r-- 1 jak jak 50M Nov 22 13:39 unstable-with-sq.tar.gz
$ diff <(tar xOf unstable.tar.gz ./var/lib/dpkg/status | grep ^Package\\\|Installed-Size) <(tar xOf unstable-with-sq.tar.gz ./var/lib/dpkg/status | grep ^Package\\\|Installed-Size) -U0
diff --git dev/fd/63 dev/fd/62
--- dev/fd/63
+++ dev/fd/62
@@ -29,2 +29,4 @@ Installed-Size: 109
-Package: gpgv
-Installed-Size: 509
+Package: gpgv-from-sq
+Installed-Size: 14
+Package: gpgv-sq
+Installed-Size: 8167
@@ -110,0 +113,2 @@ Installed-Size: 368
+Package: libsqlite3-0
+Installed-Size: 1833
There are a bunch of toolchain/crate-splitting issues here: Rust
overlinks, neither do we use all 8MB of the code that's linked
into gpgv-sq, nor does it actually need libsqlite3-0. This happens
because gpgv-sq uses gpg-sq crate, if they were split up, things
should go down quite a bit is my understanding.

I also have an sopv implementation in progress. The issues there
are quite funny:

- If verification fails we more or less can't tell you why. You'll
get

Verification with sopv failed with code 3:
No acceptable signatures found

- No tests

- Can't set a crypto policy on it, aka enforce hashes or algorithms,
need to rely on backend.

But aside from that, it's only 2 MB with sqopv as the implementation.

[Holger Levsen]

Post by Julian Andres Klode
We currently see a size increase of 8% (9MB uncompressed, 4MB gzipped) in an

^^ that's trixie, in sid they're a bit smaller now and will probably soon
shrink further:

***@debian-work:/tmp $ schroot -c trixie -- ls -lart /usr/bin/gpg-sq /usr/bin/gpgv-sq
-rwxr-xr-x 1 root root 8335016 Oct 23 11:06 /usr/bin/gpgv-sq
-rwxr-xr-x 1 root root 14995432 Oct 23 11:06 /usr/bin/gpg-sq
***@debian-work:/tmp $ schroot -c sid -- ls -lart /usr/bin/gpg-sq /usr/bin/gpgv-sq
-rwxr-xr-x 1 root root 6220512 Nov 25 19:29 /usr/bin/gpgv-sq
-rwxr-xr-x 1 root root 10785096 Nov 25 19:29 /usr/bin/gpg-sq
--
cheers,
Holger

⢀⣎⠟⠻⢶⣊⠀
⣟⠁⢠⠒⠀⣿⡁ holger@(debian|reproducible-builds|layer-acht).org
⢿⡄⠘⠷⠚⠋⠀ OpenPGP: B8BF54137B09D35CF026FE9D 091AB856069AAA1C
⠈⠳⣄

warum ist symbolischer Klimaschutz immer sowas wie „es bringt zwar nicht viel,
aber die Strohhalme sind jetzt aus Pappe“ und nie „es bringt zwar nicht viel,
aber Privatjets sind jetzt verboten“ (elhotzo)

[Andreas Metzler]

On 2024-11-21 Julian Andres Klode <***@debian.org> wrote:
[...]

Post by Julian Andres Klode
An optimal mechanism would instea

[...]

Something seems to be missing here.

cu Andreas

[Julian Andres Klode]

Post by Andreas Metzler
[...]

Post by Julian Andres Klode
An optimal mechanism would instea

[...]
Something seems to be missing here.
cu Andreas

Apologies, I started that thought and didn't quite finish it
and forgot it was still there when sending.

It's a bit surprising perhaps for users if they start with
gpgv-from-sq in the debootstrap, then later they manually
install gpgv, and /usr/bin/gpgv still points to gpgv-sq
due to the divert.

But on the other hand, we do need co-installability,
otherwise the test suite of apt can't test both :D

[Julian Andres Klode]

Post by Julian Andres Klode
I've just finished more or less, adjusting the APT test suite
to test gpgv-sq. I plan to upload APT that tests gpgv-sq
tomorrow. This ensures full compatibility between apt and
gpgv-sq going forward.
After that migrates to testing next week, I want to make
the switch: APT by default should use gpgv-sq. Previous
discussions with the security team did not reveal any
blockers for that, despite the strenuous nature of
security updates for Rust packages.

I have been informed I did not include the reasons and it's become
clear not everyone already knows the background here:

1. The GnuPG upstream forked the OpenPGP standard into his own
thing called LibrePGP, and GnuPG 2.4 implements that new thing
and is by default incompatible with other OpenPGP implementations.

2. GnuPG 2.4 is in experimental and patching out the LibrePGP
stuff is kind of necessary for it to be acceptable for release.

3. GnuPG 2.2 which is in unstable reaches its end of life in a couple
of weeks.

4. The GnuPG implementation quality has issues, such as silently
ignoring options not relevant to the current operation/mode,
producing no clear errors on expired signatures (they show
up as valid, just not as "good", but not as "bad" either),
and some features are very much unsafe, for example, the
new --assert-pubkey-algo feature accepts
<operator><name><size>
as the syntax, so it looks at >=ed448 and accepts ed25519
as being stronger because 25519 >= 448, whereas it is
the weaker curve.

Switching to gpgv-sq gets us out of this hole now while
we are waiting for the Stateless OpenPGP standard and
implementations of it to mature such that we can switch
to sqopv (| rsopv | sopv-gpgv | gosop).

Also it's written in a memory safe language which might make
the OpenPGP packet parsing safer :D

[Frank Guthausen]

Post by Julian Andres Klode
1. The GnuPG upstream forked the OpenPGP standard into his own
thing called LibrePGP, and GnuPG 2.4 implements that new thing
and is by default incompatible with other OpenPGP implementations.

Which kind of default incompatibility is implemented in GnuPG 2.4?

kind regards
Frank

[Sune Vuorela]

Post by Frank Guthausen

Post by Julian Andres Klode
1. The GnuPG upstream forked the OpenPGP standard into his own
thing called LibrePGP, and GnuPG 2.4 implements that new thing
and is by default incompatible with other OpenPGP implementations.

Which kind of default incompatibility is implemented in GnuPG 2.4?

GnuPG people withdrew from the OpenPGP standardization process because
of irreconcilable differences between parts of the working group, and
has gone a different way of standardizing what's being done, called
LibrePGP.

LWN did an article in december about it.

One of the biggest issues in the newest version of OpenPGP standard is,
according to GnuPG people, the need in the OpenPGP standard to have 3
diferent ways of doing AEAD.
One of them being quite more complex than the others while not as such
better except if your business model involves storing user's private
keys on your servers, which I consider a bit questionable.
The OpenPGP standard only has one of the 3 as required (the theoretical
best one), but the others are optional to implement.
OpenPGP.js defaults to the complex one that is optional.

LibrePGP supported by GnuPG and RNP (the pgp component of Thunderbird)
The new OpenPGP spec is supported by Seqouia and the libraries done by
ProtonMail.

/Sune

[Frank Guthausen]

On Mon, 25 Nov 2024 08:20:43 -0000 (UTC)

Post by Sune Vuorela

Post by Frank Guthausen
Which kind of default incompatibility is implemented in GnuPG 2.4?

[...]
LWN did an article in december about it.

Do you mean the schism article[1]? I'll take this
one as a starting point to dive into the matter.

[1] https://lwn.net/Articles/953797/

kind regards
Frank

[Sune Vuorela]

Post by Julian Andres Klode
As for ports without gpgv-sq, this does not affect them,
they can be served by the gpgv alternative. Once a gpgv-sq
is available, it's important to note that no migration happens

To me it looks like we will sligthly grow the base system and have
different kind of bugs per architecture depending on availability of
rust.

That doesn't sound great from a software testing and bug compatibility
perspective.

I'm not sure I think it is a great idea.

/Sune

[Julian Andres Klode]

Post by Sune Vuorela

Post by Julian Andres Klode
As for ports without gpgv-sq, this does not affect them,
they can be served by the gpgv alternative. Once a gpgv-sq
is available, it's important to note that no migration happens

To me it looks like we will sligthly grow the base system and have
different kind of bugs per architecture depending on availability of
rust.

All release architectures support Rust. We should not accept
release architectures without Rust support.

A minor set of ports architectures does not have Rust support
yet.

[Jonas Smedegaard]

Quoting Julian Andres Klode (2024-11-22 17:36:17)

Post by Julian Andres Klode

Post by Sune Vuorela

Post by Julian Andres Klode
As for ports without gpgv-sq, this does not affect them,
they can be served by the gpgv alternative. Once a gpgv-sq
is available, it's important to note that no migration happens

To me it looks like we will sligthly grow the base system and have
different kind of bugs per architecture depending on availability of
rust.

All release architectures support Rust. We should not accept
release architectures without Rust support.
A minor set of ports architectures does not have Rust support
yet.

Rust is unsupported on i386 and patched to silently assume i686 - see
DEP-3 references in this patch for discussions about that, and the patch
itself for a way to more loudly make reverse dependencies aware that
code using SSE2 *must* be compiled without optimizations on i386:
https://salsa.debian.org/debian/rust-wide/-/blob/debian/latest/debian/patches/2001_fail_non-sse2-x86.patch

Beware that Rust team build routines run tests without optimizations,
regardless of DEB_BUILD_OPTIONS=noopt, so for libraries maintained by
them the issue may go unnoticed until reverse dependencies run into the
issue *and* test for it, otherwise it might go unnoticed until users
report it.

- Jonas

[Chris Hofstaedtler]

Post by Jonas Smedegaard

Post by Julian Andres Klode
All release architectures support Rust. We should not accept
release architectures without Rust support.
A minor set of ports architectures does not have Rust support
yet.

Rust is unsupported on i386 and patched to silently assume i686

i686 is not a problem, as that's the arch baseline for our i386
arch since bookworm.

Post by Jonas Smedegaard
- see
DEP-3 references in this patch for discussions about that, and the patch
itself for a way to more loudly make reverse dependencies aware that
https://salsa.debian.org/debian/rust-wide/-/blob/debian/latest/debian/patches/2001_fail_non-sse2-x86.patch
Beware that Rust team build routines run tests without optimizations,
regardless of DEB_BUILD_OPTIONS=noopt, so for libraries maintained by
them the issue may go unnoticed until reverse dependencies run into the
issue *and* test for it, otherwise it might go unnoticed until users
report it.

So maybe it's time to raise the baseline to i686+sse2.

C.

[Jonas Smedegaard]

Quoting Chris Hofstaedtler (2024-11-23 04:16:29)

Post by Chris Hofstaedtler

Post by Jonas Smedegaard

Post by Julian Andres Klode
All release architectures support Rust. We should not accept
release architectures without Rust support.
A minor set of ports architectures does not have Rust support
yet.

Rust is unsupported on i386 and patched to silently assume i686

i686 is not a problem, as that's the arch baseline for our i386
arch since bookworm.

Post by Jonas Smedegaard
- see
DEP-3 references in this patch for discussions about that, and the patch
itself for a way to more loudly make reverse dependencies aware that
https://salsa.debian.org/debian/rust-wide/-/blob/debian/latest/debian/patches/2001_fail_non-sse2-x86.patch
Beware that Rust team build routines run tests without optimizations,
regardless of DEB_BUILD_OPTIONS=noopt, so for libraries maintained by
them the issue may go unnoticed until reverse dependencies run into the
issue *and* test for it, otherwise it might go unnoticed until users
report it.

So maybe it's time to raise the baseline to i686+sse2.

As I understand the situation with Rust, it is *not* that compiled code
fails to run on old non-SSE2 hardware. Instead, the problem is that the
Rust compiler produces code that is *ALWAYS* broken regardless if target
hardware supports SSE2 or not.

Yes, your final remark is a "solution" regardless, I just wanted to
emphasize that the problem affects the whole architecture, not only
outdated parts of it.

...unless I have misunderstand the situation, obciously.

- Jonas

[Fabian Grünbichler]

Post by Jonas Smedegaard
Quoting Chris Hofstaedtler (2024-11-23 04:16:29)

Post by Chris Hofstaedtler

Post by Jonas Smedegaard

Post by Julian Andres Klode
All release architectures support Rust. We should not accept
release architectures without Rust support.
A minor set of ports architectures does not have Rust support
yet.

Rust is unsupported on i386 and patched to silently assume i686

i686 is not a problem, as that's the arch baseline for our i386
arch since bookworm.

yeah, this is not related to that particular change, but to this:

https://wiki.debian.org/ArchitectureSpecificsMemo#i386

in combination with a bad interaction between rustc and LLVM..

Post by Jonas Smedegaard

Post by Chris Hofstaedtler

Post by Jonas Smedegaard
- see
DEP-3 references in this patch for discussions about that, and the patch
itself for a way to more loudly make reverse dependencies aware that
https://salsa.debian.org/debian/rust-wide/-/blob/debian/latest/debian/patches/2001_fail_non-sse2-x86.patch
Beware that Rust team build routines run tests without optimizations,
regardless of DEB_BUILD_OPTIONS=noopt, so for libraries maintained by
them the issue may go unnoticed until reverse dependencies run into the
issue *and* test for it, otherwise it might go unnoticed until users
report it.

So maybe it's time to raise the baseline to i686+sse2.

As I understand the situation with Rust, it is *not* that compiled code
fails to run on old non-SSE2 hardware. Instead, the problem is that the
Rust compiler produces code that is *ALWAYS* broken regardless if target
hardware supports SSE2 or not.
Yes, your final remark is a "solution" regardless, I just wanted to
emphasize that the problem affects the whole architecture, not only
outdated parts of it.
...unless I have misunderstand the situation, obciously.

AFAICT, this seems to be true. I was (so far) under the impression so far that this only affects FP arithmetics (i.e., the usual issue one runs into this is test code that does FP operations and then checks that the result is as expected). given that it's not limited to that (see https://github.com/rust-lang/rust/issues/114479#issuecomment-2072052116 and https://github.com/rust-lang/compiler-team/issues/808), the situation is a lot worse!

so yeah, I guess we should either

A) move i386 rustc to Rust's i586 target (which doesn't have SSE out of the box), instead of the i686-with-SSE2-disabled it currently uses
B) bump the i386 baseline in Debian to require SSE2, and stop disabling SSE2 there in rustc
C) disable all optimizations for Rust code on i386 (not really an option I think, just here for completeness sake)

one downside of the i586 approach is that its not a tier 1 target upstream, but a tier 2 without host tools one, similar to armel or mips64el. that means both less testing upstream, and no prebuilt stage0 for (re)bootstrapping, which makes jumping versions more involved. this might affect things like cargo-web/rustc-web on stable, or the ability to quickly catch up with upstream if we lag behind version wise.

both A) and B) are rather invasive changes.. if the RT (CCed) has a strong preference I'll of course follow their lead in this matter. otherwise, I'll see how involved the changes for A) are and maybe upload a build of that to experimental in the next week, so that we can do an archive-wide rebuild and see whether things break.

p.s.: if you become aware of such an issue with any toolchain, it would be nice to file a bug report *with the corresponding toolchain package in Debian* to make its maintainer (me in this case) aware as well. we could have had 2-3 months head start on tackling this issue if you had done so..

[Jonas Smedegaard]

[individuals stripped as recipients, as per Debian mailinglist policy]

Quoting Fabian GrÃŒnbichler (2024-11-23 15:14:42)

Post by Fabian Grünbichler
p.s.: if you become aware of such an issue with any toolchain, it
would be nice to file a bug report *with the corresponding toolchain
package in Debian* to make its maintainer (me in this case) aware as
well. we could have had 2-3 months head start on tackling this issue
if you had done so..

Duly noted. Sorry!

I genuinely thought that I had not uncovered a new bug but simply
noticed yet another case of the Rust team approaching Debian in ways
unusual to me but in principle not wrong nor broken nor buggy, and
therefore only a reason for filing bugreports if I wanted to be annoying
which I don't.

I shall try to be more noisy in future.

- Jonas

[Fabian Grünbichler]

Post by Fabian Grünbichler

Post by Jonas Smedegaard
Quoting Chris Hofstaedtler (2024-11-23 04:16:29)

Post by Chris Hofstaedtler

Post by Jonas Smedegaard

Post by Julian Andres Klode
All release architectures support Rust. We should not accept
release architectures without Rust support.
A minor set of ports architectures does not have Rust support
yet.

Rust is unsupported on i386 and patched to silently assume i686

i686 is not a problem, as that's the arch baseline for our i386
arch since bookworm.

https://wiki.debian.org/ArchitectureSpecificsMemo#i386
in combination with a bad interaction between rustc and LLVM..

Post by Jonas Smedegaard

Post by Chris Hofstaedtler

Post by Jonas Smedegaard
- see
DEP-3 references in this patch for discussions about that, and the patch
itself for a way to more loudly make reverse dependencies aware that
https://salsa.debian.org/debian/rust-wide/-/blob/debian/latest/debian/patches/2001_fail_non-sse2-x86.patch
Beware that Rust team build routines run tests without optimizations,
regardless of DEB_BUILD_OPTIONS=noopt, so for libraries maintained by
them the issue may go unnoticed until reverse dependencies run into the
issue *and* test for it, otherwise it might go unnoticed until users
report it.

So maybe it's time to raise the baseline to i686+sse2.

As I understand the situation with Rust, it is *not* that compiled code
fails to run on old non-SSE2 hardware. Instead, the problem is that the
Rust compiler produces code that is *ALWAYS* broken regardless if target
hardware supports SSE2 or not.
Yes, your final remark is a "solution" regardless, I just wanted to
emphasize that the problem affects the whole architecture, not only
outdated parts of it.
...unless I have misunderstand the situation, obciously.

AFAICT, this seems to be true. I was (so far) under the impression so
far that this only affects FP arithmetics (i.e., the usual issue one
runs into this is test code that does FP operations and then checks
that the result is as expected). given that it's not limited to that
(see
https://github.com/rust-lang/rust/issues/114479#issuecomment-2072052116
and https://github.com/rust-lang/compiler-team/issues/808), the
situation is a lot worse!
so yeah, I guess we should either
A) move i386 rustc to Rust's i586 target (which doesn't have SSE out of
the box), instead of the i686-with-SSE2-disabled it currently uses
B) bump the i386 baseline in Debian to require SSE2, and stop disabling SSE2 there in rustc
C) disable all optimizations for Rust code on i386 (not really an
option I think, just here for completeness sake)

read the full upstream issues now - downgrading to i586 doesn't solve the issue at all, upstream just doesn't care about it there because that target is not part of their fully supported tier.

so that means option A) is effectively off the table (other than we save ourselves some further patching once the "don't allow to disable SSE2 on i686" patches hit an upstream release), and just leaves B) if we want to really solve it, and C) if we want to do an incomplete hacky papering over.

Post by Fabian Grünbichler
The unsoundness is not just theoretical; the LLVM IR Rust compiles f32 and f64 operations to has the desired Rust semantics, whereas the LLVM non-SSE x86 backend compiles that IR to machine code that violates the semantics of the IR. This means e.g. LLVM optimisations that (correctly) operate presuming the LLVM IR semantics with regards to evaluation precision can cause the LLVM non-SSE x86 backend introduce out-of-bounds reads/writes in safe code (see this earlier comment for a code example). The NaN quietening issue also violates the semantics of LLVM IR and can cause the emitted binary to mutate the value of non-float types (see this earlier comment for the code sample, details in this comment).
Because this is a miscompilation at the LLVM IR -> machine code stage, as opposed to the Rust -> LLVM IR stage, miscompilations can occur in other programming languages that use LLVM as a codegen backend. For example, llvm/llvm-project#89885 contains an example of a miscompilation from C. Ultimately what matters are the semantics of the LLVM IR; not everything that is permitted by the IEEE 754 specification is permitted by LLVM IR (and vice versa).
The return value ABI issue is tracked separately in #115567, and affects all 32-bit x86 targets, not just those with SSE/SSE2 disabled. It is possible to manually load/store a f32/f64 signalling NaN to/from an x87 register without quietening it (see e.g. the code in #115567 (comment)), but currently neither LLVM nor Rust do so. The "Rust" ABI (which doesn't have any stability guarantees) is being changed to avoid x87 registers completely in #123351.
To my knowledge, nobody on the LLVM side really cares about this. So until that changes it is unlikely that we'll be able to improve things in Rust here. Telling programmers "on this hardware your program may randomly explode for no fault of your own" is not very useful. (I mean, it is a useful errata of course, but it doesn't make sense to make this the spec.)

so this is actually an LLVM+x87 bug, not limited to rustc, and not limited to Debian's rustc variant either, it's just that upstream's i686 rustc doesn't disable SSE2, whereas ours (and upstream's i586) do, and thus expose the issue. I tend to agree with upstream that the *only proper* solution is to enable/require SSE2.

[Chris Hofstaedtler]

Post by Fabian Grünbichler
A) move i386 rustc to Rust's i586 target (which doesn't have SSE out of the box), instead of the i686-with-SSE2-disabled it currently uses
B) bump the i386 baseline in Debian to require SSE2, and stop disabling SSE2 there in rustc
C) disable all optimizations for Rust code on i386 (not really an option I think, just here for completeness sake)

D) follow the other teams and stop building Rust on i386.

Chris

[Johannes Schauer Marin Rodrigues]

Quoting Chris Hofstaedtler (2024-11-23 17:50:25)

Post by Chris Hofstaedtler

Post by Fabian Grünbichler
A) move i386 rustc to Rust's i586 target (which doesn't have SSE out of the box), instead of the i686-with-SSE2-disabled it currently uses
B) bump the i386 baseline in Debian to require SSE2, and stop disabling SSE2 there in rustc
C) disable all optimizations for Rust code on i386 (not really an option I think, just here for completeness sake)

D) follow the other teams and stop building Rust on i386.

And that would leave i386 without mesa and python-cryptography both of which
not only have an extremely high popcon but are also the dependencies of some
other rather important things...

I know you know, just for the casual reader. :)

cheers, josch

[Fabian Grünbichler]

Post by Chris Hofstaedtler

Post by Fabian Grünbichler
B) bump the i386 baseline in Debian to require SSE2, and stop disabling SSE2 there in rustc
C) disable all optimizations for Rust code on i386 (not really an option I think, just here for completeness sake)

D) follow the other teams and stop building Rust on i386.

I am not personally invested in i386 at all, but that is still not a call that I'd want to make as toolchain maintainer. If the release team prefers it over B) and over risking running into the miscompilation issues in practice, then I won't object.

FWIW, my (personal!) current preference would be B), followed by either documenting the issue and accepting it or D). Given the far-reaching implications of both B and D, I don't think this is a decision that either I or the Rust team can make alone.

And as Josch indicated, dropping Rust from i386 would remove quite a lot of packages/affect quite a lot more teams than just the Rust team.

[Sylvestre Ledru]

Post by Fabian Grünbichler

Post by Chris Hofstaedtler

Post by Fabian Grünbichler
B) bump the i386 baseline in Debian to require SSE2, and stop disabling SSE2 there in rustc
C) disable all optimizations for Rust code on i386 (not really an option I think, just here for completeness sake)

D) follow the other teams and stop building Rust on i386.

I am not personally invested in i386 at all, but that is still not a call that I'd want to make as toolchain maintainer. If the release team prefers it over B) and over risking running into the miscompilation issues in practice, then I won't object.
FWIW, my (personal!) current preference would be B), followed by either documenting the issue and accepting it or D). Given the far-reaching implications of both B and D, I don't think this is a decision that either I or the Rust team can make alone.

+1 for B). I would be happy to do the same (enable ss2 extensions) for
llvm too.

S

[Iustin Pop]

Post by Fabian Grünbichler

Post by Chris Hofstaedtler

Post by Fabian Grünbichler
B) bump the i386 baseline in Debian to require SSE2, and stop disabling SSE2 there in rustc
C) disable all optimizations for Rust code on i386 (not really an option I think, just here for completeness sake)

D) follow the other teams and stop building Rust on i386.

I am not personally invested in i386 at all, but that is still not a call that I'd want to make as toolchain maintainer. If the release team prefers it over B) and over risking running into the miscompilation issues in practice, then I won't object.
FWIW, my (personal!) current preference would be B), followed by either documenting the issue and accepting it or D). Given the far-reaching implications of both B and D, I don't think this is a decision that either I or the Rust team can make alone.

I've been watching this thread from the sideline so far, but after
thinking some more, I don't understand why B) is simply not the chosen
path forward.

Are there viable/realistic platforms today that are i386 for real, and
don't support SSE2? In other words, if Debian as a whole does B) (not
only for rust), what is the practical impact on any non-toy hardware?

regards,
iustin

[Simon McVittie]

Post by Iustin Pop
Are there viable/realistic platforms today that are i386 for real, and
don't support SSE2? In other words, if Debian as a whole does B) (not
only for rust), what is the practical impact on any non-toy hardware?

SSE2 is well over 20 years old: in the Intel world, SSE2 was new in the
Pentium 4 (2000), and in the AMD world, SSE2 was new in the first
AMD64 CPUs (2003).

According to Wikipedia, the x86 CPUs without SSE2 since 2000 are:

- 32-bit AMD CPUs (Athlon XP and similar)
- VIA C3
- Transmeta Crusoe
- Intel Quark

We no longer release i386 kernels (since linux_6.11-1~exp1) and it has
already been announced that Debian 13 will not have i386 installation
media, so it is already the case that it is only possible to run trixie
on 32-bit x86 by upgrading from bookworm and keeping the bookworm kernel
(or installing a locally-built kernel).

If we raise the baseline to include SSE2, then we would lose the ability
to do that on Athlon XP, C3, Crusoe or Quark. I don't think that's a
practical problem in 2024: those are all designs from the early 2000s,
except for the Quark, which is a discontinued embedded processor that
we already cannot support in a reasonable way as a result of #738575.

According to Wikipedia, after raising the baseline it would still be
possible in principle to run i386 trixie using a bookworm or locally-built
kernel on some of the later 32-bit Intel CPUs (Pentium 4, Pentium M,
Atom), or on Transmeta Efficeon or VIA C7.

i386 in trixie continues to be available as a multiarch foreign
architecture on x86_64 CPUs, or as an architecture for containers or
chroots running on an x86_64 kernel, but a baseline that includes SSE2
would not harm that use-case, because MMX, SSE and SSE2 are part of the
x86_64 baseline already.

My advice to anyone still running Debian on a 32-bit x86 machine would
be to look into acquiring a second-hand 64-bit laptop, many of which are
already sufficiently old that they're most likely to be disposed of as
e-waste if unsold (so replacing an existing 32-bit machine would not be
a net increase in the amount of e-waste needing disposal). As an example,
here in the UK, eBay tells me I could receive a working Lenovo X200 next
week for £50, or a newer, faster and likely more power-efficient X280
for around £100.

smcv

[Andrey Rakhmatullin]

Post by Iustin Pop

Post by Fabian Grünbichler

Post by Chris Hofstaedtler

Post by Fabian Grünbichler
B) bump the i386 baseline in Debian to require SSE2, and stop disabling SSE2 there in rustc
C) disable all optimizations for Rust code on i386 (not really an option I think, just here for completeness sake)

D) follow the other teams and stop building Rust on i386.

I am not personally invested in i386 at all, but that is still not a call that I'd want to make as toolchain maintainer. If the release team prefers it over B) and over risking running into the miscompilation issues in practice, then I won't object.
FWIW, my (personal!) current preference would be B), followed by either documenting the issue and accepting it or D). Given the far-reaching implications of both B and D, I don't think this is a decision that either I or the Rust team can make alone.

I've been watching this thread from the sideline so far, but after
thinking some more, I don't understand why B) is simply not the chosen
path forward.

Nobody has made the choice or asked relevant people to make a choice, and
relevant people haven't commented on this, to my knowledge (RT by their
own 2023 update are *not* relevant people wrt raising the baseline).
And among people who cannot make this choice but voice their opinion (or
explicitly decline to have one) many are either conservative or think that
the project is, or should be, conservative. In other words, I feel like in
this specific case having many people on mailing lists hesitating about
this doesn't mean the project won't do that in the near future.

[Andrey Rakhmatullin]

Post by Fabian Grünbichler
B) bump the i386 baseline in Debian to require SSE2, and stop disabling SSE2 there in rustc

As suggested by Chris, and a popular opinion/request in general.

[Bastien Roucariès]

Post by Andrey Rakhmatullin

Post by Fabian Grünbichler
B) bump the i386 baseline in Debian to require SSE2, and stop disabling SSE2 there in rustc

It will break I suppose imagemagick on i386 and scientific software testsuite

i386 means FPU so excess precision.

SSE is good and even better but excess precision is worked arround thus FTBFS

Bastien

Post by Andrey Rakhmatullin
As suggested by Chris, and a popular opinion/request in general.

[Jeremy Bícha]

Post by Bastien Roucariès

Post by Fabian Grünbichler
B) bump the i386 baseline in Debian to require SSE2, and stop disabling SSE2 there in rustc

It will break I suppose imagemagick on i386 and scientific software testsuite
i386 means FPU so excess precision.
SSE is good and even better but excess precision is worked arround thus FTBFS

Debian's i386 already has excess precision:
https://wiki.debian.org/ArchitectureSpecificsMemo#i386

If you mean something else, could you be… more precise?

Thank you,
Jeremy Bícha

[Bastien Roucariès]

Post by Fabian Grünbichler

Post by Bastien Roucariès

Post by Fabian Grünbichler
B) bump the i386 baseline in Debian to require SSE2, and stop disabling SSE2 there in rustc

It will break I suppose imagemagick on i386 and scientific software testsuite
i386 means FPU so excess precision.
SSE is good and even better but excess precision is worked arround thus FTBFS

https://wiki.debian.org/ArchitectureSpecificsMemo#i386
If you mean something else, could you be
 more precise?

if SSE2 is enables gcc will not use i387 thus i386 will loss excess precision and have double like amd64

i387 => excess precision
SSE2 no excess precision

Bastien

Post by Fabian Grünbichler
Thank you,
Jeremy Bícha

[Andrey Rakhmatullin]

Post by Bastien Roucariès

Post by Fabian Grünbichler

Post by Bastien Roucariès

Post by Fabian Grünbichler
B) bump the i386 baseline in Debian to require SSE2, and stop disabling SSE2 there in rustc

It will break I suppose imagemagick on i386 and scientific software testsuite
i386 means FPU so excess precision.
SSE is good and even better but excess precision is worked arround thus FTBFS

https://wiki.debian.org/ArchitectureSpecificsMemo#i386
If you mean something else, could you be
 more precise?

if SSE2 is enables gcc will not use i387 thus i386 will loss excess precision and have double like amd64
i387 => excess precision
SSE2 no excess precision

Which is a good thing normally.
Just remove the workarounds you mentioned.

[Bastien Roucaries]

Post by Andrey Rakhmatullin

Post by Bastien Roucariès

Post by Fabian Grünbichler

Post by Bastien Roucariès

Post by Fabian Grünbichler
B) bump the i386 baseline in Debian to require SSE2, and stop
disabling SSE2 there in rustc> > >

It will break I suppose imagemagick on i386 and scientific software testsuite
i386 means FPU so excess precision.
SSE is good and even better but excess precision is worked arround
thus FTBFS> >

https://wiki.debian.org/ArchitectureSpecificsMemo#i386
If you mean something else, could you be… more precise?

if SSE2 is enables gcc will not use i387 thus i386 will loss excess
precision and have double like amd64
i387 => excess precision
SSE2 no excess precision

Which is a good thing normally.
Just remove the workarounds you mentioned.

I do not said it was a bad thing, I said just does we have time before release
?

[Fabian Grünbichler]

Post by Fabian Grünbichler

Post by Bastien Roucariès

Post by Fabian Grünbichler
B) bump the i386 baseline in Debian to require SSE2, and stop disabling SSE2 there in rustc

It will break I suppose imagemagick on i386 and scientific software testsuite
i386 means FPU so excess precision.
SSE is good and even better but excess precision is worked arround thus FTBFS

https://wiki.debian.org/ArchitectureSpecificsMemo#i386
If you mean something else, could you be… more precise?

Its very much possible that things relying on x87 floats on i386 will need to adapt if some underlying implementation switches to SSE, just like the inverse happens now (like testsuites expecting SSE behaviour/results that need to be adapted to x87).

I don't think that is a problem per se. And you can still use x87 floats if those work for your package, even if we go down that road. Well not with rust, but given that those are apparently broken already anyway...

[Bastien Roucariès]

Post by Fabian Grünbichler

Post by Fabian Grünbichler

Post by Bastien Roucariès

Post by Fabian Grünbichler
B) bump the i386 baseline in Debian to require SSE2, and stop disabling SSE2 there in rustc

It will break I suppose imagemagick on i386 and scientific software testsuite
i386 means FPU so excess precision.
SSE is good and even better but excess precision is worked arround thus FTBFS

https://wiki.debian.org/ArchitectureSpecificsMemo#i386
If you mean something else, could you be
 more precise?

Its very much possible that things relying on x87 floats on i386 will need to adapt if some underlying implementation switches to SSE, just like the inverse happens now (like testsuites expecting SSE behaviour/results that need to be adapted to x87).
I don't think that is a problem per se. And you can still use x87 floats if those work for your package, even if we go down that road.

In theory yes. In practice I do not know, I am quite unease to x87 passed to sse due to library then passed back to main program...

It will give us some interesting result

Post by Fabian Grünbichler
Well not with rust, but given that those are apparently broken already anyway...

[Andrew M.A. Cater]

Post by Andrey Rakhmatullin

Post by Fabian Grünbichler
B) bump the i386 baseline in Debian to require SSE2, and stop disabling SSE2 there in rustc

As suggested by Chris, and a popular opinion/request in general.
--
WBR, wRAR

Considering that for Trixie we won't have an i386 installer, particularly:
it seems evident to me that as more software depends on Rust, less of it
will be usable on i386 (and possibly something like armhf in due course.)
This whether or not we rebase the hardware requirements for an architecture.

I think that it is inevitable that i386 won't get the same amount of
software as amd64: Rust might be the deal breaker. (In the same way
that some large/complex apps don't get built for all architectures /
memory sizes. - Not everything can happily build/run Firefox, for example).

Just a thought

Andy Cater
(***@debian.org)

[Andrey Rakhmatullin]

Post by Andrew M.A. Cater

Post by Andrey Rakhmatullin

Post by Fabian Grünbichler
B) bump the i386 baseline in Debian to require SSE2, and stop disabling SSE2 there in rustc

As suggested by Chris, and a popular opinion/request in general.

it seems evident to me that as more software depends on Rust, less of it
will be usable on i386 (and possibly something like armhf in due course.)
This whether or not we rebase the hardware requirements for an architecture.

Sorry, I don't get it.

[Andrew M.A. Cater]

Post by Andrey Rakhmatullin

Post by Andrew M.A. Cater

Post by Andrey Rakhmatullin

Post by Fabian Grünbichler
B) bump the i386 baseline in Debian to require SSE2, and stop disabling SSE2 there in rustc

As suggested by Chris, and a popular opinion/request in general.

it seems evident to me that as more software depends on Rust, less of it
will be usable on i386 (and possibly something like armhf in due course.)
This whether or not we rebase the hardware requirements for an architecture.

Sorry, I don't get it.

Hi - sorry I might have conflated more than one thing: let's see if I can make
myself more clear.

1. Rust itself is fast-moving and difficult to package/deal with. There's a
couple of threads on LWN.net on this at the moment. There's a further
distinction between Rust in general and Rust in the Linux kernel itself -
which is further behind Rust head development. In general: Support in
Debian is difficult for fast-moving software without guaranteed backward
compatibility long term.

2. Upstream Rust doesn't particularly care about 32 bit i386 - so it's a
Debian thing to try and produce appropriate binaries including Rust wherever
they are.

3. There are packages including Rust that can't be built on 32 bit / can't be
built on some of Debian's ports. Firefox is one, I think - and there's the
general problem that some larger packages like libreoffice are marginal on
some architectures, without considering Rust. That situation can only get
worse with time.

4. The consensus seems to be that i386 in Trixie won't have an installer
built for it - there's no kernel support in the most recent kernels. The
architecture is being maintained primarily for compatibility with older
software (also the reason for no time_t64 transition on i386).

5. If we're moving hardware baselines for the sake of Rust (or any other
software on this architecture) it's already too late.

Does this help? I don't have a horse in this race, so this is purely an
opinion to be considered,

All the very best, as ever,

Andy

Post by Andrey Rakhmatullin
--
WBR, wRAR

[Paul Gevers]

Hi,

Post by Andrew M.A. Cater
5. If we're moving hardware baselines for the sake of Rust (or any other
software on this architecture) it's already too late.

Huh? Why?

[Putting my Release Team hat off] Personally I think Debian should be
raising the baseline for i386. I'm not sure about to which level, but
I've seen proposals in this thread.

Given that
1) we're no longer supporting i386 as a full architecture (no kernel, no
installer, only in chroots or as multiarch)
2) we don't clearly have i386 porters
maybe we should seek consensus in this thread and go with that.

[Release Team hat on] I would take consensus for a decision on the topic.

Paul

[Andrey Rakhmatullin]

Post by Paul Gevers

Post by Andrew M.A. Cater
5. If we're moving hardware baselines for the sake of Rust (or any other
software on this architecture) it's already too late.

Huh? Why?

I think the Andrew's idea in that line is "some software shouldn't be, and
some software can't be, in the future i386 port, and changing the baseline
won't change that fact for most of that software".
As I've just written in my other email, it's unclear to me if doing this,
among other things, is indeed useless.

Post by Paul Gevers
[Putting my Release Team hat off] Personally I think Debian should be
raising the baseline for i386. I'm not sure about to which level, but I've
seen proposals in this thread.
Given that
1) we're no longer supporting i386 as a full architecture (no kernel, no
installer, only in chroots or as multiarch)
2) we don't clearly have i386 porters
maybe we should seek consensus in this thread and go with that.
[Release Team hat on] I would take consensus for a decision on the topic.

(my opinion on this, already stated in this thread, is "match the amd64
one" and I think when we bump the amd64 one we should also bump the i386
one)

[Peter Pentchev]

Post by Paul Gevers
Hi,

Post by Andrew M.A. Cater
5. If we're moving hardware baselines for the sake of Rust (or any other
software on this architecture) it's already too late.

Huh? Why?
[Putting my Release Team hat off] Personally I think Debian should be
raising the baseline for i386. I'm not sure about to which level, but I've
seen proposals in this thread.
Given that
1) we're no longer supporting i386 as a full architecture (no kernel, no
installer, only in chroots or as multiarch)
2) we don't clearly have i386 porters
maybe we should seek consensus in this thread and go with that.
[Release Team hat on] I would take consensus for a decision on the topic.

FWIW from a mostly-lurking small-time DD who often wishes he would
allocate more time to help with Rust packaging in Debian: go for it.

(not sure if you were actually asking for lurkers to pipe in;
thought I would on the off chance of "everyone who agrees is silent")

G'luck,
Peter

[Holger Levsen]

Post by Paul Gevers
[Putting my Release Team hat off] Personally I think Debian should be
raising the baseline for i386. I'm not sure about to which level, but I've
seen proposals in this thread.

same here.

Post by Paul Gevers
[Release Team hat on] I would take consensus for a decision on the topic.

I for one, agree to raise the baseline.
--
cheers,
Holger

⢀⣎⠟⠻⢶⣊⠀
⣟⠁⢠⠒⠀⣿⡁ holger@(debian|reproducible-builds|layer-acht).org
⢿⡄⠘⠷⠚⠋⠀ OpenPGP: B8BF54137B09D35CF026FE9D 091AB856069AAA1C
⠈⠳⣄

You can cut all the flowers, but you can’t keep spring from coming.
#transdayofremembrance #berlin (2024)

[Chris Hofstaedtler]

Post by Paul Gevers
[Release Team hat on] I would take consensus for a decision on the topic.

Based on my casual reading up on what other distros did before us,
many years ago, ISTM raising the baseline to at least SSE2 would be
a very good idea.

Next up: raising the amd64 baseline.

Chris

[Fabian Grünbichler]

Post by Paul Gevers
Hi,

Post by Andrew M.A. Cater
5. If we're moving hardware baselines for the sake of Rust (or any other
software on this architecture) it's already too late.

Huh? Why?
[Putting my Release Team hat off] Personally I think Debian should be raising the baseline for i386. I'm not sure about to which level, but I've seen proposals in this thread.
Given that
1) we're no longer supporting i386 as a full architecture (no kernel, no installer, only in chroots or as multiarch)
2) we don't clearly have i386 porters
maybe we should seek consensus in this thread and go with that.
[Release Team hat on] I would take consensus for a decision on the topic.

Probably obvious given my other replies in this thread, but just FTR - I'd support that :)

[Jeremy Bícha]

Post by Paul Gevers
[Release Team hat on] I would take consensus for a decision on the topic.

If the excess precision issue goes away by bumping the i386 baseline,
that would make my life easier.

Thank you,
Jeremy Bícha

[Fabian Grünbichler]

Post by Paul Gevers
Hi,

Post by Andrew M.A. Cater
5. If we're moving hardware baselines for the sake of Rust (or any other
software on this architecture) it's already too late.

Huh? Why?
[Putting my Release Team hat off] Personally I think Debian should be
raising the baseline for i386. I'm not sure about to which level, but
I've seen proposals in this thread.
Given that
1) we're no longer supporting i386 as a full architecture (no kernel, no
installer, only in chroots or as multiarch)
2) we don't clearly have i386 porters
maybe we should seek consensus in this thread and go with that.
[Release Team hat on] I would take consensus for a decision on the topic.

been a month and a half now, with only pro voices in response to this (although not that many). how do we (want to) proceed? is this just a matter of the release time deciding that the baseline has been raised and communicating that, or is there some formal process involved (i.e., should I file a bug somewhere with a concrete baseline proposal?)?

the next rustc release is in 6 days, the one after (which I also very much would like to land in Trixie, freeze timeline and policies permitting ;)) 6 weeks after that. if the baseline is raised I'd drop the corresponding patches in rustc (I already did a test build of 1.83 and don't expect any immediate issues there), but the sooner I can do that the more time we have to handle fallout on the crate packages side (which is mostly dropping patches as well, since upstream normally conforms to stock semantics, and not to the Debian i386 ones).

thanks for your consideration!
Fabian

[Andrey Rakhmatullin]

Post by Andrew M.A. Cater

Post by Andrey Rakhmatullin

Post by Andrew M.A. Cater

Post by Andrey Rakhmatullin

Post by Fabian Grünbichler
B) bump the i386 baseline in Debian to require SSE2, and stop disabling SSE2 there in rustc

As suggested by Chris, and a popular opinion/request in general.

it seems evident to me that as more software depends on Rust, less of it
will be usable on i386 (and possibly something like armhf in due course.)
This whether or not we rebase the hardware requirements for an architecture.

Sorry, I don't get it.

Hi - sorry I might have conflated more than one thing

That's what I thought :)

Post by Andrew M.A. Cater
1. Rust itself is fast-moving and difficult to package/deal with. There's a
couple of threads on LWN.net on this at the moment. There's a further
distinction between Rust in general and Rust in the Linux kernel itself -
which is further behind Rust head development. In general: Support in
Debian is difficult for fast-moving software without guaranteed backward
compatibility long term.
2. Upstream Rust doesn't particularly care about 32 bit i386 - so it's a
Debian thing to try and produce appropriate binaries including Rust wherever
they are.

Yeah. Bumping the i386 baseline will certainly only fix some of the
problems, talking here not just about Rust but in general. I agree that
i386 is not a priority for most upstreams and the baseline doesn't matter
in many of those cases.

Post by Andrew M.A. Cater
3. There are packages including Rust that can't be built on 32 bit / can't be
built on some of Debian's ports. Firefox is one, I think - and there's the
general problem that some larger packages like libreoffice are marginal on
some architectures, without considering Rust. That situation can only get
worse with time.
4. The consensus seems to be that i386 in Trixie won't have an installer
built for it - there's no kernel support in the most recent kernels. The
architecture is being maintained primarily for compatibility with older
software (also the reason for no time_t64 transition on i386).
5. If we're moving hardware baselines for the sake of Rust (or any other
software on this architecture) it's already too late.

I think it's something like that:
1. Considering the likely future path for the i386 Debian arch we should
mostly care about libraries and maybe agressively disable building leaf
executable packages at some point. I don't remember anyone suggesting to
instead make a closed "allowlist" and drop everything else, and that could
be an interesting idea, but maybe it's too early for it.
2. It seems that we won't be able to maintain everything we want for i386,
and will need to drop random things after they are no longer possible to
maintain there.
3. Still, we may want to do some things we can do in order to maintain
more "libraries" for longer.

So yeah, it may be possible that the people responsible will say "it's not
possible for us to keep Rust on i386". But it's unclear if it's worth it
to keep it or not in the possible future "legacy apps support only" i386
if it is possible to keep it.

[Fabian Grünbichler]

Post by Andrew M.A. Cater

Post by Andrey Rakhmatullin

Post by Fabian Grünbichler
B) bump the i386 baseline in Debian to require SSE2, and stop disabling SSE2 there in rustc

As suggested by Chris, and a popular opinion/request in general.
--
WBR, wRAR

it seems evident to me that as more software depends on Rust, less of it
will be usable on i386 (and possibly something like armhf in due course.)
This whether or not we rebase the hardware requirements for an architecture.

There's a big difference between
- some software doesn't build on i386 anymore (cause of memory/.. constraints)
- all Rust software is potentially subtly broken on i386

The former won't go away by raising the baseline, the latter would..

[Bastien Roucariès]

Post by Andrey Rakhmatullin

Post by Fabian Grünbichler
B) bump the i386 baseline in Debian to require SSE2, and stop disabling SSE2 there in rustc

As suggested by Chris, and a popular opinion/request in general.

Hi,

First if we bump ABI could we target "pABI v0" aka amd64 baseline
instead of only SSE2 ?

It is perfectly equivalent (except minor point about FXSR).

Moreover you could test cpu support using this https://wiki.debian.org/InstructionSelection#Isa_support_package

I have made qemu-good qemu-bad wrapper to test if needed

In this case, for your user I will add a wrapper that at preinst time warn user about unsupported instrution for next debian version and backport to bookworm PU

I suppose dpkg could be made pre depends on this wrapper and we could break the upgrade if instruction is not supported

Bastien

[Fabian Grünbichler]

Post by Bastien Roucariès

Post by Andrey Rakhmatullin

Post by Fabian Grünbichler
B) bump the i386 baseline in Debian to require SSE2, and stop disabling SSE2 there in rustc

As suggested by Chris, and a popular opinion/request in general.

Hi,
First if we bump ABI could we target "pABI v0" aka amd64 baseline
instead of only SSE2 ?
It is perfectly equivalent (except minor point about FXSR).

I don't have any opinion on that :)

Post by Bastien Roucariès
Moreover you could test cpu support using this
https://wiki.debian.org/InstructionSelection#Isa_support_package

that doesn't really help. existing packages are already possibly broken in bookworm as well, and they are broken irrespective of whether the system they are installed on supports SSE2 or not.

Post by Bastien Roucariès
I have made qemu-good qemu-bad wrapper to test if needed
In this case, for your user I will add a wrapper that at preinst time
warn user about unsupported instrution for next debian version and
backport to bookworm PU
I suppose dpkg could be made pre depends on this wrapper and we could
break the upgrade if instruction is not supported

see above, this only helps for non-SSE systems to prevent them from upgrading since they are no longer supported if we raise the baseline, but it doesn't actually solve the issue at hand if we don't. unless we raise the baseline only for Rust, but I don't really see a point in that.

[Bastien Roucariès]

Post by Fabian Grünbichler

Post by Bastien Roucariès

Post by Andrey Rakhmatullin

Post by Fabian Grünbichler
B) bump the i386 baseline in Debian to require SSE2, and stop disabling SSE2 there in rustc

As suggested by Chris, and a popular opinion/request in general.

Hi,
First if we bump ABI could we target "pABI v0" aka amd64 baseline
instead of only SSE2 ?
It is perfectly equivalent (except minor point about FXSR).

I don't have any opinion on that :)

Post by Bastien Roucariès
Moreover you could test cpu support using this
https://wiki.debian.org/InstructionSelection#Isa_support_package

that doesn't really help. existing packages are already possibly broken in bookworm as well, and they are broken irrespective of whether the system they are installed on supports SSE2 or not.

Post by Bastien Roucariès
I have made qemu-good qemu-bad wrapper to test if needed
In this case, for your user I will add a wrapper that at preinst time
warn user about unsupported instrution for next debian version and
backport to bookworm PU
I suppose dpkg could be made pre depends on this wrapper and we could
break the upgrade if instruction is not supported

see above, this only helps for non-SSE systems to prevent them from upgrading since they are no longer supported if we raise the baseline,

yes it is the goal do not upgrade if we raise the baseline

Bastien

Post by Fabian Grünbichler
but it doesn't actually solve the issue at hand if we don't. unless we raise the baseline only for Rust, but I don't really see a point in that.

[Andrey Rakhmatullin]

Post by Bastien Roucariès

Post by Andrey Rakhmatullin

Post by Fabian Grünbichler
B) bump the i386 baseline in Debian to require SSE2, and stop disabling SSE2 there in rustc

As suggested by Chris, and a popular opinion/request in general.

Hi,
First if we bump ABI could we target "pABI v0" aka amd64 baseline
instead of only SSE2 ?

It makes total sense.

[Shengjing Zhu]

On Sat, Nov 23, 2024 at 10:30 PM Fabian Grünbichler

Post by Fabian Grünbichler
A) move i386 rustc to Rust's i586 target (which doesn't have SSE out of the box), instead of the i686-with-SSE2-disabled it currently uses
B) bump the i386 baseline in Debian to require SSE2, and stop disabling SSE2 there in rustc
C) disable all optimizations for Rust code on i386 (not really an option I think, just here for completeness sake)

For reference, golang had a similar problem 3 years ago[1]. We decided
to downgrade the baseline for golang on i386.

[1] https://lists.debian.org/debian-devel/2021/04/msg00165.html

[Fabian Grünbichler]

Post by Shengjing Zhu
On Sat, Nov 23, 2024 at 10:30 PM Fabian Grünbichler

Post by Fabian Grünbichler
A) move i386 rustc to Rust's i586 target (which doesn't have SSE out of the box), instead of the i686-with-SSE2-disabled it currently uses
B) bump the i386 baseline in Debian to require SSE2, and stop disabling SSE2 there in rustc
C) disable all optimizations for Rust code on i386 (not really an option I think, just here for completeness sake)

For reference, golang had a similar problem 3 years ago[1]. We decided
to downgrade the baseline for golang on i386.
[1] https://lists.debian.org/debian-devel/2021/04/msg00165.html

downgrading the baseline for Rust is what got us into this mess ;) there is no 32-bit x86 softfloat target upstream. there is a 64-bit one for kernel/.. usage: https://doc.rust-lang.org/beta/rustc/platform-support/x86_64-unknown-none.html

[Bastien Roucariès]

Post by Fabian Grünbichler

On Sat, Nov 23, 2024 at 10:30 PM Fabian GrÃŒnbichler

Post by Fabian Grünbichler
A) move i386 rustc to Rust's i586 target (which doesn't have SSE out of the box), instead of the i686-with-SSE2-disabled it currently uses
B) bump the i386 baseline in Debian to require SSE2, and stop disabling SSE2 there in rustc
C) disable all optimizations for Rust code on i386 (not really an option I think, just here for completeness sake)

For reference, golang had a similar problem 3 years ago[1]. We decided
to downgrade the baseline for golang on i386.
[1] https://lists.debian.org/debian-devel/2021/04/msg00165.html

downgrading the baseline for Rust is what got us into this mess ;) there is no 32-bit x86 softfloat target upstream. there is a 64-bit one for kernel/.. usage: https://doc.rust-lang.org/beta/rustc/platform-support/x86_64-unknown-none.html

How much effort it will be to use SIMDeverywhere on i386 ?

https://wiki.debian.org/SIMDEverywhere

Bastien

[Ben Hutchings]

Post by Jonas Smedegaard
Quoting Julian Andres Klode (2024-11-22 17:36:17)

Post by Julian Andres Klode

Post by Sune Vuorela

Post by Julian Andres Klode
As for ports without gpgv-sq, this does not affect them,
they can be served by the gpgv alternative. Once a gpgv-sq
is available, it's important to note that no migration happens

To me it looks like we will sligthly grow the base system and have
different kind of bugs per architecture depending on availability of
rust.

All release architectures support Rust. We should not accept
release architectures without Rust support.
A minor set of ports architectures does not have Rust support
yet.

Rust is unsupported on i386 and patched to silently assume i686 - see
DEP-3 references in this patch for discussions about that, and the patch
itself for a way to more loudly make reverse dependencies aware that
https://salsa.debian.org/debian/rust-wide/-/blob/debian/latest/debian/patches/2001_fail_non-sse2-x86.patch

[...]

That's unfortunate. However, I think this a short-term problem. Since
we no longer ship an i386 kernel, after the trixie release we should be
able to raise the baseline for i386 to include the features that are
architectural in amd64.

Ben.

[Julian Andres Klode]

Post by Julian Andres Klode
I've just finished more or less, adjusting the APT test suite
to test gpgv-sq. I plan to upload APT that tests gpgv-sq
tomorrow. This ensures full compatibility between apt and
gpgv-sq going forward.

2.9.14 in unstable contains the test suite fixes and prefers
gpgv-sq *if installed* but does not change any dependencies,
so you can now easily try out gpgv-sq in APT just by installing
it, without affecting other gpgv uses.

That said I of course Recommend you to install gpgv-from-sq;
gpgv in particular has a small interface and there should not
be anything missing or broken.

The broader gpg replacement meanwhile lacks editing and deleting
keys and possibly some other stuff people actually use, much of
which they can do using native sq.

If you want to learn more about Sequoia and the Chameleon
in particular, watch Holger's talk at this year's DebConf:

https://debconf24.debconf.org/talks/16-chameleon-the-easy-way-to-try-out-sequoia-openpgp-written-in-rust/

[Holger Levsen]

Post by Julian Andres Klode
If you want to learn more about Sequoia and the Chameleon
https://debconf24.debconf.org/talks/16-chameleon-the-easy-way-to-try-out-sequoia-openpgp-written-in-rust/

thanks, Julian.

the first 7min of
https://toulouse2024.mini.debconf.org/talks/20-lightning-talks/
have an updated version of that talk and regarding Sequoia in
general I recommened
https://debconf24.debconf.org/talks/127-sequoia-pgp-sq-gpg-from-sq-v6-openpgp-and-debian/
from Justus Winter, one of Sequoia's upstream developers.
--
cheers,
Holger

⢀⣎⠟⠻⢶⣊⠀
⣟⠁⢠⠒⠀⣿⡁ holger@(debian|reproducible-builds|layer-acht).org
⢿⡄⠘⠷⠚⠋⠀ OpenPGP: B8BF54137B09D35CF026FE9D 091AB856069AAA1C
⠈⠳⣄

In Germany we don‘t say „Happy Valentine‘s Day, I love you“, we say „ich werde
diesen vom Markt kreierten, konsumorientierten Trend des Kapitalismus nicht
unterstÃŒtzen,“ and I think that’s beautiful. (Hazel Brugger)

[Julian Andres Klode]

Post by Julian Andres Klode
I've just finished more or less, adjusting the APT test suite
to test gpgv-sq. I plan to upload APT that tests gpgv-sq
tomorrow. This ensures full compatibility between apt and
gpgv-sq going forward.
After that migrates to testing next week, I want to make
the switch: APT by default should use gpgv-sq. Previous
discussions with the security team did not reveal any
blockers for that, despite the strenuous nature of
security updates for Rust packages.

This has been delayed. There's ongoing investigation into
sqv and sqopv, which are smaller verifiers from Sequoia,
measuring only 2MB and without an SQLite dependency, hence
saving about 6MB.

We also identified issues with keys that are using SHA-1
binding signatures. I'm in the process of adding a crypto
policy override to APT to re-allow these until 2026; but
we also do not have a way to warn people about upcoming
revocations, so lots of stuff will suddenly fail on a new
year.

I don't want unstable users to switch multiple times
if possible.

There's limitations

1. gpgv-sq - this essentially mostly works as gpgv
now, so we get the feedback we are used to

2. sqopv - produces no usable feedback at this time
if signature verification failed, and also does
not respect SEQUOIA_CRYPTO_POLICY.

3. sqv - produces very detailed feedback, but does not
have supported for clearsigned files. This is _less_
of a blocker: APT detaches signatures itself anyhow
for historical reasons - gpgv lacked the ability to
do so before, and we need to ensure that the verified
bits actually match what we read later on.

RHEL 10 is adopting sq and sqv as the tooling to ship;
sqopv is technically much more powerful than sqv, though,
this may be an odd decision that they went with sqv, given
it also does not share an interface with `sq verify`.

I want to spend a day on my 4 day weekend to see if I
can whack sqopv and sqv into apt test suite compliance.

Post by Julian Andres Klode
My plan here is to use
Depends: gpgv-from-sq | gpgv-sq | gpgv
Recommends: gpgv-sq

I'm considering adding architecture restrictions to
allow opt-in on architectures, giving ports the ability
to migrate when they deem ready.

In light of the discussion above about sq(op)v, this
may change into

Depends: sqv [architecture list],
gpgv [!architecture list]

We had some further discussion about gpgv not being
in widespread use and removing it from the default
bootstrap possibly being a reasonable way forward.

[Julian Andres Klode]

Post by Julian Andres Klode

Post by Julian Andres Klode
I've just finished more or less, adjusting the APT test suite
to test gpgv-sq. I plan to upload APT that tests gpgv-sq
tomorrow. This ensures full compatibility between apt and
gpgv-sq going forward.
After that migrates to testing next week, I want to make
the switch: APT by default should use gpgv-sq. Previous
discussions with the security team did not reveal any
blockers for that, despite the strenuous nature of
security updates for Rust packages.

This has been delayed. There's ongoing investigation into
sqv and sqopv, which are smaller verifiers from Sequoia,
measuring only 2MB and without an SQLite dependency, hence
saving about 6MB.

An sqv backend is now available in apt-team/apt!409 and in
experimental in apt 2.9.17+exp1.

Note that the experimental upload only supports architectures
with sqv available. There is no fallback yet.

The plan is to detect if sqv is available at build time, by
build-depending on sqv for the correct set of architectures,
and then generate a `Depends: sqv` for those architectures,
and `Depends: gpgv` for other (ports) architectures.

The sqv binary is about 2MB large when optimized for size,
and provides good feedback when a key cannot be verified.

[Julian Andres Klode]

Post by Julian Andres Klode

Post by Julian Andres Klode

Post by Julian Andres Klode
I've just finished more or less, adjusting the APT test suite
to test gpgv-sq. I plan to upload APT that tests gpgv-sq
tomorrow. This ensures full compatibility between apt and
gpgv-sq going forward.
After that migrates to testing next week, I want to make
the switch: APT by default should use gpgv-sq. Previous
discussions with the security team did not reveal any
blockers for that, despite the strenuous nature of
security updates for Rust packages.

This has been delayed. There's ongoing investigation into
sqv and sqopv, which are smaller verifiers from Sequoia,
measuring only 2MB and without an SQLite dependency, hence
saving about 6MB.

An sqv backend is now available in apt-team/apt!409 and in
experimental in apt 2.9.17+exp1.
Note that the experimental upload only supports architectures
with sqv available. There is no fallback yet.
The plan is to detect if sqv is available at build time, by
build-depending on sqv for the correct set of architectures,
and then generate a `Depends: sqv` for those architectures,
and `Depends: gpgv` for other (ports) architectures.
The sqv binary is about 2MB large when optimized for size,
and provides good feedback when a key cannot be verified.

The Sequoia sqv backend is now the default backend in unstable
for architectures that have it (all release architectures, most
ports).

2.9.19 also replaces internal GnuTLS and gcrypt use with OpenSSL,
and all use of GnuPG in the test suite with Sequoia's `sq` command.

There is a backwards-incompatible change: Signed-By can no
longer contain an exact subkey match (suffix "!"). That
information is - rightly so - not available in the sqv
output.

[Julian Andres Klode]

Post by Julian Andres Klode

Post by Julian Andres Klode

Post by Julian Andres Klode

Post by Julian Andres Klode
I've just finished more or less, adjusting the APT test suite
to test gpgv-sq. I plan to upload APT that tests gpgv-sq
tomorrow. This ensures full compatibility between apt and
gpgv-sq going forward.
After that migrates to testing next week, I want to make
the switch: APT by default should use gpgv-sq. Previous
discussions with the security team did not reveal any
blockers for that, despite the strenuous nature of
security updates for Rust packages.

This has been delayed. There's ongoing investigation into
sqv and sqopv, which are smaller verifiers from Sequoia,
measuring only 2MB and without an SQLite dependency, hence
saving about 6MB.

An sqv backend is now available in apt-team/apt!409 and in
experimental in apt 2.9.17+exp1.
Note that the experimental upload only supports architectures
with sqv available. There is no fallback yet.
The plan is to detect if sqv is available at build time, by
build-depending on sqv for the correct set of architectures,
and then generate a `Depends: sqv` for those architectures,
and `Depends: gpgv` for other (ports) architectures.
The sqv binary is about 2MB large when optimized for size,
and provides good feedback when a key cannot be verified.

The Sequoia sqv backend is now the default backend in unstable
for architectures that have it (all release architectures, most
ports).
2.9.19 also replaces internal GnuTLS and gcrypt use with OpenSSL,
and all use of GnuPG in the test suite with Sequoia's `sq` command.
There is a backwards-incompatible change: Signed-By can no
longer contain an exact subkey match (suffix "!"). That
information is - rightly so - not available in the sqv
output.

Space consumption, with apt from experimental:

105M experimental.min.tar
192M experimental.tar
114M unstable.min.tar
196M unstable.tar

i.e. we see a 9MB saving for essential+apt, and a 4MB saving
for a default mmdebstrap. Something still pulls in gpgv there
which is unfortunate, we lack a 5MB savings.

More savings can be achieved by building sqv using openssl,
then we stop pulling in nettle.

[Holger Levsen]

Post by Julian Andres Klode
i.e. we see a 9MB saving for essential+apt, and a 4MB saving
for a default mmdebstrap.

very nice!

Post by Julian Andres Klode
Something still pulls in gpgv there
which is unfortunate, we lack a 5MB savings.
More savings can be achieved by building sqv using openssl,
then we stop pulling in nettle.

/me nods, something for 2025.

which reminds me something to share here: in the last weeks I've enabled
size optimisations for chameleon and sqv bringing down the sizes (in
bytes) from

6208216 /usr/bin/gpgv-sq
10780992 /usr/bin/gpg-sq
2958984 /usr/bin/sqv

to

4533016 /usr/bin/gpgv-sq
7319936 /usr/bin/gpg-sq
1718816 /usr/bin/sqv

with hardly an impact on performance.

Doing the same for sq is my agenda as well as packaging sq 1.0 :)
--
cheers,
Holger

⢀⣎⠟⠻⢶⣊⠀
⣟⠁⢠⠒⠀⣿⡁ holger@(debian|reproducible-builds|layer-acht).org
⢿⡄⠘⠷⠚⠋⠀ OpenPGP: B8BF54137B09D35CF026FE9D 091AB856069AAA1C
⠈⠳⣄

It's the end of the world as we know it - and I feel fine.

[Chris Hofstaedtler]

Post by Julian Andres Klode
Something still pulls in gpgv there
which is unfortunate, we lack a 5MB savings.

dpkg-dev Depends: gpgv | sq | ...

That seems odd. Maybe it wants gpgv | sqv | ...
instead?

If its not dpkg-dev, I don't see whats pulling gpgv.

Chris

[Guillem Jover]

Hi!

Post by Chris Hofstaedtler

Post by Julian Andres Klode
Something still pulls in gpgv there
which is unfortunate, we lack a 5MB savings.

I think that would be gpgv being Priority: important, which makes
debootstrap and friends pull it in by default. I guess that might
need to be swapped now.

Post by Chris Hofstaedtler
dpkg-dev Depends: gpgv | sq | ...
That seems odd. Maybe it wants gpgv | sqv | ...
instead?

I do have a branch to add support for sqv, should get in with the next
dpkg upload. And probably can now swap the order of preference there
too.

Thanks,
Guillem

[Julian Andres Klode]

Post by Guillem Jover
Hi!

Post by Chris Hofstaedtler

Post by Julian Andres Klode
Something still pulls in gpgv there
which is unfortunate, we lack a 5MB savings.

I think that would be gpgv being Priority: important, which makes
debootstrap and friends pull it in by default. I guess that might
need to be swapped now.

This is being tracked in 1091200 (the demotion). I didn't suggest
a swap, after all, apt's dependencies will pull in the right one.

Post by Guillem Jover

Post by Chris Hofstaedtler
dpkg-dev Depends: gpgv | sq | ...
That seems odd. Maybe it wants gpgv | sqv | ...
instead?

I do have a branch to add support for sqv, should get in with the next
dpkg upload. And probably can now swap the order of preference there
too.

Sweet!