Rebuilds to enable PAC and BTI support on arm64

[Sebastian Ramacher]

Hi,

since dpkg 1.22.0 the additional hardening flags to enable Pointer
Authentication (PAC) and Branch Target Identification (BTI)
on arm64 are enabled by default. See [1] for the discussion to enable
these flags.

To have the desired effect for the next release and have some time
to catch regressions, I have started with scheduling rebuilds of
packages that have not been built since the change in the default flags.
While the change of flags only affects arm64, packages building
Multi-Arch: same binaries require consistent versions on all
architectures. For those packages, the rebuilds have been scheduled on
all architectures.

Note that all builds have been scheduled with build priority -50, so
builds of uploads have higher priority and will be picked up by the
buildds before PAC/BTI rebuilds.

Thanks to Emanuele Rocca for identifying the list of packages that have
to be rebuilt to gain PAC/BTI support.

Cheers

[1] https://lists.debian.org/debian-dpkg/2022/05/msg00022.html

[Holger Levsen]

Post by Sebastian Ramacher
since dpkg 1.22.0 the additional hardening flags to enable Pointer
Authentication (PAC) and Branch Target Identification (BTI)
on arm64 are enabled by default. See [1] for the discussion to enable
these flags.

/me likes

Post by Sebastian Ramacher
To have the desired effect for the next release and have some time
to catch regressions, I have started with scheduling rebuilds of
packages that have not been built since the change in the default flags.
While the change of flags only affects arm64, packages building
Multi-Arch: same binaries require consistent versions on all
architectures. For those packages, the rebuilds have been scheduled on
all architectures.

/me likes very much! background: even though snapshot.d.o has been
fixed now, so that it's become generally usable again, several many
snapshots from 2023 and 2024 are missing, thus making it impossible
to recreate the build environments used needed for reproducible builds
of trixie.

these mass rebuilds will help reduce that gap.

Post by Sebastian Ramacher
Thanks to Emanuele Rocca for identifying the list of packages that have
to be rebuilt to gain PAC/BTI support.

thank you both! :)
--
cheers,
Holger

⢀⣎⠟⠻⢶⣊⠀
⣟⠁⢠⠒⠀⣿⡁ holger@(debian|reproducible-builds|layer-acht).org
⢿⡄⠘⠷⠚⠋⠀ OpenPGP: B8BF54137B09D35CF026FE9D 091AB856069AAA1C
⠈⠳⣄

Gendern ist wie Wurst ohne Fleisch: Fortschritt.

[Emanuele Rocca]

Post by Sebastian Ramacher
since dpkg 1.22.0 the additional hardening flags to enable Pointer
Authentication (PAC) and Branch Target Identification (BTI)
on arm64 are enabled by default.

Some more background and an update on this.

Both PAC and BTI are enabled by adding -mbranch-protection=standard to
the compiler flags. The defaults in Debian sid include such flag since
August 2023 (dpkg 1.22.0) as Sebastian said.

However PAC and BTI differ in the way they are enabled. For PAC, simply
building a program with -mbranch-protection=standard results in PAC to
be enabled. When it comes to BTI, all execution units (ie: all object
files) linked together need to have BTI in order for the resulting ELF
file to have BTI turned on. Since pretty much every program in the world
uses crtbeginS.o and crtendS.o from GCC as well as crti.o, Scrt1.o and
crtn.o from glibc, this means that only packages built with a
BTI-enabled GCC and glibc get the feature. In sid, we enabled BTI
support in gcc-14 14.1.0-4 (2024-07-10) and glibc 2.39-5 (2024-07-22).
See https://wiki.debian.org/ToolChain/PACBTI for more details.

I performed a local archive rebuild to get the list of all packages that
don't currently have BTI on, but would get it with a simple rebuild
(binNMU). I added the date of "last build" to the output just to verify
that no package was built after the end of July 2024, those should have
had BTI already. To my surprise some of the packages in the list where
last built in 2014, which is... well a long time ago!
https://people.debian.org/~ema/pac-bti/arm64-binNMUs.log

When the binNMUs started (thank you Sebastian) we had 10204 binary
packages with BTI turned on, and we are now at 18348. Once the rebuilds
are over I'll check the situation again. There's likely going to be a
long tail of packages that don't get BTI with a simple rebuild for many
reasons, including for example not using the default compiler flags.

As a final thought, given that new toolchain versions bring multiple
improvements over the years it's perhaps worth thinking about rebuilding
the archive on some sort of regular basis to make sure we get the
benefits?

ema

[Andrey Rakhmatullin]

Post by Emanuele Rocca
As a final thought, given that new toolchain versions bring multiple
improvements over the years it's perhaps worth thinking about rebuilding
the archive on some sort of regular basis to make sure we get the
benefits?

"Let's at least force rebuilds all packages not rebuilt since stable
before every freeze starts" is a popular opinion.

[Holger Levsen]

Post by Andrey Rakhmatullin
"Let's at least force rebuilds all packages not rebuilt since stable
before every freeze starts" is a popular opinion.

true. and "let's not do that" is even more popular, else why haven't we
done this in three decades?
--
cheers,
Holger

⢀⣎⠟⠻⢶⣊⠀
⣟⠁⢠⠒⠀⣿⡁ holger@(debian|reproducible-builds|layer-acht).org
⢿⡄⠘⠷⠚⠋⠀ OpenPGP: B8BF54137B09D35CF026FE9D 091AB856069AAA1C
⠈⠳⣄

If you upload your address book to "the cloud", I don't want to be in it.

[Andrey Rakhmatullin]

Post by Holger Levsen

Post by Andrey Rakhmatullin
"Let's at least force rebuilds all packages not rebuilt since stable
before every freeze starts" is a popular opinion.

true. and "let's not do that" is even more popular, else why haven't we
done this in three decades?

Doing is harder than talking, and doing something like that is much
harder than, say, fixing and uploading a package. Where can one start?

[Andreas Tille]

Post by Holger Levsen

Post by Andrey Rakhmatullin
"Let's at least force rebuilds all packages not rebuilt since stable
before every freeze starts" is a popular opinion.

true. and "let's not do that" is even more popular, else why haven't we
done this in three decades?

Changing something is just some effort. How do you want to distinguish
the "let's not do that" opinion from the "no spare time to trigger a
change" "opinion"?

Finally we have those regular archive wide rebuilds that trigger lots of
FTBFS bugs. I could imagine to simply upload those that pass the
rebuild tests. Otherwise the package needs manual intervention by the
maintainer which also will end up in an upload or a testing removal.

Kind regards
Andreas.

[Guillem Jover]

Hi!

Post by Andrey Rakhmatullin

Post by Emanuele Rocca
As a final thought, given that new toolchain versions bring multiple
improvements over the years it's perhaps worth thinking about rebuilding
the archive on some sort of regular basis to make sure we get the
benefits?

"Let's at least force rebuilds all packages not rebuilt since stable
before every freeze starts" is a popular opinion.

Of course, as Emanuele mentions doing mass rebuilds, which seems to
be more common nowadays, can bring benefits from toolchain improvements
(potentially all layers from compilers, to packaging tools).

But routinely rebuilding and _uploading_ the results also comes with
its drawbacks. For one it will hide ABI breaks. Having packages not get
upgraded during a Debian release upgrade has also been a good and easy
indicator for end users that those packages are stale and perhaps it's
time to look for alternatives.

Thanks,
Guillem