Building packages in the future.

[Santiago Vila]

Hello.

This is an update for my previous MBF announcement here:

https://lists.debian.org/debian-devel/2024/05/msg00414.html

I did another test rebuild and found 11 new packages failing
in the not-so-distant future. I also found another package
for which the fix was lost and the bug had to reopened.

The tag for the bugs being reported is here:

https://bugs.debian.org/cgi-bin/pkgreport.cgi?users=debian-***@lists.debian.org;tag=ftbfs-during-trixie-support-period

I was told that it was ok to consider those bugs as "RC for trixie"
but I was also requested to be nice when reporting those bugs, so I have
been reporting them as severity:normal (except when the future
suddenly arrives, like the few ones that started to fail on 2025-01-01).

Can the Release Managers suggest a calendar for raising those bugs,
first to important, then to serious?

Thanks.

[Otto Kekäläinen]

Hi!

Post by Santiago Vila
https://lists.debian.org/debian-devel/2024/05/msg00414.html
I did another test rebuild and found 11 new packages failing
in the not-so-distant future. I also found another package
for which the fix was lost and the bug had to reopened.

Did you use libfaketime in this round of rebuilds?
(https://lists.debian.org/debian-devel/2024/05/msg00422.html)? Do you
think it works well, should we add an extra job in Salsa CI that does
the build or runs autopkgtests under 'faketime 2028-06-30'? Or even
'faketime 2038-06-30' to cover 32-bit issues too?

[Santiago Vila]

Post by Otto Kekäläinen

Post by Santiago Vila
https://lists.debian.org/debian-devel/2024/05/msg00414.html
I did another test rebuild and found 11 new packages failing
in the not-so-distant future. I also found another package
for which the fix was lost and the bug had to reopened.

Did you use libfaketime in this round of rebuilds?
(https://lists.debian.org/debian-devel/2024/05/msg00422.html)? Do you
think it works well, should we add an extra job in Salsa CI that does
the build or runs autopkgtests under 'faketime 2028-06-30'? Or even
'faketime 2038-06-30' to cover 32-bit issues too?

Yes, it would be *really* nice to have an extra job in Salsa CI for that!

No, I did not use libfaketime yet (sorry). If you want to make a job
in Salsa you can just test with any of the affected packages above and
the (failed) build log should be very similar to the one I provided
in each of the bugs.

I would be more than happy if we could release trixie
without time-bombs, but of course if we can also test
for 2038-06-30, the better.

Maybe a common job which may be fine-tuned using a variable
for the cut date would allow to do that easily.

Thanks.

[Santiago Ruano Rincón]

Post by Santiago Vila

Post by Otto Kekäläinen

Post by Santiago Vila
https://lists.debian.org/debian-devel/2024/05/msg00414.html
I did another test rebuild and found 11 new packages failing
in the not-so-distant future. I also found another package
for which the fix was lost and the bug had to reopened.

Did you use libfaketime in this round of rebuilds?
(https://lists.debian.org/debian-devel/2024/05/msg00422.html)? Do you
think it works well, should we add an extra job in Salsa CI that does
the build or runs autopkgtests under 'faketime 2028-06-30'? Or even
'faketime 2038-06-30' to cover 32-bit issues too?

Yes, it would be *really* nice to have an extra job in Salsa CI for that!
No, I did not use libfaketime yet (sorry). If you want to make a job
in Salsa you can just test with any of the affected packages above and
the (failed) build log should be very similar to the one I provided
in each of the bugs.

JFTR, faketime causes issues in the reprotest job:

https://salsa.debian.org/salsa-ci-team/pipeline/#faketime-is-currently-disabled

Post by Santiago Vila
I would be more than happy if we could release trixie
without time-bombs, but of course if we can also test
for 2038-06-30, the better.
Maybe a common job which may be fine-tuned using a variable
for the cut date would allow to do that easily.
Thanks.

[Otto Kekäläinen]

Thanks for encouragement. I filed
https://salsa.debian.org/salsa-ci-team/pipeline/-/issues/411 and will
continue research on libfaketime/datefudge in CI there.

[Chris Hofstaedtler]

Hi,

Post by Otto Kekäläinen
Thanks for encouragement. I filed
https://salsa.debian.org/salsa-ci-team/pipeline/-/issues/411 and will
continue research on libfaketime/datefudge in CI there.

as we've seen in the time_t-64 transition, programs that interpose
library calls like this are extremely hard to get right and very
brittle.

I would strongly suggest to not put new load-bearing stuff on top of
such a program, especially when the rest of Debian is trying to get
rid of fakeroot.

Maybe you can consider using a time namespace (unshare -T) and
change the system date/time in that namespace.

Chris

[Holger Levsen]

Post by Chris Hofstaedtler
Maybe you can consider using a time namespace (unshare -T) and
change the system date/time in that namespace.

I'd also strongly suggest to do a full archive rebuild in such namespace
comparing that with a full archive rebuild in the presense to get data
how much failures there are, before putting this into CI on salsa or elsewhere..
--
cheers,
Holger

⢀⣎⠟⠻⢶⣊⠀
⣟⠁⢠⠒⠀⣿⡁ holger@(debian|reproducible-builds|layer-acht).org
⢿⡄⠘⠷⠚⠋⠀ OpenPGP: B8BF54137B09D35CF026FE9D 091AB856069AAA1C
⠈⠳⣄

The greatest danger in times of turbulence is not the turbulence;
it is to act with yesterdays logic. (Peter Drucker)

[Simon McVittie]

Post by Chris Hofstaedtler
as we've seen in the time_t-64 transition, programs that interpose
library calls like this are extremely hard to get right and very
brittle.
I would strongly suggest to not put new load-bearing stuff on top of
such a program, especially when the rest of Debian is trying to get
rid of fakeroot.

Yes, this; but unfortunately...

Post by Chris Hofstaedtler
Maybe you can consider using a time namespace (unshare -T) and
change the system date/time in that namespace.

... this isn't a feature that time namespaces offer. Unless the
documentation in time_namespaces(7) is outdated, time namespaces currently
only namespace CLOCK_MONOTONIC* and CLOCK_BOOTTIME*, not CLOCK_REALTIME;
but it's CLOCK_REALTIME (and related clocks like CLOCK_TAI) that represent
wall-clock time, so it's those clocks that you'd need to fake in order
to answer questions like "what would happen if we recompiled this package
5 years in the future?".

Simpler/higher-level functions like time() and gettimeofday() usually
use the same clock as CLOCK_REALTIME, except for functions like
g_get_monotonic_time() that specifically say they use a monotonic clock.

This also means that time namespaces are not (yet) a way to run legacy
binaries post-2038 by telling them an earlier date.

As far as I know, both of those use-cases require either LD_PRELOAD
interposing (brittle and error-prone) or a virtual machine with its clock
intentionally set incorrectly.

smcv

[Jakub Wilk]

Maybe you can consider using a time namespace (unshare -T) and change
the system date/time in that namespace.

I don't think that would work. From time_namespaces(7):

"Note that time namespaces do not virtualize the CLOCK_REALTIME clock.
Virtualization of this clock was avoided for reasons of complexity and
overhead within the kernel."

[Bastian Blank]

Post by Chris Hofstaedtler
Maybe you can consider using a time namespace (unshare -T) and
change the system date/time in that namespace.

As those jobs already run in throw away VM, just change the system time.
You have to change it back to be able to talk TLS later, but well.

Bastian

[Holger Levsen]

Post by Santiago Vila

Post by Otto Kekäläinen
Did you use libfaketime in this round of rebuilds?

No, I did not use libfaketime yet (sorry).

so what did you use?

setting the system time to the future (like we've been doing for
tests.reproducible-builds.org/debian for many years) works nicely (with
caveats) in a dedicated setup, but is nothing which can be done for salsa
(except with dedicated future runners...)

using faketime just wont work for too many packages/languages/ecosystems.
--
cheers,
Holger

⢀⣎⠟⠻⢶⣊⠀
⣟⠁⢠⠒⠀⣿⡁ holger@(debian|reproducible-builds|layer-acht).org
⢿⡄⠘⠷⠚⠋⠀ OpenPGP: B8BF54137B09D35CF026FE9D 091AB856069AAA1C
⠈⠳⣄

No more excuses. Small actions can make a big difference. We recently switched
to paper straws on both of my private jets. (@lcamtuf)

[Santiago Vila]

[ moving technical discussion to -devel, as -release was
just to ask RM for a calendar to raise severities ]

[ Bcc to Holger on this one ]

Post by Holger Levsen
so what did you use?

I still change the system clock. So, the same you did.

[Paul Gevers]

Hi Santiago

Sorry for dropping the ball on this.

Post by Santiago Vila
I was told that it was ok to consider those bugs as "RC for trixie"
but I was also requested to be nice when reporting those bugs, so I have
been reporting them as severity:normal (except when the future
suddenly arrives, like the few ones that started to fail on 2025-01-01).
Can the Release Managers suggest a calendar for raising those bugs,
first to important, then to serious?

What I had in mind was that you'd file the bugs and after some time
(say, one or two months) raised them, giving maintainers a bit more time
than usual to react to this novel class of RC bugs as you've now started
to actively test. I consider these bugs RC as they will exhibit
themselves as FTBFS during the stable release support period. So I
suggest you mark these bugs as RC now, such that they can get fixed in
time for the trixie release.

Paul

[Santiago Vila]

Post by Santiago Vila
Can the Release Managers suggest a calendar for raising those bugs,
first to important, then to serious?

What I had in mind was that you'd file the bugs and after some time (say, one or two months) raised them, giving maintainers a bit more time than usual to react to this novel class of RC bugs as you've now started to actively test. I consider these bugs RC as they will exhibit themselves as FTBFS during the stable release support period. So I suggest you mark these bugs as RC now, such that they can get fixed in time for the trixie release.

Thanks a lot. I filed a first round of bugs on 2024-08 and a second
round on 2024-12, then raised to important on 2025-01-27 as a reminder/ping.

Considering the average time all those bugs have been open (some on 2024-08,
some on 2024-12), I think it fits the idea of waiting a little bit more time
than usual, so I've just raised all of them to serious now.

Thanks.

[Jeremy Bícha]

Thank you for this wonderful project and for raising the severity to
serious since it's clearly easier to fix the bugs in Unstable now
rather than as Stable Updates later.

I think your next iteration of this could go further. The ELTS project
aims to provide security support for 10 years for a given Debian
release. While not all packages are included in its scope, it makes
sense that many of the packages that have test SSL certificates could
be security-sensitive packages.

Thank you,
Jeremy Bícha

[Santiago Vila]

Post by Jeremy Bícha
Thank you for this wonderful project and for raising the severity to
serious since it's clearly easier to fix the bugs in Unstable now
rather than as Stable Updates later.

Thanks go to the Release Managers, more specifically Paul Gevers,
who allowed me to consider this as RC. You know, Release Managers
are the ones who ultimately decide about RC-ness of bugs.

Post by Jeremy Bícha
I think your next iteration of this could go further. The ELTS project
aims to provide security support for 10 years for a given Debian
release. While not all packages are included in its scope, it makes
sense that many of the packages that have test SSL certificates could
be security-sensitive packages.

This was a Debian-specific thing, and I don't work for the ELTS project
(nor I'm planning to do so), but of course they are free to do their
own tests and fix packages before they start to FTBFS in the LTS
distributions.

Thanks.