s390x architecture status?

[Chris Hofstaedtler]

Hello!

I want to draw attention to, from my point of view, open
questions/problems with the s390x architecture.

In short, it seems to me, that:
a) there are no porters left (to fix serious problems)
b) various packages already ignore s390x (gnome? others?)
c) general, including upstreams, interest in s390x is not there

With this context, I want to ask what the practical future of s390x
in Debian should be?

***

I acknowledge that s390x is the last big-endian release arch.
While this fact may be the cause of interest for curious people,
in general it seems to cause more problems than we need.

It also appears true that IBM has an interest in s390x, but today
they have no porters in Debian. While it's nice that a company has
an interest in their own products, that in itself does not help us.
Even if IBM staff would send patches to the BTS, someone still needs
to build, verify, upload, ... maintain the relevant packages.

My opinion on a)-c) was formed mostly by working (or avoiding
working on) bugs involving s390x. Here are three bugs which to me
sum up the current situation.

#960265: s390x install Debootstrap warning: Failure while configuring base packages

This bug was reported in June 2020, but it shows that debootstrap
for s390x was completely broken in the buster release. I'll note
that buster was released one year before the bug was reported. One
of the previous porters mentioned in that bug that clearly there can
be no real users, if such a problem gets into a stable release.

#1068119: s390-tools: FTBFS: ... -Werror=array-bounds=

New gcc upload broke building s390-tools. Time between this being
found and the gcc upload was probably long. The usual mass rebuilds
(by Lucas, Santiago, et al) cannot find this, as s390-tools is an
s390x-specific package. Upstream fixed the problem a long time ago,
but the fixes never made it into Debian. For a while this blocked
migration of mmdebstrap, which has become a relevant package to
Debian's package building stack. Ultimately the mmdebstrap
maintainer uploaded patches from upstream to make progress.

#1080317: s390-tools: FTBFS : ... -Wincompatible-pointer-types

Again, another upload broke building s390-tools. Again this was very
likely an undetected FTBFS for a long time. Upstream fixed the
problem a long time ago. Ultimately Michael Biebl uploaded these
fixes to make progress for usrmerge.

In the wake of the last bug I've opened #1084987: O: s390-tools.
I'll note the previous uploader of s390-tools removed themselves
three uploads before their last upload from Uploaders:, and the
other uploader in the list hasn't uploaded the package since 2010.

Regarding upstream interest, it is clear very often this manifests
in problems where software just does not build on s390x. Often this
is just a broken big-endian branch, easily patched in Debian. But
there are more complicated pieces of software, and they don't build,
and obviously we need help from porters to fix those.

***

Having now drawn attention to the state of s390x, I hope there will
be productive outcomes.

Maybe, motivated porters will show up and maintain the architecture and
its key packages (like s390-tools).

Otherwise I suggest we save us all a lot of time and stop carrying
an in-Debian-unmaintained architecture.


Chris

[Simon McVittie]

Post by Chris Hofstaedtler
b) various packages already ignore s390x (gnome? others?)

GNOME is currently buildable on s390x, but we have to ignore a lot of test
failures related to incorrect endianness of colour channels in image data
(for example in GTK 3, GTK 4, librsvg) and investigating those issues
tends to consume a much larger amount of maintainer time than we can
really justify.

We did temporarily remove high-level parts of GNOME (GNOME Shell
and friends) from s390x in a previous release cycle, because the
mozjs JavaScript engine was known-broken on s390x at the time, but in
current versions it appears to be working mostly as intended. However,
I'm reasonably sure that GNOME Shell has never been genuinely useful on
IBM mainframes: there is a not-entirely-hypothetical use-case for running
GTK/GNOME *apps* on one of these machines via X-forwarding or some
similar mechanism, but running a full Wayland compositor on a mainframe
seems really unlikely to be useful, particularly one like GNOME Shell
that is designed to make use of a GPU.

Post by Chris Hofstaedtler
I acknowledge that s390x is the last big-endian release arch.
While this fact may be the cause of interest for curious people,
in general it seems to cause more problems than we need.

I believe the latest status from porters outside Debian is that the
GTK and librsvg issues are believed to be caused by a regression in
src:pixman, and not actually the GNOME libraries' fault. The regression
in pixman appears to have been caused by a well-intentioned porter trying
to solve some other endianness bug but instead working around it in the
wrong layer, and last time I looked, it was believed to have been fixed
upstream but not yet fixed in Debian.

Based on fixes I've contributed to graphics-related libraries like GTK
and SDL, I've come to believe that the only way to solve endianness
issues without regressions is for each layer of the stack to have clear
documentation about the byte-order that is intended, for example
https://github.com/libsdl-org/SDL/commit/3698630bbc8e2ac501127c9c522cc0463a6c1565
which explains the meaning of each of SDL's pixel formats. Otherwise,
it's very easy for porters for a big-endian architecture to "fix" an
endianness-swap by reversing the endianness at the wrong layer of the
library stack, resulting in the correct result via "compensating errors"
for the particular library stack they are currently looking at, but with
the side-effect of regressions in a different library stack.

For example, if GTK is handling endianness wrongly, and a porter tries
to solve that by swapping the endianness in a lower-level library that
is used by both GTK and Qt, then they'll make GTK work as intended but
break Qt. Instead, the correct fix is to make each library work according
to its documentation (or, if documentation is missing, first document the
behaviour it was designed to have, and then fix any deviations from that).

Post by Chris Hofstaedtler
Maybe, motivated porters will show up and maintain the architecture and
its key packages (like s390-tools).

Another factor that is relevant to s390x is that many newer architectures
like aarch64 and riscv64 are converging on some common architecture
properties (such as booting via EFI from an NVMe, SATA or SCSI disk,
having USB connectivity, having a VGA-compatible GPU and/or a serial
console, and generally behaving a bit like a PC), but because s390x is
a mainframe architecture, it has various s390x-specific quirks such as
its own special disk and terminal devices.

This means that it will tend to need special-cased code paths in
infrastructure tools like autopkgtest-build-qemu[1], which makes it
harder for a typical Debian developer to investigate failures even if
they are willing to spend time on it.

Also, unlike typical newer ports like riscv64, the minimal s390x machine
is a large, power-hungry, expensive mainframe: an interested developer
can't simply set up a dev board on their desk like they can for embedded
or "PC-like" ports.

smcv

[1] https://lists.debian.org/debian-devel/2024/10/msg00284.html

[Marco d'Itri]

Post by Chris Hofstaedtler
It also appears true that IBM has an interest in s390x, but today

I wonder if their interest could actually be just in Debian providing
a base for the Ubuntu port (which I understand used to be funded by
IBM).
And if this is still true now that IBM owns Red Hat.

You have been clear enough in documenting how this port has no porters
and probably also very few users. I think that somebody who actually has
an interest in it needs to step up very soon to maintain it, or else the
first step would be to declare it not a release architecture anymore.

[Elizabeth K. Joseph]

Post by Marco d'Itri

Post by Chris Hofstaedtler
It also appears true that IBM has an interest in s390x, but today

I wonder if their interest could actually be just in Debian providing
a base for the Ubuntu port (which I understand used to be funded by
IBM).
And if this is still true now that IBM owns Red Hat.
You have been clear enough in documenting how this port has no porters
and probably also very few users. I think that somebody who actually has
an interest in it needs to step up very soon to maintain it, or else the
first step would be to declare it not a release architecture anymore.

It is still true, and as a data point the IBM LinuxONE Open Source
Cloud has Debian as one of the operating systems that we offer to open
source software projects doing development, due to popular demand.

I work at IBM and have been in touch with Berli Gayathri as she gets
up to speed with the Debian community and is learning the intricacies
of being involved with the project. We're also looking internally at
IBM to see if any of the Debian Developers who already work here are
able to help us continue maintaining this port. It's taking some time,
but we are still eager to see the support continue, and not just as a
base for Ubuntu.

[Alberto Garcia]

Post by Chris Hofstaedtler
b) various packages already ignore s390x (gnome? others?)

WebKitGTK still builds in s390x, but the Skia graphics library does
not support big-endian machines so if the Cairo backend is ever
dropped then we probably won't be able to support s390x any longer.

Berto

[1] https://github.com/WebKit/WebKit/blob/webkitgtk-2.47.1/Source/ThirdParty/skia/include/private/base/SkLoadUserConfig.h#L56