apt running on crashed gnome-terminal

[Enrico Zini]

Hello,

today I decided to upgrade from bookworm to trixie/testing[1][2]. I ran
the upgrade in a gnome-terminal, and of course all gnome terminals in
the system crashed halfway through the upgrade[1][3].

Unexpectedly, apt and dpkg kept doing their thing in their own headless
wonderland. I tried to track progress via top and pstree, killed a
couple of whiptail processes[4], and eventually decided to kill apt and
restart things from a new terminal.

I'm not sure how to start reporting bugs here. gnome-terminal should
clearly not have crashed (and I do not know why it did, and I'm not sure
I want to go through all the trouble to setup a bullworm[2] VM with
gnome and upgrade it to trixie to see if it crashed again.

At the same time, it was surprising that apt and dpkg continued unfazed
after the terminal died. That may be easier to reproduce: start doing
something with apt and kill gnome-terminal. I still need a bookseye[2]
VM with gnome to try that out, and I have an annoying cold, so it's not
something I'd embark on right this afternoon[5].

Not sure there's much more to be done here, given the complexity of the
whole stack, and especially if this affected just me just this once. If
it doesn't, this mail can act a friendly "hey, it wasn't just you" for a
fellow traveler.

Trixie otherwise just works, and thank us all for it! <3


Enrico

[1] One reason was to see if gnome is less infuriating[3]
[2] One reason was to finally get out of release names starting with
'b', which have been confusing the hell out of me for half of a
decade
[3] Spoiler warning: not particularly
[4] Likely warnings that this or that go or rust program or qt chromium
embedding monster cannot be supported by the Security Team, for
which I have heartfelt support and which however stopped the
upgrade flow at least 5 times
[5] Do we have a thing that clones the running system into a throwaway
VM?

[Mo Zhou]

Hi Enrico,

Dist upgrade is something I'd prefer to do in a terminal muxer, e.g.,
tmux, screen, zellij, or wrappers like byobu. Even if the whole
desktop environment crashed during the upgrade, you can still check
the status in tty without losing the process handling the stdout/err
of apt.

I think apt/dpkg are well-designed to be failure/fault-proof. In my
memory I did lots of reboot/poweroff during apt upgrade since forgotten.
Powerloss is also safe in my experience, which is also discussed in the
dpkg fsync() on -devel concurrently.

Back to gnome crash -- I did not encounter desktop crash during the
past 6 years of using Debian Sid as daily drive. Maybe there is
something wrong in detecting which service (like gdm) needs a restart?

Post by Enrico Zini
Hello,
today I decided to upgrade from bookworm to trixie/testing[1][2]. I ran
the upgrade in a gnome-terminal, and of course all gnome terminals in
the system crashed halfway through the upgrade[1][3].
Unexpectedly, apt and dpkg kept doing their thing in their own headless
wonderland. I tried to track progress via top and pstree, killed a
couple of whiptail processes[4], and eventually decided to kill apt and
restart things from a new terminal.
I'm not sure how to start reporting bugs here. gnome-terminal should
clearly not have crashed (and I do not know why it did, and I'm not sure
I want to go through all the trouble to setup a bullworm[2] VM with
gnome and upgrade it to trixie to see if it crashed again.
At the same time, it was surprising that apt and dpkg continued unfazed
after the terminal died. That may be easier to reproduce: start doing
something with apt and kill gnome-terminal. I still need a bookseye[2]
VM with gnome to try that out, and I have an annoying cold, so it's not
something I'd embark on right this afternoon[5].
Not sure there's much more to be done here, given the complexity of the
whole stack, and especially if this affected just me just this once. If
it doesn't, this mail can act a friendly "hey, it wasn't just you" for a
fellow traveler.
Trixie otherwise just works, and thank us all for it! <3
Enrico
[1] One reason was to see if gnome is less infuriating[3]
[2] One reason was to finally get out of release names starting with
'b', which have been confusing the hell out of me for half of a
decade
[3] Spoiler warning: not particularly
[4] Likely warnings that this or that go or rust program or qt chromium
embedding monster cannot be supported by the Security Team, for
which I have heartfelt support and which however stopped the
upgrade flow at least 5 times
[5] Do we have a thing that clones the running system into a throwaway
VM?

[Richard Lewis]

Post by Enrico Zini
today I decided to upgrade from bookworm to trixie/testing[1][2]. I ran
the upgrade in a gnome-terminal, and of course all gnome terminals in
the system crashed halfway through the upgrade[1][3].

not much consolation, but: i supose this is why the release-notes
recommend upgrading inside screen (see also:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1035401)

[Noah Meyerhans]

Post by Richard Lewis

Post by Enrico Zini
today I decided to upgrade from bookworm to trixie/testing[1][2]. I ran
the upgrade in a gnome-terminal, and of course all gnome terminals in
the system crashed halfway through the upgrade[1][3].

not much consolation, but: i supose this is why the release-notes
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1035401)

Yes, but the discussion in #1035401 and the corresponding MR shows that
this has limits as well.

IMO, an upgrade between releases should happen with as few layers of
software involved as possible. Local tty login or serial console is
ideal. This is, of course, not always feasible.

noah

1. https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1035401
2. https://salsa.debian.org/ddp-team/release-notes/-/merge_requests/213

[Hakan Bayındır]

Post by Enrico Zini
Hello,

Hi,

Post by Enrico Zini
today I decided to upgrade from bookworm to trixie/testing[1][2]. I ran
the upgrade in a gnome-terminal, and of course all gnome terminals in
the system crashed halfway through the upgrade[1][3].

This is strange. I’m not a GNOME user myself, but in the KDE land, Konsole (or any other app) never crashed during between similar large dist-upgrades.

Post by Enrico Zini
Unexpectedly, apt and dpkg kept doing their thing in their own headless
wonderland. I tried to track progress via top and pstree, killed a
couple of whiptail processes[4], and eventually decided to kill apt and
restart things from a new terminal.
I'm not sure how to start reporting bugs here. gnome-terminal should
clearly not have crashed (and I do not know why it did, and I'm not sure
I want to go through all the trouble to setup a bullworm[2] VM with
gnome and upgrade it to trixie to see if it crashed again.

From my perspective, the GNOME terminal shouldn’t behave like that. Probably recreating the crash and debugging it, and solving is eventually is the key. apt did what it was designed to do, and did it well, apparently.

Post by Enrico Zini
At the same time, it was surprising that apt and dpkg continued unfazed
after the terminal died. That may be easier to reproduce: start doing
something with apt and kill gnome-terminal. I still need a bookseye[2]
VM with gnome to try that out, and I have an annoying cold, so it's not
something I'd embark on right this afternoon[5].

apt has some superpowers which allows it to continue when the terminal it has been reporting to disappears. This allows safe upgrades over flaky SSH sessions. The best practice is to run a terminal multiplexer before these long and big operations, but apt is more resilient than it looks.

Post by Enrico Zini
Not sure there's much more to be done here, given the complexity of the
whole stack, and especially if this affected just me just this once. If
it doesn't, this mail can act a friendly "hey, it wasn't just you" for a
fellow traveler.
Trixie otherwise just works, and thank us all for it! <3

Trixie is already in a pretty good shape from my experience, yes, and thanks everyone for it, too.

Have a nice year,

Hakan

Post by Enrico Zini
Enrico
[1] One reason was to see if gnome is less infuriating[3]
[2] One reason was to finally get out of release names starting with
'b', which have been confusing the hell out of me for half of a
decade
[3] Spoiler warning: not particularly
[4] Likely warnings that this or that go or rust program or qt chromium
embedding monster cannot be supported by the Security Team, for
which I have heartfelt support and which however stopped the
upgrade flow at least 5 times
[5] Do we have a thing that clones the running system into a throwaway
VM?
--

[Richard Lewis]

Post by Enrico Zini
[5] Do we have a thing that clones the running system into a throwaway
VM?

systemd-nspawn claims to support this -- https://0pointer.net/blog/running-an-container-off-the-host-usr.html

[Jochen Sprickerhof]

Hi Enrico,

Post by Enrico Zini
Unexpectedly, apt and dpkg kept doing their thing in their own headless
wonderland. I tried to track progress via top and pstree, killed a
couple of whiptail processes[4], and eventually decided to kill apt and
restart things from a new terminal.

You could also just do:

tail -f /var/log/apt/term.log

Cheers Jochen

[Tzafrir Cohen]

Hi,

Post by Enrico Zini
At the same time, it was surprising that apt and dpkg continued unfazed
after the terminal died. That may be easier to reproduce: start doing
something with apt and kill gnome-terminal. I still need a bookseye[2]
VM with gnome to try that out, and I have an annoying cold, so it's not
something I'd embark on right this afternoon[5].

Not that you can install it with an apt install running, but reptyr
helped me recover somewhat similar issues and seems to be a tool worth
having installed. I had not had a chance to try using it to recover an
apt install.

[Gabor Gombas]

Post by Enrico Zini
today I decided to upgrade from bookworm to trixie/testing[1][2]. I ran
the upgrade in a gnome-terminal, and of course all gnome terminals in
the system crashed halfway through the upgrade[1][3].

Foreword: I'm pretty happy with Gnome Terminal and use it exclusively
for many many years now. That said...

Gnome Terminal nowadays has a server architecture. That means even if
you open many terminals, there is still just a single backend process
controlling all. So "all gnome terminals" really refers to that single
backend process (well, one per user, but if you were doing a dist
upgrade, then I doubt you had multiple users logged in via different
locally connected monitors/keyboards).

If that single process quit unexpectedly, then you should see systemd
recording that in the journal (journalctl --user), and if it was a true
crash instead of a controlled abort(), then the kernel should also have
logged it.

And it's also possible that there was nothing wrong with Gnome Terminal
itself. Nowadays there is a lot of infrastructure around it - and I'm
sure nobody tested all the potential error paths which could be hit if
some parts of that infrastructure disappear/restart and the new version
of a given piece is not necessarily 100% compatible with the previous
version.

Regards,
Gabor