Discussion:
Improvement of headless server upgrades
(too old to reply)
Helmut K. C. Tessarek
2025-03-05 02:40:01 UTC
Permalink
This is my first mail in a Debian mailing list and I hope I've chosen
the correct one. There are way too many lists thus please direct me to
the correct one, in case I messed up.

I would like to make a suggestion for release upgrades. It should not be
a huge effort to implement and I can work with someone to make this happen.

In the last few days I ran into a serious issue when upgrading to newer
releases on 3 headless servers: the network connection went dead.
In the first situation the interface name changed from eth0 to end0 and
after the reboot my adapter got a link-local address.
In the second situation dhcpcd was replaced by Network Manager and once
again the network was dead.

Please don't get me wrong, I am not against keeping up with the times
and use new technology. Far from it.
But in both cases I had to connect my headless machines to a monitor and
keyboard and fix the network issues. Usually this is not a problem, but
sometimes it might be impossible.
In both cases I did not even know that these things would happen. The
dist-upgrade made the changes without my input and I was left in the
dark. Iamgine my surprise when I couldn't connect to my boxes anymore.

Both network "outages" could have been prevented by adding a note at the
end of the dist-upgrade output.

e.g. something like the following (monospace font required for the
"Attention" text):

_ _____ _____ _____ _ _ _____ ___ ___ _ _
/ \|_ _|_ _| ____| \ | |_ _|_ _/ _ \| \ | |
/ _ \ | | | | | _| | \| | | | | | | | | \| |
/ ___ \| | | | | |___| |\ | | | | | |_| | |\ |
/_/ \_\_| |_| |_____|_| \_| |_| |___\___/|_| \_|

Network interface name changed: please update config files before reboot.

_ _____ _____ _____ _ _ _____ ___ ___ _ _
/ \|_ _|_ _| ____| \ | |_ _|_ _/ _ \| \ | |
/ _ \ | | | | | _| | \| | | | | | | | | \| |
/ ___ \| | | | | |___| |\ | | | | | |_| | |\ |
/_/ \_\_| |_| |_____|_| \_| |_| |___\___/|_| \_|

Network subsystem changed: please add a system connection before reboot.

Is this something that makes sense? Often you have a remote console like
an iLO or whatever cloud systems provide. But in some cases there is
nothing. No console, no monitor, no keyboard. Only a power and a network
cable.

Cheers,
K. C.

P.S.: I go by KC, trying to avoid the Spaceballs Dark Helmet mixup.


--
regards Helmut K. C. Tessarek KeyID 0x172380A011EF4944
Key fingerprint = 8A55 70C1 BD85 D34E ADBC 386C 1723 80A0 11EF 4944

/*
Thou shalt not follow the NULL pointer for chaos and madness
await thee at its end.
*/
Fabio Fantoni
2025-03-05 09:30:01 UTC
Permalink
Post by Helmut K. C. Tessarek
This is my first mail in a Debian mailing list and I hope I've chosen
the correct one. There are way too many lists thus please direct me to
the correct one, in case I messed up.
I would like to make a suggestion for release upgrades. It should not
be a huge effort to implement and I can work with someone to make this
happen.
In the last few days I ran into a serious issue when upgrading to
newer releases on 3 headless servers: the network connection went dead.
In the first situation the interface name changed from eth0 to end0
and after the reboot my adapter got a link-local address.
In the second situation dhcpcd was replaced by Network Manager and
once again the network was dead.
Hi, when you do upgrade to a new major version you should before read
release note for important changes, also read NEWS on packages upgrade
(that contain important changes, including the one that can require
manual changes). It is also good to test upgrade on a test server or on
a minor one for the first time on any major version upgrade, so you can
spot other possible unforeseen events.

These network changes are documented and well known, I have done many
Debian server upgrades over the years and for example those network
changes I had seen before by documenting myself and I had made sure to
modify correctly before the upgrade and reboot to avoid unexpected
events. Alternatively you can also modify to force the old names.
Post by Helmut K. C. Tessarek
Please don't get me wrong, I am not against keeping up with the times
and use new technology. Far from it.
But in both cases I had to connect my headless machines to a monitor
and keyboard and fix the network issues. Usually this is not a
problem, but sometimes it might be impossible.
In both cases I did not even know that these things would happen. The
dist-upgrade made the changes without my input and I was left in the
dark. Iamgine my surprise when I couldn't connect to my boxes anymore.
Both network "outages" could have been prevented by adding a note at
the end of the dist-upgrade output.
e.g. something like the following (monospace font required for the
    _  _____ _____ _____ _   _ _____ ___ ___  _   _
   / \|_   _|_   _| ____| \ | |_   _|_ _/ _ \| \ | |
  / _ \ | |   | | |  _| |  \| | | |  | | | | |  \| |
 / ___ \| |   | | | |___| |\  | | |  | | |_| | |\  |
/_/   \_\_|   |_| |_____|_| \_| |_| |___\___/|_| \_|
Network interface name changed: please update config files before reboot.
    _  _____ _____ _____ _   _ _____ ___ ___  _   _
   / \|_   _|_   _| ____| \ | |_   _|_ _/ _ \| \ | |
  / _ \ | |   | | |  _| |  \| | | |  | | | | |  \| |
 / ___ \| |   | | | |___| |\  | | |  | | |_| | |\  |
/_/   \_\_|   |_| |_____|_| \_| |_| |___\___/|_| \_|
Network subsystem changed: please add a system connection before reboot.
Is this something that makes sense? Often you have a remote console
like an iLO or whatever cloud systems provide. But in some cases there
is nothing. No console, no monitor, no keyboard. Only a power and a
network cable.
Cheers,
  K. C.
P.S.: I go by KC, trying to avoid the Spaceballs Dark Helmet mixup.
Marc Haber
2025-03-05 15:10:01 UTC
Permalink
On Tue, 4 Mar 2025 20:39:43 -0500, "Helmut K. C. Tessarek"
Post by Helmut K. C. Tessarek
Both network "outages" could have been prevented by adding a note at the
end of the dist-upgrade output.
e.g. something like the following (monospace font required for the
_ _____ _____ _____ _ _ _____ ___ ___ _ _
/ \|_ _|_ _| ____| \ | |_ _|_ _/ _ \| \ | |
/ _ \ | | | | | _| | \| | | | | | | | | \| |
/ ___ \| | | | | |___| |\ | | | | | |_| | |\ |
/_/ \_\_| |_| |_____|_| \_| |_| |___\___/|_| \_|
Network interface name changed: please update config files before reboot.
Why would people who do not read the NEWS.Debian or the Release Notes
read that? And how would you solve the issue of people wanting more
and more of those attention banners?

Greetings
Marc
--
----------------------------------------------------------------------------
Marc Haber | " Questions are the | Mailadresse im Header
Rhein-Neckar, DE | Beginning of Wisdom " |
Nordisch by Nature | Lt. Worf, TNG "Rightful Heir" | Fon: *49 6224 1600402
Michael Banck
2025-03-05 15:20:01 UTC
Permalink
Hi,
Post by Marc Haber
On Tue, 4 Mar 2025 20:39:43 -0500, "Helmut K. C. Tessarek"
Post by Helmut K. C. Tessarek
Both network "outages" could have been prevented by adding a note at the
end of the dist-upgrade output.
e.g. something like the following (monospace font required for the
_ _____ _____ _____ _ _ _____ ___ ___ _ _
/ \|_ _|_ _| ____| \ | |_ _|_ _/ _ \| \ | |
/ _ \ | | | | | _| | \| | | | | | | | | \| |
/ ___ \| | | | | |___| |\ | | | | | |_| | |\ |
/_/ \_\_| |_| |_____|_| \_| |_| |___\___/|_| \_|
Network interface name changed: please update config files before reboot.
Why would people who do not read the NEWS.Debian or the Release Notes
read that? And how would you solve the issue of people wanting more
and more of those attention banners?
Presumably because it will be written directly to their terminal.

Things in NEWS.Debian or the release notes probably only can caution
users to be aware of possible problems; if we are able to detect that
there will be a problem and warn the person running the command, that
sounds sensible to me, but I have not looked at the details.


Michael
Bjørn Mork
2025-03-05 16:50:01 UTC
Permalink
Post by Michael Banck
Post by Marc Haber
On Tue, 4 Mar 2025 20:39:43 -0500, "Helmut K. C. Tessarek"
Post by Helmut K. C. Tessarek
Both network "outages" could have been prevented by adding a note at the
end of the dist-upgrade output.
e.g. something like the following (monospace font required for the
_ _____ _____ _____ _ _ _____ ___ ___ _ _
/ \|_ _|_ _| ____| \ | |_ _|_ _/ _ \| \ | |
/ _ \ | | | | | _| | \| | | | | | | | | \| |
/ ___ \| | | | | |___| |\ | | | | | |_| | |\ |
/_/ \_\_| |_| |_____|_| \_| |_| |___\___/|_| \_|
Network interface name changed: please update config files before reboot.
Why would people who do not read the NEWS.Debian or the Release Notes
read that? And how would you solve the issue of people wanting more
and more of those attention banners?
Presumably because it will be written directly to their terminal.
apt install apt-listchanges


Bjørn
Matthias Urlichs
2025-03-05 18:30:02 UTC
Permalink
Post by Bjørn Mork
apt install apt-listchanges
To be fair, apt-listchanges lists a whole lot of changes, esp. when you
do a dist-upgrade.

Noticing the one change among the umpteen more-or-less-major NEWS
entries that actually affects the ability of your system to safely
reboot with the same network configuration is not a trivial task, even
for reasonably experienced sysadmins.

--
-- regards
--
-- Matthias Urlichs
Helmut K. C. Tessarek
2025-03-05 19:40:01 UTC
Permalink
Post by Matthias Urlichs
Noticing the one change among the umpteen more-or-less-major NEWS
entries that actually affects the ability of your system to safely
reboot with the same network configuration is not a trivial task, even
for reasonably experienced sysadmins.
I couldn't agree more.

--
regards Helmut K. C. Tessarek KeyID 0x172380A011EF4944
Key fingerprint = 8A55 70C1 BD85 D34E ADBC 386C 1723 80A0 11EF 4944

/*
Thou shalt not follow the NULL pointer for chaos and madness
await thee at its end.
*/
Marc Haber
2025-03-05 19:40:02 UTC
Permalink
Post by Matthias Urlichs
Noticing the one change among the umpteen more-or-less-major NEWS
entries that actually affects the ability of your system to safely
reboot with the same network configuration is not a trivial task, even
for reasonably experienced sysadmins.
It is also a challenge for a package maintainer to judge WHEN to drop
a higher priority NEWS entry.

I think that our release notes recommend one or another way for console
access for upgrades for exactly this reason.

Greetings
Marc
--
-----------------------------------------------------------------------------
Marc Haber | "I don't trust Computers. They | Mailadresse im Header
Leimen, Germany | lose things." Winona Ryder | Fon: *49 6224 1600402
Nordisch by Nature | How to make an American Quilt | Fax: *49 6224 1600421
Soren Stoutner
2025-03-05 20:00:01 UTC
Permalink
Post by Marc Haber
Post by Matthias Urlichs
Noticing the one change among the umpteen more-or-less-major NEWS
entries that actually affects the ability of your system to safely
reboot with the same network configuration is not a trivial task, even
for reasonably experienced sysadmins.
It is also a challenge for a package maintainer to judge WHEN to drop
a higher priority NEWS entry.
I think that our release notes recommend one or another way for console
access for upgrades for exactly this reason.
I would not be opposed to some type of special warning that an upgrade would
break remote access on a particular machine if it were possible to produce the
warning reliably, but I don't think it would be that easy to implement.
Sometimes it might be easy to detect, but I think a lot of times it isn't easy
for Debian to know when the change will cause breakage or not, and if people
become dependent on these warnings instead of actually reading the release
notes and the NEWS entries, then I can see this causing more problems than it
solves.

If you are doing an upgrade on a remote or headless system, it probably
behooves your time to read over the release notes and the NEWS entries to
understand how changes may affect your connectivity. As a human you are much
more likely to know if the change will break remote access than any automated
check Debian can produce.

Now, if any of the breakages being discussed were not sufficiently documented in
the release notes or NEWS entries, that is a different issue and one which
should probably be addressed directly to the package that made the breaking
change.
--
Soren Stoutner
***@debian.org
Bjørn Mork
2025-03-05 20:10:01 UTC
Permalink
Post by Matthias Urlichs
Post by Bjørn Mork
apt install apt-listchanges
To be fair, apt-listchanges lists a whole lot of changes, esp. when
you do a dist-upgrade.
Noticing the one change among the umpteen more-or-less-major NEWS
entries that actually affects the ability of your system to safely
reboot with the same network configuration is not a trivial task, even
for reasonably experienced sysadmins.
I agree. But I don't see what makes the proposal any better.

We can (and do, AFAIK) discuss which items belong in NEWS. But there
are many more-or-less-major changes which might require attention when
you do a dist-upgrade. Filtering this list down to one or two items is
not realistic. And prefixing the list with ATTENTION won't make the job
any easier.


Bjørn
Marc Haber
2025-03-05 20:40:01 UTC
Permalink
I doubt there are more than one or at max two items that would break
network connectivity.
And which of the millions of changes would that be that would break YOUR
system?
--
-----------------------------------------------------------------------------
Marc Haber | "I don't trust Computers. They | Mailadresse im Header
Leimen, Germany | lose things." Winona Ryder | Fon: *49 6224 1600402
Nordisch by Nature | How to make an American Quilt | Fax: *49 6224 1600421
Matthias Urlichs
2025-03-05 21:30:01 UTC
Permalink
Post by Marc Haber
And which of the millions of changes would that be that would break
YOUR system?
The number of changes that can adversely impact your ability to boot
*or* your network connectivity (but which won't fail the upgrade!) can
safely be assumed to be smaller than five or so.

Anything else can be fixed post-upgrade.

--
-- regards
--
-- Matthias Urlichs
Marc Haber
2025-03-06 08:00:04 UTC
Permalink
Post by Matthias Urlichs
Post by Marc Haber
And which of the millions of changes would that be that would break
YOUR system?
The number of changes that can adversely impact your ability to boot
*or* your network connectivity (but which won't fail the upgrade!) can
safely be assumed to be smaller than five or so.
And which of the millions are those five? Any small number of "false
alerts" already will cause users to stop reading.

Maybe we could introduce a new header field where a package might create
itself as "probably harmful to the reboot" (like grub, initramfs-tools,
network-manager, systemd etc) and have their NEWS entries sorted first
by apt-listchanges.

But I think that it is unrealistic to expect maintainers to classify
their changes whether they might affect reboot.

Running a headless server without console access is a challenge. You
_really_ need to know your way to debug such a situation. I don't think
that we as a distribtion can change that.

In the current case, it's only a matter of hooking up a display and
keyboard to a physically accessible system. It could be a housing server
in a datacenter next time (been there, done that).

Greetings
Marc
--
-----------------------------------------------------------------------------
Marc Haber | "I don't trust Computers. They | Mailadresse im Header
Leimen, Germany | lose things." Winona Ryder | Fon: *49 6224 1600402
Nordisch by Nature | How to make an American Quilt | Fax: *49 6224 1600421
Helmut K. C. Tessarek
2025-03-05 20:40:01 UTC
Permalink
Post by Bjørn Mork
We can (and do, AFAIK) discuss which items belong in NEWS. But there
are many more-or-less-major changes which might require attention when
you do a dist-upgrade. Filtering this list down to one or two items is
not realistic. And prefixing the list with ATTENTION won't make the job
any easier.
I doubt there are more than one or at max two items that would break
network connectivity.

In both cases it was only one item that broke my ability to connect to
the rebooted machine.
After the dist-upgrade finishes, people would usually type reboot and
hit enter.
If above the current prompt (where you enter 'reboot'), something like

_ _____ _____ _____ _ _ _____ ___ ___ _ _
/ \|_ _|_ _| ____| \ | |_ _|_ _/ _ \| \ | |
/ _ \ | | | | | _| | \| | | | | | | | | \| |
/ ___ \| | | | | |___| |\ | | | | | |_| | |\ |
/_/ \_\_| |_| |_____|_| \_| |_| |___\___/|_| \_|

or

____ _____ ___ ____ ____ _____ ___ ____
/ ___|_ _/ _ \| _ \ / ___|_ _/ _ \| _ \
\___ \ | || | | | |_) | \___ \ | || | | | |_) |
___) || || |_| | __/ ___) || || |_| | __/
|____/ |_| \___/|_| |____/ |_| \___/|_|

shows up, I am pretty sure it catches your attention.

Cheers,
K. C.

--
regards Helmut K. C. Tessarek KeyID 0x172380A011EF4944
Key fingerprint = 8A55 70C1 BD85 D34E ADBC 386C 1723 80A0 11EF 4944

/*
Thou shalt not follow the NULL pointer for chaos and madness
await thee at its end.
*/
Helmut K. C. Tessarek
2025-03-05 19:40:02 UTC
Permalink
Hello,

Thaank you for all the replies so far.
Post by Michael Banck
Presumably because it will be written directly to their terminal.
Yes, this is one of the reasons. During a dist-upgrade you receive the
changes, but it's many, many pages long and spotting the entry that
renders your system dead is not easy.

I am not asking to create multiple ATTENTION banners at the end. All
issues can be fixed as long as I can connect to the machine.
Thus anything that would prevent the machine from booting (e.g. if it's
known that the machine would not boot unless a grub-install is done) or
having a functioning network connection, should be mentioned explicitly
at the end.

The subject specifically points out an improvement for headless server
upgrades. Chances are that these "special" notifications only happen now
and then.

But it would certainly make the upgrade experience better.

Cheers,
K. C.

P.S.: I admit that reading NEWS/UPGRADE files and articles is also
important and that doing test migrations is best practices. On the other
hand, it's not always possible to have a test bed for every single setup.

--
regards Helmut K. C. Tessarek KeyID 0x172380A011EF4944
Key fingerprint = 8A55 70C1 BD85 D34E ADBC 386C 1723 80A0 11EF 4944

/*
Thou shalt not follow the NULL pointer for chaos and madness
await thee at its end.
*/
Marc Haber
2025-03-06 09:10:02 UTC
Permalink
On Wed, 5 Mar 2025 14:36:20 -0500, "Helmut K. C. Tessarek"
Post by Helmut K. C. Tessarek
On the other
hand, it's not always possible to have a test bed for every single setup.
It is way easier to have a test bed or to arrange for console access
than even trying to cater for every situation that might happen. Rest
assured that package maintainers already try to avoid situations like
that to the best of their knowledge.

Greetings
Marc
--
----------------------------------------------------------------------------
Marc Haber | " Questions are the | Mailadresse im Header
Rhein-Neckar, DE | Beginning of Wisdom " |
Nordisch by Nature | Lt. Worf, TNG "Rightful Heir" | Fon: *49 6224 1600402
Marc Haber
2025-03-06 14:50:01 UTC
Permalink
P.S.: This failure mode isn't even documented in the release notes.
But they do, don't they, recommend to have console access available
during an upgrade right?

Greetings
Marc
--
----------------------------------------------------------------------------
Marc Haber | " Questions are the | Mailadresse im Header
Rhein-Neckar, DE | Beginning of Wisdom " |
Nordisch by Nature | Lt. Worf, TNG "Rightful Heir" | Fon: *49 6224 1600402
Richard Lewis
2025-03-07 00:50:01 UTC
Permalink
Post by Marc Haber
On Tue, 4 Mar 2025 20:39:43 -0500, "Helmut K. C. Tessarek"
Post by Helmut K. C. Tessarek
Network interface name changed: please update config files before reboot.
P.S.: This failure mode isn't even documented in the release notes.
This does seem to be correct given that the OP was upgrading from buster
to bullseye. however i dont recall seeing any bugs about this in those
releases, and it seems unlikely such a big issue was missed?

changes to network naming were documented in previous release-notes including:
- buster release notes at 4.1.6 (https://www.debian.org/releases/buster/amd64/release-notes.en.txt)
- the stretch release notes at 2.2.9. (https://www.debian.org/releases/stretch/amd64/release-notes.en.txt)

so i wonder if this is perhaps fall-out from not following the
instructions on earlier upgrades?

Every version of the release notes also has sections advising on how to
upgrade over ssh (4.1.4) and probably other places in part 4 (which i
always find harder to read than expected, but the advice is quite good)
Helmut Grohne
2025-03-06 05:50:01 UTC
Permalink
This is my first mail in a Debian mailing list and I hope I've chosen the
correct one. There are way too many lists thus please direct me to the
correct one, in case I messed up.
As the matter is intersecting multiple packages, it isn't the worst
choice.
In the last few days I ran into a serious issue when upgrading to newer
releases on 3 headless servers: the network connection went dead.
In the first situation the interface name changed from eth0 to end0 and
after the reboot my adapter got a link-local address.
I am surprised to see this happen. Back in older times, interfaces used
to be named like eth0, but that should not be happening since quite a
number of stable releases. Those old systems that still use eth0 today
tend to do it due to a file /etc/udev/rules.d/70-persistent-net.rules.
Does that exist and list your interface on the affected system? If not,
can you try figuring out why it was still named eth0 before upgrading?

In principle, I expect that this is an unusual circumstance and
therefore do not think it is worth mitigating.
In the second situation dhcpcd was replaced by Network Manager and once
again the network was dead.
Can you try figuring out what dependency caused Network Manager to be
installed? Your apt or dpkg logs may be helpful here.
Both network "outages" could have been prevented by adding a note at the end
of the dist-upgrade output.
There are two difficulties in this proposal. One is finding a
responsible package for printing this note and the other is properly
detecting the changed network interface name before doing the reboot
that actually changes it.

As a result of these, I suggest investing time into making the described
situations less likely rather than printing a warning.

Helmut
Helmut K. C. Tessarek
2025-03-06 23:10:01 UTC
Permalink
Post by Helmut Grohne
I am surprised to see this happen. Back in older times, interfaces used
to be named like eth0, but that should not be happening since quite a
number of stable releases. Those old systems that still use eth0 today
tend to do it due to a file /etc/udev/rules.d/70-persistent-net.rules.
Does that exist and list your interface on the affected system? If not,
can you try figuring out why it was still named eth0 before upgrading?
It was buster to bullseye. A machine I had forgotten since it was
running perfectly.
It was my fault that I didn't read the 3000 lines of NEWS for the
upgrade, but I have done buster upgrades before and I didn't lose
network connectivity then. I forgot that the ifname would change.
Post by Helmut Grohne
Can you try figuring out what dependency caused Network Manager to be
installed? Your apt or dpkg logs may be helpful here.
I am looking into it.

Cheers,
K. C.


--
regards Helmut K. C. Tessarek KeyID 0x172380A011EF4944
Key fingerprint = 8A55 70C1 BD85 D34E ADBC 386C 1723 80A0 11EF 4944

/*
Thou shalt not follow the NULL pointer for chaos and madness
await thee at its end.
*/
Holger Levsen
2025-03-06 08:50:01 UTC
Permalink
Both network "outages" could have been prevented by adding a note at the end
of the dist-upgrade output.
they could also have been prevented by reading the release notes and following
their advice.

that would also have prevented this wonderful bikeshedding thread.

sorry to state the obvious.
--
cheers,
Holger

⢀⣎⠟⠻⢶⣊⠀
⣟⠁⢠⠒⠀⣿⡁ holger@(debian|reproducible-builds|layer-acht).org
⢿⡄⠘⠷⠚⠋⠀ OpenPGP: B8BF54137B09D35CF026FE9D 091AB856069AAA1C
⠈⠳⣄

It blows my mind that the people who rage about the spike proteins don't rage
about governments encouraging the spread of self-replicating auto-mutating
aerosolised spike proteins. (@1goodtern)
Fabio Fantoni
2025-03-06 10:30:01 UTC
Permalink
Post by Holger Levsen
Both network "outages" could have been prevented by adding a note at the end
of the dist-upgrade output.
they could also have been prevented by reading the release notes and following
their advice.
that would also have prevented this wonderful bikeshedding thread.
sorry to state the obvious.
I think the things to do can be summed up as follows: read release note
of new major version and NEWS entries

If you are upgrading important or critical systems try also:

- test before in a testing system, or vm. or even just start from a less
critical one

- try to have additional access in case of unforeseen events (there are
not only the network ones mentioned in this discussion that can prevent
the boot but also other undocumented and/or specific ones to the system
or to a part of it). I mean for example an access from the host in case
of vm or an ipmi system in case of physical server.

- if you don't have much time to read the whole release note read at
least the "Upgrade specific items" part, for example for buster there is
"5.1.6. Migrating from legacy network interface names".

- if you don't have time to read NEWS every update do it at least for
the first time so you can see all the common ones on the basic packages,
for example however it can be useful to have a quick look at each update
and read those of specific programs to be aware of important or
essential changes and save time rather than having problems later,
perhaps not noticing the problems immediately or not being able to find
the cause immediately


Regarding possible improvements I think the only thing that can be added
is a check to the upgrade operations and if it is detected that you are
upgrading to packages of the next major version add a simple initial
warning (but not a big "image" as suggested in topic start) in which it
is recommended to read the release notes, especially the "Upgrade
specific items" part, and the NEWS of the packages.
Wookey
2025-03-06 14:20:01 UTC
Permalink
Post by Holger Levsen
Both network "outages" could have been prevented by adding a note at the end
of the dist-upgrade output.
they could also have been prevented by reading the release notes and following
their advice.
Would it? Do we know why these things happened? K.C. does not say what
he was upgrading from/to (so it wasn't a very useful report in that
regard), but there has certainly been a long-term expectation that
headless upgrades will work (and in my experience of 25 years now they
always do (well done everyone - I am regularly impressed at how this
usually doesn't break).

Which entry in which release notes will warn that this time
(presumably - was this an upgrade to unstable K.C. or to bookworm, or
something else?) the (pretty old now) eth0 -> 'annoying, unmemorable,
but ordered and unique', renaming will/might actually break your
config? Or that dhcpd will be replaced by network manager in such a
way that things break? (Is the mechanism here that the server was
running dhcpd to dish out addresses so now in fact the server is
working but other machines are not getting addresses?

I do read the release notes before the first upgrade to a new release,
but I wouldn't be expecting either of the mentioned things to break so
I'm not (yet) convinced that 'RTFM' is a fair response here. I do
vaguely recall that one version of DHCP (isc-dhcp?) was being retired
(did that happen for bookworm?). But normally debian upgrades do not
replace your existing packages, precisely because they might be doing
something important.

I've just looked over the release notes for upgrading to bookworm and
whilst here is loads of good advice about checking for obsolete
packages, noting removals, making backups etc, it is largely generic
and relies on the user knowing what removed packages do. I didn't see
anything specific which warned about the 2 issues noted, and the guide
is pretty long these days, so some skimming the 3rd time you upgrade
is inevitable. So yeah, would RTFM really have avoided these problems?
Maybe, maybe not.

In general I would echo Helmut's response: it is better to work out
why this happenned and try to prevent it, than to add more notices, as
we do have some notice mechanisms already. But they are old, and
expectations change so it's not crazy to ask if they are still sufficient.

But equally, starting with 'it's your fault' seems unhelpful and
possibly unfair. I look forward to some answers in response to Helmut,
and we'll find out if there are real issues to address or not.

Wookey
--
Principal hats: Debian, Wookware
http://wookware.org/
Andrey Rakhmatullin
2025-03-06 16:40:01 UTC
Permalink
Post by Wookey
Which entry in which release notes will warn that this time
(presumably - was this an upgrade to unstable K.C. or to bookworm, or
something else?) the (pretty old now) eth0 -> 'annoying, unmemorable,
but ordered and unique', renaming will/might actually break your
config?
4.1.6 in https://www.debian.org/releases/buster/amd64/release-notes.en.txt
(I cannot link to a HTML version because it doesn't look like it exists
anymore, or I cannot Google it, which is the same thing in our case).
--
WBR, wRAR
Bjørn Mork
2025-03-06 17:00:01 UTC
Permalink
Post by Wookey
Which entry in which release notes will warn that this time
(presumably - was this an upgrade to unstable K.C. or to bookworm, or
something else?) the (pretty old now) eth0 -> 'annoying, unmemorable,
but ordered and unique', renaming will/might actually break your
config?
There probably hasn't been a warning since buster, when the release
notes [1] warned:

4.1.6. Verify network interface name support

Systems upgraded from older releases that still use network
interfaces with names like eth0 or wlan0 are at risk of losing
networking once they switch to buster; see Section 5.1.6,
“Migrating from legacy network interface names” for migration
instructions.


Maybe all release notes after this should have warned about using
systemd unpredictable interface names on headless systems? Particularily
on systems with a single network interface, where such breakage is
completely unnecessary and "eth0" is guaranteed to be stable from kernel
to kernel.

We should probably also advise users of headless systems with multiple
interfaces that they need to manually configure names for their
interfaces. Personally I prefer more descriptive names like "uplink",
"backend" etc. Avoids the systemd breakage and makes network config
easier to read/debug.


Bjørn

[1] - https://www.debian.org/releases/buster/amd64/release-notes.en.txt
Helmut K. C. Tessarek
2025-03-06 23:00:01 UTC
Permalink
Post by Wookey
Would it? Do we know why these things happened? K.C. does not say what
he was upgrading from/to (so it wasn't a very useful report in that
regard), but there has certainly been a long-term expectation that
headless upgrades will work (and in my experience of 25 years now they
always do (well done everyone - I am regularly impressed at how this
usually doesn't break).
It was a Debian Buster on armv7l that I had forgotten, because it was
running perfectly until now, but apps complained that the OS is no
longer supported. That upgrade changed the ifname.
(The other issue happened either from bullseye to bookworm or after.)

I have done upgrades from buster to bullseye before (on amd64) and I
never lost network connectivity even though the network interface name
changed.
Maybe because those machines already used a different network setup. I
don't really know. (It's been a while.)
This was the reason I was so startled that I couldn't connect anymore.
Yes, I had done updates where the ifname changed, but sorry that I
forgot that fact, since it was rather a long time ago (epecially since I
still had network access in my previous upgradres where that happened).

But I can only agree and I am equally impressed as Wookey how well
dist-upgrades work.

I maybe should have mentioned that I have done countless updates before
and never ran into any issues. Or at least I knew beforehand that I had
to make changes that something like that might not happen.

Please note that I was not trying to blame Debian or package managers or
anyone really. This was clearly my fault. I started this thread to begin
a discussion whether there is an option to prevent something like that.
An option that doesn't require me to read through 3000 lines of text. If
not, it's fine.
If yes, it might be awesome to do so. That's all.

Cheers,
K. C.

--
regards Helmut K. C. Tessarek KeyID 0x172380A011EF4944
Key fingerprint = 8A55 70C1 BD85 D34E ADBC 386C 1723 80A0 11EF 4944

/*
Thou shalt not follow the NULL pointer for chaos and madness
await thee at its end.
*/
Helmut K. C. Tessarek
2025-03-06 23:20:01 UTC
Permalink
Post by Helmut K. C. Tessarek
It was a Debian Buster on armv7l that I had forgotten, because it
was running perfectly until now, but apps complained that the OS is
no longer supported.
To clarify my previous statement:
The upgrade to bullseye changed the ifname.

Buster used eth0.
After the rebooot, bulsseye used a different ifname.


--
regards Helmut K. C. Tessarek KeyID 0x172380A011EF4944
Key fingerprint = 8A55 70C1 BD85 D34E ADBC 386C 1723 80A0 11EF 4944

/*
Thou shalt not follow the NULL pointer for chaos and madness
await thee at its end.
*/
Loading...