Niels Thykier
2024-11-29 10:10:01 UTC
Reply
PermalinkWe would like to propose a change in default for `dpkg`, which will
enable `rootless` by default (changing it from "opt-in" to "opt-out").
This would be a considerable milestone bringing us to into 90-99%
territory of removing the need for root (`fakeroot` or otherwise)
in the Debian packaging stack.
While my long term goal is the complete removal of `fakeroot` as a
mandatory piece in our packaging stack, it is *not* the goal of this
MBF. For this MBF, the focus is on removing blockers to
switching the default rather than removing use of `fakeroot` in
general. This is also why one the solutions we propose to the "busy
maintainer" is an "opt-out" rather than "fix the root cause".
# The quick summary
* About 10000 packages build successfully with this change.
* About 225 bugs (~2%) would be filed at severity important due to
problems detected with these changes.
* About 650 packages (~6.5%) had an unrelated FTBFS in sid when the
test where done that could hide other failures.
* Four packages could not build on the test infrastructure.
* Two packages were identified to misbuild. These already have bugs
filed (one misbuild without the change due to a bug in packaging;
the other will misbuild with the change).
# The bug template used
Severity: important
User: ***@thykier.net
Usertags: rrr-no-as-default-issue
Dear maintainer,
During a test rebuild for building packages "Rules-Requires-Root: no" as
the default in `dpkg`, {{ src.name }} failed to rebuild.
Log Summary:
-------------------------------------------------------------------------------
[...]
{{ log_snippet }}
-------------------------------------------------------------------------------
The above is just how the build ends and not necessarily the most
relevant part. If required, the full build log is available here (for
the next 30
days):
https://people.debian.org/~nthykier/rrr-no-as-default/logs/{{
build.log_artifact_id }}.gz
You can find common solutions at
https://people.debian.org/~nthykier/rrr-no-as-default/docs/solutions.md
If this is really a bug in one of the build-depends, please use
reassign and affects, so that this is still visible in the BTS web
page for this package.
If this package is listed in
https://people.debian.org/~nthykier/rrr-no-as-default/docs/static-ownership.list,
then please just set `Rules-Requires-Root: binary-targets` to the source
stanza of `debian/control` as a fix to this bug.
If this package is listed in
https://people.debian.org/~nthykier/rrr-no-as-default/docs/maybe-misbuilds.list,
then the package was deemed at risk for misbuilding (having wrong
ownership) but had a FTBFS problem we tested it. Please test whether the
package works with `Rules-Requires-Root: no` validating that the
resulting deb has the correct ownership for all paths in the deb.
Thanks,
# Affected packages
We have sorted the affected packages into the following groups:
1) Packages with static non-`root:root` ownership. These will
generally need a `Rules-Requires-Root: binary-targets` for now, and
then they are done.
2) Packages that do not have any non-`root:root` ownership. These can
often be converted to `Rules-Requires-Root: no` with a bit of
tweaking (see attached solutions.md). However, in a pinch, you can
just set `Rules-Requires-Root: binary-targets` to get off the list
of blockers and come back to it later.
3) Packages that were deemed at risk of misbuilding in a manual review,
but FTBFS for an unrelated reason when we did the test rebuild.
The "all-affected.dd-list" and "all-affected.list" contains packages
from all groups and defines the list of packages that we will file
bugs against. The "static-ownership.dd-list" (and
"static-ownership.list") only contains the packages with static
non-`root:root` ownership.
The packages in "static-ownership.list" should just be uploaded with
`Rules-Requires-Root: binary-targets` added to the source stanza of
`debian/control`. For all other packages, it should be possible to
convert it to use `Rules-Requires-Root: no` if the maintainer has
the capacity to do so. See solutions.md for common issues and how
to solve them.
Finally, the packages in the "maybe-builds.dd-list" and
"maybe-misbuilds.list" are the packages in group 3. We are filing a bug
against these packages as a precaution rather than a confirmed problem.
See "Findings of manual review" below for details.
# Background, context, and other details
This section is for people that want more details on what we are doing,
why I am pushing for it, the testing method using, etc. You can skip
this if you are already convinced.
## Motivation for the change
Changing the default is a major undertaking and the change should have
long term benefits that should outweigh the short term costs. To that
effect, my motivation for removing `fakeroot` is that the tool is a
fragile liability that costs us unnecessary complexity and resources.
* Assembling debian packages should not require `root` in the first
place. What we need is to assemble some archives with correct
metadata; nothing more. This is possible to do with userspace tools
as a normal user and for over 99% of the archive, `dpkg-deb` can
now this without needing `fakeroot`.
- Reminder: While my argument is for the 100% case, this MBF is
aimed at optimizing for the common case rather than the 1% case.
* Like many other `LD_PRELOAD` hacks, `fakeroot` plays a game of
whack-a-mole to intercept all the library calls relevant for its
task. When it loses the whack-a-mole game, we end up with silent
misbuilds, where installing the deb package potentially becomes
a security issue since "random" paths on the system is now owned
by a non-root user. The wrong file/dir with incorrect ownership
is a freebie "local-user to root" security bug.
A recent example being #1023286 which caused -dbgsym packages to
have a directory owned by a non-root user. The `fakeroot` fix
was then blocked for months due to FTBFS on mipsel (#1024544).
The primary reason why this was such a small issue where:
1) Wide-spread use of `Rules-Requires-Root: no` to limit which
packages could be affected (about 50% of the archive had
migrated away from `fakeroot`).
2) A quick fix to `debhelper` to remove the reliance on
`fakeroot` when assembling `-dbgsym` packages.
* The use of the `fakeroot` tool is not transparent even when
it fails to intercept critical library calls:
- #534879 (chmod race with tcp engine from 2009; unfixed)
- #606064 (segfault in i386 chroot on amd64 host)
- #948522 (`file` with sandbox enabled broke under `fakeroot`)
- #239057 (setgid directories are not correctly emulated)
- #799858 (SELinux labels are not initialized properly)
- #802612 (fakeroot fails in user namespaces)
This a small curated list of some of the open issues we have had
with fakeroot in the Debian bug tracker. These are basically
technical debt plus upkeep to have `fakeroot` match the changing
Linux landscape.
These are my primary arguments for us moving away from `fakeroot` in
general. With this MBF, we would change the default to "no fakeroot"
rather than "with fakeroot".
Note, despite my arguments against `fakeroot`, its maintainers
have my respect and sympathy. The tool has been instrumental
to Debian being where it is today (this is our version of the
Python's GIL problem), and `fakeroot` cannot have been easy
to maintain.
## Cost of the change / Key results
217 packages will FTBFS with the change. 2 will misbuild (bugs already
filed for these) and 10078 will "just work(tm)".
645 packages failed during the build testing for unrelated reasons
(including OOM on the build host).
Note that `debftbfs` (devscripts) identified existing FTBFS bugs for
422 packages in the "to be tested set".
## What happens after the bugs are fixed and the default is changed?
The endgame (from the Motivation part) is migrating all packages away
from `Rules-Requires-Root: binary-targets`. There will be two fronts
to work on:
* Packages with all `root/root` ownerships in their debs that would
need its root cause fixed, so they can build without `fakeroot`
(this is usually removing unnecessary chown'ing).
* Packages with static non-`root/root` ownerships in their debs.
There is no rootless workflow for these with `dpkg-deb` supported
features.
Note: Packages will still be able to Build-Depends on `fakeroot`,
which might be relevant for running some testsuites (etc.). This
targeted use of `fakeroot` is a lot more palatable, since it
reduces the scope for problems and makes us immune to wrong
ownership problems.
## Method and failure modes
This section is only relevant if you are curious about how we found
these packages, or you are concerned we missed something. If you agree
with the premise and just want to fix your packages, feel free to skip
this section.
While planning this default change, Guillem and I considered different
possible failure scenarios for how changing the default could
break. Based on our thoughts on this topic, we devised a testing plan
to identify these potential problems and how to resolve them.
### Failure modes
We identified the following likely outcomes of building a package
without root despite it not opting into `Rules-Requires-Root: no`:
1) Successful build. This is the most common case due to widespread
use of `dh` and its ability to enable rootless builds
automatically when available.
2) FTBFS. Any manual `chown`'ing (or `install -o/-g`) will generally
cause the build to fail, since both requires root user privileges.
Any legacy `checkroot` targets will also fall into this, since
they check the current user is root and abort otherwise. Though,
it is important to note here that `dh_testroot` does not fail here
(since it expects the `debhelper` stack to react accordingly).
3) Silent misbuild. Basically, anything in group 2) where the error
code of the command/systemcall is ignored leading to a
"successful" build as far as exit code goes, but incorrect content
in the package.
This comes in two subgroups:
a) Non-root owned paths in the package.
b) All `root:root` owned paths in the package, when at least
one path should have been non-`root:root` (like `root:bin`)
The primary concern here was the "Silent misbuild", since they would
go unnoticed in a mass-rebuild and likely hide among successful
builds.
### Risky packages
Secondly, we looked at which kind of packages were most likely to have
this kind of failure. These are grouped:
pg1) Packages with no standardized build system (The `other` on
trends.d.n). We considered this group to be very likely to have
problems with misbuilds if that would occur at all.
pg2) Packages with non-`root:root` static ownership.
We considered this group to be possible to have problems, since
one way of implementing static ownership is letting upstream's
build system do the `chown`'ing and then doing `dh_fixperms -X` to
keep the special ownership/mode. If the upstream build system
ignores errors, this style could cause a problem.
pg3) Packages using `classic debhelper`. These were originally
classified as unlikely. However, a codesearch.d.n on `dpkg(-deb)`
in `debian/rules` usage was done in the end just to be safe.
pg4) Packages using `cdbs` or `dh`. These have been assumed to be fine.
In theory, they can be made unsafe, but the maintainer would have
to go out of their way to replace the `dh_builddeb` with
`dpkg-deb --build`. Though, obvious uses of that command would
have been caught be the codesearch done as part of `pg3)`.
All the packages in `pg1)` plus `pg2)` plus the packages matched by the
codesearch listed in `pg3)` were manually reviewed for failure modes.
Note that the codesearch is not flawless, since packages can include
their packaging from other files in the `debian` directory. When the
included file does not start with `debian/rules` (such as
`include debian/implicit`), then the codesearch will have missed its
contents.
### Findings of manual review
Two cases of "successful misbuilds" have been identified:
* libjama (#1086841). This is from the `other` build system list.
* binutils-mipsen (#1082298). This was mentioned by (I think) Helmut
during our analysis discussions in #dpkg-devel and not found
directly. Note this one is technically a "false-positive", since
the package already has `Rules-Requires-Root: no` (that is, this
is bug is not about the default changing, but an uploader applying
`Rules-Requires-Root: no` without confirming it worked correctly)
Additionally, the following packages were found to be suspected of
silent misbuilds, but failed for unrelated reasons in our build
testing:
* cross-toolchain-base{,-mipsen,-ports}
* gcc-*-cross{,-mipsen,-ports}
Their packaging is non-trivial and was not fully analyzed, so they may
be safe. One of these groups chains into a `dpkg-buildpackage` (as in
`debian/rules` calls `dpkg-buildpackage` and the other group seems to
manually pack or repack debs via `dpkg-deb`. This was deemed too unusual
to spend the non-trivial effort to analyze fully given they could not be
build in the test infrastructure for unrelated reasons.
### What was rebuild on debusine.d.n
When we did the rebuild, the following packages were rebuild:
1) All packages from pg1
2) All packages from pg2
3) All packages tagged by `lintian` with
`silent-on-rules-requiring-root`
The first two groups were rebuilt mostly confirm the manual review and
to validate the rebuild setup. This also weeded out a few
false-positives in the manual review.
All the builds was done on arm64 (it had more capacity available).
### The modified dpkg
The rebuild was done with `dpkg` from this branch
https://git.hadrons.org/cgit/debian/dpkg/dpkg.git/commit/?h=next/rrr-default-no
(HEAD being 82cafddd936946b88f67b1e76601b04ca8a81586 with a `gbp dch -S`
on top)
### Full build results
A total of 10940 source packages were rebuild.
* 10078 build successful.
- Of these, 2 misbuild successfully (both noted above)
* 862 failures
- Of these 217 failed with an error related to this change
- The remaining failed for other reasons (including running
out of memory on the host, etc.)
The `check-logs` script (attached) has the regexes used for classifying
the logs for the curious.
# Thanks
Thanks to the Debusine team for providing the test infrastructure.
Thanks to Stefano for his rebuild tooling stack, which easily saved me a
week of work.
Thanks to Helmut for making us aware of Debusine being able to help with
this purpose and helping with discussions in #dpkg-devel.
Best regards,
Guillem and Niels