we need to talk about gitattributes

[Julien Plissonneau Duquène]

TL;DR can we please fix salsa-ci, gbp and dgit?

Good morning,

This is certainly not the most pleasant subject, but as several critical
tools and services are currently taking a direction that is, in my
opinion, misguided, a discussion here may help to steer that ship in a
way that will prevent further damages.

For a bit of context and history, gitattributes were introduced [1] with
git 1.5.2 released on 2007-05-20 [2], almost 18 years ago.

As with many things with git, proper use of various gitattributes
features is not always intuitive and requires carefully reading the
documentation and eventually experimenting to make sure they won't bite
you later. As such, it is expected that some upstream projects committed
some problematic settings at some point in time and eventually fixed the
settings or the repository later. In the meantime, it is possible that
these problematic settings caused or still cause build issues in Debian
packages. If you happen to know some examples of such past or present
breakage, please kindly share the references here so we can discuss the
failure modes and possible appropriate mitigations, especially if you
were using a gbp or dgit-based workflow at that time.

The current mitigations are implemented as a feature of gbp
setup-gitattributes named "defuse-gitattributes" and released with
version 0.9.23 on 2021-06-08 [3] (a bit less than 4 years ago), inspired
after a feature of dgit named setup-gitattributes that appeared with
dgit 3.3 [4], released on 2017-01-16 (8 years ago).

These mitigations are fundamentally flawed because they don't — and
can't — cover all legitimate use cases, resulting in builds breaking in
non-obvious ways, as is documented in salsa issues, even when using the
"right" tools (e.g. gbp) as documented. They are also based on wrong
assumptions about git's design.

The author of that dgit feature states his beliefs pretty explicitly in

This isn't compatible with dgit's core invariant, which is that the
git tree object is precisely the same as the content of the source
package. (The alternative invariant would be that source package is
identical to content of the working tree *as transformed by
gitattributes* - but the gitattributes are typically
context-sensitive, lossy, and very complex, so that isn't workable.)
I agree that this whole situation is not optimal. To be honest,
I think the whole gitattributes system in git is a mistake.

Unfortunately the only correct git invariant, as also very explicitly
stated by the designers of git (see [1] and the discussions that
follow), is the alternative one that was rejected by dgit's author.

My opinion is that the author above and subsequent adopters of that
approach underestimate the issues caused by using git in a way that is
basically incompatible with all other git tools in existence, including
upstream repository hosting, git itself and all current IDEs, and that
this approach qualifies as yet another wicked way to misuse
gitattributes configuration (and thus contributing to the set of
problems they may cause) rather than as a "workable" way to avoid them.
It actually only works in very specific workflows.

A codesearch.d.n query [6] finds 804 unique source packages that include
.gitattribute files with settings possibly influencing text file
conversion on checkout, so this is not quite a niche issue either.

So far 4 packages [7] had to explicitly set
SALSA_CI_DISABLE_GBP_SETUP_GITATTRIBUTES to 1 as the default
configuration causes the build to fail in a non obvious way. I had to
ask on #salsa-ci what could cause an affected build [8] to fail, as it
never occured to me that simply using gbp clone could mess with git
configuration in a way that would cause an otherwise perfectly fine
repository to systematically fail the builds.

Reporting issues against salsa-ci [9] [10] and gbp [11] [12] was so far
reminiscent of the famous "ABC game" of bureaucracies [42]: A — we do it
this way because B and C do it this way; B — we do it this way because A
and C do it this way — and then I didn't bother opening an issue with C
as I don't use it and don't plan to in the near future.

Is the Debian project already committed to make a widespread use of
"dgit" repositories that look like git repositories but are not designed
to be compatible with the git repositories that are used everywhere else
— sometimes they are until they aren't — and document that fact and the
very limited set of workflows in which these dgit repositories are
guaranteed to work so contributors can avoid some of the traps; or are
there still other possible outcomes?

Cheers,


[1]:
https://lore.kernel.org/git/***@woody.linux-foundation.org/
[2]:
https://lore.kernel.org/git/***@assigned-by-dhcp.cox.net/

[3]:
https://salsa.debian.org/agx/git-buildpackage/-/blob/debian/0.9.23/debian/changelog?ref_type=tags#L25

[4]:
https://browse.dgit.debian.org/dgit.git/commit/?h=debian/3.3&id=2a43d50b3d1f62400dfc404d378b0cd9fcb27c85

[5]: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1079434#10

[6]:
https://codesearch.debian.net/search?q=%5E%5B%5E%23%5D.*%5B%5Ea-z%5D%28crlf%7Ceol%7Ctext%29%28%5B%5Ea-z%5D%7C%24%29+path%3A%5C.gitattributes&literal=0

[7]:
https://codesearch.debian.net/search?q=SALSA_CI_DISABLE_GBP_SETUP_GITATTRIBUTES&literal=1

[8]: https://salsa.debian.org/jpd/kotlin/-/jobs/7093478#L3479

[9]: https://salsa.debian.org/salsa-ci-team/pipeline/-/issues/409
[10]:
https://salsa.debian.org/salsa-ci-team/pipeline/-/merge_requests/581

[11]: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1092800
[12]: https://salsa.debian.org/agx/git-buildpackage/-/merge_requests/38

[42]: https://archive.org/details/msdos_Bureaucracy_1987

[Sean Whitton]

Dear Julien,

You already started a discussion about this salsa some time ago.[1]

You have now come to debian-devel and posted a highly inflammatory
message accusing all sorts of things of being broken, while actually
arguing in fairly vague terms.

I don't think any productive discussion can be had in response to your
message.

To others reading, if you'd like to see the response by the "author of
that dgit feature" then please see [1]. The discussion should probably
be continued there, if anywhere.

And for the avoidance of doubt: both gbp and dgit are committed to being
usable in as many situations as possible; they are not fundamentally
broken; they are not setting up some sort of parallel git ecosystem.
Not at all.

[1] https://salsa.debian.org/salsa-ci-team/pipeline/-/issues/409

[Julien Plissonneau Duquène]

Hi Sean,

Post by Sean Whitton
You have now come to debian-devel and posted a highly inflammatory
message accusing all sorts of things of being broken, while actually
arguing in fairly vague terms.

On the discussion in question there are several references of actual
cases where the current handling of gitattributes breaks things. There
is also a merge request with a test case that reliably reproduces the
issue. I do not think this qualifies for "vague terms".

On the other hand, I have yet to see actual cases where not disabling
gitattributes would break things, despite asking several times. I would
also like to see the very real problems that are reported acknowledged
by the sponsors of "gitattributes defusing".

Post by Sean Whitton
I don't think any productive discussion can be had in response to your
message.

Casually dismissing the concerns and avoiding this necessary discussion
is not a helpful attitude.

Post by Sean Whitton
And for the avoidance of doubt: both gbp and dgit are committed to being
usable in as many situations as possible; they are not fundamentally
broken;

I'm not saying anything else here. My point is that it's the way
gitattributes "defusing" is implemented that is fundamentally broken,
and that this makes it unsuitable as a configuration default considering
the goals above.

Post by Sean Whitton
they are not setting up some sort of parallel git ecosystem.
Not at all.
Note that enabling gitattributes will make the tree incompatible with
tag2upload.

Cheers,


[1]:
https://salsa.debian.org/salsa-ci-team/pipeline/-/merge_requests/581#note_587974