Simpler git workflow for packaging with upstreamless repositories

[Kari Pahula]

With all the recent talk about increasing the use of git for
packaging, I'd like to address one thing: git-buildpackage is very
complex to use. As an alternative, I'd like to propose a simpler
setup:

- Only store debian/ in the repository and use origtargz for the rest.

- One branch per distribution you target.

- Only tag debian revisions.

That's it, less is more. The master branch would be just a single
debian directory. What it specifically wouldn't have is an upstream
branch and no extracted upstream sources in any permanent branch.

The workflow on a new upstream version would be:

1. debchange (or equivalent) to bump debian/changelog.

2. Download orig.tar with origtargz.

3. Use pristine-lfs commit on the orig.tar.

pristine-tar wouldn't work with this setup since there wouldn't be an
upstream branch anymore. It wouldn't be too hard to have tooling to
do the three steps on one go. Step one could also include a commit.

gbp dch would still be useful with this workflow. gbp pq could be
made to work with this if you extracted orig.tar and committed it to a
temporary local working branch and used it against it. I'm not sure
if there's much else in gbp that'd be applicable anymore if you do
away with having an upstream branch.

This workflow is basically what the Haskell team is using [1] except
they don't commit orig.tar files to lfs but rely on having them
available from Hackage (Haskell's central community package hub).

I guess interfacing with upstream's git with gbp has its uses but it
just seems to me that the capability comes with a hefty cost. If you
can maintain a package with having orig.tar files be your interface to
upstream's releases then it simplifies things on our side a whole lot.

I've been a DD longer than git-buildpackage has been around and I've
been looking at using it at various times but pretty much every time
my reaction's been "must I?" I know a lot of people do use gbp daily
for work with good effect but somehow I suspect even they had quite a
steep learning curve to get into it. I wouldn't fancy the idea of
trying to explain how to use it to someone on d-mentors.

I must say that I haven't been that impressed with the DEP-18
discussion I've seen. It seems like the message has pretty
consistently been "if you don't use git you're the problem" and I'm
sick of it and I can tell you it only makes me want to ignore you. If
you ask me it's git-buildpackage's irreducible difficulty of use
that's the largest road block to increase the use of git for
packaging.

If gbp works for your team then good for you but you'll have to excuse
me if I'm not that enthusiastic about sending patches to your way. I
expect it's going to be contentious to come to the list to essentially
say that a widely used tool that was first released 18 years ago
sucks.

[1]: https://wiki.debian.org/Teams/DebianHaskellGroup/CollabMaint/GettingStarted#How_To_Update_the_Debianization_of_a_Package_Already_Under_DHG_Control

[Soren Stoutner]

Kari,

Post by Kari Pahula
I've been a DD longer than git-buildpackage has been around and I've
been looking at using it at various times but pretty much every time
my reaction's been "must I?" I know a lot of people do use gbp daily
for work with good effect but somehow I suspect even they had quite a
steep learning curve to get into it. I wouldn't fancy the idea of
trying to explain how to use it to someone on d-mentors.

I don’t want to necessarily disagree with anything you have said, but if you
want an example of me explaining how to use gbp on Debian Mentors you can take
a look at:

https://lists.debian.org/debian-mentors/2024/09/msg00057.html

And yes, it is a steep learning curve.

[Holger Levsen]

hi,

fwiw, I'm also not using git-buildpackage and I don't want to. (Step learning
curve, too much magic, hard to debug and I dont see benefits for the packages
I maintain.) I'm also not a beginner. And I love gbp-dch.

just as another data point. (the above.)

please dont enforce git-buildpackage on everyone.

emacs might be better than nano for many usecases, but not all. if someone
would force me to use emacs, I'd be out. (FWIW, i'm a vim user, this is an
analogy.)
--
cheers,
Holger

⢀⣎⠟⠻⢶⣊⠀
⣟⠁⢠⠒⠀⣿⡁ holger@(debian|reproducible-builds|layer-acht).org
⢿⡄⠘⠷⠚⠋⠀ OpenPGP: B8BF54137B09D35CF026FE9D 091AB856069AAA1C
⠈⠳⣄

Segregation was legal. Slavery was legal. Don't use legality as a guide to
morality.

[Otto Kekäläinen]

Hi!

Post by Holger Levsen
fwiw, I'm also not using git-buildpackage and I don't want to. (Step learning
curve, too much magic, hard to debug and I dont see benefits for the packages
I maintain.) I'm also not a beginner. And I love gbp-dch.

I think git-buildpackage is great, and I am a happy user. However, it
took me quite a while to figure out what is the optimal way to use it,
as it has so many options. Can you elaborate what is the specific
action/workflow you think git-buildpackage you were frustrated with?
Perhaps I can help you discover the optimal workflow and you can skip
the learning curve?

Post by Holger Levsen
please dont enforce git-buildpackage on everyone.

I am not enforcing git-buildpackage on everyone. I am accelerating the
convergence of workflows so that it is easier for maintainers to
collaborate, so we can have more code reviews, more new maintainers,
and in the long run perhaps decrease the number of single-maintainer
packages for packages where the maintainer does not want to be the
sole maintainer. Currently the learning curve on how to contribute
prevents many people from doing it. As part of this I help people make
the most out of git-buildpackage, which is by far the most popular
tool for this, and to my knowledge the only tool that supports having
upstream branch in the same repository, cherry-picking upstream
commits easily, rebasing Debian patches on upstream, tracking
supply-chain from upstream git tags and tarballs (or both) to Debian
releases and so forth. There is a debian/gbp.conf in 13573 packages in
Debian, and probably twice as many in total use git-buildpackage
already. Almost all team policies I read have converged on
git-buildpackage. If we don't converge on what the majority is already
using, what then?

Post by Holger Levsen
emacs might be better than nano for many usecases, but not all. if someone
would force me to use emacs, I'd be out. (FWIW, i'm a vim user, this is an
analogy.)

The analogy with an editor does not work, as the editor is your
personal tool. A better analogy would be interactive pair-programming,
and in such a setting would need to agree with your pair to use for
example Pulsar or Zed so that there is a common system that shows on
your screens what the other person is typing.

If you had bad experiences with git-buildpackage, please share them
with me, I can help you find the optimal workflow, and you can perhaps
give git-buildpackage a second chance?

[Holger Levsen]

Post by Otto Kekäläinen

Post by Holger Levsen
fwiw, I'm also not using git-buildpackage and I don't want to. (Step learning
curve, too much magic, hard to debug and I dont see benefits for the packages
I maintain.) I'm also not a beginner. And I love gbp-dch.

I think git-buildpackage is great, and I am a happy user. However, it
took me quite a while to figure out what is the optimal way to use it,
as it has so many options. Can you elaborate what is the specific
action/workflow you think git-buildpackage you were frustrated with?

sigh. see above.

in addition to the above: gbp is great if it works, if it doesn't I can
start to learn to debug a complex system, while on the contrary, if I use
git and debuild/pbuilder/sbuild manually all the time, I know those tools,
they are rather easy to debug etc.

Post by Otto Kekäläinen
I am not enforcing git-buildpackage on everyone. I am accelerating the
convergence of workflows so that it is easier for maintainers to
collaborate, so we can have more code reviews, more new maintainers,
and in the long run perhaps decrease the number of single-maintainer
packages for packages where the maintainer does not want to be the
sole maintainer.

and you seem to think adding another Debian specific complex tool will
attrack new people? Most people^wcontributors out there know git already,
for them a simple git only workflow is *much* more inviting than having
to learn gbp *only* to contribute to Debian. gbp is useless outside
Debian (and derivatives. Yet, thats only a part of the free software world.)

Post by Otto Kekäläinen
I think this will be more successful if you frame this as good patterns
to follow for people who wish to opt-in on the premise of adopting this
workflow. That framing follows other DEP's. Now it reads as a mandate
for everything. Acknowledging existance of exceptions to the rules
within the rules may also help to gain acceptance.

This!

...and hopefully last post on this thread from me. I've said everything I
wanted to say.
--
cheers,
Holger

⢀⣎⠟⠻⢶⣊⠀
⣟⠁⢠⠒⠀⣿⡁ holger@(debian|reproducible-builds|layer-acht).org
⢿⡄⠘⠷⠚⠋⠀ OpenPGP: B8BF54137B09D35CF026FE9D 091AB856069AAA1C
⠈⠳⣄

The upcoming clima apocalypse is the big elephant in every room now.

[Chris Hofstaedtler]

Post by Holger Levsen
and you seem to think adding another Debian specific complex tool will
attrack new people? Most people^wcontributors out there know git already,
for them a simple git only workflow is *much* more inviting than having
to learn gbp *only* to contribute to Debian.

This. Exactly this.

This is the reason I can contribute small improvements to Alpine and
other projects with reasonable effort for small improvements. The
tools are the same tools I already use elsewhere, and stuff is as
transparent as it needs to be for outside contributors.

Time to get build people where they are, and not where we are.

Chris

[Otto Kekäläinen]

Hi,

Post by Chris Hofstaedtler

Post by Holger Levsen
and you seem to think adding another Debian specific complex tool will
attrack new people? Most people^wcontributors out there know git already,
for them a simple git only workflow is *much* more inviting than having
to learn gbp *only* to contribute to Debian.

This. Exactly this.
This is the reason I can contribute small improvements to Alpine and
other projects with reasonable effort for small improvements. The
tools are the same tools I already use elsewhere, and stuff is as
transparent as it needs to be for outside contributors.
Time to get build people where they are, and not where we are.

As Andrey wrote, sure this would be better, but also impossible.

How .dsc/.deb/.orig.tar.gz and occasional modified .tar.gz work in
Debian is fundamentally very different from how in e.g. Alpine a
single APKBUILD defines a the upstream source a just an URL and a
sha512 to verify the download was correct. Maybe one day in the future
we change the Debian source format to not have upstream tarball
bundled, but that is a totally different discussion from what we are
doing now in DEP-18: recognizing the current most common workflows and
elevating them as the recommended baseline for those who are keen to
collaborate more efficiently.

In Debian/DEP-18 for all the normal daily packaging work like pulling
and pushing, making commits, amending commits, creating branches,
rebasing branches etc you can and should use just regular git commits.
Git-buildpackage is there only to tell people that is the tool to
interact with repos. Git-buildpackage does what it does out of
necessity to be able to produce the *.changes and *.dsc files.
Therefore the upstream import and building packages need to be done in
a certain way and there are 2 or 3 branches in the git repositories.
Luckily, these 3 branch names are unified to be debian/latest,
upstream/latest and pristine-tar thanks to DEP-14. Now with DEP-18 we
can move along and make it easier for new people and also tenured
maintainers to land on the same best practices.

Going back to the example of Alpine, they enforce heavily one single
workflow that includes one single VCS (git), one single hosting place
(https://gitlab.alpinelinux.org/) and to my knowledge also one single
tool to build and update packages. DEP-18 does not enforce anything,
but if you want to have a situation that people voluntarily move
towards adopting the same base workflow for undifferentiated Debian
packaging, you should support DEP-18 and not dismiss it.

[Andrey Rakhmatullin]

Post by Holger Levsen

Post by Otto Kekäläinen
I am not enforcing git-buildpackage on everyone. I am accelerating the
convergence of workflows so that it is easier for maintainers to
collaborate, so we can have more code reviews, more new maintainers,
and in the long run perhaps decrease the number of single-maintainer
packages for packages where the maintainer does not want to be the
sole maintainer.

and you seem to think adding another Debian specific complex tool will
attrack new people? Most people^wcontributors out there know git already,
for them a simple git only workflow is *much* more inviting than having
to learn gbp *only* to contribute to Debian. gbp is useless outside
Debian (and derivatives. Yet, thats only a part of the free software world.)

Yes, changing Debian so that the required workflows are more simple is
much better but also impossible.

[Otto Kekäläinen]

Hi,

Post by Holger Levsen

Post by Otto Kekäläinen

Post by Holger Levsen
fwiw, I'm also not using git-buildpackage and I don't want to. (Step learning
curve, too much magic, hard to debug and I dont see benefits for the packages
I maintain.) I'm also not a beginner. And I love gbp-dch.

I think git-buildpackage is great, and I am a happy user. However, it
took me quite a while to figure out what is the optimal way to use it,
as it has so many options. Can you elaborate what is the specific
action/workflow you think git-buildpackage you were frustrated with?

sigh. see above.

I obviously saw above and I think asking for more details I think is
fair so we can discuss if the thing you experienced as "hard to learn"
or "had too much magic" was fundamentally bad and can't be fixed, or
perhaps if better docs or better error handling etc can make that
frustration go away. I feel we should try to polish and improve the
existing tooling that the majority uses, rather than giving up on it
and demand an entirely new tool or workflow.

Post by Holger Levsen
in addition to the above: gbp is great if it works, if it doesn't I can
start to learn to debug a complex system, while on the contrary, if I use
git and debuild/pbuilder/sbuild manually all the time, I know those tools,
they are rather easy to debug etc.

Personally I think gbp is very easy to debug. You just add --verbose
to any command and it will print out what it is doing. Example:

± gbp import-orig --uscan --verbose
gbp:debug: ['git', 'rev-parse', '--show-cdup']
gbp:debug: ['git', 'rev-parse', '--is-bare-repository']
gbp:debug: ['git', 'rev-parse', '--git-dir']
gbp:debug: ['git', 'for-each-ref', '--format=%(refname:short)', 'refs/heads/']
gbp:debug: ['git', 'show-ref', '--verify', 'refs/heads/upstream']
gbp:error:
Repository does not have branch 'upstream' for upstream sources. If
there is none see
file:///usr/share/doc/git-buildpackage/manual-html/gbp.import.html#GBP.IMPORT.CONVERT
on howto create it otherwise use --upstream-branch to specify it.

I am sure you already knew this, but pasting here an example for the
sake of discussion.

The output and debuggability can for sure further be improved still, I
am not saying gbp is perfect, but it is by far the best we have and
deserves to be the thing any team or large group standardize on today
to have an easier time collaborating.

Post by Holger Levsen

Post by Otto Kekäläinen
I think this will be more successful if you frame this as good patterns
to follow for people who wish to opt-in on the premise of adopting this
workflow. That framing follows other DEP's. Now it reads as a mandate
for everything. Acknowledging existance of exceptions to the rules
within the rules may also help to gain acceptance.

This!

Thanks for the feedback, I will try to further iterate on the
introduction part in DEP-18 to make is more clear what is the purpose
and that a DEP-18 is not a policy and is opt-in like all the other
DEPs.

[Simon Josefsson]

Post by Otto Kekäläinen

Post by Holger Levsen
in addition to the above: gbp is great if it works, if it doesn't I can
start to learn to debug a complex system, while on the contrary, if I use
git and debuild/pbuilder/sbuild manually all the time, I know those tools,
they are rather easy to debug etc.

Personally I think gbp is very easy to debug. You just add --verbose
± gbp import-orig --uscan --verbose

...

I thought about the gbp aspect of the lets-move-things-to-salsa and I
suggest to consider if they are better left orthogonal. Could you make
one DEP for lets-move-to-salsa and one DEP for
lets-document-a-build-workflow? The former would not talk a bout local
builds at all. The latter would not talk about the Salsa workflow at
all. They both intersect on git branch naming, and maybe some other
minor aspects, but don't we have that already in DEP 14?

Compare making packaging contributions to Debian to packaging
contributions to Homebrew, Alpine, OpenWRT, ArchLinux, Guix, etc. There
is a possibility to reach a completely git-based Salsa workflow for
Debian packages that doesn't involve any local builds on people's
laptops at all. This is comparable to Homebrew contributions, I find
them a joy to work with in comparison and can contribute even without
having any Homebrew development environment setup, or even without
physical access to a Mac. There is no reason Debian packaging
contributions couldn't work like that. Yes, old timers will continue
build stuff on their laptop, and that has to be fine. We need to
resolve how to authenticate archive uploads chaining back to a DD
instead of PGP on tarball, but that is doable.

/Simon

[Soren Stoutner]

Post by Otto Kekäläinen
I am not enforcing git-buildpackage on everyone. I am accelerating the
convergence of workflows so that it is easier for maintainers to
collaborate, so we can have more code reviews, more new maintainers,
and in the long run perhaps decrease the number of single-maintainer
packages for packages where the maintainer does not want to be the
sole maintainer. Currently the learning curve on how to contribute
prevents many people from doing it.

I am all for this. I don’t know if we will ever get the number of Debian
workflows down to 1, but I would like to see it somewhere lower than 1,000
(that is only a slight exaggeration) in my lifetime.

[Peter Pentchev]

Post by Kari Pahula
With all the recent talk about increasing the use of git for
packaging, I'd like to address one thing: git-buildpackage is very
complex to use. As an alternative, I'd like to propose a simpler
- Only store debian/ in the repository and use origtargz for the rest.
- One branch per distribution you target.
- Only tag debian revisions.
That's it, less is more. The master branch would be just a single
debian directory. What it specifically wouldn't have is an upstream
branch and no extracted upstream sources in any permanent branch.

Let me start by saying that I understand where you're coming from;
any tool that allows different use scenarios will almost inevitably
grow more and more complex with time, and become more and more
difficult to use by beginners, unless there are good tutorials and
step-by-step recipes for the most common cases. TBH, I think that
the git-buildpackage manual is relatively easy to read and
understand, but, of course, that opinion of mine is tainted by
the fact that I cannot exactly be called a beginner :) (although
there are new things I learn about Debian packaging every month)

However... different people are used to different workflows.
I personally really, really like the fact that I have the Git history of
all the upstream files in the same repo and I don't have to go over to
a different repo to figure out what changed when. Also, I like that
immediately after `gbp import-orig` I can run `git show upstream` to
review the diff (TBH, me being very pedantic, I usually have already
given it a quick glance using `diff -urN` on unpacked tarballs before
even importing it, if only to figure out if there are new files that
I need to exclude, but that's not always the case), and that I can
at any time later run `git diff upstream/2.17.18..upstream/3.14.15 path`
or similar commands. I have even been known to use `git bisect` in
a Debian package directory in some weird cases.

And yes, all of that can be handled in a separate Git repo with
a clean checkout of the upstream repository without any of
the Debian fluff; however, that would require me switching between
terminals/tmux panes/whatever, and sometimes that seems like too much
work when I can have it all in a single repository :)

So... yes, a simpler setup would work for some people, and it may be
better for beginners. However, there are some benefits to a full
repository containing both the upstream source and the Debian changes,
and some people like to use them every now and then.

Still, thanks for your desire to make working on Debian easier and
better!

G'luck,
Peter

[Kari Pahula]

Post by Peter Pentchev
However... different people are used to different workflows.
I personally really, really like the fact that I have the Git history of
all the upstream files in the same repo and I don't have to go over to
a different repo to figure out what changed when. Also, I like that
immediately after `gbp import-orig` I can run `git show upstream` to
review the diff (TBH, me being very pedantic, I usually have already
given it a quick glance using `diff -urN` on unpacked tarballs before
even importing it, if only to figure out if there are new files that
I need to exclude, but that's not always the case), and that I can
at any time later run `git diff upstream/2.17.18..upstream/3.14.15 path`
or similar commands. I have even been known to use `git bisect` in
a Debian package directory in some weird cases.
And yes, all of that can be handled in a separate Git repo with
a clean checkout of the upstream repository without any of
the Debian fluff; however, that would require me switching between
terminals/tmux panes/whatever, and sometimes that seems like too much
work when I can have it all in a single repository :)

Preface, just in case: I list a few git commands, don't try running
them if you have uncommitted changes.

Okay, I could amend what I proposed originally with this option: Do
the debian work in a fork of upstream's repository, but never merge
the debian branch and upstream branch. That is, the start of Debian
maintenance would be by cloning upstream and then with "git checkout
--orphan debian" followed by "git reset --hard".

Do you think you could do all of the above with that model, with a
command like "git checkout debian -- ." to insert the debian contents
to an upstream branch?

I'm thinking that it's the merging of debian and upstream branches and
maintaining it that really causes the warts in gbp and I'm not at all
sure if there are any actual workflows that require having that.
"upstreamless" as I described it may be a bit too much for general use
but could there be a case for doing everything with a mergeless model
instead?

Furthermore, I think import-orig should really be only about
establishing a particular byte string as the orig.tar. Think of
object storage. If there's a connection to a particular commit hash
or release tag in the repository it would better be represented as a
separate entry. Perhaps as a text file like debian/upstream-commits
that would be a list of upstream version/commit hash or tag pairs.
From where I stand, the way these disparate concerns have been merged
seems to be one root cause for all sorts of unintended consequences.

Post by Peter Pentchev
So... yes, a simpler setup would work for some people, and it may be
better for beginners. However, there are some benefits to a full
repository containing both the upstream source and the Debian changes,
and some people like to use them every now and then.

I don't agree with framing this as a beginner/expert issue.

Post by Peter Pentchev
Still, thanks for your desire to make working on Debian easier and
better!

You don't really sound willing to discuss anything with that but I'm
still going to try.

[Andrey Rakhmatullin]

Post by Kari Pahula

Post by Peter Pentchev
However... different people are used to different workflows.
I personally really, really like the fact that I have the Git history of
all the upstream files in the same repo and I don't have to go over to
a different repo to figure out what changed when. Also, I like that
immediately after `gbp import-orig` I can run `git show upstream` to
review the diff (TBH, me being very pedantic, I usually have already
given it a quick glance using `diff -urN` on unpacked tarballs before
even importing it, if only to figure out if there are new files that
I need to exclude, but that's not always the case), and that I can
at any time later run `git diff upstream/2.17.18..upstream/3.14.15 path`
or similar commands. I have even been known to use `git bisect` in
a Debian package directory in some weird cases.
And yes, all of that can be handled in a separate Git repo with
a clean checkout of the upstream repository without any of
the Debian fluff; however, that would require me switching between
terminals/tmux panes/whatever, and sometimes that seems like too much
work when I can have it all in a single repository :)

Preface, just in case: I list a few git commands, don't try running
them if you have uncommitted changes.
Okay, I could amend what I proposed originally with this option: Do
the debian work in a fork of upstream's repository, but never merge
the debian branch and upstream branch. That is, the start of Debian
maintenance would be by cloning upstream and then with "git checkout
--orphan debian" followed by "git reset --hard".
Do you think you could do all of the above with that model, with a
command like "git checkout debian -- ." to insert the debian contents
to an upstream branch?
I'm thinking that it's the merging of debian and upstream branches and
maintaining it that really causes the warts in gbp and I'm not at all
sure if there are any actual workflows that require having that.
"upstreamless" as I described it may be a bit too much for general use
but could there be a case for doing everything with a mergeless model
instead?
Furthermore, I think import-orig should really be only about
establishing a particular byte string as the orig.tar. Think of
object storage. If there's a connection to a particular commit hash
or release tag in the repository it would better be represented as a
separate entry. Perhaps as a text file like debian/upstream-commits
that would be a list of upstream version/commit hash or tag pairs.
From where I stand, the way these disparate concerns have been merged
seems to be one root cause for all sorts of unintended consequences.

You are talking about the upstream git history while Peter is talking
about simply importing orig tarballs.

Maybe what is causing warts in gbp *for you* is some *specific* gbp
workflow, one of the many it supports?

[Blair Noctis]

Post by Kari Pahula
I'm thinking that it's the merging of debian and upstream branches and
maintaining it that really causes the warts in gbp and I'm not at all
sure if there are any actual workflows that require having that.
"upstreamless" as I described it may be a bit too much for general use
but could there be a case for doing everything with a mergeless model
instead?

Could we turn the current packaging format:

foo-1.2.3/ -> upstream source
src/
...
debian/ -> Debian inserted things

where debian/ must reside *inside* the upstream source; inside out:

foo-1.2.3-debian/
... Debian things
upstream/ -> upstream source
src/
...

where "upstream source" could be extracted from a tarball, a cloned Git repository checked out at a specified commit, a Git submodule, etc. as the maintainer pleases. However it's produced, it is now a "guest" of the packaging (which now becomes the "host"), allowing the host to do more, unlike the current way around, where the packaging is the guest and pretty limited.

So we no longer have the problem of "merging of debian and upstream branches"?

Post by Kari Pahula
Yes, changing Debian so that the required workflows are more simple is
much better but also impossible.

This whimsical "4.0 (inside-out)" format is only for your entertainment. :>

[Andrey Rakhmatullin]

Post by Kari Pahula
With all the recent talk about increasing the use of git for
packaging, I'd like to address one thing: git-buildpackage is very
complex to use. As an alternative, I'd like to propose a simpler

We want fewer allowed setups, not more. Though it's clear to me at this
point that if this become more than a dream, we will need to allow more
than one.

Post by Kari Pahula
- Only store debian/ in the repository and use origtargz for the rest.
- One branch per distribution you target.
- Only tag debian revisions.

--git-overlay probably supports this, but it's not really documented so I
don't really know.

Post by Kari Pahula
That's it, less is more. The master branch would be just a single
debian directory. What it specifically wouldn't have is an upstream
branch and no extracted upstream sources in any permanent branch.

Many people want to have version-tracked upstream sources so this can't be
the only allowed workflow.

Post by Kari Pahula
gbp dch would still be useful with this workflow. gbp pq could be
made to work with this if you extracted orig.tar and committed it to a
temporary local working branch and used it against it.

"How do I create and update patches" is actually a big part of a good
workflow, and this way doesn't sound easy.
Though of course there is no way to have 3.0 (quilt), git and easy patch
workflows at the same time.

Post by Kari Pahula
I guess interfacing with upstream's git with gbp has its uses but it
just seems to me that the capability comes with a hefty cost. If you
can maintain a package with having orig.tar files be your interface to
upstream's releases then it simplifies things on our side a whole lot.

It's not necessarily interfacing with upstream's git, just
version-tracking tarballs is already very useful. Maybe more useful than
the upstream git in certain use cases.

Post by Kari Pahula
I've been a DD longer than git-buildpackage has been around and I've
been looking at using it at various times but pretty much every time
my reaction's been "must I?" I know a lot of people do use gbp daily
for work with good effect but somehow I suspect even they had quite a
steep learning curve to get into it. I wouldn't fancy the idea of
trying to explain how to use it to someone on d-mentors.

I have many interconnected thoughts about this:

1. git-buildpackage is complicated because it supports many different
workflows, some of which don't have anything in common. This simply
represents a bigger problem in Debian and is not specific to
git-buildpackage.
2. git-buildpackage is bad. This is unavoidable, because any tool that
tries to wrap the inherently incompatible Debian packages workflow into a
VCS will be bad in some way. So we don't and, unavoidably, can't have a
better tool and must work with what we have.
3. git-buildpackage has a steep learning curve, but I don't expect it to
be that steep for people who are already proficient both with Debian
packaging and with git. Which new users aren't, at least because of the
former, but it's far from the only thing with a steel learning curve and
not enough docs they will hit their heads against.
4. git itself has a steep learning curve, bad UI, bad docs etc. etc., but
we clearly don't have a better VCS to store packages in, and we really
need to do that. And, at least, git knowledge is both useful outside
Debian and possible to already have before becoming a Debian package,
unlike almost everything else you need to know in Debian packaging.
5. *Debian packaging itself* is notoriously hard to learn, and as long as
it stays as hard as it is now not a lot of things on top of it could make
the overall experience harder and some of them may make it *easier* for
many people.
6. Whatever workflow one needs to learn, even if it's just a single one,
will be hard for a newbie in a large part because of the disconnect
between the VCS concept and the "source package as a first class citizen"
concept, even before we talk about patches, uscan and upstreams without
tarballs.

Long story short, I don't envy our newbie contributors and I don't think
it will be easier for them in my lifetime.

Post by Kari Pahula
I must say that I haven't been that impressed with the DEP-18
discussion I've seen. It seems like the message has pretty
consistently been "if you don't use git you're the problem" and I'm
sick of it and I can tell you it only makes me want to ignore you.

I'm afraid I don't see a peaceful way forward for the project. Most likely
we will continue stagnating and losing people but in a less probably
scenario with big changes many people will resign.

Post by Kari Pahula
If you ask me it's git-buildpackage's irreducible difficulty of use
that's the largest road block to increase the use of git for packaging.

:-/

[Otto Kekäläinen]

Hi!

Post by Kari Pahula
With all the recent talk about increasing the use of git for
packaging, I'd like to address one thing: git-buildpackage is very
complex to use. As an alternative, I'd like to propose a simpler

..

Post by Kari Pahula
sick of it and I can tell you it only makes me want to ignore you. If
you ask me it's git-buildpackage's irreducible difficulty of use
that's the largest road block to increase the use of git for
packaging.

I think git-buildpackage is great, and I am a happy user.

It did however take me quite a while to figure out what is the best
practice to use it, as it has so many options. I see you are also
using it at https://salsa.debian.org/science-team/chuffed so you seem
to have learned to use it. Can you elaborate, is there a specific
thing you think git-buildpackage falls short on?

Instead of trying to write a new tool from scratch, I hope more people
would offer help to Guido to maintain and polish git-buildpackage and
docs. I am for example proposing doc improvements in
https://salsa.debian.org/agx/git-buildpackage/-/merge_requests/6 and I
tried make git-buildpackage use DEP-14 branches (debian/latest and
upstream/latest) by default in
https://salsa.debian.org/agx/git-buildpackage/-/merge_requests/1,
although I ran out of steam with this latter one as it turned out to
be quite complex. I was also planning to suggest updates the the
Developers Reference on how to use git-buildpackage in the context of
daily package maintenance tasks so people would more easily discover
them (draft at https://salsa.debian.org/debian/developers-reference/-/merge_requests/63).

To make it easier for contributors I publish in my packages
debian/gbp.conf, so the settings are automatically correct, and also
README.source(.md) so contributors can discover the correct gbp
comands easily. See examples at
https://salsa.debian.org/debian/entr/-/blob/debian/latest/debian/gbp.conf
and https://salsa.debian.org/debian/entr/-/blob/debian/latest/debian/README.source.md.

I am contemplating if I should also make videos on Debian packaging
best practices, or have start a new Matrix channel specifically to
help maintainers setup their git repositories correctly and find the
optimal git-buildpackage commands for each thing they want to do.

[Johannes Schauer Marin Rodrigues]

Hi,

Quoting Otto KekÀlÀinen (2024-11-19 07:34:45)

I am contemplating if I should also make videos on Debian packaging best
practices, or have start a new Matrix channel specifically to help
maintainers setup their git repositories correctly and find the optimal
git-buildpackage commands for each thing they want to do.

I'd describe myself of being in the camp of "I don't care about the commands, i
don't care how my git branches are named, just document one thing so that my
packages can do the same thing that most other packages do". For that reason I
so far created new package repos with "gbp import-dsc" and used all the
defaults that come with that. I don't think I care much for what these defaults
are at least I didn't see myself reading past discussions about this and
thinking "nooooo don't name this branch debian/latest instead of debian/foo" or
something. I like to read of other people's workflows but then I often do not
see how their workflows can possibly fit my packages. There seem to be many
people who have the upstream git as part of their packaging git. I'm happy that
works for them but I don't see how I can leave my tarball-centered workflows
(even though all my upstream work in git) if all my upstreams ship DFSG
non-free material which I have to remove from their tarballs first.

So given all this, the point I wanted to make was: I'd like to watch your
videos if you make them. But at least for me you do not have to go through the
trouble of shooting videos. Just some HTML docs would be just fine for me.
Currently, I'm missing a long-term maintained document which explains "the"
(haha) recommended Debian git workflow through the lifetime of a package from
its initial creation to backports or stable updates.

Thank you for your work!

cheers, josch

[Simon Josefsson]

I like to read of other people's workflows but then I often do not see
how their workflows can possibly fit my packages. There seem to be
many people who have the upstream git as part of their packaging
git. I'm happy that works for them but I don't see how I can leave my
tarball-centered workflows (even though all my upstream work in git)
if all my upstreams ship DFSG non-free material which I have to remove
from their tarballs first.

This works fairly well these days: use Files-Excluded: in d/copyright
and "dversionmangle=s/\+ds\d*$//,repacksuffix=+ds" in d/watch. Tools
like gbp, uscan, origtargz seems to do the right thing automatically.

So given all this, the point I wanted to make was: I'd like to watch your
videos if you make them. But at least for me you do not have to go through the
trouble of shooting videos. Just some HTML docs would be just fine for me.
Currently, I'm missing a long-term maintained document which explains "the"
(haha) recommended Debian git workflow through the lifetime of a package from
its initial creation to backports or stable updates.

Yes please! There are so many details (library transitions? non-dfsg
files? experimental->sid migration? go reverse-rebuilds? etc) that
are not well documented. I have my own big README with various workflow
commands but I deal with packages following perhaps 10+ different styles
today, and I try to respect the traditional style used in a package.

/Simon

[Johannes Schauer Marin Rodrigues]

Quoting Simon Josefsson (2024-11-22 12:18:36)

I like to read of other people's workflows but then I often do not see how
their workflows can possibly fit my packages. There seem to be many people
who have the upstream git as part of their packaging git. I'm happy that
works for them but I don't see how I can leave my tarball-centered
workflows (even though all my upstream work in git) if all my upstreams
ship DFSG non-free material which I have to remove from their tarballs
first.

This works fairly well these days: use Files-Excluded: in d/copyright and
"dversionmangle=s/\+ds\d*$//,repacksuffix=+ds" in d/watch. Tools like gbp,
uscan, origtargz seems to do the right thing automatically.

That's what I'm doing. But that works with tarballs not with upstream as git.
If upstream (deliberately, so this will not change) has DSFG-non-free content
in it, then I should not copy that into a git packaging repo that is targeting
main. Removing the problematic parts from the upstream git repos would rewrite
their history, invalidate tags etc, so the result would not be very useful
anymore. Thus, I usually have one directory on my PC with the upstream git and
another with my Debian packaging git. The packaging git does not have the
upstream git with non-free content int it. Instead my packaging git regularly
imports new upstream releases as tarballs and removes the non-free content via
Files-Excluded and dversionmangle/repacksuffix in d/watch. I don't see how I
can get rid of the tarballs here but instead embrace the (non-free) upstream
git. Maybe I'm missing something?

Thanks!

cheers, josch

[Sean Whitton]

Hello Kari,

Have you seen:

https://wiki.debian.org/GitPackagingSurvey

You may find something suitable for what you want there.

[Stefano Rivera]

Hi Kari (2024.11.18_15:40:55_+0000)

Post by Kari Pahula
- Only store debian/ in the repository and use origtargz for the rest.

I used to feel strongly this way, too. However,

A big advantage of storing the upstream sources in git is that you can
use git to manage the quilt patch queue. I used to be pretty good at
editing patches to get them to apply after upstream changed the code,
but its painful and slow work. gbp-pq rebase is *infinitely* better.
You need to be comfortable in git rebasing. But if you use git, you are
probably already a git rebase ninja. You can use those same skills in
Debian patch queue maintenance.

I'd suggest giving gbp-pq a good try before writing off the gbp stack.
Maintaining a complex patch stack is *much* easier with it. I look
after packages in both layouts, and I am sold on storing upstream
sources in git.

Stefano