Bug#1086878: python-catalogue: 2.1.0 was yanked - what version scheme should we use for 2.0.10?

[Colin Watson]

Source: python-catalogue
Version: 2.1.0-6
Severity: normal
X-Debbugs-Cc: Andreas Tille <***@debian.org>, debian-***@lists.debian.org

https://pypi.org/project/catalogue/#history shows that 2.1.0 was yanked
from PyPI, but it's what we currently have in Debian. Some of the more
recent releases (which I think are really in the same version series -
it's just that upstream changed their minds about bumping the patch
version) contain fixes that we should have in Debian; in particular
while trying to fix python-srsly I ran into a problem which I think is
fixed by
https://github.com/explosion/catalogue/commit/75f5e9c24e93b5fcc2b3e9f324d9328bc871abad.

I'm happy to do the work to get us onto a newer version, but I wanted to
check what to use for the version scheme. Should we use
2.1.0+really2.0.10-1 or 1:2.0.10-1? I think there'd be some
justification for an epoch here since the upstream version numbering
scheme changed (cf.
https://www.debian.org/doc/debian-policy/ch-controlfields.html#s-f-version),
but I'm copying debian-devel as per policy on epoch proposals.

Thanks,

[Guillem Jover]

Hi!

Post by Colin Watson
Source: python-catalogue
Version: 2.1.0-6
Severity: normal
https://pypi.org/project/catalogue/#history shows that 2.1.0 was yanked
from PyPI, but it's what we currently have in Debian. Some of the more
recent releases (which I think are really in the same version series -
it's just that upstream changed their minds about bumping the patch
version) contain fixes that we should have in Debian; in particular
while trying to fix python-srsly I ran into a problem which I think is
fixed by
https://github.com/explosion/catalogue/commit/75f5e9c24e93b5fcc2b3e9f324d9328bc871abad.
I'm happy to do the work to get us onto a newer version, but I wanted to
check what to use for the version scheme. Should we use
2.1.0+really2.0.10-1 or 1:2.0.10-1? I think there'd be some
justification for an epoch here since the upstream version numbering
scheme changed (cf.
https://www.debian.org/doc/debian-policy/ch-controlfields.html#s-f-version),
but I'm copying debian-devel as per policy on epoch proposals.

The way I see it, this does not look like a version schema change to
me, upstream released a version that they then retracted:

https://github.com/explosion/catalogue/issues/46

Given that I assume the current (non-retracted) upstream version is
going to be close to surpass the retracted one, I'd go for the +really
hack. In this case invalidating relationships for external
dependencies would not seem like a big issue, because it looks like
the yanked version is the only one that has ever been in Debian, but
it avoids the ugliness and confusion of epochs (people tend to forget
to add the epoch in relationships for example) and its stickiness,
going forward.

The other question that comes to mind is why the yanked version was
uploaded, as from that issue above it seems at that time it should
have already been marked as yanked. Perhaps we have some automated
tool that does not honor the yanked markings, which might deserve a bug
report? Andreas do you recall what tool or process you used for that?

Thanks,
Guillem

[Colin Watson]

Post by Guillem Jover

Post by Colin Watson
https://pypi.org/project/catalogue/#history shows that 2.1.0 was yanked
from PyPI, but it's what we currently have in Debian. Some of the more
recent releases (which I think are really in the same version series -
it's just that upstream changed their minds about bumping the patch
version) contain fixes that we should have in Debian; in particular
while trying to fix python-srsly I ran into a problem which I think is
fixed by
https://github.com/explosion/catalogue/commit/75f5e9c24e93b5fcc2b3e9f324d9328bc871abad.
I'm happy to do the work to get us onto a newer version, but I wanted to
check what to use for the version scheme. Should we use
2.1.0+really2.0.10-1 or 1:2.0.10-1? I think there'd be some
justification for an epoch here since the upstream version numbering
scheme changed (cf.
https://www.debian.org/doc/debian-policy/ch-controlfields.html#s-f-version),
but I'm copying debian-devel as per policy on epoch proposals.

The way I see it, this does not look like a version schema change to
https://github.com/explosion/catalogue/issues/46

I guess you might or might not describe that as a change in the upstream
version numbering scheme, depending on how you interpret policy's
language. *shrug*

Post by Guillem Jover
Given that I assume the current (non-retracted) upstream version is
going to be close to surpass the retracted one, I'd go for the +really
hack. In this case invalidating relationships for external
dependencies would not seem like a big issue, because it looks like
the yanked version is the only one that has ever been in Debian, but
it avoids the ugliness and confusion of epochs (people tend to forget
to add the epoch in relationships for example) and its stickiness,
going forward.

I don't really have any information on whether upstream plans a 2.1.1 or
similar, but it's true it might well happen.

Post by Guillem Jover
The other question that comes to mind is why the yanked version was
uploaded, as from that issue above it seems at that time it should
have already been marked as yanked. Perhaps we have some automated
tool that does not honor the yanked markings, which might deserve a bug
report? Andreas do you recall what tool or process you used for that?

I'd initially misread it as being just a day or two after the yanked
version, but you're right, it was months later. I suspect it was simply
uscan - it's using the GitHub tags rather than looking at PyPI, and the
tag was never removed, so it's hard to see how it could have known any
better.

This does leave the question of how to hide that version from uscan in
the future, since uscan doesn't make it easy to ignore specific upstream
versions and I'd prefer to avoid using opaque regex constructions to do
it. My best idea is to use uversionmangle to turn 2.1.0 into something
like 2.0.8~pre1, but is there a better idiom?

[Guillem Jover]

Hi!

Post by Colin Watson

Post by Guillem Jover
Given that I assume the current (non-retracted) upstream version is
going to be close to surpass the retracted one, I'd go for the +really
hack. In this case invalidating relationships for external
dependencies would not seem like a big issue, because it looks like
the yanked version is the only one that has ever been in Debian, but
it avoids the ugliness and confusion of epochs (people tend to forget
to add the epoch in relationships for example) and its stickiness,
going forward.

I don't really have any information on whether upstream plans a 2.1.1 or
similar, but it's true it might well happen.

Right, see below…

Post by Colin Watson

Post by Guillem Jover
The other question that comes to mind is why the yanked version was
uploaded, as from that issue above it seems at that time it should
have already been marked as yanked. Perhaps we have some automated
tool that does not honor the yanked markings, which might deserve a bug
report? Andreas do you recall what tool or process you used for that?

I'd initially misread it as being just a day or two after the yanked
version, but you're right, it was months later. I suspect it was simply
uscan - it's using the GitHub tags rather than looking at PyPI, and the
tag was never removed, so it's hard to see how it could have known any
better.
This does leave the question of how to hide that version from uscan in
the future, since uscan doesn't make it easy to ignore specific upstream
versions and I'd prefer to avoid using opaque regex constructions to do
it. My best idea is to use uversionmangle to turn 2.1.0 into something
like 2.0.8~pre1, but is there a better idiom?

Ah I assumed the sources were being taken from pypi, if they are taken
from GitHub, then that explains yes. Perhaps using
https://pypi.org/project/catalogue/#files as the URL for uscan (if uscan
is happy with that one), would solve that problem? (And if it does,
then perhaps python packages should be progressively transitioned to
use pypi URLs to avoid this kind of problem?)

But I'm thinking that, perhaps the best option is to ask upstream
directly, whether they are going to release a 2.1.x release soon, or
if they could do that now, and/or whether they could perhaps
remove/rename the git tag perhaps (with the implied issues with messing
with history and git tags being sticky on cloned repos)? As I assume
other downstreams might be in the same/similar situation?

Thanks,
Guillem

[Colin Watson]

Post by Guillem Jover

Post by Colin Watson
I'd initially misread it as being just a day or two after the yanked
version, but you're right, it was months later. I suspect it was simply
uscan - it's using the GitHub tags rather than looking at PyPI, and the
tag was never removed, so it's hard to see how it could have known any
better.
This does leave the question of how to hide that version from uscan in
the future, since uscan doesn't make it easy to ignore specific upstream
versions and I'd prefer to avoid using opaque regex constructions to do
it. My best idea is to use uversionmangle to turn 2.1.0 into something
like 2.0.8~pre1, but is there a better idiom?

Ah I assumed the sources were being taken from pypi, if they are taken
from GitHub, then that explains yes. Perhaps using
https://pypi.org/project/catalogue/#files as the URL for uscan (if uscan
is happy with that one), would solve that problem? (And if it does,
then perhaps python packages should be progressively transitioned to
use pypi URLs to avoid this kind of problem?)

Some do, but it can't be done systematically: if we get the orig.tar
from PyPI then it's the "sdist", which is built from the upstream
repository, but Python's build tooling unfortunately means it's a bit
easier than ideal for the sdist to be accidentally lacking some files,
such as documentation or tests. Examples from a quick look in my
browser history:

https://github.com/RKrahl/pytest-dependency/pull/79
https://github.com/pgjones/quart-trio/pull/19
https://github.com/jendrikseipp/vulture/pull/368

Post by Guillem Jover
But I'm thinking that, perhaps the best option is to ask upstream
directly, whether they are going to release a 2.1.x release soon, or
if they could do that now, and/or whether they could perhaps
remove/rename the git tag perhaps (with the implied issues with messing
with history and git tags being sticky on cloned repos)? As I assume
other downstreams might be in the same/similar situation?

Indeed. I've filed https://github.com/explosion/catalogue/issues/74
asking if they're willing to help out here.

[Colin Watson]

Post by Colin Watson

Post by Guillem Jover
But I'm thinking that, perhaps the best option is to ask upstream
directly, whether they are going to release a 2.1.x release soon, or
if they could do that now, and/or whether they could perhaps
remove/rename the git tag perhaps (with the implied issues with messing
with history and git tags being sticky on cloned repos)? As I assume
other downstreams might be in the same/similar situation?

Indeed. I've filed https://github.com/explosion/catalogue/issues/74
asking if they're willing to help out here.

Upstream hasn't responded yet, so I'm going to do the ugly uscan hack
for now:

diff --git a/debian/watch b/debian/watch
index f75b80b..fe03f69 100644
--- a/debian/watch
+++ b/debian/watch
@@ -1,4 +1,4 @@
version=4

-opts="filenamemangle=s%(?:.*?)?v?(\d[\d.]*)\.tar\.gz%@PACKAGE@-$1.tar.gz%" \
- https://github.com/explosion/catalogue/tags (?:.*?/)?v?@ANY_VERSION@\.tar\.gz
+opts="uversionmangle=s/(2\.0\..*)/2.1.0+really$1/,filenamemangle=s%(?:.*?)?v?(\d[\d.]*)\.tar\.gz%@PACKAGE@-$1.tar.gz%" \
+ https://github.com/explosion/catalogue/tags (?:.*?/)?v?(?!2\.1\.0)(\d[\-+\.:\~\d]*)\.tar\.gz

After a bit more digging, I did manage to track down why this happened
in the first place: upstream initially moved some configuration handling
from python-thinc to python-catalogue (using the Debian source package
names here for clarity), but that apparently "[caused] some issues with
other dependencies" (https://github.com/explosion/thinc/issues/408) and
they ended up creating python-confection instead. That's packaged in
Debian and it doesn't look as though our python-thinc package needs the
functionality that was removed from python-catalogue, so it should be
safe for me to go ahead with the +really hack here.

[Andreas Tille]

Hi,

Post by Guillem Jover
Ah I assumed the sources were being taken from pypi, if they are taken
from GitHub, then that explains yes. Perhaps using
https://pypi.org/project/catalogue/#files as the URL for uscan (if uscan
is happy with that one), would solve that problem? (And if it does,
then perhaps python packages should be progressively transitioned to
use pypi URLs to avoid this kind of problem?)

I have *not* inspected the situation personally. However, you might
want to check the difference between the PyPI tarball and the tarball
from Github. In lots of cases these are different and the maintainer
might have picked Github for a reason (which is actually the
recommendation I've read several times on the Debian Python list).
Common cases are missing test suite inside PyPI tarball but maybe other
things as well.

Post by Guillem Jover
But I'm thinking that, perhaps the best option is to ask upstream
directly, whether they are going to release a 2.1.x release soon, or
if they could do that now, and/or whether they could perhaps
remove/rename the git tag perhaps (with the implied issues with messing
with history and git tags being sticky on cloned repos)? As I assume
other downstreams might be in the same/similar situation?

Sounds sensible - otherwise I'd probably go with the override proposed
by Colin.

Kind regards
Andreas.