General Questions about Translations and what a package maintainer has to do

[Marc Haber]

tl;dr: We need more docs about best practices to handle translations as
a package maintainer


My fellow developers,

there are two things in being a DD that I truly despise. The one is
keeping the machine readable debian/copyright up to date, the other is
handling of translations, regardless of whether it's po-debconf, manpage
translations or program translations (when I am also the upstream).

I might rant about debian/copyright when I blow my fuse next time, but
today it's going to be translations. For me, it seems impossible to
support translator's work without putting a significant burden of
additional work to put on oneself. Especially when one uses version
control and does not do all development in Mast^wdebian/latest, dealing
with translations is a nightmare when it comes to merge. As the Newbie¹
DD I am, I keep running into either nightmare merges, or unnecessarily
fuzzed or even destroyed translations, in all cases feeling even more
stupid and incompetent when some translator points out my mistakes. I
have felt being sent back and forth between different workflows (all of
them wrong) by following random advice without being able to find
authoritative explanation.

I might have a fundamantal misunderstandig of procedures, but all
documentation one finds, including Chapter 8 in the Developer's
Reference (which links to a document by Tomohiro KUBOTA which is no
longer there), elaborate on how one would do actual translations, but
doesn't go as far as giving best practice documentation about what a
package maintainer is supposed to do to make translation blend into a
normal packaging workflow without being a nuisance ("put them into the
po directory and build the package" doesn't fit a modern packaging
workflow using version control).

My example package is adduser, but I think that my questions might apply
to other packages as well. adduser has both its program messages and the
manual pages translatable, the latter being done with po4a. I am aware
that there are also translations for debconf templates, but adduser
doesn't have those (any more). I think the problems that show with
debconf template translations are similar to the pain one feels with
documentation and program translations. I actively avoid using debconf
in my packages because I don't want to go through the pain that handling
translations causes, and many parts of Debian consider it bad practice
to use debconf. But that's an entirely different rant.

Handling translations does hurt when the sources are stored in version
control because I constantly end up having changes to pot and po files
in commits where they don't belong, or uncommitted changes that prevent
me, for example, from doing rebases. I have tried to build a workflow
that doesn't hurt me as package maintainer as much, but it has turned
out that this doesn't work because many translators don't care.

Please don't take me wrong, I know this is a rant, and I know that
you'll notice that I am typing this with my fists clenched. But my time
and my nerves and my mental health as as important as it is to be nice
to my translators. I do care, but sometimes it's really hard to maintain
a straight and friendly face while cursing our tools and docs inside.

Whenever I am angry about something in Debian, I start writing docs. So
I try this here, but here I don't know enough to be really helpful. I
hope that this rant will start a positive discussion with actual results
that I could pour into a Wiki page that might actually help with the
pain I am feeling, assuming that many other maintainers feel as well.

Let me try to summarize what I have understood regarding translations
and what my problems are with that.

(1)
When writing software, docs or debconf templates, the respective author
marks certain strings as translateable. There is a number of conventions
to do so which are language dependent. Let's assume that has been done
the right way, there are docs about this.

(2)
There is some point in the development process when it is time to ask
for translations. Translators need a POT file which contains all the
translatable strings, and they make a PO file from that which contains
the actual translation.

(3)
Some program (xgettext for program translations, po4a for manual pages
and some podebconf tool for debconf templates) is used to pull the
translatable strings from the source code and to create a POT file.

xgettext doesn't even try to create a meaningful header and overwrites
whatever one has written into the previous version of the POT file, so a
wrapper is already needed to have a header that translators can fill
(which they usually don't do).

For Adduser's program translations, my call to xgettext is:
xgettext --keyword=mtx --keyword=gtx --omit-header -o "$TEMP_FILE"
--from-code=UTF-8 -L perl adduser deluser $(find . -name "*.pm")².
TEMP_FILE then gets the generated header prepended to result in
adduser.pot.

I have seen this being done in debian/rules' clean target which, in
in-tree builds, causes the POT file to be changed as well and I don't
understand at which step of the packaging process it would be a good
idea to commit that POT file. If I build my package out of tree (like I
do out of tradition of svn-buildpackage, I have gbp configured to use
../build-area), the POT file ends up newly generated in the source
package but never gets updated in git. Adduser had POT files from 2022
in git until just recently because I just never noticed. There is no
lintian check and no check inside tracker.d.o for this.

In other packages, there is a dedicated m4 macro to call xgettext which
doesn't make things easier to understand.

(4)
Then,
msgmerge --update --backup=none --no-fuzzy-matching "${PO_FILE}" "${POT_FILE}"
is called for every existing PO file. This doesn't move the header from
the POT file to the existing PO file so stupidities like "# COPYRIGHT
THE PACKAGE CREATOR" just never get fixed because the translators don't
seem to care.

If a po file for a language already exists during this step, the already
existing translation gets merged into the new po file. In some
circumstances that I have not understood yet, the translation gets
"fuzzied", which I have been told causes a lot of unnecessary and
repeated work for the translators which I am supposed to avoid by doing
manual work myself which I don't understand. Not doing this work is
condemned as "not being nice to translators".

Basically the same applies for this step than for the POT generation
step, with the additional hardship that the PO files are generated,
being written to by a program AND STILL contain a significant part of
human work. I never know how much work of other people I am destroying
by calling msgmerge out of line. In which stage of package build do I do
msgmerge? Do I commit the merged po files, when do I commit them, what
do I do with them during git merge when a feature branch is merged?

(5)
podebconf-report-po is used to generate the calls for translation. One
message is sent to this mailing list with the pot file attached, and for
each existing po file, the translators listed in that file get an
individual mail with just the respective po file attached.

If the msgmerge step is forgotten, they get an already translated po
file that doesn't match the pot and therefore is useless.

In theory, for an already existing package, the POT file is not needed,
right?

(6)
Depending on the age of the existing translations, about half of the
messages I send to individual translators are going to bounce. Am I
supposed to report that to debian-***@l.d.o as a followup to the
general translation request so that new translators can take up the
outdated translatorless translations? Or am I supposed to send the
general translation request to debian-i18n last so that I can explicitly
mention the translatorless languages there?

(8)
When a translator does a translation, they send me a new po file
containing the actual translation. If it's a new language, they start
with the pot file that hopefully has the correct header, and if it's an
existing language, they start with the old po file, which almost always
has a historically grown header that is in more or less dire need of
streamlining and cleaning. They either take the PO/POT file from the
e-mail attachment, use a package the pulled from the archive³ or they
pull the PO/POT file from git.

They usually don't bother about the header or copyright, so things like
package name, licenseª, Project-ID-Version and PO-Revision-Date are
often questionable, unclear, just plain wrong or cause extra work to
package maintainer because, for example, a different license was chosen
than the actual package is licensed under either out of incompetence or
not caring.

Am I supposed to fix those headers in the po file myself? Am I supposed
to ask the translator to fix the headers? Or am I supposed to just
ignore all of that and just accept whatever I get sent? I often feel
like a smart-mouthed know-it-all when I ask a translator to improve the
headers of their PO file.

(9)
I then commit the po file the translator sent me to version control

(10)
And then I eventually release the package.

In theory, it would probably be good to do all that regeneration when
preparing a package for release. Why don't we have a debian/rules target
like debian/rules prepare-release that might be useful for that? How
could we protect us against uploading a package with outdated POT/PO
files? People make mistakes.

How am I supposed to handle the unavoidable differences between git
branches, that are probably easier to solve when I am just merging a
feature branch but can be a major pain when merging suite branches like
experimental, stable, unstable where translation work has already been
done?

There must be some smarter method when merging to mas^wdebian/unstable
than (1) move away all po files, (2) merge, (3) ignore all merge
conflicts in po files, (4) regenerate POT, (5) restore po files moved
away in step 1, (6) msgmerge, (7) do a dedicated translation commit
(one? or one per file?).

I have caused enough breakage in adduser in the last weeks and have
wasted enough time of both translators and myself. For the time being, I
am halting all my efforts to be "nice to translators" to avoid breaking
more things and to keep the chaos in my package trees and version
control low. You can read from my git histories pretty well that I don't
know zilch about what I am doing.

This has to stop as long as being nice on translators means multiplying
my own degree of unthanked busy work.

This happens through a crucial part of adduser development since we are
nearing the freeze, but first I need to build knowledge that I should
have built 25 years ago but noone bothered to document. I really don't
know how translations in Debian have come up to THIS point in the
absence of serious docs. Maybe my fellow DDs are smarter than I am with
all tools involved.

Thanks for reading up to this point. Writing this message alone has cost
me three hours of my time that I'd rather have put in productive
packaging work, and a sleepless night. You know, when I blow a fuse, I
rant, and then I start writing docs. I guess when I put the result of
this discussion in a wiki page, it should be under i18n, right? I am
inclined to put on https://wiki.debian.org/I18n a dedicated chapter
titled "for package maintainers", probably between "Keyboard input
infrastructure support" and "Meetings" as this is a matter beyond
interna of the translation teams and the i18n effort. Am I on the
correct track with that?

Ich habe fertigǂ. Thanks in advance,
Marc Haber

¹ I have only been a DD for a bit more than two decades
² adduser has strings that get used in both translated and untranslated
form, making sure that messages written to the console are translated
and messages written to syslog are written in English to make handling
bug repors easier
³ I have received translations that were obviously done against the POT
file from stable.
ª I have received translations that placed the translation under the
same copyright as $SOME_OTHER_PACKAGE.
ǂ https://en.wikipedia.org/wiki/Giovanni_Trapattoni#In_popular_culture

[Simon Richter]

Hi,

I'm on mobile, so only a quick reply: merging should be done with msgmerge as well — you need to call it twice, once with the po files from both branches, and once again with the pot file to fix all the location comments and fuzzy flags.

Alternatively, you can define a filter in git to remove(!) locations on checkout and restore them from the pot file on commit, that solves the majority of conflicts.

   Simon

[Marc Haber]

Post by Simon Richter
I'm on mobile, so only a quick reply: merging should be done with
msgmerge as well — you need to call it twice, once with the po files
from both branches, and once again with the pot file to fix all the
location comments and fuzzy flags.

That sounds really interesting. Did anybody include this in git like for
the excellent dpkg-mergechangelogs, so that it can be done
automatically?

If doing that automatically doesn't work, I would love to see a
complete usecase.

Post by Simon Richter
Alternatively, you can define a filter in git to remove(!) locations on
checkout and restore them from the pot file on commit, that solves the
majority of conflicts.

How would that look like? git filters are a mystery for those wizards.

Greetings
Marc

[Maytham Alsudany]

Hi Marc,

Here are some of my thoughts based on my (relatively little) experience
with both translating to Arabic and receiving translations.

It might also be worthwhile to forward your message to
debian-***@l.d.o, since translators and i18n people are more likely to
be subscribed there and less likely to be subscribed here.

Post by Marc Haber
(3)
Some program (xgettext for program translations, po4a for manual pages
and some podebconf tool for debconf templates) is used to pull the
translatable strings from the source code and to create a POT file.
xgettext doesn't even try to create a meaningful header and overwrites
whatever one has written into the previous version of the POT file, so a
wrapper is already needed to have a header that translators can fill
(which they usually don't do).
xgettext --keyword=mtx --keyword=gtx --omit-header -o "$TEMP_FILE"
--from-code=UTF-8 -L perl adduser deluser $(find . -name "*.pm")².
TEMP_FILE then gets the generated header prepended to result in
adduser.pot.

xgettext comes with a ton of options to help you. Have a look at the
diff I've attached for what I've been able to do.

Note that you shouldn't define the plural stuff in the POT file, that's
something that's set on a per-language basis. There should also be a
newline between the header and first message block.

Post by Marc Haber
I have seen this being done in debian/rules' clean target which, in
in-tree builds, causes the POT file to be changed as well and I don't
understand at which step of the packaging process it would be a good
idea to commit that POT file. If I build my package out of tree (like I
do out of tradition of svn-buildpackage, I have gbp configured to use
../build-area), the POT file ends up newly generated in the source
package but never gets updated in git. Adduser had POT files from 2022
in git until just recently because I just never noticed. There is no
lintian check and no check inside tracker.d.o for this.
In other packages, there is a dedicated m4 macro to call xgettext which
doesn't make things easier to understand.

Usually, all this stuff with generating and updating POT & PO files is
upstream's responsibility to deal with, hence why you'll find little
documentation for translating anything other than debconf templates. Since this
is a native package, it's up to you to do what you want. My suggestion is to
run this script before release; the most important thing is that it is run
after the program's messages are updated and _finalised_, and before sending it
to translators.

Post by Marc Haber
(4)
Then,
msgmerge --update --backup=none --no-fuzzy-matching "${PO_FILE}" "${POT_FILE}"
is called for every existing PO file. This doesn't move the header from
the POT file to the existing PO file so stupidities like "# COPYRIGHT
THE PACKAGE CREATOR" just never get fixed because the translators don't
seem to care.

The header is only touched when the translation is initially created.

Post by Marc Haber
If a po file for a language already exists during this step, the already
existing translation gets merged into the new po file. In some
circumstances that I have not understood yet, the translation gets
"fuzzied",

A "fuzzy" translation is when the source string has been altered but not
entirely removed, meaning that the translated string needs to be
rechecked. This often occurs when there's a grammatical change or some
rewording in the source string.

Post by Marc Haber
which I have been told causes a lot of unnecessary and
repeated work for the translators which I am supposed to avoid by doing
manual work myself which I don't understand. Not doing this work is
condemned as "not being nice to translators".

Nothing is wrong with making translations fuzzy, sometimes it's
necessary.

I've rarely seen anyone being condemned for fuzzy translations (but then
again, I work in a language team that has virtually no members). The
only reason this would happen is if a maintainer kept making pointless
changes to the source strings to the point that translators are fed up
of reviewing the same strings over and over again, when they could be
creating new translations.

Post by Marc Haber
Basically the same applies for this step than for the POT generation
step, with the additional hardship that the PO files are generated,
being written to by a program AND STILL contain a significant part of
human work. I never know how much work of other people I am destroying
by calling msgmerge out of line.

Just make sure that you're happy with the state of messages in the
program, since removing any source strings deletes them from the
translations as well. Since Git is being used, this is theoretically
reversible, but really should be avoided in the first place.

Post by Marc Haber
In which stage of package build do I do
msgmerge? Do I commit the merged po files, when do I commit them, what
do I do with them during git merge when a feature branch is merged?

In your position, I would leave the translations and the POT file
untouched on the feature branch, and only ever update them on the main
branch after merging.

Post by Marc Haber
(5)
podebconf-report-po is used to generate the calls for translation. One
message is sent to this mailing list with the pot file attached, and for
each existing po file, the translators listed in that file get an
individual mail with just the respective po file attached.
If the msgmerge step is forgotten, they get an already translated po
file that doesn't match the pot and therefore is useless.
In theory, for an already existing package, the POT file is not needed,
right?

Yes, in theory, but it's still helpful to attach it for a variety of
reasons e.g. the existing translation is an old garbled mess and
starting new is the best option.

Post by Marc Haber
(6)
Depending on the age of the existing translations, about half of the
messages I send to individual translators are going to bounce. Am I
general translation request so that new translators can take up the
outdated translatorless translations? Or am I supposed to send the
general translation request to debian-i18n last so that I can explicitly
mention the translatorless languages there?

According to the manual page (and from what I've seen on
debian-l10n-***@l.d.o), the language team listed in the PO file (which
should _always_ be debian-l10n-***@l.d.o) is Cc'ed by default, so they
will deal with inactive translators. Anyone working on translations
should be subscribed to the list for the relevant language, so it'll be
picked up. You don't need to do anything extra.

Post by Marc Haber
(8)
When a translator does a translation, they send me a new po file
containing the actual translation. If it's a new language, they start
with the pot file that hopefully has the correct header, and if it's an
existing language, they start with the old po file, which almost always
has a historically grown header that is in more or less dire need of
streamlining and cleaning. They either take the PO/POT file from the
e-mail attachment, use a package the pulled from the archive³ or they
pull the PO/POT file from git.

They really shouldn't be taking it from the archive. If you're not happy
with the version of the POT they used, you can attach the correct POT
file and ask them to fix their translations. You're not obligated to use
the file they supply.

Post by Marc Haber
They usually don't bother about the header or copyright, so things like
package name, licenseª, Project-ID-Version and PO-Revision-Date are
often questionable, unclear, just plain wrong or cause extra work to
package maintainer because, for example, a different license was chosen
than the actual package is licensed under either out of incompetence or
not caring.
Am I supposed to fix those headers in the po file myself? Am I supposed
to ask the translator to fix the headers? Or am I supposed to just
ignore all of that and just accept whatever I get sent? I often feel
like a smart-mouthed know-it-all when I ask a translator to improve the
headers of their PO file.

What you do is up to you. Translation headers are annoyingly
inconsistent from a QA perspective, so don't feel bad for asking
translators to fix headers (or fix them yourself if it's easy enough).
I've had to do this for when I recieved translations for miniflux's
debconf templates.

Post by Marc Haber
(9)
I then commit the po file the translator sent me to version control
(10)
And then I eventually release the package.
In theory, it would probably be good to do all that regeneration when
preparing a package for release. Why don't we have a debian/rules target
like debian/rules prepare-release that might be useful for that? How
could we protect us against uploading a package with outdated POT/PO
files? People make mistakes.

I've attached a rough check script in the same diff that tells you if
you're POT is outdated, based on exit code. You could use this in a
pre-commit hook or somewhere in d/rules to fail the build (like
execute_before_dh_auto_configure).

Post by Marc Haber
How am I supposed to handle the unavoidable differences between git
branches, that are probably easier to solve when I am just merging a
feature branch but can be a major pain when merging suite branches like
experimental, stable, unstable where translation work has already been
done?

Wouldn't an "theirs" merge strategy for only the translations work?
During a merge conflict, you can use `git checkout --theirs -- po/`, or
use .gitattributes with `po/* merge=theirs` (I haven't tried this).
This way new translations from, for instance, debian/experimental, will
replace the old ones in debian/unstable.

But honestly, if some changes are in debian/experimental or in a feature
branch, the translations should really be left alone, since noone will
use it anyway and the package is prone to further changes.

Post by Marc Haber
There must be some smarter method when merging to mas^wdebian/unstable
than (1) move away all po files, (2) merge, (3) ignore all merge
conflicts in po files, (4) regenerate POT, (5) restore po files moved
away in step 1, (6) msgmerge, (7) do a dedicated translation commit
(one? or one per file?).

[...]

Post by Marc Haber
Thanks for reading up to this point. Writing this message alone has cost
me three hours of my time that I'd rather have put in productive
packaging work, and a sleepless night. You know, when I blow a fuse, I
rant, and then I start writing docs. I guess when I put the result of
this discussion in a wiki page, it should be under i18n, right? I am
inclined to put on https://wiki.debian.org/I18n a dedicated chapter
titled "for package maintainers", probably between "Keyboard input
infrastructure support" and "Meetings" as this is a matter beyond
interna of the translation teams and the i18n effort. Am I on the
correct track with that?

IMO this information should probably go on i18n.d.o.

If you don't want to deal with translation stuff, I'm happy to help with
that aspect, and if you'd like you can offload that on me.

--
Maytham

[Marc Haber]

Hi,

Post by Maytham Alsudany
It might also be worthwhile to forward your message to
be subscribed there and less likely to be subscribed here.

I didn't do that on purpose because the documentation of i18n shows that
the people who are currently writing those docs don't care too much
about actually building packages. I might ask them to review my wiki
page once it is finished.

Post by Maytham Alsudany
xgettext comes with a ton of options to help you. Have a look at the
diff I've attached for what I've been able to do.

Will do and take your suggestions. But it is still a wrapper needed.

Post by Maytham Alsudany
Note that you shouldn't define the plural stuff in the POT file, that's
something that's set on a per-language basis. There should also be a
newline between the header and first message block.

Noted.

Post by Maytham Alsudany

Post by Marc Haber
I have seen this being done in debian/rules' clean target which, in
in-tree builds, causes the POT file to be changed as well and I don't
understand at which step of the packaging process it would be a good
idea to commit that POT file. If I build my package out of tree (like I
do out of tradition of svn-buildpackage, I have gbp configured to use
../build-area), the POT file ends up newly generated in the source
package but never gets updated in git. Adduser had POT files from 2022
in git until just recently because I just never noticed. There is no
lintian check and no check inside tracker.d.o for this.
In other packages, there is a dedicated m4 macro to call xgettext which
doesn't make things easier to understand.

Usually, all this stuff with generating and updating POT & PO files is
upstream's responsibility to deal with, hence why you'll find little
documentation for translating anything other than debconf templates.

The mechanism and the pain seems similar to identical. Text => POT =>
PO.

Post by Maytham Alsudany
Since this
is a native package, it's up to you to do what you want. My suggestion is to
run this script before release; the most important thing is that it is run
after the program's messages are updated and _finalised_, and before sending it
to translators.

Is it really the right thing to do that in debian/rules clean target? I
don't expect anything to change during that step that I am supposed to
commit, and it does it step in a place that is discarded after build
when doing an out of tree build.

I think that _this_ philosophy belongs in debian-devel, and I am quite
surprised that noone has an opinion about that.

Post by Maytham Alsudany

Post by Marc Haber
(4)
Then,
msgmerge --update --backup=none --no-fuzzy-matching "${PO_FILE}" "${POT_FILE}"
is called for every existing PO file. This doesn't move the header from
the POT file to the existing PO file so stupidities like "# COPYRIGHT
THE PACKAGE CREATOR" just never get fixed because the translators don't
seem to care.

The header is only touched when the translation is initially created.

I am not sure whether this is good.

Post by Maytham Alsudany
I've rarely seen anyone being condemned for fuzzy translations (but then
again, I work in a language team that has virtually no members).

That was an exaggeration. But I have experienced an instance where
somebody helping me called it "unfriendly" to not manually unfuzzy all
translations after fixing a punctuation issue.

Post by Maytham Alsudany

Post by Marc Haber
In which stage of package build do I do
msgmerge? Do I commit the merged po files, when do I commit them, what
do I do with them during git merge when a feature branch is merged?

In your position, I would leave the translations and the POT file
untouched on the feature branch, and only ever update them on the main
branch after merging.

That doesn't happen when you work on a feature branch and do in-tree
builds. One would have to be careful to NOT commit those changed files,
and people routinely using git add -a (not me) are going to have commits
of relevant changes to their code/packaging and irrelevant changes to
POT/PO files in the same commit.

Post by Maytham Alsudany
Yes, in theory, but it's still helpful to attach it for a variety of
reasons e.g. the existing translation is an old garbled mess and
starting new is the best option.

So podebconf-report-po should have an option to include the POT file
even when asking for individual translations?

Post by Maytham Alsudany

Post by Marc Haber
(6)
Depending on the age of the existing translations, about half of the
messages I send to individual translators are going to bounce. Am I
general translation request so that new translators can take up the
outdated translatorless translations? Or am I supposed to send the
general translation request to debian-i18n last so that I can explicitly
mention the translatorless languages there?

According to the manual page (and from what I've seen on
will deal with inactive translators. Anyone working on translations
should be subscribed to the list for the relevant language, so it'll be
picked up. You don't need to do anything extra.

There are languages that don't use the debian-l10n-LANG mailing list (or
the translator didn't care to put in the correct Team address). I am
seeig this for Danish, Basque, Japanese, Korean, Norwegian Bokmal,
Portuguese, Brazilian Portuguese, Slovakian, Ukrainian, and Simplified
Chinese.

Post by Maytham Alsudany

Post by Marc Haber
Am I supposed to fix those headers in the po file myself? Am I supposed
to ask the translator to fix the headers? Or am I supposed to just
ignore all of that and just accept whatever I get sent? I often feel
like a smart-mouthed know-it-all when I ask a translator to improve the
headers of their PO file.

What you do is up to you. Translation headers are annoyingly
inconsistent from a QA perspective, so don't feel bad for asking
translators to fix headers (or fix them yourself if it's easy enough).
I've had to do this for when I recieved translations for miniflux's
debconf templates.

So it is okay to fix headers in translations?

Sadly, I have some translations that have things like "Copyright SPI" or
"Copyright FSF" in their header, and _that_ is not easily removed.

Post by Maytham Alsudany

Post by Marc Haber
(9)
I then commit the po file the translator sent me to version control
(10)
And then I eventually release the package.
In theory, it would probably be good to do all that regeneration when
preparing a package for release. Why don't we have a debian/rules target
like debian/rules prepare-release that might be useful for that? How
could we protect us against uploading a package with outdated POT/PO
files? People make mistakes.

I've attached a rough check script in the same diff that tells you if
you're POT is outdated, based on exit code. You could use this in a
pre-commit hook or somewhere in d/rules to fail the build (like
execute_before_dh_auto_configure).

Sounds good. But, how does the build process distinguish whether I am
building for testing, on a feature branch, or for a release? That all
ends up with a human having to remember what to do when building for a
upload.

Post by Maytham Alsudany

Post by Marc Haber
How am I supposed to handle the unavoidable differences between git
branches, that are probably easier to solve when I am just merging a
feature branch but can be a major pain when merging suite branches like
experimental, stable, unstable where translation work has already been
done?

Wouldn't an "theirs" merge strategy for only the translations work?
During a merge conflict, you can use `git checkout --theirs -- po/`, or
use .gitattributes with `po/* merge=theirs` (I haven't tried this).
This way new translations from, for instance, debian/experimental, will
replace the old ones in debian/unstable.

Merge strategies are a mystery to me, worsened by the fact that they
work a bit (but not completely) the other way arouse when rebasing.

Post by Maytham Alsudany
But honestly, if some changes are in debian/experimental or in a feature
branch, the translations should really be left alone, since noone will
use it anyway and the package is prone to further changes.

I'd love to have automated tool support for that.

Post by Maytham Alsudany

Post by Marc Haber
Thanks for reading up to this point. Writing this message alone has cost
me three hours of my time that I'd rather have put in productive
packaging work, and a sleepless night. You know, when I blow a fuse, I
rant, and then I start writing docs. I guess when I put the result of
this discussion in a wiki page, it should be under i18n, right? I am
inclined to put on https://wiki.debian.org/I18n a dedicated chapter
titled "for package maintainers", probably between "Keyboard input
infrastructure support" and "Meetings" as this is a matter beyond
interna of the translation teams and the i18n effort. Am I on the
correct track with that?

IMO this information should probably go on i18n.d.o.

That page only has a list of languages and links to the respective
coordination pages. And it also links to wiki.d.o/I18n, and that is a
page that I can change.

Post by Maytham Alsudany
If you don't want to deal with translation stuff, I'm happy to help with
that aspect, and if you'd like you can offload that on me.

Thank you for that offer. I need to understand things to be comfortable
with tem. I think I am on a good way.

Greetings
Marc

[Marc Haber]

Post by Marc Haber

Post by Maytham Alsudany
xgettext comes with a ton of options to help you. Have a look at the
diff I've attached for what I've been able to do.

Will do and take your suggestions. But it is still a wrapper needed.

I have put this into the adduser wrapper and extended it. I also have
filed https://savannah.gnu.org/bugs/index.php?66933 against upstream
gettext to introduce a --pot-title and --content-charset option.

I would like to hear opinions about translation copyright. xgettext
generates a single copyright line to be filled in after pot generation.
But technically, the strings pulled into the potfile fall under the
copyright of the program authors (potentially all of them) as they are
pulled from the actual source code. xgettext would have to have an
option to pull in the possibly multi-line copyright string from a file
since noone would like to give tens of copyright holders on the xgettext
command line. Then, the translators would have to add themselves to the
POT file.

In addition the # FIRST AUTHOR <***@ADDRESS>, YEAR header generated
by xgettext seems redundant.

It looks to me like the comment headers generated by xgettext are not
very well thought out and impractical for software that has grown over
the years.

Am I exaggerating?

Greetings
Marc

[Julien Plissonneau Duquène]

Hi Dirk,

As I understand your question about a general workflow of translations
in all your email-cases correct, I think the short answer maybe to use
`quilt` to patch the .PO files instead of committing them directly via
Git. Also consider `gbp-pq`, could be useful, but not sure.

Thank you for the time you took to write this reply. However Mark did
not explicitly mention that fact, but he was talking about the `adduser`
package which is a (Debian) native package, which means there is no
further "upstream" and thus there is no reason to use quilt and patch
queues in this case (that would only further complicate things).

Cheers,

[Marc Haber]

Post by Julien Plissonneau Duquène

As I understand your question about a general workflow of translations
in all your email-cases correct, I think the short answer maybe to use
`quilt` to patch the .PO files instead of committing them directly via
Git. Also consider `gbp-pq`, could be useful, but not sure.

Thank you for the time you took to write this reply. However Mark did
not explicitly mention that fact, but he was talking about the
`adduser` package which is a (Debian) native package, which means
there is no further "upstream" and thus there is no reason to use
quilt and patch queues in this case (that would only further
complicate things).

Thank you for setting this straight. In fact, when I receive updated
message texts or translations for programs that have an active upstream
(such as sudo), I usually encourage the translator to send the
translation upstream so that all users of this program benefit from the
improved translation. In rare cases, I do the forwarding myself, but I
would never^wthink three times before patching an upstraem translation
in the Debian source.

Greetings
Marc

[Julien Plissonneau Duquène]

Hi,

Especially when one uses version control

Having recently worked on a (not yet mergeable as it depends on another
MR) update of the french translation of devscripts and already bracing
for the incoming pain as it's likely that I will have to rebase and
re-update that work several times until it becomes mergeable, I can
understand your state of mind.

(2)
There is some point in the development process when it is time to ask
for translations. Translators need a POT file which contains all the
translatable strings, and they make a PO file from that which contains
the actual translation.

That's for the initial translation. Once there is a .po they will update
it directly and don't need or use the .pot anymore.

(4)
Then,
msgmerge --update --backup=none --no-fuzzy-matching "${PO_FILE}" "${POT_FILE}"
is called for every existing PO file. This doesn't move the header from
the POT file to the existing PO file so stupidities like "# COPYRIGHT
THE PACKAGE CREATOR" just never get fixed because the translators don't
seem to care.

I'm not sure the --no-fuzzy-matching helps here (see below). Also I note
that msgmerge is not used at all in devscripts packaging, which only
uses po4a instead. Maybe you could look there if it could help your
case.

If a po file for a language already exists during this step, the already
existing translation gets merged into the new po file. In some
circumstances that I have not understood yet, the translation gets
"fuzzied", which I have been told causes a lot of unnecessary and
repeated work for the translators which I am supposed to avoid by doing
manual work myself which I don't understand. Not doing this work is
condemned as "not being nice to translators".

As was pointed out in another message, "fuzzied" normally means the
source message was slightly modified, e.g. an additional value for an
option was added, but msgmerge was still able to match the original
message location. In this case it means that the translation previous
translation is retained but is now out-of-date. This is actually useful
for translation work, as it makes it possible to locate translation
strings that need to be worked on.

In a few cases there is spurious "fuzzing", e.g. when the source message
uses `...' for quoting where it shoud have been using \"...\". In some
of these cases it may be possible to fix the issue in the source
message, but I believe other cases are actually bugs in msgmerge. Are
these what your translators are asking you to deal with?

In theory, for an already existing package, the POT file is not needed,
right?

It's needed to translate to a new language, and it's nice to have it in
other cases.

They usually don't bother about the header or copyright, so things like
package name, licenseª, Project-ID-Version and PO-Revision-Date are
often questionable, unclear, just plain wrong or cause extra work to
package maintainer because, for example, a different license was chosen
than the actual package is licensed under either out of incompetence or
not caring.

Some dedicated tools (e.g. poedit) automatically update some headers
(including PO-Revision-Date) and make it possible to update other
comments. They also mess up the formatting (e.g. wrapping).

On a merge request I would just post comments to ask the translator to
fix the headers metadata and licensing, but may still deal with
normalizing/rewrapping myself (e.g. by adding a commit to the MR where
possible). If this were exchanged over e-mail I would probably fix most
trivial things myself.

² adduser has strings that get used in both translated and untranslated
form, making sure that messages written to the console are translated
and messages written to syslog are written in English to make handling
bug repors easier

The "industrial" way to deal with in other (usually large) projects is
to pre/postfix all source log messages with a unique identifier e.g.
"(ADDUSER-1234)". This makes it possible to have tech-support-friendly
and end-user-friendly translated log messages, and as an added benefit
it has excellent SEO characteristics when it comes to searching for
workarounds on the web or in a knowledge base.

³ I have received translations that were obviously done against the POT
file from stable.

That's a start, I guess. Maybe in these cases you can keep the .po file
as submitted for proposed updates, merge it in unstable and nicely ask
the translator to also please work on the upcoming version.

Cheers,

[Marc Haber]

Post by Julien Plissonneau Duquène

Post by Marc Haber
(2)
There is some point in the development process when it is time to ask
for translations. Translators need a POT file which contains all the
translatable strings, and they make a PO file from that which contains
the actual translation.

That's for the initial translation. Once there is a .po they will
update it directly and don't need or use the .pot anymore.

So all love I put into the POTs header will go unnoticed and I'll need
to pay attention to individual translations' header until somebody else
cares.

Post by Julien Plissonneau Duquène

Post by Marc Haber
(4)
Then,
msgmerge --update --backup=none --no-fuzzy-matching "${PO_FILE}" "${POT_FILE}"
is called for every existing PO file. This doesn't move the header from
the POT file to the existing PO file so stupidities like "# COPYRIGHT
THE PACKAGE CREATOR" just never get fixed because the translators don't
seem to care.

I'm not sure the --no-fuzzy-matching helps here (see below).

Since --no-fuzzy-matching is just superficially documented, ...

Post by Julien Plissonneau Duquène
Also I
note that msgmerge is not used at all in devscripts packaging, which
only uses po4a instead. Maybe you could look there if it could help
your case.

po4a seems to try to do those things automatically, using msgmerge as a
subprocess.

Post by Julien Plissonneau Duquène
In a few cases there is spurious "fuzzing", e.g. when the source
message uses `...' for quoting where it shoud have been using \"...\".
In some of these cases it may be possible to fix the issue in the
source message, but I believe other cases are actually bugs in
msgmerge. Are these what your translators are asking you to deal with?

Not directly the translators, but many years ago somebody who helped me
with making a package fit for translation mentioned that such changes
are unfriendly to translators and that one should handle those manually
and unfuzz things (in a manual process that seemed clumsy and
error-prone to me even back then).

Post by Julien Plissonneau Duquène
On a merge request I would just post comments to ask the translator to
fix the headers metadata and licensing, but may still deal with
normalizing/rewrapping myself (e.g. by adding a commit to the MR where
possible). If this were exchanged over e-mail I would probably fix
most trivial things myself.

So you're saying that as package maintainer I can freely edit headers
and metadata for translations, putting the PO file into a mixed domain
between package maintainer and the respective translator?

Post by Julien Plissonneau Duquène

Post by Marc Haber
² adduser has strings that get used in both translated and untranslated
form, making sure that messages written to the console are translated
and messages written to syslog are written in English to make handling
bug repors easier

The "industrial" way to deal with in other (usually large) projects is
to pre/postfix all source log messages with a unique identifier e.g.
"(ADDUSER-1234)". This makes it possible to have tech-support-friendly
and end-user-friendly translated log messages, and as an added benefit
it has excellent SEO characteristics when it comes to searching for
workarounds on the web or in a knowledge base.

Yes, I'd rather not do that because my inner monk would first cause me
spending a few days to catalogize and abstract adduser's messages. Time
that I'd rahter not spend at the moment.

Post by Julien Plissonneau Duquène

Post by Marc Haber
³ I have received translations that were obviously done against the POT
file from stable.

That's a start, I guess. Maybe in these cases you can keep the .po
file as submitted for proposed updates, merge it in unstable and
nicely ask the translator to also please work on the upcoming version.

But translating a stable package does show a blatant non-understanding
of Debian's development mechanisms! How am I supposed to trust people
who care THIS little about the project they're contributing to?

Greetings
Marc

[Julien Plissonneau Duquène]

Post by Marc Haber

Post by Julien Plissonneau Duquène
In a few cases there is spurious "fuzzing", e.g. when the source
message uses `...' for quoting where it shoud have been using \"...\".
In some of these cases it may be possible to fix the issue in the
source message, but I believe other cases are actually bugs in
msgmerge. Are these what your translators are asking you to deal with?

Not directly the translators, but many years ago somebody who helped me
with making a package fit for translation mentioned that such changes
are unfriendly to translators and that one should handle those manually
and unfuzz things (in a manual process that seemed clumsy and
error-prone to me even back then).

It's hard to tell if this advice is still relevant without more details,
but IMO expecting the maintainers to unfuzz translations themselves is
not a sound workflow, especially if that has to be done manually.

Post by Marc Haber

Post by Julien Plissonneau Duquène
On a merge request I would just post comments to ask the translator to
fix the headers metadata and licensing, but may still deal with
normalizing/rewrapping myself (e.g. by adding a commit to the MR where
possible). If this were exchanged over e-mail I would probably fix
most trivial things myself.

So you're saying that as package maintainer I can freely edit headers
and metadata for translations, putting the PO file into a mixed domain
between package maintainer and the respective translator?

Yes. They hand you the manuscript but you are the editor and printer.
Header and metadata changes where the translator is not expected to have
any reason to disagree with your changes are fine. You may also tell the
translator that you are going to do these changes and give them a few
days to react, to be nicer.

Post by Marc Haber

Post by Julien Plissonneau Duquène

Post by Marc Haber
³ I have received translations that were obviously done against the POT
file from stable.

That's a start, I guess. Maybe in these cases you can keep the .po
file as submitted for proposed updates, merge it in unstable and
nicely ask the translator to also please work on the upcoming version.

But translating a stable package does show a blatant non-understanding
of Debian's development mechanisms! How am I supposed to trust people
who care THIS little about the project they're contributing to?

It's not necessarily that they care little, more probably they don't
have enough experience with software development or even engineering in
general and don't realize the extent of established processes that could
be in place, so they don't even search for them, or if they do they
don't know what to look for and where.

There is actually a fairly elaborate translation workflow, apparently
followed by most translation teams but AFAICT only fully documented here
[1] (that is, in french) and more specifically here [2] for Debian
native software. Don't take it personally: as an established maintainer
of i18nized software did you know about this process? Is there a page in
english or in translator's language you could refer them to so you could
kindly ask them to have their work reviewed and proofread as part of the
process (as you can't yourself check that there isn't, for example,
inappropriate language in the submitted translation), even though they
missed the initial steps?

Cheers,


[1]: https://www.debian.org/international/french/
[2]: https://www.debian.org/international/french/po

[Marc Haber]

On Wed, 19 Mar 2025 10:46:32 +0100, Julien Plissonneau Duquène

Post by Julien Plissonneau Duquène
Don't take it personally: as an established maintainer
of i18nized software did you know about this process?

No, I didn't. I am arrogant enough to expect that relevant docs are
easy to find and in English. I had four years of French in high
school, in practice that is barely enough to navigate public transport
in Paris. When I try to speak French, every second word is English and
the Grammar anyway.

Oh, my, am I going to have fun in Brest?

Greetings
Marc

[Marc Haber]

tl;dr: New Wiki Page https://wiki.debian.org/I18n/ForPackageMaintainers,
please review

Hi,

Post by Marc Haber
Whenever I am angry about something in Debian, I start writing docs. So
I try this here, but here I don't know enough to be really helpful. I
hope that this rant will start a positive discussion with actual results
that I could pour into a Wiki page that might actually help with the
pain I am feeling, assuming that many other maintainers feel as well.

Sadly, the disusscion didn't really get traction. But there was
valueable advice given. You can read up about this in the archive,
beginning with my rant,
https://lists.debian.org/debian-devel/2025/03/msg00337.html

I have put my knowledge and the result of this discussion in a wiki
page, https://wiki.debian.org/I18n/ForPackageMaintainers and have also
taken the liberty of putting a link in a prominent place of
https://wiki.debian.org/I18n

I hope this might be helpful to package maintainers in the presence of
the majority of docs that is mainly targeted at translations.

Sadly, I haven't fond a comfortable solution about handling merge and
rebase conflicts in POT and PO files. Additionally, I think that the
widely common practice of creating/updating POT and PO files in the
debian/rules clean target is somewhere between suboptimal and wrong.

I would like you people, especially you on debian-i18n¹, to review the
wiki page and give feedback (resp. direct edit the page) to make it the
authoritative documentation for package maintiners.

Thanks for helping!

Greetings
Marc

¹ I initially left out debian-i18n deliberately because I think that the
target audience is more on debian-devel, but since there are people who
think that debian-i18n can help her as well I am Ccing this to
debian-i18n. Reply-To is set to debian-devel though, feel free to ignore.
I am reading both lists.

[Helge Kreutzmann]

Hello Marc,

Post by Marc Haber
tl;dr: New Wiki Page https://wiki.debian.org/I18n/ForPackageMaintainers,
please review

I just did it. I usually updated it right away, but at one or two
points it is more like a discussion. And due to my changes some
redundancy is included, but I think this is more helpful during
development, you can streamline it of course in the next step(s).

If you have any questions on my changes it is probably best to discuss
this on list. Please keep me (or -i18n) in CC, as I'm not subscribed
to -devel.

Thanks.

Greetings

Helge

[Marc Haber]

Post by Helge Kreutzmann

Post by Marc Haber
tl;dr: New Wiki Page https://wiki.debian.org/I18n/ForPackageMaintainers,
please review

I just did it. I usually updated it right away, but at one or two
points it is more like a discussion. And due to my changes some
redundancy is included, but I think this is more helpful during
development, you can streamline it of course in the next step(s).

I appreciate your help. Thank you very much.

Post by Helge Kreutzmann
If you have any questions on my changes it is probably best to discuss
this on list. Please keep me (or -i18n) in CC, as I'm not subscribed
to -devel.

I think the only matter that needs discussion is whether and when to
commit updated PO files:

Me:
It is currently not clear how to make sure that a package maintainer
does not forget these updates without creating lots of useless commits
with new PO files that only differ in line number and date stamp
comments.

You:
Using no line numbers reduced this problem. Updated time stamps are no
worry for translators (they seldomly look at them). And they do not care
for commits - they look at master or similar or on the web pages
mentioned above and take what they find. So the most important part is
to keep this current. If you need this for your VCS, you can add code in
your build system to discard po(t) file updates which only change in the
date stamp.

My questions:
"Using no line numbers" => invoke msgmerge with --no-location?
"Web pages mentioned above" => I don't see web pages being mentioned.
That needs a name or a link
"Add code in your build system to discard po(t) that only change in the
date stamp" => that would mak ethe source package and the tag in the VCS
diverge. I don't like that at all.

Greetings
Marc

[Helge Kreutzmann]

Hello Marc,

Post by Helge Kreutzmann
If you have any questions on my changes it is probably best to discuss
this on list. Please keep me (or -i18n) in CC, as I'm not subscribed
to -devel.

I think the only matter that needs discussion is whether and when to commit
It is currently not clear how to make sure that a package maintainer does
not forget these updates without creating lots of useless commits with new
PO files that only differ in line number and date stamp comments.
Using no line numbers reduced this problem. Updated time stamps are no worry
for translators (they seldomly look at them). And they do not care for
commits - they look at master or similar or on the web pages mentioned above
and take what they find. So the most important part is to keep this current.
If you need this for your VCS, you can add code in your build system to
discard po(t) file updates which only change in the date stamp.
"Using no line numbers" => invoke msgmerge with --no-location?

Yes.

"Web pages mentioned above" => I don't see web pages being mentioned. That
needs a name or a link

I meant the references to the Debian status pages, the link I inserted
further above: https://www.debian.org/international/l10n/

Please reword this to make it more clear.

"Add code in your build system to discard po(t) that only change in the date
stamp" => that would mak ethe source package and the tag in the VCS diverge.
I don't like that at all.

This I don't understand.

At some stage you update the POT files. Then you run a (git) commit,
to place the updated files in your repository.

In manpages-l10n Tobias added code to detect, if only the time stamp
changed. If so, the time stamp is reverted to the previous value, and
a "git commit" is a noop. Then also the po files are left alone.

This "only" saves you a commit in this corner case.

It is not meant to diverge version, because in the end the po(t) files
in your package should match the po(t) files in the repository.

Greetings

Helge

[Marc Haber]

Post by Helge Kreutzmann

Post by Marc Haber
"Using no line numbers" => invoke msgmerge with --no-location?

Yes.

Post by Marc Haber
"Web pages mentioned above" => I don't see web pages being mentioned. That
needs a name or a link

I meant the references to the Debian status pages, the link I inserted
further above: https://www.debian.org/international/l10n/
Please reword this to make it more clear.

Done.

Post by Helge Kreutzmann

Post by Marc Haber
"Add code in your build system to discard po(t) that only change in the date
stamp" => that would mak ethe source package and the tag in the VCS diverge.
I don't like that at all.

This I don't understand.
At some stage you update the POT files. Then you run a (git) commit,
to place the updated files in your repository.

Yes, this should be a manual process, in my opinion.

Post by Helge Kreutzmann
In manpages-l10n Tobias added code to detect, if only the time stamp
changed. If so, the time stamp is reverted to the previous value, and
a "git commit" is a noop. Then also the po files are left alone.

That should be a feature of xgettext or the other tools that might write
the POT files: Don't regenerate the file if the only change is the time
stamp. Since xgettext is unlikely to change¹

Post by Helge Kreutzmann
This "only" saves you a commit in this corner case.
It is not meant to diverge version, because in the end the po(t) files
in your package should match the po(t) files in the repository.

The problem ist when POT and PO files are regenerated (and actually
change) in package build when you're doing an out-of-tree build.

For example, I have export-dir=../build-area set in my gbp.conf. When
building a package, gbp copies the current working copy (or exports the
requested branch) to a temporary directory under ../build-area/ and does
the actual build from there. The temporary build directory is then
discarded, leaving the current working copy unchanged. When the build
process regenerates POT and PO files, those changes never find their way
back into your working copy, making the source package and the VCS
diverge.

That's why I believe that any point in the package build is the wrong
place to regenerate PO and POT files. Either that, or we need to
strongly discourage doing out-of-tree builds.

Greetings
Marc

¹ I tried to put some of the ideas that this discussion brought into
xgettext / msgmerge bugs and they got rejected upstream quite quickly. I
will refrain from wasting my time in this regard in the future.

[Marc Haber]

I have a bit played around with `msgmerge` and found a way to display
a `diff` which contains the translations which will be destroyed if
you merge a .po file from a translator
* test-de_DE-transl.po
into your maintained one
* test-de_DE.po
in conjunction with the current generated .pot file
* test.pot
The essential code snippet looks like this
**********************************************************************
*
* msgmerge -q -C test-de_DE-transl.po test-de_DE.po test.pot \
* > test-de_DE-maint+transl.po
* msgmerge -vv -q -C test-de_DE.po test-de_DE-transl.po test.pot \
* > test-de_DE-transl+maint.po
*
* # Print out the new merged .po file
* cat test-de_DE-transl+maint.po
*
* # Show the diff of destroyed translations (including comments)
* diff -u test-de_DE-maint+transl.po test-de_DE-transl+maint.po
*
**********************************************************************

I think this is way beyond what we could expact from J Random Package
Maintainer, unless it is fully automated and can be documented and
packaged.

Greetings
Marc

[Bill Allombert]

Post by Marc Haber
tl;dr: We need more docs about best practices to handle translations as
a package maintainer

I would suggest you find someone on debian-i18n willing to handle translations for
adduser for you and give them commit access.
That is how popularity-translation translation have been handled for a long time
(thanks, Christian Perrier, I learned everything about translations
handling this way!)

Cheers,
Bill.

[Marc Haber]

On Tue, 25 Mar 2025 21:26:05 +0000, Bill Allombert

Post by Bill Allombert

Post by Marc Haber
tl;dr: We need more docs about best practices to handle translations as
a package maintainer

I would suggest you find someone on debian-i18n willing to handle translations for
adduser for you and give them commit access.

That's how I tried, that just ended up in more chaos, leading up to my
fuse blowing and trying to understand what happens.

Post by Bill Allombert
That is how popularity-translation translation have been handled for a long time
(thanks, Christian Perrier, I learned everything about translations
handling this way!)

It should stil be documented for mortals to understand.

Greetings
Marc