path: root/doc/translations_po.txt

Debian Installation Guide - notes for translations
--------------------------------------------------

This file explains how to translate the manual using .po files.
This is now officially supported and the recommended way.

!!! For general information on translating the manual, please see the
!!! translations.txt file.

Note: if you happen to modify this file, please don't forget to
      update the trailing changelog


Starting a new translation
==========================
This procedure should only be used if there is no existing translation using
XML files for your language! See the section on conversion below if you want
to switch from translating XML files to using PO files.

The template files for the manual are available in ./po/pot, so all you need
to do is create a new directory for your language (using the official ISO
code for your language) and copy all the .pot files to a .po file in that
directory.

Example for a Dutch translation:
  - mkdir ./po/nl
  - cp ./po/pot/bookinfo.pot ./po/nl/bookinfo.po
  - <po editor> ./po/nl/bookinfo.po

Use your favorite PO file editor (like lokalize or poedit) to translate the .po file.
You don't need to copy all POT files straight away, just copy each one as you
start translating them.

  
Committing your translation
===========================
The translation can be committed to d-i's git repository using:
  1. git add ./po/<lang>/*
  2. git commit -m "<some comment describing the changes>"
  2. git push

Tip: use the command 'git diff' before the git add command (step 1),
to check what you are committing.


Updating your translation from the english original
===================================================
The PO files for your language and the templates for the manual are
updated regularly when there are changes in the original English XML
files.  This is done during the "daily" builds (currently done by the i18n
coordinator); you should not have to update your PO files yourself, but
rather get the updates using 'git pull'. Changed strings will be marked
'fuzzy' and new strings will be untranslated.

After this your PO files in the ./po directory should be updated and you can
update the translation with your favorite PO file editor. To commit the
changes, follow again the procedure described in the previous section.

To update the POT and PO files manually, use the following commands:
  1. git pull
  2. ./scripts/merge_xml en
  3. ./scripts/update_pot
  4. ./scripts/update_po <your language>

NOTE
Doing this manually may result in git conflicts when you next 'git pull' with
the repository. The preferred method is to just wait until your PO files are
updated for you.
Hint: For refreshing all po and pot files in one rush, one can use the
./scripts/refresh-all-po-pot-files.sh script.


Building the manual yourself
============================
If you want to build a HTML version (or other format) of the manual yourself,
you will have to generate the XML for your language before you can run the
buildone.sh script. This can be useful, for example to debug build errors.

Use the following commands:
  1. ./scripts/merge_xml en
  2. ./scripts/update_pot
  3. ./scripts/update_po <your language>
  4. ./scripts/revert_pot
  5. ./scripts/create_xml <your language>

Steps 2 and 3 are necessary because the English XML files may have been
updated since the last daily build of the manual, so the POT files and your
PO files may be out-of-date. Step 4 makes sure you do not accidentally
update the POT files in the repository. They are not needed to create the
XML files or build the manual.

After that you can run the buildone.sh script as documented in the README in
the ./build directory. If you run buildone.sh without first generating the
XML files, you will either get an error message "unknown language" or build
from an outdated version of the XML files.

buildone_ng.sh
--------------
To provide an easier method for building the manual from po files,
a modified version of the buildone.sh script has been added as buildone_ng.sh,
which does all the needed steps including 1-5 from above to build the manual
out of your po files.
However, the update_pot and update_pot functions from above's step 1-3 are
performed in a temporary directory, so buildone_ng.sh cannot be used to update
your translation po files!
For this, you need to still follow the chapter "Updating your translation
from the english original".
The intention of buildone_ng.sh is just, to provide an easy method for
translators, to build the manual for testing their translation.


Background information
======================
There are currently two toolsets that support translating XML files using .po
files: KDE's poxml and po4a. We have chosen poxml as that gives better context
(it does not split paragraphs) for translation.
The downside of the poxml tools is that they require the .xml files to be
"proper" XML. Because some of the individual files in the English
original are not proper XML, we've had to "integrate" the files per chapter.

This means that instead of translating about 160 small files, you will be
translating about 16 large files.

Requirements
============
The following packages need to be installed on your system:
- gettext
- poxml
- gawk

If you also want to be able to build the manual, you also need to install the
build dependencies listed in build/README.

Available tools
===============
You will not be using poxml itself, but rather some scripts that call the
poxml tools.

- ./scripts/merge_xml  : merges the small .xml files into larger files
- ./scripts/update_pot : creates (new) .pot templates based on integrated
                         English .xml files
- ./scripts/revert_pot : used to revert changes in POT files to avoid
                         accidental commits
- ./scripts/update_po  : updates your .po files when there are changes in the
                         English text (the .pot templates)
- ./scripts/create_xml : creates translated .xml files from the merged English
                         .xml files and the translated .po files
- ./scripts/create_po  : used to convert an existing .xml translation to
                         using .po files
- ./scripts/set_untranslated : used for the conversion of existing translations

All scripts should be run from the ./installation-guide directory and take the language as
parameter.


Converting an existing translation from .xml files to .po files
===============================================================

!!! This paragraph only exists for historical reasons. There is no need for
!!! this operations anymore, since all translations have been converted into
!!! po files in the meantime.

It is possible to convert an existing translation of the manual (where the
individual XML files have been translated) to using PO files.
There are two important conditions:
- the translation must be up-to-date with the English XML files;
- the translation must have the same structure as the English XML files (i.e.
  the translation must have the same chapters, sections, paragraphs, tables,
  examples, etc; you can have no 'extra' paragraphs in your translation).

The first condition does not mean that everything needs to be translated!
This means you can insert the original text, that you can correct
in the PO files once they are generated.

The conversion is fairly difficult and some checks need to be done after some
of the steps. The following instructions are heavily based on the mail message
http://lists.debian.org/debian-boot/2007/07/msg00814.html, adding some caveats.
It takes German (de) as an example for the conversion.

1) Ensure the translation is _up to date_. The following command should return
   nothing.
   './scripts/doc-check de'

2) Fix any places where you know you have deviated from the original in
   your translation. The conversion will only work if the structure and
   markup of the translation and original are *identical*, at least at
   those points where the poxml utilities split out text into separate
   strings.

3) Run the following scripts:
   './scripts/merge_xml en'  # merge English XML files per chapter
   './scripts/update_pot'    # update POT files to match merged XML files
   This will create your base situation for conversion. Do _not_ 'git pull'
   after this or you may have to start from scratch.

4) Merge XML files for the translation:
   './scripts/merge_xml de'

5) Run the actual conversion:
   './scripts/create_po de'

   This may result in problems mentioned in step 7.

6) The previous step does not create the full msg headers needed in the PO
   file. For that, run the following script.
   './scripts/update_po de'

7) The result will not be perfect yet. Even if the script in
   step 5 completes successfully, you most likely do *not* yet have
   correct PO files. You need to carefully check the *complete output* and
   follow the instructions in the output of the conversion script for
   anomalies, fix them in the German translation and run the conversion
   again until there are no more anomalies. The command grep can help you
   to find the location of the strings in the translated XML files.
   Example: 'grep "term" -r -H de/'

   Note that some of the "differences" can be quite tricky to spot. You
   will probably have to start with the PO file and find the first point
   where msgid and msgstr no longer match (using clues from the output of
   the conversion script). Then check integrated original and translated
   XML files for the cause of the problem and fix it in the *translation*.
   The fixes can be done either in the regular German XML files, or in the
   "merged" ones. The last will probably be slightly easier and faster as
   it will be easier to match the errors in the PO files to the merged
   files and you won't have to repeat step 4 anymore. 

   Another source of information for structural failures in the translation
   is the appearance of two translations for the same term in English (shown
   in the output in step 5), as poxml reuses the first translation for the same
   original string when it appears more than once.
   Check carefully those messages as, at first, some will not match, and
   will offer "wrong" translations due to the structural disparity.
   You can use grep to search for any particular string in the translated
   documentation or PO files, and fix it in the XML files.

   Note: The "looks untranslated" test needs an update. It can
   currently report certain strings that are translated as untranslated.
   This is probably due to a change in the poxml tools since the scripts
   were written.

   Finally: a good method, to check the result of the conversation, is to build
   the text variant of the manual (build from your new po files), and compare 
   it against the respective text variant build from the original xml files.
   Use something like "diff -u install.de.orig.pdf install.de.pdf", to see
   the differences between the old xml-based and the new po-based manual.

8) Repeat steps 5, 6 and 7 (or 4, 5 and 6).

9) After you are happy with the conversion, run
   '.scripts/create_xml de'
   This will generate XML files in ./de.new (not in ./de as that directory
   is still in GIT). This should be without any errors.

   Possible errors:

   First, it may show there is an "extra" tag in the XML translation that may
   difficult the conversion from PO to XML. Again, use grep to find the
   "offending" tag, and fix it.
   
   This step might also show failures during XML conversion to PO:

   Another source of failure may be the gratuitous insertion of XML tags in PO
   files during 'create_po' which *should not* be present either in msgids or
   msgtrs. Once more, grep can assist you. It is also useful in this cases to
   check your file against an already PO-based translation, such as pt
   (Portuguese), in case of doubt.

10) You are almost done! Before uploading, it would be wise to build the
   translated documentation based on your new PO files. This step might show
   other inconsistencies present in the translated XML files, or in the PO
   files that resulted from errors in the conversion from XML to PO.

   It would be better if you do this in a different copy:

   'cp -r manual/ manualb && cd manualb'
   'mv de.new de'
   
   and build.

   Build both HTML and PDF formats, as the latter may show problems
   that do not (apparently) affect the building for HTML format. 

11) If all is correct, go back to the original "manual/" dir:
   'git checkout -f po/pot/'
   'git add po/de/'
   'git rm -r de'
   'git commit'
   'rm -r ./integrated/ ./de.new/'
 
Note
Do *not* update POT or PO files yourself after this point! The daily build 
scripts run by Felipe will take care of that.

Note that conversions are only done rarely and that the script used may
need updating.

--------------------------------------------------------------------------

If you find any errors in this file, please fix them.

--------------------------------------------------------------------------
Changelog:

2019-12-21 Holger Wansing
 - Update content (there are no xml-based translations anymore, po-based is
   the default).

2018/06/03 Holger Wansing
           Samuel Thibault
 - Updating (most) content on git migration

2017-04-12 Holger Wansing
 - Document new buildone_ng.sh script, to build manual from po files
   with one command.
 - Remove old 'Limitations' paragraph (from Sarge period).

2010-10-05 Omar Campagne
 - Document steps for conversion from XML to PO-based translations.

2010-05-01 Tapio Lehtonen
 - Typo fixes

2007-07-31 Frans Pop
 - Add link to full conversion instructions

2005-04-19 Frans Pop
 - Finally rewrite this file following official implementation of PO file
   translation over a month ago...
 - Reflect the changed file structure in the ./po directory

2005-02-15 Frans Pop
 - Corrected the location where generated XML files are written

2005-02-14 Frans Pop
 - Corrected an error in the name of one script
 - Clarified the difference between starting a new translation and
   converting an existing one

2005-02-12 Frans Pop
 - First version