Merge pull request MarlinFirmware#3754 from gudnimg/languages-gudni

Update language script to run on Windows

lang/README.md

@@ -8,9 +8,9 @@ Firmware support is controlled by the ``LANG_MODE`` define in the configuration,
 
 ### Required tools
 
-Python 3 with the ``regex``, ``pyelftools`` and ``polib`` modules as well as ``gettext`` and ``dos2unix``. On a debian-based distribution, install the required packages with:
+Python 3 is the main tool. To install the required packages run the following command in the `lang` folder:
 
-    sudo apt-get install python3-regex python3-pyelftools python3-polib gettext dos2unix
+    pip install -r requirements.txt
 
 ### Main summary
 
@@ -26,8 +26,8 @@ High-level tools:
 * ``config.sh``: Language selection/configuration
 * ``fw-build.sh``: Builds the final multi-language hex file into this directory
 * ``fw-clean.sh``: Cleanup temporary files left by ``fw-build.sh``
-* ``update-pot.sh``: Extract internationalized strings from the sources and place them inside ``po/Firmware.pot``
-* ``update-po.sh``: Refresh po file/s with new translations from the main pot file.
+* ``update-pot.py``: Extract internationalized strings from the sources and place them inside ``po/Firmware.pot``
+* ``update-po.py``: Refresh po file/s with new translations from the main pot file.
 
 Lower-level tools:
 
@@ -47,6 +47,28 @@ This step is already performed for you when using ``build.sh`` or ``PF-build.sh`
 
 ### Updating an existing translation
 
+#### How to update `.pot` file
+
+Run 
+
+     python update-pot.py
+
+to regenerate ``po/Firmware.pot`` and verify that the annotation has been picked up correctly. You can stop here if you only care about the annotation.
+
+#### How to update `.po` file
+
+To update a single `.po` file:
+
+    python update-po.py --file Firmware_XY.po
+
+This will propagate the new strings to your language. This will merge the new strings, update references/annotations as well as marking unused strings as obsolete.
+
+To update all .po files at once:
+
+    python update-po.py --all
+
+
+
 #### Typo or incorrect translation in existing text
 
 If you see a typo or an incorrect translation, simply edit ``po/Firmware_XY.po`` and make a pull request with the changes.
@@ -65,25 +87,23 @@ to preview all translations as formatted on the screen.
 
 If some text is missing, but there is no reference text in the po file, you need to refresh the translation file by picking up new strings and annotations from the template.
 
-Run ``./update-po.sh po/Firmware_XY.po`` to propagate the new strings to your language. This will merge the new strings, update references/annotations as well as marking unused strings as obsolete.
-
-Update the translations, then proceed as for [typo or incorrect translation](#typo-or-incorrect-translation-in-existing-text).
+See section [how to update .po file](#how-to-update-.po-file) to update the translations, then proceed as for [typo or incorrect translation](#typo-or-incorrect-translation-in-existing-text).
 
 ### Fixing an incorrect screen annotation or english text
 
 The screen annotations as well as the original english text is extracted from the firmware sources. **Do not change the main pot file**. The ``pot`` and ``po`` file contains the location of the annotation to help you fix the sources themselves.
 
-Run ``./update-pot.sh`` to regenerate ``po/Firmware.pot`` and verify that the annotation has been picked up correctly. You can stop here if you only care about the annotation.
+* See section [how to update .pot file](#how-to-update-.pot-file) to update the reference file.
 
-Run ``./update-po.sh po/Firmware_XY.po`` otherwise to propagate the annotation to your language, then proceed as for [typo or incorrect translation](#typo-or-incorrect-translation-in-existing-text).
+* To sync one language: See section [how to update .po file](#how-to-update-.po-file); to propagate the annotation from the `.pot` file to your language, then proceed as for [typo or incorrect translation](#typo-or-incorrect-translation-in-existing-text).
 
 ### Adding a new language
 
 Each language is assigned a two-letter ISO639-1 language code.
 
 The firmware needs to be aware of the language code. It's probably necessary to update the "Language codes" section in ``Firmware/language.h`` to add the new code as a ``LANG_CODE_XY`` define as well as add the proper language name in the function ``lang_get_name_by_code`` in ``Firmware/language.c``.
 
-It is a good idea to ensure the translation template is up-to-date before starting to translate. Run ``./update-pot.sh`` to regenerate ``po/Firmware.pot`` if possible.
+It is a good idea to ensure the translation template is up-to-date before starting to translate. See section [how to update .pot file](#how-to-update-.pot-file).
 
 Copy ``po/Firmware.pot`` to ``po/Firmware_XY.po``. The *same* language code needs to be used for the "Language" entry in the metadata. Other entries can be customized freely.
 

lang/lang-extract.py

@@ -4,8 +4,19 @@
 import codecs
 import polib
 import regex
+import os
 import sys
 import lib.charset as cs
+from pathlib import Path
+
+# Absolute path
+BASE_DIR: Path = Path.absolute(Path(__file__).parent)
+PO_DIR: Path = BASE_DIR / "po"
+
+# Pathlib can't change the working directory yet
+# The script is currently made to assume the working
+# directory is ./lang/po
+os.chdir(PO_DIR)
 
 def line_warning(path, line, msg):
     print(f'{path}:{line}: {msg}', file=sys.stderr)
@@ -43,7 +54,7 @@ def index_to_line(index, lines):
 
 
 def extract_file(path, catalog, warn_skipped=False):
-    source = open(path).read()
+    source = open(path, encoding="utf-8").read()
     newlines = newline_positions(source)
 
     # match internationalized quoted strings
@@ -144,7 +155,7 @@ def extract_file(path, catalog, warn_skipped=False):
 
 
 def extract_refs(path, catalog):
-    source = open(path).read()
+    source = open(path, encoding="utf-8").read()
     newlines = newline_positions(source)
 
     # match message catalog references to add backrefs

lang/requirements.txt

@@ -0,0 +1,3 @@
+polib==1.1.1
+pyelftools==0.29
+regex==2022.9.13

lang/update-po.py

@@ -0,0 +1,50 @@
+#!/usr/bin/env python3
+
+"""
+Portable script to update po files on most platforms
+"""
+
+import argparse
+from sys import stderr, exit
+import shutil
+from pathlib import Path
+import polib
+from polib import POFile
+
+BASE_DIR: Path = Path.absolute(Path(__file__).parent)
+PO_DIR: Path = BASE_DIR / "po"
+PO_FILE_LIST: list[Path] = []
+POT_REFERENCE: POFile = polib.pofile(PO_DIR/'Firmware.pot')
+
+
+def main():
+    global PO_FILE_LIST
+    ap = argparse.ArgumentParser()
+    group = ap.add_mutually_exclusive_group(required=True)
+    group.add_argument('-f', '--file', help='File path for a single PO file to update Example: ./po/Firmware_cs.po')
+    group.add_argument('--all', action='store_true', help='Update all PO files at once')
+    args = ap.parse_args()
+
+    if args.all:
+        PO_FILE_LIST = sorted(PO_DIR.glob('**/*.po'))
+    elif args.file:
+        if Path(args.file).is_file():
+            PO_FILE_LIST.append(Path(args.file))
+        else:
+            print("{}: file does not exist or is not a regular file".format(args.file), file=stderr)
+            return 1
+
+    for po_file in PO_FILE_LIST:
+        # Start by creating a back-up of the .po file
+        po_file_bak = po_file.with_suffix(".bak")
+        shutil.copy(PO_DIR / po_file.name, PO_DIR / po_file_bak.name)
+        po = polib.pofile(po_file)
+        po.merge(POT_REFERENCE)
+        po.save(po_file)
+
+
+if __name__ == "__main__":
+    try:
+        main()
+    except KeyboardInterrupt:
+        exit(-1)

lang/update-pot.py

@@ -0,0 +1,87 @@
+#!/usr/bin/env python3
+
+"""
+Script updates the Firmware.pot file.
+
+The script does the following:
+1. Current Firmware.pot is backed up with a copy, Firmware.pot.bak
+2. Runs lang-extract.py with all the correct arguments.
+"""
+
+import sys
+import os
+from pathlib import Path, PurePath, PurePosixPath
+import shutil
+import subprocess
+from subprocess import CalledProcessError
+
+# Constants
+BASE_DIR: Path = Path.absolute(Path(__file__).parent)
+PROJECT_DIR: Path = BASE_DIR.parent
+PO_DIR: Path = BASE_DIR / "po"
+
+# Regex pattern to search for source files
+SEARCH_REGEX: str = "[a-zA-Z]*.[ch]*"
+
+# Folders to search for messages
+SEARCH_PATHS: list[str] = ["./Firmware", "./Firmware/mmu2"]
+
+
+def main():
+    # List of source files to extract messages from
+    FILE_LIST: list[Path] = []
+
+    # Start by creating a back-up of the current Firmware.pot
+    shutil.copy(PO_DIR / "Firmware.pot", PO_DIR / "Firmware.pot.bak")
+
+    # Get the relative prepend of Project directory relative to ./po directory
+    # This should be something like '../../'
+    # Note: Pathlib's relative_to() doesn't handle this case yet, so let's use os module
+    rel_path = os.path.relpath(PROJECT_DIR, PO_DIR)
+
+    # We want to search for the C/C++ files relative to the .po/ directory
+    # Lets append to the search path an absolute path.
+    for index, search_path in enumerate(SEARCH_PATHS.copy()):
+        try:
+            # Example: Converts ./Firmware to ../../Firmware
+            SEARCH_PATHS[index] = PurePath(rel_path).joinpath(search_path)
+
+            # Example: Convert ../../Firmware to ../../Firmware/[a-zA-Z]*.[ch]*
+            SEARCH_PATHS[index] = PurePosixPath(SEARCH_PATHS[index]).joinpath(
+                SEARCH_REGEX
+            )
+        except ValueError as error:
+            print(error)
+
+    # Extend the glob and append all found files into FILE_LIST
+    for pattern in SEARCH_PATHS:
+        for file in sorted(PO_DIR.glob(str(pattern))):
+            FILE_LIST.append(file)
+
+    # Convert the path to relative and use Posix format
+    for index, absolute_path in enumerate(FILE_LIST.copy()):
+        FILE_LIST[index] = PurePosixPath(absolute_path).relative_to(PO_DIR)
+
+    # Run the lang-extract.py script
+    SCRIPT_PATH = BASE_DIR.joinpath("lang-extract.py")
+    try:
+        subprocess.check_call(
+            [
+                "python",
+                SCRIPT_PATH,
+                "--no-missing",
+                "-s",
+                "-o",
+                "./Firmware.pot",
+                *FILE_LIST,
+            ]
+        )
+    except CalledProcessError as error:
+        print(error)
+
+
+if __name__ == "__main__":
+    try:
+        main()
+    except KeyboardInterrupt:
+        sys.exit(-1)