unicode on Windows

[schaeferc-si]

Dev Effort

1D

Description

I'm running fido 1.3.4 on Ubuntu and on Windows 10. On Ubuntu, fido handles non-ascii file names correctly. On Windows fido will not process file/directory names with unicode (presumably utf16) names.

[schaeferc-si]

Additional details... Here's output for one run:

FIDO v1.3.4 (formats-v88.xml, container-signature-20160121.xml, format_extensions.xml)
OK,198,fmt/394,"DS_store file (MAC)","DS_Store",6148,"R:\Repository\Accessions\2008-2013\09-228_R\source\box_7\China_Constructs-Song_Dong\D_1.DS_Store","None","signature"
bad repeat interval
bad repeat interval
FIDO: Error in identify_file: Path is R:\Repository\Accessions\2008-2013\09-228_R\source\box_7\China_Constructs-Song_Dong\D_1????
FIDO: Error in identify_file: Path is R:\Repository\Accessions\2008-2013\09-228_R\source\box_7\China_Constructs-Song_Dong\D_1?????
FIDO: Processed 1 files in 290.67 msec, 3 files/sec

The first directory ('????') is 人民建筑.
The second directory ('?????') is 尹秀珍讲座

In Fido.list_files, converting 'root' to unicode before the call to 'os.walk(root)' seems to allow the walk proceed; however, there is still an issue outputting results using the default handler 'handle_matches'.

[bitsgalore]

FWIW, I remember we once had the same problems in jpylyzer, which were solved by (if I remember well):

Path walk

Instead of root = os.path.normpath(root), use root = unicode(root, 'utf-8') (but this is only needed in Python 2.x; in Python 3.x root = os.path.normpath(root) should work fine).

Output

For the outputting it is important that the encoding is explicitly set. In jpylyzer we use this:

# Set encoding of the terminal to UTF-8
if sys.version.startswith("2"):
    out = codecs.getwriter("UTF-8")(sys.stdout)
elif sys.version.startswith("3"):
    out = codecs.getwriter("UTF-8")(sys.stdout.buffer)

And then write to stdout using something like this:

out.write(whatever)

See also the code in jpylyzer.py.

[sromkey]

@bitsgalore has also pointed out, it might be best to address this after the Python 3 upgrade, rather than before.

[carlwilson]

closed by #200 in v1.6