Special character treatment

[Tazinho]

Systematical Addition of cases to support, similar to german umlauts. Also decide of and how to replace in replace Special characters argument

[Tazinho]

already included:
ä -> ae, ö -> oe, ü -> ue, ß -> ss, @ -> at, ` -> "", ' -> ""

[Tazinho]

review this issue also regarding encoding stuff, after this topic is included into the new advanced r version

[Tazinho]

basics about encoding:
https://www.joelonsoftware.com/2003/10/08/the-absolute-minimum-every-software-developer-absolutely-positively-must-know-about-unicode-and-character-sets-no-excuses/

-ascii: nr 32-127 space = 32, A = 65, etc. -> 7 bits (2^7 = 128) (most were using 8 bit bytes)
so one bit was free for any usage
<32 "unprintable" -> controlcharacters (i.e. 7 = "beep", 12 = "print",...)
-> everything good for english alphabet.

-1 byte = 8 bits = possibilities: 0-255 -> 128-255 still free

• ibm oem with some european accented characters
• many other oem character sets evolved
  -> so encoding displayed on different displays turned to different character outputs

ANSI standard -> 0-127 have the same meaning for all encodings regarding this standard

= 128 non-uniform ("code pages", different nr for diff countries for example)

• problems in asia -> DBCS "double byte character set" (some in 1, some in 2)(hard to ... -> wind: AnsiNext AnsiPrev)
• problem only if change computer or language

-> Unicode!!!

• effort to single character set including all! reasonable writing sytems
  unicode != 16-bit code (16 bit = 65.536)
  before: someting like this: A -> 0100 0001, ...
  unicode: A -> code point (theoretical concept?) completely other story how this is represented in mem or disk
  A (for example) is platonic ideal. A is different to B or a, but the same as italic or bold A or any font.
  unicode cosortium: figured out, which characters are really different (ß not ss and also similar looks are not same characters...)
  Codepoints: U+0639 (for example; U+ for unicode and then a number)
• there is no limit do space of unicode characters definitely not two bites (there are alr. more than 65.536)
• Hello in Unicode U+0048 U+0065 U+006C U+006C U+006F

Endocings (to store unicode)

• different ways appeared to store unicode. problem was with ofthen zeroes, which often unnecessarily
  doubled the amount of memory. So many people didn't use it.
  -> UTF-8: 0-127 werden in 1 byte, darüber in 2,3,4,5 bzw. 6 byte gespeichert.
  -> für english: utf8 = ascii
  Hello in unicode: U+0048 U+0065 U+006C U+006C U+006F, in utf8 UND ascii UND ansi UND all oem ch sets:
  48 65 6C 6C 6F

• 3 possibilities: ucs-2 (two bytes + find out if hidh/low endian), utf-16 (16 bits), UTF-8 (well with old ascii)

• others: utf-7 (better utf8?), utf-4 (4byte utf, but too memory expensive)

• translation from unicode to others is ok for english alphabets, but results in ? or � if not available.

• popular englisch-text encodings: windows-1252, iso-8859-1 aka latin-1 (both for west europe, not russian,...)

• utf7,8,16, 32 can store every codepoint

• need ALWAYS to know encoding of strings. plain text is not ASCII. TJER AIN'T NO SUCH THING AS PLAIN TEXT
  so say if it is in utf-8, ascii, latin-1, windows 1252, ... more than 100 endocings, all different above 127

• std metadata for encoding:

  • email: Content-Type: text/plain; charset=”UTF-8″
  • web servers: metatag:

• when these info is missing, different ways are gone. For example micr. int. exp. guesses.

• however, you can also guess encodings yourself and try a bit.

[Tazinho]

see also here:
https://stat.ethz.ch/R-manual/R-devel/library/base/html/Encoding.html

Maybe it is best to set the encoding at the beginning via

Encoding(x) <- "UTF-8"

which will result in "UTF-8" (for words with special characters) and "unknown" for words without special characters (ASCII 0-127).

[Tazinho]

or maybe better enc2utf8 or iconv

[Tazinho]

added enc2utf8 as first step into the parsing. Currently only into unstable development and experimental
edit: resolved this, since stringr (based on stringi) seem to always encode 2 utf8 anyway

[Tazinho]

propaply replace_special_characters should take a character string of the form
c("countryset1", "countryset2", ...) instead of a logical.
The countrysets should be handled in order of their supplyment to replace_special_characters.
The replacement should result in only 0-127 Ascii letters, so that there can't be any collisions of between several special character replacements.

[Tazinho]

sth like mentioned here https://unix.stackexchange.com/questions/171832/converting-a-utf-8-file-to-ascii-best-effort
by radovan garbik will do it...

[Tazinho]

need to check out this
http://stackoverflow.com/questions/17517319/r-replacing-foreign-characters-in-a-string
in detail. especiall stringi::stri_trans_general

[Tazinho]

stringi::stri_trans_general(, "Latin-ASCII") might be the best one can get...have to look up latin

[Tazinho]

or maybe just wrap the stri_trans_general function and supply the id as an argument to to any case

stri_trans_general("gro\u00df", "latin-ascii")
stri_trans_general("stringi", "latin-greek")
stri_trans_general("stringi", "latin-cyrillic")
stri_trans_general("stringi", "upper") # see stri_trans_toupper
stri_trans_general("\u0104", "nfd; lower") # compound id; see stri_trans_nfd
stri_trans_general("tato nie wraca ranki wieczory", "pl-pl_FONIPA")

[Tazinho]

have to look for a language and symbol agnostic transcription for unicode...icu doesnt seem to work for everything (Latin-ASCII has some problems and ü to ue is not possible in any icu conversion). maximally it could be added as an extrafeature...

[Tazinho]

Use stringi::stri_trans_general(), when next update of stringi is available on cran

[Tazinho]

implemented now, update, when I find a nice source for additional transliteration dictionaries.
Currently implemented (besides stri_trans_general()): "german"