Add support for multibyte characters to $IFS

[JohnoKing]

This pull request fixes BUG_MULTIBIFS, which had two bug reports in the ksh2020 branch. The modernish regression test suite now only reports eight test failures.

• Backport Eric Scrivner's fix for multibyte IFS characters (slightly modified for compatibility with C89). Explanation from Fix expansion of multibyte IFS characters att/ast#737:

  Previously, the varsub method used for the macro expansion of $param, ${param}, and ${param op word} would incorrectly expand the internal field separator (IFS) if it was a multibyte character. This was due to truncation based on the incorrect assumption that the IFS would never be larger than a single byte.

  This change fixes this issue by carefully tracking the number of bytes that should be persisted in the IFS case and ensuring that all bytes are written during expansion and substitution.

  Bug report: ksh93: "$*" joins positional parameters on the first byte of $IFS instead of first character att/ast#13

• Fixed another bug that caused multibyte characters with the same initial byte to be treated as the same character by the IFS. This bug was occurring because the initial byte wasn't being written to the stack when the IFS delimiter and multibyte character had the same initial byte:

  $ IFS=£
  $ v='§'
  $ set -- $v
  $ v="${1-}"
  $ echo "$v" | hd # The first byte should be c2, but it isn't due to the bug
  00000000  a7 0a                                             |..|
  00000002

  The code in question was skipping past the initial byte with continue:
  

  ksh/src/cmd/ksh93/sh/macro.c

  Lines 2403 to 2406 in 8c16f38

  	if(n==S_MBYTE)
  	{
  	if(sh_strchr(mp->ifsp,cp-1)<0)
  	continue;

  
  This problem is fixed by putting the initial byte on the stack with sfputc before the continue.

  Bug report: IFS: UTF-8 support is incomplete att/ast#1372

[McDutchie]

Did you figure out the fix for field splitting (the second fix)? Nice find!

Re:

The modernish regression test suite now only reports eight test failures.

I would hope those are xfails, not actual test failures…

[JohnoKing]

I would hope those are xfails, not actual test failures…

No need to worry, the test failures are just xfails for bugs like BUG_BRACQUOT.