Keyboard layouts are ignored (all platforms)

[cknoerle]

fvim 0.2 and also previous version do not handle my standard German keyboard layout correctly.
For example a ":" is not on "SHIFT-." but on "SHIFT-ü"

AFAIR this also happened to me in previous builds of fvim.

Windows 1809 build 17763.615 settings:

• Region: Germany
• Regional format: German (Germany)
• Windows display language: English (United States)

Let me know what else you need.
Thanks in advance!

[yatli]

This might be an upstream Avalonia bug.
I also notice some problems for Mac keyboards -- for a Japanese layout keyboard with different arrangement of symbols -- but fvim treats the keystrokes as if it's on a standard US keyboard (yeah, got shift+"something" for ':', but on the actual keyboard it's somewhere else)

Does this match the problem you're encountering?
I can file another issue to the upstream (or like the last time, send a PR over).

[cknoerle]

Yes, sounds exactly like my problem. I already had the feeling it was standard US layout for some but not all the keys. Am 23.07.2019 um 08:57 schrieb Yatao Li <[email protected]>: This might be an upstream Avalonia bug. I also notice some problems for Mac keyboards -- for a Japanese layout keyboard with different arrangement of symbols -- but fvim treats the keystrokes as if it's on a standard US keyboard (yeah, got shift+"something" for ':', but on the actual keyboard it's somewhere else) Does this match the problem you're encountering? I can file another issue to the upstream (or like the last time, send a PR over). — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

[yatli]

Alright. Let me make another attempt at the Avalonia key mapping problem.
I'll let you know when I make progress on this.

[yatli]

The issue is reproduced on my end.

OS: Windows 10 1903
FVim: HEAD
Locale: English(United States)
Language: Deutsch(Deutschland)
Language pack is not installed, just basic typing.
Physical keyboard: QWERTY standard 104 key

Open up notepad.exe, type [;'
Got result: üöä

From the above result, I figure it'd a QWERTZ layout.

Open up FVim.exe, type [;'
Got result: ;``'

Which is a bit puzzling -- it's neither QWERTY, nor the German QWERTZ.
y and z are swapped indeed.

[Thanood]

Hi @yatli
First of all, thanks for creating a neovim ui on the .Net platform. 😃

I'm experiencing the same issue.
I've had a little dive into your code and traces while I've tried fvim key commands and I may be able to at least give some pointers.

Please take the following with a grain of salt since I'm neither an F# expert nor an Avalonia expert.

I'm using a German keyboard layout where ":" is on "SHIFT-." (so, SHIFT-period). The triggered input is ">".
When typing this key combination, "SHIFT-period" is exactly what is logged:

model: grid #1: OnInput: Key (Shift,OemPeriod)

So keystroke detection seems to work, right?
I've found that trace here: https://github.com/yatli/fvim/blob/master/model.fs#L398 so I attached a breakpoint.

The pressed key combination is then transformed here: https://github.com/yatli/fvim/blob/master/model.fs#L410

which seems to trigger the match here: https://github.com/yatli/fvim/blob/master/model.fs#L205
(sorry for all the links 😉 )

"SHIFT-period" is then explicitly transformed to nvim_input ">" here: https://github.com/yatli/fvim/blob/master/model.fs#L299

Of course, changing

        |  Key(HasFlag(KeyModifiers.Shift), Key.OemPeriod)            -> Normal ">"

to

        |  Key(HasFlag(KeyModifiers.Shift), Key.OemPeriod)            -> Normal ":"

"solves" the issue (for users of a German keyboard layout, therefore the air quotes 😃).

Edit: the full trace seems to be even more interesting:

model: grid #1: OnInput: Key (Shift,OemPeriod)
neovim.process: call: 40 -> [0,40,"nvim_input",[">"]]
model: grid #1: OnInput: Key (Shift,OemPeriod)
model: grid #1: OnInput: TextInput ":"
model: rejected: TextInput ":"
model: grid #1: OnInput: TextInput ":"
model: rejected: TextInput ":"

The TextInput match(?) returns the correct character.

[fc1943s]

+1
Having the same problem with the ABNT2 layout.

Is this something that must be implemented by hand or the layout translation should be handled by the OS itself?

[krissen]

+1
Same problem. MacOS. Feels like the keys which are actually entered correspond to the standard US layout, as has been noted above. I'm using a Swedish keyboard layout.

[jonoll123]

Same problem for me as well, Windows 10 using a Swedish keyboard. The resulting key presses appear to be neither Swedish nor US. If I switch to US layout in Windows it works as expected, but I would rather use the Swedish layout...

[sphynx79]

Same problem for me Windows 10 using a Italian keyboard layout, but fvim use English layout.

[yatli]

I've done another round of research on this.

Windows: https://docs.microsoft.com/en-us/windows/win32/api/winuser/nf-winuser-mapvirtualkeyexa
macOS: https://stackoverflow.com/questions/18408565/map-nsevent-keycode-to-virtual-key-code#:~:text=To%20translate%20from%20the%20virtual,kTISPropertyUnicodeKeyLayoutData%20as%20the%20property%20key.
Linux: https://tronche.com/gui/x/xlib/input/keyboard-encoding.html

As the number of users with this problem increases I'd start to feel guilty for pushing this one back for other stuff..

A question for you guys, how do you expect the vim key mapping to behavior? For example if I want to go INSERT, should I press "i", or whatever key located at where the en-US "i" is?

[yatli]

Change of plan -- instead of solving this upstream I'll see if I can do it from fvim directly and then ask them to backport.

Edit: after second thought... It doesn't make sense, because this is platform-specific, and Avalonia already hides the info we need and instead present us with Avalonia key sequences.

[krissen]

@yatli My two cents, in short: keys should be mapped according to the system settings, not according to en-US.

Longer version: (i would probably be mapped at the same key in most keyboard layouts. : is an example that has been mentioned above, which is mapped differently on different keyboard layouts.) On a keyboard which is set by system to Swedish, I currently have to press shift-ö in fvim to get a :. When in fact I would expect to get a : when I press what, in Swedish keyboards, is mapped to : (shift-.). That is, following the system keyboard settings.

[yatli]

understood.

[yatli]

@krissen any comments on symbol processing? Like, is there any symbol that doesn't exist on the en-US keyboard?

[krissen]

@yatli There are. Just speaking from my own workflow/vim-config, those symbols would not be important with regards to commands and such (seldom used in binds), but naturally are essential for writing text. That is, I could not use an editor which would not be able to input symbols such as åäö. Other keyboard layouts would have other symbols, particular to their languages. I believe there has been german examples earlier in this issue-thread.

The key thing (pun intended) really is, for a key-press to be predictable. That is, for key-presses in one app to be the same in every other: that which is set by the system.

[yatli]

åäö

Ah, I mean punctuations. :)

[krissen]

Oh, I see. Nonesuch that I am aware of.

[yatli]

There's a problem wrt punct rewriting.
According to https://docs.microsoft.com/en-us/windows/win32/api/winuser/nf-winuser-mapvirtualkeya :

MAPVK_VK_TO_CHAR 2 | uCode is a virtual-key code and is translated into an unshifted character value in the low-order word of the return value. Dead keys (diacritics) are indicated by setting the top bit of the return value. If there is no translation, the function returns 0.

This is the only entry to translate the key codes into mapped keys -- but it translates into unshifted keys.
I don't understand how can I go from unshifted to a shifted character here... Thoughts?

[yatli]

hmm. this may help: https://stackoverflow.com/questions/5825820/how-to-capture-the-character-on-different-locale-keyboards-in-wpf-c

[yatli]

Ok PR is up: AvaloniaUI/Avalonia#4447

[yatli]

@gibboni do you build yourself? If so, please try to run in debug mode and post some logs.
My guess is that AltGr is regarded as yet-another-alt (or the OS doesn't even send us the correct TextInput event)

[yatli]

Wow. <AltGr-key> with FRA layout is actually mapped to <LeftCtrl-Alt-key>
What is the numerical key for ^?

[yatli]

In FRA ^ is 9 -- and AltGr-9 puts 9^ into FVim

[gibboni]

Yes, I do build fvim myself. I got the following error when I pressed AltGr: input: rejected: (Key (None, None), "")

[gibboni]

Interesting. If I press the keys that got swallowed I got nothing. No errors.

[gibboni]

It seems that the AltGr is working even though log states "Key (None, None, ""). If I press AltGr-+ I get in the logs:
input: rejected: (Key (None, None), "")
input: rejected: (Key (None, OemPlus), "")
input: OnInput: key: "\"

The problem now seems to be, that some of the keys are completely dead and don't show anything in the log output.

[yatli]

@gibboni the internal logic is like this:

As mentioned earlier in the thread, for mapped keys, both KeyDown and TextInput events will be triggered.
I'm using some heuristic to determine whether a key is mapped, and reject the KeyDown event accordingly.
For some keys (e.g. AltGr+9), the KeyDown events are more complex than this -- FVim first receives Key(Ctrl), then Key(Ctrl+Alt, 9) -- and it cannot distinguish that from a enUS keyboard actually pressing Ctrl+Alt+9 -- so both the KeyDown and TextInput gets through.

[krissen]

Latest artifacts here: https://v-yadli.visualstudio.com/fvim/_build/results?buildId=653&view=artifacts&pathAsName=false&type=publishedArtifacts

Tried going there to fetch it; I get to the page, click on drop-osx and then nothing happens. 🤷‍♂️ Attempted from a couple of browsers etc.

So thought I'd give building from source a go. ^HEAD from master has the keyboard issue, ofc. But was built and worked as expected.

Checked out ^HEAD (d70d90a) from branch keyboard_layout. Built successfully (dotnet build -c Release). Ran OK. But nothing happens regardless of what keys I press. The window just sits there. Tried building 3014958 but didn't make a difference.

Am new to dotnet, so perhaps I'm running the wrong commands or something? Or attempting to build the wrong branch?

Let me know what I'm doing wrong -- or if the above means you've got testing results. ;-) Am on macOS, sv-SE.

[yatli]

@krissen https://v-yadli.visualstudio.com/862b0001-b236-4376-baf4-7c9655c99ba7/_apis/build/builds/653/artifacts?artifactName=drop-osx&api-version=6.0&%24format=zip

[krissen]

@krissen https://v-yadli.visualstudio.com/862b0001-b236-4376-baf4-7c9655c99ba7/_apis/build/builds/653/artifacts?artifactName=drop-osx&api-version=6.0&%24format=zip

Got it. As with the version built from source, I get an unresponsive application window. That is it.

If I build it from source, it reads my vim dotfiles, and seems to have the appearance I'd expect etc. The "artifacts version" does not. (If I remove .vim, .vimrc and .config/fvim the FVim application window looks identical between built-from-source and artifacts.) But that's a different thing; the point to make in relation to the thread is that the application does not react to any input.

[yatli]

Thanks for testing.
This is probably OSX specific -- I remember Avalonia team talked about this.
So perhaps I'll have to dive into that issue myself... (again!)

[yatli]

@krissen did you try to run in debug mode? dotnet run directly.
What's the log like?

[krissen]

@krissen did you try to run in debug mode? dotnet run directly. What's the log like?

I think the relevant parts would be the following:

input: rejected: (Key (None, I), "")
input: rejected: (Key (Shift, RightShift), "S")
input: rejected: (Key (None, U), "")
input: rejected: (Key (None, Q), "")
input: rejected: (Key (None, R), "")
input: rejected: (Key (None, D2), "")
input: rejected: (Key (None, D1), "")
input: rejected: (Key (None, W), "")
input: rejected: (Key (None, D3), "")
input: rejected: (Key (None, H), "")
input: rejected: (Key (None, G), "")
input: rejected: (Key (None, Space), "")
input: rejected: (Key (None, C), "")
input: rejected: (Key (None, H), "")
input: rejected: (Key (None, J), "")
input: rejected: (Key (None, D4), "")
input: rejected: (Key (None, G), "")
input: rejected: (Key (None, Space), "")
input: rejected: (Key (None, K), "")
input: rejected: (Key (None, J), "")
input: rejected: (Key (None, G), "")
input: rejected: (Key (None, D4), "")
input: rejected: (Key (None, Space), "")

That is, all input is rejected.

EDIT: Except Shift.

EDIT2: Just for the fun of it, I changed system keyboard layout to en-GB. Output is, I guess unsurprisingly, identical.

[krissen]

I could add, that it interprets keypresses according to an English keyboard layout.

That is, if I press å, ä, ö, respectively, it reads it as:

input: rejected: (Key (None, Oem4), "")
input: rejected: (Key (None, OemQuotes), "")
input: rejected: (Key (None, OemSemicolon), "")

[yatli]

Yeah, these are raw scancodes.
The TextInput events are missing. Game over for macOS. :(

[krissen]

Ah well. At least we played.

[yatli]

Next step attack vectors:

1. Avalonia, macOS, TextInput
2. Figure out better heuristic for AltGr translation

[yatli]

@krissen all hope is not lost!
When working on a input method-related bug I discovered a potential fix for macOS. Would you like to try it out?

The code is on merged to master, and the artifacts will be ready shortly.

[yatli]

This is the critical patch for macOS:

9f975b6

An Avalonia dev (thanks @maxkatz6 !) reminded me that in macOS, if a raw key is marked as "handled", it won't be translated to text input, which is confirmed by @krissen 's debug log.

[krissen]

Built it from master; so far, works like a charm! 🎉

Looking forward to trying it out some more!

[yatli]

Guys, v0.3.512 has been released with keyboard layout fix merged in.
This issue will self-destruct in a week :p

[gibboni]

Hi! Was this fix meant only for macOS? Because on Windows nothing that requires AltGr still doesn't work.

[yatli]

This fix should cover all platforms.
AltGr will need some finishing touches.
I recently discovered in other software like WindowsTerminal, there's an option called "AltGr aliasing" -- meaning that AltGr on the keyboard will actually produce a Ctrl+Alt.
Funnily, there's no way for the OS to tell AltGr from Ctrl+Alt.
For example I have mapped to "search everything" and when I press AltGr+S, the search window pops up.
We should have an option to allow reinterpreting Ctrl+Alt.

[krissen]

(MacOS doesn't differentiate between Alt (or 'Opt' in Mac-speak) and AltGR, rather there is only (left-)Alt and (right-)Alt.)

[yatli]

v0.3.513 adds a new command FVimKeyAltGr v:true to enable aliasing.
I'm happily surprised to see it in action, that 7, shift-7, altgr-7 produces 3 different things :D

(not too surprised because some people here use this layout:

[yatli]

0.3.513 is published. Case closed. 😎

[hexacera]

@yatli I'm using the Dvorak layout and experiencing issues with the latest build on macOS 12.3, if I press in insert mode (raw key codes being Ctrl + J on QWERTY), FVim interprets it as and inserts a newline instead of deleting one character back, same for all other shortcuts with Ctrl (e.g. inserts a tab instead of exiting to normal mode). I tried with and without FVimKeyAltGr v:true without success.

Shift shortcuts appears to be fine though.

[yatli]

@VVVFO hey thanks for reporting. Yes I'm aware of this problem and I'm not sure how to solve this problem yet.

Let's open up a new thread for that.
#222