New keyboard API for macOS

[ArturKovacs]

• Tested on all platforms changed
• Compilation warnings were addressed
• cargo fmt has been run on this branch
• cargo doc builds successfully
• Added an entry to CHANGELOG.md if knowledge of this change could be valuable to users
• Updated documentation to reflect any user-facing changes, including notes of platform-specific behavior
• Created or updated an example program if it would help users understand this functionality
• Updated feature matrix, if new features were added or implemented

This is the macOS implementation of #753 and it also brings the Android backed up to date with the new API.

This should be merged after #1788.

Side-note: the IME implementation is based on #1669 but as I don't know any language which naturally uses IME, it's difficult for me to know if what I ended up implementing is correct.

[Kethku]

I think there is an issue here where ModifiersState reports alt as held down when using a keyboard layout where the right alt key is actually AltGr. This leads to incorrect keybindings when using the modifiers state to determine if alt is held in combination with another character.

[Kethku]

Ok I've been discussing with some folks and my comment is incorrect.
The problem is that for certain keyboard layouts on macos holding option should modify the key sent via the text event. However it appears that the modification isn't reproduced in the text field of the key event

Example:
For a german keyboard layout, holding option and pressing 8 should result in the character '{'.
However in my testing the key I get back is just 8 with a modifier alt reported via the modifiers changed event.

I believe this represents a bug, but I could be wrong. Any insight would be greatly appreciated.

[ArturKovacs]

@Kethku thank you for reporting this, I completely missed this bug!

It should be fixed by the commit I just pushed. Test for this behaviour and potential regressions would be greatly appreciated.

[Kethku]

Edit: adding @ArturKovacs because this request is mostly targeted at you who designed the change

I will test this soon. Over all the new design is a huge improvement. One problem I'm fighting though is that as far as I can tell it's not possible to determine whether a given modifier was used in the construction of a character. My usecase is in a vim client so if the user binds the key binding ctrl-!, it's not correct to send ctrl-shift-! Because the shift was "used up". This becomes more complicated with bindings where control or alt modify the character.

I don't know if it's possible, but an api which allows you to compare the values with or without individual (not just all modifiers) modifiers would make this perfect. But that's just my two cents. Amazing improvement regardless

[ArturKovacs]

One problem I'm fighting though is that as far as I can tell it's not possible to determine whether a given modifier was used in the construction of a character.

The intended usage is that you listen to WindowEvent::ModifiersChanged events and keep track of the pressed modifiers yourself. Then you can use this in any event you want (e.g. mouse input, device events)

The reason for not putting the modifier info into the KeyEvent is to avoid having multiple sources of truth. And to avoid appending this info to a bunch of events when the application itself could instead keep track of the modifiers. Here's one of the places where this argument was formulated: #1124 (comment)

[ArturKovacs]

I don't know if it's possible, but an api which allows you to compare the values with or without individual (not just all modifiers) modifiers would make this perfect.

If I understand you correctly, this is already possible:

match event {
    Event::WindowEvent {
        event: WindowEvent::ModifiersChanged(new_modifiers),
        ..
    } => {
        if new_modifiers.contains(winit::keyboard::ModifiersState::CONTROL) {
            // Do the thing
        }
    }
    _ => ()
}

[Kethku]

That's not quite the problem. Knowing whether a modifier is held is easy like you've shown. Knowing whether a particular modifier contributed to the selected character is not easy though.

Due to a quirk in neovim's keybinding code, shift-{character} is different from {character} so shift-! would be handled differently than ! even though the shift key is necessary to produce the ! character. This problem extends to other modifiers such as ctrl. I'm told that some keyboard layouts require ctrl to input given characters.

So the thing that would help is a way to tell if a given modifier contributed to the selection of the resulting character and so should be omitted from the keybinding reporting or if the modifier could have been omitted to produce the same character. I don't know if such an api is possible on all platforms so it may be a moot point but I figured I'd ask and see

[ArturKovacs]

Knowing whether a particular modifier contributed to the selected character is not easy though.

That is true, in fact I doubt it's possible on macOS or Windows. However, assuming that all active modifiers contributed to the generated character could be a useable approximation. Also note that on desktop platforms you have access to the pressed key as if there was no modifiers active, using key_without_modifiers.

If I understand correctly, you have keybindings that you want to trigger in response to the corresponding input. And you want to allow this "corresponding input" to be either

1. the exact input specified by the binding
2. and any other input where additionnal modifiers are held down which do not contribute to the produced character.

I don't understand why 2 is a requirement. For example I would not expect that my ctrl+A keybinding gets triggered when I press cmd+ctrl+A. In fact I would hope it doesn't get triggered in that case.

In any case, if you want to allow 2, then what you could do, is to check if the current input is exactly the same as the specified binding, then remove cmd from the current input and check if it matches the binding that way. This assumes that cmd is the only key which doesn't influece the pressed key but that could be a decent approximation for your use case.

[Kethku]

Interesting can you elaborate on what you mean by remove the modifier? Are you suggesting I construct a new key event somehow?

[Kethku]

I think you mean submit the keybinding to neovim with and without the modifier?

I think the conclusion I'm coming to is that a change needs to happen in neovim to handle with and without bindings the same.

I will think some more on whether that's possible. From neovim's perspective, they can't know whether a key used modifiers to produce a given character, so they have to assume that shift-! shouldn't be handled the same as ! because a given keyboard layout may input the ! without it.

I guess what I sorta need is an api to look up predicively what a given set of keys would produce. Again though that might not be possible...

[ArturKovacs]

Alright it took me a while, but I think now I understand what you want to achieve. Correct me if I'm wrong but what you want seems to be that Shift+1=! should behave the same as !=!. Again, it seems that you could achieve something like this by assuming that Shift always infulences the character (unless it's a non-charater input). In practice this would mean that you define all keybindings without Shift in neovim, and then send all input to neovim without telling neovim that Shift is held down. There may be some collision between keybindings on certain keyboard layouts, but I actually can't think of a specific example of such a collision.

I guess what I sorta need is an api to look up predicively what a given set of keys would produce.

Something similar is probably possible on desktop platforms, but I would like to merge the current API into master before we make further additions to it, as it takes a very long time to come to an agreement about and finish the implementation for such additions.

[TobTobXX]

Sooo.... is the code on this branch still a candidate for master? If yes, can you do a rebase?

(The folks from the neovide project would like to use the new keyboard API.)

Thank you!

[kchibisov]

closing as we have mono branch.