Bug: kanata breaks SDL2-based games #567

tpstevens · 2023-09-24T22:43:56Z

Requirements

I've searched issues to see if this has not been reported before.

Describe the bug

On Windows 10, kanata sends events that get treated by SDL (and probably the OS?) as text input events but not keyup and keydown events.

Without kanata running:

[2023-09-24 15:23:44.279] [info] SDL_KEYDOWN
[2023-09-24 15:23:44.279] [info] SDL_TEXTINPUT
[2023-09-24 15:23:44.398] [info] SDL_KEYUP

With kanata running:

[2023-09-24 15:39:12.135] [info] SDL_TEXTINPUT

I've tested with my own games built with SDL2 (which is how I generated the above logs) and other games that use SDL2. I'm happy to test on Debian as well.

Relevant kanata config

No response

To Reproduce

Start an SDL2-based game that takes keyboard input (Ocean's Heart, Divinity: Original Sin 2, probably Counter-Strike, etc)
Try to move with WASD
Don't profit

Expected behavior

kanata should send keydown and keyup events. I'd expect that the keydown might come with a little delay after kanata has decided what the key should resolve to.

Kanata version

1.4.0

Debug logs

No response

Operating system

Windows 10

Additional context

No response

The text was updated successfully, but these errors were encountered:

tpstevens · 2023-09-24T23:02:16Z

Update: it works properly on Debian 12 using XFCE4.

jtroo · 2023-09-24T23:08:10Z

I suspect there's nothing that can be done about this, at least from the Kanata side, with the LLHOOK mechanism.

The SDL2 library probably detects the synthetic inputs and decides to handle them differently than real inputs.

The only workaround would likely be use interception (* see warnings).

Unless of course someone with more SDL2 knowledge can correct me, maybe there are other knobs to turn.

tpstevens · 2023-09-24T23:20:39Z

I'm not super familiar with SDL2's inner workings, but I'll see what I can find out. I'm surprised nobody has run into this so far -- SDL is a pretty popular framework for building games and game engines on.

tpstevens · 2023-09-24T23:37:47Z

I naively reviewed SDL2's source, and it reads separate Windows events for keydown, keyup, and text input. Where in kanata's source does it generate the event for Windows?

jtroo · 2023-09-24T23:49:32Z

kanata/src/oskbd/windows/mod.rs

Line 62 in b585a5e

fn send_key_sendinput(code: u16, is_key_up: bool) {

VictorLemosR · 2023-10-13T13:39:47Z

@tpstevens did you solve this problem?

I've been using kanata for 3 months and this has been an issue for me. Usually, I just close kanata which is far from ideal. Interception is also not an option for me, because some games ban it (from what I've researched, many bots use it).

@jtroo I do not know much about LLHOOK or SDL2, but, in my case, the inputs from kanata were recognized, it's just action keys (if I may call that) like enter or space, for example, did not work as intended. I even tested writing some text inside the applications and all keys were normal, including space. I did not test w, a, s, d keys, but I can test it, if it's relevant.
What tpstevens said about inputs being read as text makes sense.
Do you think is there a way to solve this? Like forcing to seng the input? I saw you linked the 'send_key_sendinput', but my knowledge is limited, I can't do anything with that

jtroo · 2023-10-13T21:50:40Z

One could try playing around with adding the Unicode or Scancode flags: https://learn.microsoft.com/en-us/windows/win32/api/winuser/ns-winuser-keybdinput

And then compile and test.

VictorLemosR · 2023-10-16T20:02:15Z

One could try playing around with adding the Unicode

I've started with your suggestion about unicode, however encountered a problem from the start, which sounds basic, but couldn't find anywhere in code, docs or discussions: how to insert a unicode from the hex value? Is it even possible?

I've tested the common ways for utf-8 and utf-16, but no success. For example, to space character: \u0020, \u0020, u0020, \u{0020}, "\u0020". All of them gave the error "unicode expects exactly one unicode character as an argument"

jtroo · 2023-10-16T20:07:25Z

One could try playing around with adding the Unicode

I've started with your suggestion about unicode, however encountered a problem from the start, which sounds basic, but couldn't find anywhere in code, docs or discussions: how to insert a unicode from the hex value? Is it even possible?

I've tested the common ways for utf-8 and utf-16, but no success. For example, to space character: \u0020, \u0020, u0020, \u{0020}, "\u0020". All of them gave the error "unicode expects exactly one unicode character as an argument"

Based on your comment, my understanding is you're trying the Unicode action in the Kanata config? My comment was more playing with the code using these flags:

Flag	description
KEYEVENTF_SCANCODE 0x0008	If specified, wScan identifies the key and wVk is ignored.
KEYEVENTF_UNICODE 0x0004	If specified, the system synthesizes a VK_PACKET keystroke. The wVk parameter must be zero. This flag can only be combined with the KEYEVENTF_KEYUP flag. For more information, see the Remarks section.

At this code:

kanata/src/oskbd/windows/mod.rs

Line 62 in b585a5e

fn send_key_sendinput(code: u16, is_key_up: bool) {

VictorLemosR · 2023-10-16T20:11:32Z

Based on your comment, my understanding is you're trying the Unicode action in the Kanata config?

I was trying exactly that.

My comment was more playing with the code using these flags:

Ok, I'll give a look into this, thanks

VictorLemosR · 2023-10-17T22:03:59Z

@jtroo your hint was on point. Problem solved.
For future users, change kb_input.wVk = code; into

        let mut special_keys_hash:HashMap<u16, u32> = HashMap::new();
        special_keys_hash.insert(0x20, 0x0020); //space key
        special_keys_hash.insert(0x0D, 0x000D); //enter key
        special_keys_hash.insert(0x1B, 0x001B); //escape key

        if special_keys_hash.contains_key(&code) {
            let code_u32 = special_keys_hash.get(&code).copied().unwrap();
            kb_input.dwFlags |= KEYEVENTF_SCANCODE;
            kb_input.wScan = MapVirtualKeyA(code_u32, 0) as u16;
        } else {
            kb_input.wVk = code;
        }

To get the values of each key, look into this site: Virtual key codes
Full function:

fn send_key_sendinput(code: u16, is_key_up: bool) {
    unsafe {
        let mut kb_input: KEYBDINPUT = mem::zeroed();
        if is_key_up {
            kb_input.dwFlags |= KEYEVENTF_KEYUP;
        }
        //kb_input.wVk = code;
        let mut special_keys_hash:HashMap<u16, u32> = HashMap::new();
        special_keys_hash.insert(0x20, 0x0020); //space key
        special_keys_hash.insert(0x0D, 0x000D); //enter key
        special_keys_hash.insert(0x1B, 0x001B); //escape key

        if special_keys_hash.contains_key(&code) {
            let code_u32 = special_keys_hash.get(&code).copied().unwrap();
            kb_input.dwFlags |= KEYEVENTF_SCANCODE;
            kb_input.wScan = MapVirtualKeyA(code_u32, 0) as u16;
        } else {
            kb_input.wVk = code;
        }

        let mut inputs: [INPUT; 1] = mem::zeroed();
        inputs[0].type_ = INPUT_KEYBOARD;
        *inputs[0].u.ki_mut() = kb_input;
        SendInput(1, inputs.as_mut_ptr(), mem::size_of::<INPUT>() as _);
    }
}

jtroo · 2023-10-17T22:46:09Z

Thanks for doing the investigation work.

It's probably worth investigating at some point in the future if using KEYEVENTF_SCANCODE for all (or most, if not possible for all) keys would be better than what kanata is doing now.

tpstevens · 2023-10-18T00:42:01Z

Will that properly handle holding a key down for a long time and then releasing?

…

On Tue, Oct 17, 2023 at 3:46 PM, jtroo ***@***.***(mailto:On Tue, Oct 17, 2023 at 3:46 PM, jtroo <<a href=)> wrote: Thanks for doing the investigation work. It's probably worth investigating at some point in the future if using KEYEVENTF_SCANCODE for all (or most, if not possible for all) keys would be better than what kanata is doing now. — Reply to this email directly, [view it on GitHub](#567 (comment)), or [unsubscribe](https://github.com/notifications/unsubscribe-auth/ADA56O7EA5CVC2OHSO45RKTX74DDZAVCNFSM6AAAAAA5FHNENCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONRXGMYTCMJUHE). You are receiving this because you were mentioned.Message ID: ***@***.***>

jtroo · 2023-10-18T04:32:30Z

Not sure what the context is around that, I'm not aware of any issue around holding a key for a long time.

tpstevens · 2023-10-18T04:46:25Z

The original issue that I discovered was that kanata wasn't sending keydown and keyup events (at least how SDL2 receives them). But I think I misread the sample code above -- it looks like the SendInput function is sent for the up and down event, while I originally thought it was only for keys up. So I'd expect all keys to work properly if KEVENTF_SCANCODE was applied.

VictorLemosR · 2023-10-18T13:04:00Z

It's probably worth investigating at some point in the future if using KEYEVENTF_SCANCODE for all (or most, if not possible for all) keys would be better than what kanata is doing now.

wScan seems to simulate a key more realistically than wVk, it does not send directly a virtual code like wVk. It sends a code conversion from the virtual code to what the hardware expects. The conversion is made from MapVirtualKeyA, which I don't know exactly how it is done. Not even sure if it varies between hardware. As an addition, I'd guess it works for all keys, since you can convert all of virtual keys.

I have no idea about performance. For sure my test implementation is not the best, I'm even creating a hash on every key press, although it was not enough for me to notice any change in time at all.

What I didn't solve yet is how to use modifier keys (ctrl+c, for example) which are still bugged in my app. However, it's still so specific for my use that I just didn't care to learn how kanata does it and translate from wvk to wscan. From what I saw, it just seems to use an extended version table of virtual key

Will that properly handle holding a key down for a long time and then releasing?

Yes. You can hold down and it'll repeat the key like expected. Below is an example from windows_key_tester of me holding space key

VictorLemosR · 2023-10-30T18:37:10Z

Edit: windows key was bugged, so I had to come wih an if for it. Also changed code

Btw, I've been testing the new method for all keys for the past 3 days and didn't notice any change at all besides everything also working on the app I wanted (keys like enter, space and all modifiers works normal now).
Here is the new code for send_key_sendinput function

fn send_key_sendinput(code: u16, is_key_up: bool) {
    unsafe {
        let mut kb_input: KEYBDINPUT = mem::zeroed();
        if is_key_up {
            kb_input.dwFlags |= KEYEVENTF_KEYUP;
        }
        //if for windows left key
        if code == 0x5B {
            kb_input.wVk = code;
        } else {
            let code_u32 = code as u32;
            kb_input.dwFlags |= KEYEVENTF_SCANCODE;
            kb_input.wScan = MapVirtualKeyA(code_u32, 0) as u16;
        }

        let mut inputs: [INPUT; 1] = mem::zeroed();
        inputs[0].type_ = INPUT_KEYBOARD;
        *inputs[0].u.ki_mut() = kb_input;
        SendInput(1, inputs.as_mut_ptr(), mem::size_of::<INPUT>() as _);
    }
}

jtroo · 2024-02-11T04:56:03Z

I have added the SCANCODE-using code under a feature flag win_sendinput_send_scancodes for now and plan to build an extra binary with it for some number of releases so that people can experiment with it if desired.

VictorLemosR · 2024-03-06T20:35:15Z

Hey, @jtroo. Just saw your comment, not that active on git. 😓
I've been using the scan method since then and made some tweaks along the way. The last change was in december and everything seems fine. The win key, for example, is working.

Here are the comments I left to my future self so I could remember what I was doing:
"All the keys that are extended are on font 1, inside the table on column 'Scan 1 Make' and start with '0xE0'.
To obtain the scancode, one could just print 'kb_input.wScan' from the function below.
Font 1: https://learn.microsoft.com/en-us/windows/win32/inputdev/about-keyboard-input#scan-codes
To obtain a virtual key code, one could just print 'code' from the function below for a key or see font 2
Font 2: https://learn.microsoft.com/en-us/windows/win32/inputdev/virtual-key-codes

For example, left arrow and 4 from numpad. For the scancode, they have the same low byte,
but not the same high byte, which is 0xE0. LeftArrow = 0xE04B, keypad 4 = 0x004B. For the virtual code,
left arrow is 0x25 and 4 from numpad is 0x64.
There is a windows function called 'MapVirtualKeyA' that can be used to convert a virtual key code to a scancode."

What I imagine is that SDL2-based apps use the scancode, not sure why they use the wrong one though (I even read some problems very similar to ours with people programming games, because the inputs were being translated wrong). With my last code posted, I converted the virtualcode to scancode, but passed only the low byte to 'kb_input.wScan' and the program assumed 0x00 as the high byte for all the keys, resulting in all the extended keys with the wrong output.

I've made an array with the virtualcodes to be able to identify the extended keys and change the high byte. The array was made from looking into font 1 to know whick keys were extended and got the virtualcode inside font 2.

If there is any question, don't mind asking. Here is the new code:

fn send_key_sendinput(code: u16, is_key_up: bool) {
    const EXTENDED_KEYS: [u16; 35] = [
        0xb1, 0xb0, 0xa3, 0xad, 0x8c, 0xb3, 0xb2, 0xae, 0xaf, 0xac, 0x6f,
        0x2c, 0xa5, 0x24, 0x26, 0x21, 0x25, 0x27, 0x23, 0x28, 0x22, 0x2d,
        0x2e, 0x5b, 0x5c, 0x5d, 0x5f, 0xaa, 0xa8, 0xa9, 0xa7, 0xa6, 0xac,
        0xb4, 0x13, 
    ];

    unsafe {
        let mut kb_input: KEYBDINPUT = mem::zeroed();
        if is_key_up {
            kb_input.dwFlags |= KEYEVENTF_KEYUP;
        }

        let code_u32 = code as u32;
        kb_input.dwFlags |= KEYEVENTF_SCANCODE;
        kb_input.wScan = MapVirtualKeyA(code_u32, 0) as u16;
        

        let is_extended_key: bool = EXTENDED_KEYS.contains(&code);
        if is_extended_key {
            kb_input.wScan = (0xE0 << 8) | kb_input.wScan;
            kb_input.dwFlags |= KEYEVENTF_EXTENDEDKEY;
        }
        let mut inputs: [INPUT; 1] = mem::zeroed();
        inputs[0].type_ = INPUT_KEYBOARD;
        *inputs[0].u.ki_mut() = kb_input;
        SendInput(1, inputs.as_mut_ptr(), mem::size_of::<INPUT>() as _);
    }
}

jtroo · 2024-03-15T05:59:55Z

Thanks for the updated code!

Lawrence-of-AnKing · 2024-04-09T11:21:41Z

Not work in Terraria, any key in layer can not accept by game. I can only close Katana to solve it. So sad...

Lawrence-of-AnKing · 2024-04-09T11:23:14Z

How about keep a black name list, in which kanata not active in those program to solve it?

jtroo · 2024-04-09T14:33:24Z

Not work in Terraria, any key in layer can not accept by game. I can only close Katana to solve it. So sad...

Which variant of kanata did you try? Please try one of: winiov2 or scancode from the release page, if you have not tried it.

Lawrence-of-AnKing · 2024-04-10T06:45:27Z

Not work in Terraria, any key in layer can not accept by game. I can only close Katana to solve it. So sad...

Which variant of kanata did you try? Please try one of: winiov2 or scancode from the release page, if you have not tried it.

I tried the winiov2 edition, it works well, thanks for you work!

gerhard-h · 2024-04-10T07:11:35Z

how to compile the winiov2 version myself?

same question for the scancode version, maybe --features win_sendinput_send_scancodes, win_llhook_read_scancodes ?

jtroo · 2024-04-10T12:46:51Z

how to compile the winiov2 version myself?

same question for the scancode version, maybe --features win_sendinput_send_scancodes, win_llhook_read_scancodes ?

You can find the commands used to build them in the justfile.

tpstevens added the bug Something isn't working label Sep 24, 2023

tpstevens assigned jtroo Sep 24, 2023

jtroo added question Further information is requested windows Issue pertains to Windows only labels Sep 24, 2023

jtroo added the llhook Pertains to the standard version of Kanata on Windows label Sep 24, 2023

jtroo removed the question Further information is requested label Oct 17, 2023

jtroo mentioned this issue Oct 29, 2023

Bug: mapped keys seen as unmapped in WinCompose compose sequences #597

Open

1 task

jtroo closed this as completed Feb 11, 2024

jtroo mentioned this issue Mar 12, 2024

Send extended scancodes for keys that require them #817

Closed

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bug: kanata breaks SDL2-based games #567

Bug: kanata breaks SDL2-based games #567

tpstevens commented Sep 24, 2023

tpstevens commented Sep 24, 2023

jtroo commented Sep 24, 2023 •

edited

Loading

tpstevens commented Sep 24, 2023

tpstevens commented Sep 24, 2023

jtroo commented Sep 24, 2023

VictorLemosR commented Oct 13, 2023

jtroo commented Oct 13, 2023 •

edited

Loading

VictorLemosR commented Oct 16, 2023 •

edited

Loading

jtroo commented Oct 16, 2023 •

edited

Loading

VictorLemosR commented Oct 16, 2023

VictorLemosR commented Oct 17, 2023

jtroo commented Oct 17, 2023

tpstevens commented Oct 18, 2023 via email

jtroo commented Oct 18, 2023

tpstevens commented Oct 18, 2023

VictorLemosR commented Oct 18, 2023 •

edited

Loading

VictorLemosR commented Oct 30, 2023 •

edited

Loading

jtroo commented Feb 11, 2024

VictorLemosR commented Mar 6, 2024

jtroo commented Mar 15, 2024

Lawrence-of-AnKing commented Apr 9, 2024

Lawrence-of-AnKing commented Apr 9, 2024 •

edited

Loading

jtroo commented Apr 9, 2024

Lawrence-of-AnKing commented Apr 10, 2024

gerhard-h commented Apr 10, 2024

jtroo commented Apr 10, 2024

Bug: kanata breaks SDL2-based games #567

Bug: kanata breaks SDL2-based games #567

Comments

tpstevens commented Sep 24, 2023

Requirements

Describe the bug

Relevant kanata config

To Reproduce

Expected behavior

Kanata version

Debug logs

Operating system

Additional context

tpstevens commented Sep 24, 2023

jtroo commented Sep 24, 2023 • edited Loading

tpstevens commented Sep 24, 2023

tpstevens commented Sep 24, 2023

jtroo commented Sep 24, 2023

VictorLemosR commented Oct 13, 2023

jtroo commented Oct 13, 2023 • edited Loading

VictorLemosR commented Oct 16, 2023 • edited Loading

jtroo commented Oct 16, 2023 • edited Loading

VictorLemosR commented Oct 16, 2023

VictorLemosR commented Oct 17, 2023

jtroo commented Oct 17, 2023

tpstevens commented Oct 18, 2023 via email

jtroo commented Oct 18, 2023

tpstevens commented Oct 18, 2023

VictorLemosR commented Oct 18, 2023 • edited Loading

VictorLemosR commented Oct 30, 2023 • edited Loading

jtroo commented Feb 11, 2024

VictorLemosR commented Mar 6, 2024

jtroo commented Mar 15, 2024

Lawrence-of-AnKing commented Apr 9, 2024

Lawrence-of-AnKing commented Apr 9, 2024 • edited Loading

jtroo commented Apr 9, 2024

Lawrence-of-AnKing commented Apr 10, 2024

gerhard-h commented Apr 10, 2024

jtroo commented Apr 10, 2024

jtroo commented Sep 24, 2023 •

edited

Loading

jtroo commented Oct 13, 2023 •

edited

Loading

VictorLemosR commented Oct 16, 2023 •

edited

Loading

jtroo commented Oct 16, 2023 •

edited

Loading

VictorLemosR commented Oct 18, 2023 •

edited

Loading

VictorLemosR commented Oct 30, 2023 •

edited

Loading

Lawrence-of-AnKing commented Apr 9, 2024 •

edited

Loading