Freezing NVDA causes keyboard to stop responding

[tspivey]

STR:

1. In the Python console, execute:
   import re;wx.CallLater(2500, re.search, 'a+b', 'a'*20000000)
2. Press escape to close the console before that runs.

Result:
NVDA freezes, as expected. However, I can't do anything with the keyboard to restart NVDA.

From discussion on IRC, I tried commenting out nvdaControllerInternal_typedCharacterNotify in nvdaHelper/remote/typedCharacter.cpp and was able to use the keyboard to restart if NVDA froze this way.

It looks like when that part of NVDAHelper calls back into Python, NVDAHelper freezes.

[tspivey]

In this case, re doesn't release the GIL.

[jcsteh]

Note that this can only occur if something (like re) doesn't release the GIL. Most C extensions (including ctypes) do release it while doing long-running processing.

I'm also wondering whether this could happen if NVDA's process gets suspended for some reason; e.g. while debugging. @feerrenrut, you mentioned you saw keyboard lockups, etc. when debugging recently. Was this when attaching a debugger to NVDA, some other process or both?

[feerrenrut]

I did see something like this when debugging in windows 7 with VS 2015, attaching the debugger to an already running nvda and then pausing execution. This locked up the whole system (including visual studio, start menu, and even task manager). I had to ctrl + alt + del and log out.

I deleted the devEnv app module, and I was able to use Visual Studio again, however I was still not able to work with other applications (eg Notepad++, start menu)

At different points in time I was running from installed copies, directly from the launcher, and from source. I don't remember the exact timeline with relation to the keyboard lockups. We will need to try this again at some point and take note of the details.

[feerrenrut]

I noticed today, rather than removing the devEnv app module, the difference in behaviour seems to come from running Visual studio with administrator privileges. This is when running from source.

[feerrenrut]

P2 to investigate how we could work around this. To be re-prioritised when more information is available.

@jcsteh Could you please document what you already know about this, re: in proc key listeners / GIL lock.

[dkager]

My keyboard stops responding at least once a day when typing in Word/Outlook with braille enabled. Reproduced on both Windows 7 and 10, no other screen readers installed. Only Ctrl+Alt+Del appears to help.
Is this related, or a separate issue? Can I do anything to track down the cause?

[jcsteh]

Info on why this occurs:

1. NVDA has an in-process hook to monitor typed characters. We do have a keyboard hook as well, but typed characters are more than just keys pressed; e.g. dead keys (where the first key does nothing and a second key types a character influenced by the first key), typing Unicode characters using alt plus numpad keys, etc. Windows doesn't provide a way for us to access this information out-of-process.
2. When a typed character is received, we call an RPC function in NVDA.
3. The RPC function executes in an RPC thread in NVDA and queues an event to the main thread.
4. If the main thread is busy, there is no problem because the function executes on a different thread, queues to the main thread and returns immediately.
5. However, because the RPC function executes Python code, it does block if another thread is holding the GIL. Normally, the GIL isn't held for very long. However, some modules (such as re) don't release the GIL at all while they're processing.

Possible solutions:

1. We can truncate anything sent to speech beyond a reasonable limit. There are probably other reasons to do this anyway. Some synths might not cope so well with such long strings. This is complicated somewhat by the fact that spoken text is not a single string; it could be scattered among commands.
2. We can somehow make the typed character RPC function C++ only. That avoids the GIL. However, we'd need to find some way to notify Python which doesn't involve calling any Python code (and which doesn't involve polling).
3. I came across the new regex module which will eventually replace the re module. It apparently fixes this GIL bug.

I think option 1 actually has the most benefit. Options 2 and 3 do fix the GIL issue, but they're much harder to implement and they also don't stop the main thread from freezing or synths from choking.

I'm also concerned this might happen if NVDA's process gets suspended; e.g. by a debugger. In this case, it's nothing to do with the GIL, but remote processes don't know NVDA has been suspended and will still try to make RPC calls which will block until NVDA resumes. None of the above solutions fix this problem. The only way we could fix that is to make the notification to NVDA asynchronous. One way would be using Asynchronous RPC.

[Laughingthunder]

Hi, If I am understanding you correctly, you are suggesting truncation of speech synthesis? If so, what do you consider to be a "reasonable limit"? Just wondering.

…

On 2/6/2017 4:39 PM, James Teh wrote: Info on why this occurs: 1. NVDA has an in-process hook to monitor typed characters. We do have a keyboard hook as well, but typed characters are more than just keys pressed; e.g. dead keys (where the first key does nothing and a second key types a character influenced by the first key), typing Unicode characters using alt plus numpad keys, etc. Windows doesn't provide a way for us to access this information out-of-process. 2. When a typed character is received, we call an RPC function in NVDA. 3. The RPC function executes in an RPC thread in NVDA and queues an event to the main thread. 4. If the main thread is busy, there is no problem because the function executes on a different thread, queues to the main thread and returns immediately. 5. However, because the RPC function executes Python code, it does block if another thread is holding the GIL. Normally, the GIL isn't held for very long. However, some modules (such as |re|) don't release the GIL at all while they're processing. Possible solutions: 1. We can truncate anything sent to speech beyond a reasonable limit. There are probably other reasons to do this anyway. Some synths might not cope so well with such long strings. This is complicated somewhat by the fact that spoken text is not a single string; it could be scattered among commands. 2. We can somehow make the typed character RPC function C++ only. That avoids the GIL. However, we'd need to find some way to notify Python which doesn't involve calling any Python code (and which doesn't involve polling). 3. I came across the new regex module <https://pypi.python.org/pypi/regex> which will eventually replace the |re| module. It apparently fixes this GIL bug. I think option 1 actually has the most benefit. Options 2 and 3 do fix the GIL issue, but they're much harder to implement and they also don't stop the main thread from freezing or synths from choking. I'm also concerned this might happen if NVDA's process gets suspended; e.g. by a debugger. In this case, it's nothing to do with the GIL, but remote processes don't know NVDA has been suspended and will still try to make RPC calls which will block until NVDA resumes. None of the above solutions fix this problem. The only way we could fix that is to make the notification to NVDA asynchronous. One way would be using Asynchronous RPC <https://msdn.microsoft.com/en-us/library/windows/desktop/aa373550%28v=vs.85%29.aspx>. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#6463 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/APElWdT9EK9K4xWv13H02SkBPo0NeCYoks5rZ6EcgaJpZM4KWq4v>.

[LeonarddeR]

Could this somehow be related to #6291?

@jcsteh commented on 6 Feb 2017, 23:39 CET:

1. We can truncate anything sent to speech beyond a reasonable limit.

I'm probably missing something, but I do not yet see how truncating speech can help in solving issues with the typed characters hook. Aren't these unrelated issues, at least when character/word echo is off?

[jcsteh]

I'm probably missing something, but I do not yet see how truncating speech can help in solving issues with the typed characters hook. Aren't these unrelated issues, at least when character/word echo is off?

As noted in #6463 (comment), the hook calls an RPC function in NVDA. Under normal circumstances, that works just fine, since it runs on another thread. However, if something is holding the Python GIL (remembering that the GIL is a global lock; only one thread can hold it at a time), the RPC function will block until the GIL is released. We use regular expressions for speech processing, and unfortunately, the re module seems to hold the GIL for the entire duration of its processing. So, if a massive string is spoken and it's still being processed by the re module when the typed character hook runs, the RPC function will freeze, thus freezing the remote process that initiated the RPC function.

[LeonarddeR]

@jcsteh: thanks for the clarification.

I wonder whether aasync RPC as you mentioned in #6463 (comment) might be yet a way to investigate further, as there are most likely more situations where this could be beneficial. At least, this applies to every case where a process could get locked up like this.

[LeonarddeR]

For anyone experiencing hangs in Outlook without UIA enabled, I created a prototype try build that executes Word in process cals using watchdog.cancellableExecute. It will most likely raise CallCancelled in huge documents where the object model is pretty slow, but in theory, it will at least avoid that the main thread gets stuck for ever.

See https://ci.appveyor.com/api/buildjobs/322p7yv21r4e3jew/artifacts/output%2Fnvda_snapshot_try-wordWatchdog-18159%2Cda148151.exe

cc @bramd @dkager

[LeonarddeR]

Small update about the try build I mentioned in #6463 (comment). I'm afraid it degrades performance even more.

[Adriani90]

Not sure if it helps, but while I was reproducing this issue with NvDA 2024.1 on Windows 11 23 H2, I started Narator which didn't really work until I moved the mouse. For what ever reason, Narator generated a crashdump while the keyboard was not responding.
I attach it here.
Narrator.exe.15228.dmp
Narrator.exe.15228.dmp.zip

[jcsteh]

2. When a typed character is received, we call an RPC function in NVDA.

To be more accurate, we call an RPC function in nvdaHelperLocal, which then calls into NVDA. This just led me to another possible solution. Instead of calling a Python function here, we could shove it in a C++ queue and use PostMessage to notify the NVDA main thread to wake up, which could then consume the queue. We would probably want to do this with all RPC functions. That would mean they would never run synchronously though and their might be consequences I don't know about for some of these functions. Also, this still doesn't solve the problem when the NVDA process is suspended by a debugger.

[jcsteh]

This C++ queue + PostMessage solution might also solve #14764.

[jcsteh]

Strictly speaking, it doesn't have to be the main thread which consumes the queue. It could be a background thread too, which we would wake with SetEvent or similar. Either way, though, the Python code would not run synchronously in the RPC callback.

[dpy013]

What is the progress on current issues?