UI Automation in Windows Console: Make UIA the default for Windows 10…

… 1803 and later (PR #9771)

NVDA's UIA console support significantly improves performance and stability, so we should enable it by default.

While UIA in consoles was added in Windows 10 1709, the initial implementation left much to be desired, particularly concerning caret movement and expansion to character.

A new `UIAUtils.shouldUseUIAConsole` function has been added which determines whether to use UIA in consoles:
* If the user has explicitly chosen to use UIA or legacy console support, the function returns `True` or `False` respectively.
* Otherwise, if the user is running at least Windows 10 1803, the function returns `True`, or `False` if not.

The "use UI Automation in the Windows Console when available" GUI setting has been changed to expose all options in the config spec (i.e. to automatically detect UIA/legacy or override this autodetection).

builds on #9614
closes #673
closes #6291

source/NVDAObjects/UIA/__init__.py

@@ -26,6 +26,7 @@
 import textInfos
 from logHandler import log
 from UIAUtils import *
+from UIAUtils import shouldUseUIAConsole
 from NVDAObjects.window import Window
 from NVDAObjects import NVDAObjectTextInfo, InvalidNVDAObject
 from NVDAObjects.behaviors import ProgressBar, EditableTextWithoutAutoSelectDetection, Dialog, Notification, EditableTextWithSuggestions, ToolTip
@@ -871,7 +872,7 @@ def findOverlayClasses(self,clsList):
 		# Support Windows Console's UIA interface
 		if (
 			self.windowClassName == "ConsoleWindowClass"
-			and config.conf['UIA']['winConsoleImplementation'] == "UIA"
+			and shouldUseUIAConsole()
 		):
 			from . import winConsoleUIA
 			winConsoleUIA.findExtraOverlayClasses(self, clsList)

source/NVDAObjects/UIA/winConsoleUIA.py

@@ -11,7 +11,6 @@
 
 from comtypes import COMError
 from UIAUtils import isTextRangeOffscreen
-from winVersion import isWin10
 from . import UIATextInfo
 from ..behaviors import KeyboardHandlerBasedTypedCharSupport
 from ..window import Window
@@ -24,30 +23,28 @@ class consoleUIATextInfo(UIATextInfo):
 	#: to do much good either.
 	_expandCollapseBeforeReview = False
 
-	def __init__(self,obj,position,_rangeObj=None):
+	def __init__(self, obj, position, _rangeObj=None):
 		super(consoleUIATextInfo, self).__init__(obj, position, _rangeObj)
 		# Re-implement POSITION_FIRST and POSITION_LAST in terms of
 		# visible ranges to fix review top/bottom scripts.
-		if position==textInfos.POSITION_FIRST:
+		if position == textInfos.POSITION_FIRST:
 			visiRanges = self.obj.UIATextPattern.GetVisibleRanges()
 			firstVisiRange = visiRanges.GetElement(0)
 			self._rangeObj = firstVisiRange
 			self.collapse()
-		elif position==textInfos.POSITION_LAST:
+		elif position == textInfos.POSITION_LAST:
 			visiRanges = self.obj.UIATextPattern.GetVisibleRanges()
 			lastVisiRange = visiRanges.GetElement(visiRanges.length - 1)
 			self._rangeObj = lastVisiRange
 			self.collapse(True)
 
-	def collapse(self,end=False):
-		"""Works around a UIA bug on Windows 10 1903 and later."""
-		if not isWin10(1903):
-			return super(consoleUIATextInfo, self).collapse(end=end)
+	def collapse(self, end=False):
+		"""Works around a UIA bug on Windows 10 1803 and later."""
 		# When collapsing, consoles seem to incorrectly push the start of the
 		# textRange back one character.
 		# Correct this by bringing the start back up to where the end is.
-		oldInfo=self.copy()
-		super(consoleUIATextInfo,self).collapse(end=end)
+		oldInfo = self.copy()
+		super(consoleUIATextInfo, self).collapse(end=end)
 		if not end:
 			self._rangeObj.MoveEndpointByRange(
 				UIAHandler.TextPatternRangeEndpoint_Start,
@@ -157,9 +154,7 @@ def expand(self, unit):
 			return super(consoleUIATextInfo, self).expand(unit)
 
 	def _get_isCollapsed(self):
-		"""Works around a UIA bug on Windows 10 1903 and later."""
-		if not isWin10(1903):
-			return super(consoleUIATextInfo, self)._get_isCollapsed()
+		"""Works around a UIA bug on Windows 10 1803 and later."""
 		# Even when a console textRange's start and end have been moved to the
 		# same position, the console incorrectly reports the end as being
 		# past the start.
@@ -215,9 +210,9 @@ def _getWordOffsetsInThisLine(self, offset, lineInfo):
 			min(end.value, max(1, len(lineText) - 2))
 		)
 
-	def __ne__(self,other):
+	def __ne__(self, other):
 		"""Support more accurate caret move detection."""
-		return not self==other
+		return not self == other
 
 
 class consoleUIAWindow(Window):

source/NVDAObjects/behaviors.py

@@ -413,7 +413,7 @@ def event_typedCharacter(self, ch):
 				config.conf['keyboard']['speakTypedCharacters']
 				or config.conf['keyboard']['speakTypedWords']
 			)
-			and not config.conf['UIA']['winConsoleSpeakPasswords']
+			and not config.conf['terminals']['speakPasswords']
 			and self._supportsTextChange
 		):
 			self._queuedChars.append(ch)

source/UIAUtils.py

@@ -1,12 +1,14 @@
-#A part of NonVisual Desktop Access (NVDA)
-#Copyright (C) 2015-2016 NV Access Limited
-#This file is covered by the GNU General Public License.
-#See the file COPYING for more details.
+# A part of NonVisual Desktop Access (NVDA)
+# Copyright (C) 2015-2019 NV Access Limited, Bill Dengler
+# This file is covered by the GNU General Public License.
+# See the file COPYING for more details.
 
 import operator
 from comtypes import COMError
+import config
 import ctypes
 import UIAHandler
+from winVersion import isWin10
 
 def createUIAMultiPropertyCondition(*dicts):
 	"""
@@ -220,3 +222,21 @@ def getValue(self,ID,ignoreMixedValues=False):
 			raise UIAMixedAttributeError
 		return val
 
+
+def shouldUseUIAConsole(setting=None):
+	"""Determines whether to use UIA in the Windows Console.
+@param setting: the config value to base this check on (if not provided,
+it is retrieved from config).
+	"""
+	if not setting:
+		setting = config.conf['UIA']['winConsoleImplementation']
+	if setting == "legacy":
+		return False
+	elif setting == "UIA":
+		return True
+	# #7497: Windows 10 Fall Creators Update has an incomplete UIA
+	# implementation for console windows, therefore for now we should
+	# ignore it.
+	# It does not implement caret/selection, and probably has no
+	# new text events.
+	return isWin10(1803)

source/_UIAHandler.py

@@ -356,9 +356,10 @@ def IUIAutomationNotificationEventHandler_HandleNotificationEvent(self,sender,No
 
 	def _isBadUIAWindowClassName(self, windowClass):
 		"Given a windowClassName, returns True if this is a known problematic UIA implementation."
-		# #7497: Windows 10 Fall Creators Update has an incomplete UIA implementation for console windows, therefore for now we should ignore it.
-		# It does not implement caret/selection, and probably has no new text events.
-		if windowClass == "ConsoleWindowClass" and config.conf['UIA']['winConsoleImplementation'] != "UIA":
+		if (
+			windowClass == "ConsoleWindowClass"
+			and not UIAUtils.shouldUseUIAConsole()
+		):
 			return True
 		return windowClass in badUIAWindowClassNames
 

source/config/configSpec.py

@@ -11,11 +11,11 @@
 #: provide an upgrade step (@see profileUpgradeSteps.py). An upgrade step does not need to be added when
 #: just adding a new element to (or removing from) the schema, only when old versions of the config 
 #: (conforming to old schema versions) will not work correctly with the new schema.
-latestSchemaVersion = 2
+latestSchemaVersion = 3
 
 #: The configuration specification string
 #: @type: String
-configSpecString = ("""# NVDA Configuration File
+configSpecString = (f"""# NVDA Configuration File
 schemaVersion = integer(min=0, default={latestSchemaVersion})
 [general]
 	language = string(default="Windows")
@@ -188,9 +188,9 @@
 	enabled = boolean(default=true)
 	useInMSWordWhenAvailable = boolean(default=false)
 	winConsoleImplementation= option("auto", "legacy", "UIA", default="auto")
-	winConsoleSpeakPasswords = boolean(default=false)
 
 [terminals]
+	speakPasswords = boolean(default=false)
 	keyboardSupportInLegacy = boolean(default=True)
 
 [update]
@@ -224,7 +224,7 @@
 
 [development]
 	enableScratchpadDir = boolean(default=false)
-""").format(latestSchemaVersion=latestSchemaVersion)
+""")
 
 #: The configuration specification
 #: @type: ConfigObj

source/config/profileUpgradeSteps.py

@@ -45,3 +45,16 @@ def upgradeConfigFrom_1_to_2(profile):
 	else:
 		del profile["braille"]["cursorShape"]
 		profile["braille"]["cursorShapeFocus"] = cursorShape
+
+
+def upgradeConfigFrom_2_to_3(profile):
+	# The winConsoleSpeakPasswords option has been moved to the terminals section of the config.
+	try:
+		speakPasswords = profile["UIA"]["winConsoleSpeakPasswords"]
+	except KeyError:
+		# Setting does not exist, no need for upgrade of this setting
+		log.debug("winConsoleSpeakPasswords not present, no action taken.")
+		pass
+	else:
+		del profile["UIA"]["winConsoleSpeakPasswords"]
+		profile["terminals"]["speakPasswords"] = speakPasswords

source/gui/settingsDialogs.py

@@ -1,9 +1,11 @@
 # -*- coding: UTF-8 -*-
-#settingsDialogs.py
-#A part of NonVisual Desktop Access (NVDA)
-#Copyright (C) 2006-2019 NV Access Limited, Peter Vágner, Aleksey Sadovoy, Rui Batista, Joseph Lee, Heiko Folkerts, Zahari Yurukov, Leonard de Ruijter, Derek Riemer, Babbage B.V., Davy Kager, Ethan Holliger
-#This file is covered by the GNU General Public License.
-#See the file COPYING for more details.
+# settingsDialogs.py
+# A part of NonVisual Desktop Access (NVDA)
+# Copyright (C) 2006-2019 NV Access Limited, Peter Vágner, Aleksey Sadovoy,
+# Rui Batista, Joseph Lee, Heiko Folkerts, Zahari Yurukov, Leonard de Ruijter,
+# Derek Riemer, Babbage B.V., Davy Kager, Ethan Holliger, Bill Dengler
+# This file is covered by the GNU General Public License.
+# See the file COPYING for more details.
 
 import logging
 from abc import abstractmethod
@@ -42,6 +44,7 @@
 import inputCore
 from . import nvdaControls
 from driverHandler import *
+from UIAUtils import shouldUseUIAConsole
 import touchHandler
 import winVersion
 import weakref
@@ -2053,20 +2056,43 @@ def __init__(self, parent):
 		self.UIAInMSWordCheckBox.SetValue(config.conf["UIA"]["useInMSWordWhenAvailable"])
 		self.UIAInMSWordCheckBox.defaultValue = self._getDefaultValue(["UIA", "useInMSWordWhenAvailable"])
 
-		# Translators: This is the label for a checkbox in the
-		#  Advanced settings panel.
-		label = _("Use UI Automation to access the Windows Console when available")
-		consoleUIADevMap = True if config.conf['UIA']['winConsoleImplementation'] == 'UIA' else False
-		self.ConsoleUIACheckBox=UIAGroup.addItem(wx.CheckBox(self, label=label))
-		self.ConsoleUIACheckBox.SetValue(consoleUIADevMap)
-		self.ConsoleUIACheckBox.defaultValue = self._getDefaultValue(["UIA", "winConsoleImplementation"])
-
-		# Translators: This is the label for a checkbox in the
-		#  Advanced settings panel.
-		label = _("Speak &passwords in UIA consoles (may improve performance)")
-		self.winConsoleSpeakPasswordsCheckBox=UIAGroup.addItem(wx.CheckBox(self, label=label))
-		self.winConsoleSpeakPasswordsCheckBox.SetValue(config.conf["UIA"]["winConsoleSpeakPasswords"])
-		self.winConsoleSpeakPasswordsCheckBox.defaultValue = self._getDefaultValue(["UIA", "winConsoleSpeakPasswords"])
+		# Translators: This is the label for a combo box for selecting the
+		# active console implementation in the advanced settings panel.
+		# Choices are automatic, prefer UIA, and legacy.
+		consoleComboText = _("Windows C&onsole support:")
+		consoleChoices = [
+			# Translators: A choice in a combo box in the advanced settings
+			# panel to have NVDA determine its Windows Console implementation
+			# automatically.
+			_("Automatic"),
+			# Translators: A choice in a combo box in the advanced settings
+			# panel to have NVDA use UIA in the Windows Console when available.
+			_("Prefer UIA"),
+			# Translators: A choice in a combo box in the advanced settings
+			# panel to have NVDA use its legacy Windoes Console support
+			# in all cases.
+			_("Legacy")
+		]
+		#: The possible console config values, in the order they appear
+		#: in the combo box.
+		self.consoleVals = (
+			"auto",
+			"UIA",
+			"legacy"
+		)
+		self.consoleCombo = UIAGroup.addLabeledControl(consoleComboText, wx.Choice, choices=consoleChoices)
+		self.consoleCombo.Bind(
+			wx.EVT_CHOICE,
+			self.enableConsolePasswordsCheckBox,
+			self.consoleCombo
+		)
+		curChoice = self.consoleVals.index(
+			config.conf['UIA']['winConsoleImplementation']
+		)
+		self.consoleCombo.SetSelection(curChoice)
+		self.consoleCombo.defaultValue = self.consoleVals.index(
+			self._getDefaultValue(["UIA", "winConsoleImplementation"])
+		)
 
 		# Translators: This is the label for a group of advanced options in the
 		#  Advanced settings panel
@@ -2078,6 +2104,13 @@ def __init__(self, parent):
 		sHelper.addItem(terminalsGroup)
 		# Translators: This is the label for a checkbox in the
 		#  Advanced settings panel.
+		label = _("Speak &passwords in Windows Console (may improve performance)")
+		self.speakPasswordsCheckBox = terminalsGroup.addItem(wx.CheckBox(self, label=label))
+		self.speakPasswordsCheckBox.SetValue(config.conf["terminals"]["speakPasswords"])
+		self.speakPasswordsCheckBox.defaultValue = self._getDefaultValue(["terminals", "speakPasswords"])
+		self.enableConsolePasswordsCheckBox()
+		# Translators: This is the label for a checkbox in the
+		#  Advanced settings panel.
 		label = _("Use the new t&yped character support in legacy Windows consoles when available")
 		self.keyboardSupportInLegacyCheckBox=terminalsGroup.addItem(wx.CheckBox(self, label=label))
 		self.keyboardSupportInLegacyCheckBox.SetValue(config.conf["terminals"]["keyboardSupportInLegacy"])
@@ -2156,6 +2189,13 @@ def __init__(self, parent):
 		]
 		self.Layout()
 
+	def enableConsolePasswordsCheckBox(self, evt=None):
+		return self.speakPasswordsCheckBox.Enable(
+			shouldUseUIAConsole(self.consoleVals[
+				self.consoleCombo.GetSelection()
+			])
+		)
+
 	def onOpenScratchpadDir(self,evt):
 		path=config.getScratchpadDir(ensureExists=True)
 		os.startfile(path)
@@ -2168,8 +2208,8 @@ def haveConfigDefaultsBeenRestored(self):
 			self._defaultsRestored and
 			self.scratchpadCheckBox.IsChecked() == self.scratchpadCheckBox.defaultValue and
 			self.UIAInMSWordCheckBox.IsChecked() == self.UIAInMSWordCheckBox.defaultValue and
-			self.ConsoleUIACheckBox.IsChecked() == (self.ConsoleUIACheckBox.defaultValue=='UIA') and
-			self.winConsoleSpeakPasswordsCheckBox.IsChecked() == self.winConsoleSpeakPasswordsCheckBox.defaultValue and
+			self.consoleCombo.GetSelection() == self.consoleCombo.defaultValue and
+			self.speakPasswordsCheckBox.IsChecked() == self.speakPasswordsCheckBox.defaultValue and
 			self.keyboardSupportInLegacyCheckBox.IsChecked() == self.keyboardSupportInLegacyCheckBox.defaultValue and
 			self.autoFocusFocusableElementsCheckBox.IsChecked() == self.autoFocusFocusableElementsCheckBox.defaultValue and
 			self.caretMoveTimeoutSpinControl.GetValue() == self.caretMoveTimeoutSpinControl.defaultValue and
@@ -2180,8 +2220,8 @@ def haveConfigDefaultsBeenRestored(self):
 	def restoreToDefaults(self):
 		self.scratchpadCheckBox.SetValue(self.scratchpadCheckBox.defaultValue)
 		self.UIAInMSWordCheckBox.SetValue(self.UIAInMSWordCheckBox.defaultValue)
-		self.ConsoleUIACheckBox.SetValue(self.ConsoleUIACheckBox.defaultValue=='UIA')
-		self.winConsoleSpeakPasswordsCheckBox.SetValue(self.winConsoleSpeakPasswordsCheckBox.defaultValue)
+		self.consoleCombo.SetSelection(self.consoleCombo.defaultValue == 'UIA')
+		self.speakPasswordsCheckBox.SetValue(self.speakPasswordsCheckBox.defaultValue)
 		self.keyboardSupportInLegacyCheckBox.SetValue(self.keyboardSupportInLegacyCheckBox.defaultValue)
 		self.autoFocusFocusableElementsCheckBox.SetValue(self.autoFocusFocusableElementsCheckBox.defaultValue)
 		self.caretMoveTimeoutSpinControl.SetValue(self.caretMoveTimeoutSpinControl.defaultValue)
@@ -2192,11 +2232,11 @@ def onSave(self):
 		log.debug("Saving advanced config")
 		config.conf["development"]["enableScratchpadDir"]=self.scratchpadCheckBox.IsChecked()
 		config.conf["UIA"]["useInMSWordWhenAvailable"]=self.UIAInMSWordCheckBox.IsChecked()
-		if self.ConsoleUIACheckBox.IsChecked():
-			config.conf['UIA']['winConsoleImplementation'] = "UIA"
-		else:
-			config.conf['UIA']['winConsoleImplementation'] = "auto"
-		config.conf["UIA"]["winConsoleSpeakPasswords"]=self.winConsoleSpeakPasswordsCheckBox.IsChecked()
+		consoleChoice = self.consoleCombo.GetSelection()
+		config.conf['UIA']['winConsoleImplementation'] = (
+			self.consoleVals[consoleChoice]
+		)
+		config.conf["terminals"]["speakPasswords"] = self.speakPasswordsCheckBox.IsChecked()
 		config.conf["terminals"]["keyboardSupportInLegacy"]=self.keyboardSupportInLegacyCheckBox.IsChecked()
 		config.conf["virtualBuffers"]["autoFocusFocusableElements"] = self.autoFocusFocusableElementsCheckBox.IsChecked()
 		config.conf["editableText"]["caretMoveTimeoutMs"]=self.caretMoveTimeoutSpinControl.GetValue()

user_docs/en/userGuide.t2t

@@ -1686,11 +1686,21 @@ This includes in Microsoft Word itself, and also the Microsoft Outlook message v
 However, There may be some information which is either not exposed, or exposed incorrectly in some versions of Microsoft Office, which means this UI automation support cannot always be relied upon.
 We still do not recommend that  the majority of users turn this on by default, though we do welcome users of Office 2016/365 to test this feature and provide feedback.
 
-==== Use UI Automation to access the Windows Console when available ====[AdvancedSettingsConsoleUIA]
-When this option is enabled, NVDA will use a new, work in progress version of its support for Windows Console which takes advantage of [accessibility improvements made by Microsoft https://devblogs.microsoft.com/commandline/whats-new-in-windows-console-in-windows-10-fall-creators-update/]. This feature is highly experimental and is still incomplete, so its use is not yet recommended. However, once completed, it is anticipated that this new support will become the default, improving NVDA's performance and stability in Windows command consoles.
+==== Windows Console support ====[AdvancedSettingsConsoleUIA]
+This option selects how NVDA interacts with the Windows Console used by command prompt, PowerShell, and the Windows Subsystem for Linux.
+It does not affect the modern Windows Terminal.
+In Windows 10 version 1709, Microsoft [added support for its UI Automation API to the console https://devblogs.microsoft.com/commandline/whats-new-in-windows-console-in-windows-10-fall-creators-update/], bringing vastly improved performance and stability for screen readers that support it.
+In situations where UI Automation is unavailable or known to result in an inferior user experience, NVDA's legacy console support is available as a fallback.
+The Windows Console support combo box has three options:
+- Automatic: Uses UI Automation in consoles on Windows 10 version 1803 and later. This option is recommended and set by default.
+- Prefer UIA: Uses UI Automation in consoles if available, even on Windows versions with incomplete or buggy implementations. While this limited functionality may be useful (and even sufficient for your usage), use of this option is entirely at your own risk and no support for it will be provided.
+- Legacy: UI Automation in the Windows Console will be completely disabled, so the legacy fallback will always be used.
+-
 
-==== Speak passwords in UIA consoles ====[AdvancedSettingsWinConsoleSpeakPasswords]
-This setting controls whether characters are spoken by [speak typed characters #KeyboardSettingsSpeakTypedCharacters] or [speak typed words #KeyboardSettingsSpeakTypedWords] in situations where the screen does not update (such as password entry) in the Windows Console with UI automation support enabled. For security purposes, this setting should be left disabled. However, you may wish to enable it if you experience performance issues or instability with typed character and/or word reporting while using NVDA's new experimental console support.
+==== Speak passwords in Windows Console ====[AdvancedSettingsWinConsoleSpeakPasswords]
+This setting controls whether characters are spoken by [speak typed characters #KeyboardSettingsSpeakTypedCharacters] or [speak typed words #KeyboardSettingsSpeakTypedWords] in situations where the screen does not update (such as password entry) in some terminal programs, such as the Windows Console with UI automation support enabled and Mintty.
+For security purposes, this setting should be left disabled.
+However, you may wish to enable it if you experience performance issues or instability with typed character and/or word reporting in consoles, or work in trusted environments and prefer password announcement.
 
 ==== Use the new typed character support in legacy Windows consoles when available ====[AdvancedSettingsKeyboardSupportInLegacy]
 This option enables an alternative method for detecting typed characters in legacy Windows consoles.