Skip to content

08. Common pitfalls; tips and tricks

Jump to bottom
Descolada edited this page Sep 27, 2023 · 4 revisions

Preamble

Using UIA can sometimes prove quite tricky, and bugs or unforeseen erratic behaviour can be encountered. This is for multiple reasons:

  1. The userbase for UIA (both this library and UIAutomation itself) is niche and small, and the amount of people testing it is also small, so bugs are more likely.
  2. Originally UIA was created accessibility (people with vision or hearing problems), and accordingly some parts of UIA get more attention than others: patterns sometimes don't work as expected, methods like ElementFromHandle or TreeWalkers might not always give the expected results etc.
  3. UIA properties/methods sometimes need to be implemented by the program creators themselves, who might not pay much (or any) attention to accessibility. Most common controls (buttons, checkboxes etc) work fine, but specialized controls might not be accessible at all. Also some types of windows are sometimes very poorly accessible, for example Qt windows. If you encounter any weird bugs or issues, be sure to raise a issue on this GitHub page, or the AutoHotkey forums thread.

Common pitfalls

Window not visible

  • If a window is minimized it is not accessible by UIA, so consider adding a check for that (with WinGet).
  • Some programs hide their UIA tree when the window is not visible to save memory. Chromium applications in particular do this, so be careful to activate the window before using UIA calls.

Chromium windows

Sometimes Chromium-based windows (such as Chrome, Microsoft Teams, Skype, Discord) cannot be accessed like normal with ElementFromHandle and the content isn't accessible. In this case you will need to use ElementFromChromium:

#include <UIA>
skypeEl := UIA.ElementFromChromium("ahk_exe skype.exe")
MsgBox skypeEl.DumpAll()

You know you are dealing with a Chromium window if Window Spy shows either ahk_class Chrome_WidgetWin_1 for the ahk_class, or ClassNN Chrome_RenderWidgetHostHWND1 for the content.

Also, Chromium might not have accessibility enabled by default, which means that regular ElementFrom... methods might not work and the content of the app might not show. We can send Chromium the signal for accessibility by sending a special message to it (WM_GETOBJECT to Chrome_RenderWidgetHostHWND1), which activates UIA and makes the content accessible. UIA.ElementFrom... methods (ElementFromHandle etc) do this by default since there are just so many Chromium/Electron apps, but to get maximum performance this can be turned off by setting activateChromiumAccessibility to False (eg UIA.ElementFromHandle("A",, False)).

Tips and tricks

My element of interest doesn't have any characteristics to find it

Usually an element can be found using Name, ControlType, AutomationId, ClassName, or a combination of these. For example, there might be multiple elements with the name "Username", but only one Edit field with that name, so FindElement({Name:"Username", Type:"Edit"}) can be used. In some cases though, the element really doesn't have anything to discern it by. In these cases there are a few options:

  1. First find a "reference" element, then move in the tree with WalkTree to the element of interest. For example, if a webpage has a Text element with the name "Username:" but the corresponding Edit field doesn't have an unique name, then if the Edit element is the next sibling for the Text element in the UIA tree, use element.FindElement({Name:"Username", Type:"Edit"}).WalkTree("+1").
  2. Find a container element inside the main element in which FindElement will return your element of interest. For example a window might have many elements with the name "Username:", but if the element of interest is inside a container named "Login fields", then first find the container and then use FindElement on that container instead of the window element.
  3. Locate the element by path with ElementFromPath. You can get the UIA path or condition path with UIAViewer. Note that if the structure of the window changes (for example with a software update, or the user changes it), then the path will also change.

Speed improvements

UIA is generally slower than MSAA/Acc, but if properly used it should be faster than a human. The main advice here is to limit searches of Current (live) elements as much as possible, which means limiting the amount of FindElement and FindElements calls by batching conditions, and using caching.

  1. Use FindElement if looking for only 1-2 elements, but consider using FindElements with condition batching if searching for 3+ elements. Avoid using FindElements unnecessarily since it will generally look through the whole tree whereas FindElement stops searching once a match is found.
  2. Instead of multiple (3+) FindElement calls, consider using FindElements and batching conditions in such a way that all desired elements are returned with one call, and then the individual elements can be determined by their properties. For example, if searching for fields named "Username" and "Password", instead of doing user := FindElement({Name:"Username"}) and pass := FindElement({Name:"Password"), instead do els := FindElements([{Name:"Username"}, {Name:"Password"}]) and then user := els[1].Name = "Password" ? els[2] : els[1]. The speed improvement gained by this is influenced by where the elements are in the tree. If the Username and Password are at the end of the tree, then one FindElements call will be as fast as one FindElement call and thus be twice as fast (since we are doing just one call instead of two). If the elements are at the beginning of the tree then it might be faster to do two FindElement calls. Generally it's better to use FindElements if looking for 3+ matches.
  3. Use a smaller search scope. The default is TreeScope.Descendants (= 4), but in some cases this might be very slow. If one of the first branches of the tree is huge but a later branch is very small, and your element of interest is in the later branch, it might be better to first get that later branch with FindElement TreeScope.Children, and then use FindElement starting from that. For example, in Firefox every tabs' content is displayed in the UIA tree, so searches in Firefox might get very slow unless the search scope is limited (they would search all tabs!). In Firefox's case the search could be limited to the active tab with activeTab := firefox.FindElement({IsOffscreen:0, AutomationId:"panel-", mm:2, scope:"Children"}) and then doing subsequent searches with activeTab.FindElement(...). The key here is search scope "Children", which avoids going through any tab content while looking for the active tab panel.
  4. Depending on the position of the element in the tree, setting the "order" parameter of FindElement might also speed up the search. For example, if the element is at the end of the tree, then set it to "LastToFirstOrder" (or alternatively set index to -1) so the search will start from the end of the tree instead.
  5. Caching may improve the search speed significantly if multiple searches are needed, because the searches could be done with the "Cached" variant of search functions (eg FindCachedElement), which are much faster than the regular ones. If you don't need to interact with the element, then using UIA.AutomationElementMode.None as the CacheRequest mode will speed the search up even more. UIA.AutomationElementMode.None will fetch only a small part of the element (the properties defined in the CacheRequest), whereas without it the whole element will be returned (which is slow).
  6. For multiple searches consider using the "startingElement" argument. By setting that argument the search will start from that element, instead of from the beginning of the tree. For example if you already have found an "Username" element and want to get the "Password" element, start the search from the previously found element: WindowEl.FindElement({Name:"Password", startingElement:usernameElement})

Debugging

Is the element accessible?

Using UIAViewer or Accessibility Insights make sure that the element is actually accessible by UIA, and check what properties, methods, and patterns it has available.

Did you get the correct element?

  1. Make sure with UIAViewer that you have the correct data for your search condition: for example when finding by name, "Value" is not the same as "Value " with a space at the end.
  2. Use element.Dump(), DumpAll(), or Highlight() methods to check if the element you got is the correct one
  3. If working with patterns, make sure that you actually got the pattern by checking for a property of the pattern.

Is the element accessible from a control?

Sometimes elements only show up if ElementFromHandle is used on a control handle, not the main window itself. You can check for that by using UIATreeViewer and expanding the window control tree on the left hand side. If the element is indeed in a control UIA tree then use ControlGetHwnd to get the handle for the control and use it in ElementFromHandle.

Does the pattern work?

If trying to use pattern methods (like InvokePattern Invoke()), then make sure the method actually works with the element using Accessibility Insights or UIAViewer. Sometimes the pattern shows as "available", but isn't properly implemented and actually doesn't work.

Events

When working with events, check the element and its parent with Accessibility Insights to see if the target event is actually triggered. For example, it's relatively common that PropertyChangedEvent isn't actually sent by the program, whereas StructureChanged event might be sent. This means you would have to use StructureChanged instead. In Accessibility Insights first inspect the target element, then click on the "..." button to open Listen to Events:

Clone this wiki locally