Guidance for touch devices

[ZoeBijl]

From @LJWatson on May 25, 2015 12:30

We need to address the problem of design pattern accessibility on touch screen devices. Touch ATs intercept all touch gestures. Only click/blur/focus are handled reliably - and they're not sufficient to support interaction with complex widgets like trees.

We should look to include guidance that surfaces these shortcomings, and indicate which of the design patterns will/won't work on touch devices with ATs enabled.

Copied from original issue: w3c/aria#60

[ZoeBijl]

From @patrickhlauke on May 25, 2015 13:43

For context (as the above is paraphrasing my recent ranting on twitter), this relates to further thinking around this bug/comment I filed https://lists.w3.org/Archives/Public/public-pfwg-comments/2015AprJun/0005.html / https://www.w3.org/Bugs/Public/show_bug.cgi?id=28694

[ZoeBijl]

From @stevefaulkner on May 25, 2015 13:56

Have set up a google spreadsheet to record compatibility issues with design patterns
https://docs.google.com/spreadsheets/d/1gN9oRZPdrJxLDNtT6nVO4fn7E7sn1061L9Xl3__slZ4/edit?usp=sharing

[ZoeBijl]

From @patrickhlauke on May 25, 2015 13:56

A simple, concrete example: take an implementation of a "faked" slider like http://hanshillen.github.io/jqtest/#goto_slider - to my knowledge, this implementation follows the advice provided in http://www.w3.org/WAI/PF/aria-practices/#slider which defines the expected keyboard behavior (use of left/right arrow, home/end, page up/down, tab to get in/out of the slider widget itself). Now, as with many other patterns, this: a) assumes that it's working fine for mouse users (so nothing special is defined/mentioned about how a mouse user would need to interacts with it) and b) defines behavior for keyboard users, which the author is expected to include by hooking into keydown/keypress events, check for those defined expected keystrokes, and trigger relevant JS in response.

This simple slider is unusuable for users with touchscreen interfaces and AT. Simply taking the example of iOS/VoiceOver, a user is able to set focus to the slider "knob". VoiceOver announces (because of the role="slider") that it's "adjustable" and that a user can "swipe up or down with one finger to adjust the value". However, doing this has no effect - as the actual functionality that fakes this is attached via keydown events listening for arrow/home/end/etc, and swiping up/down does NOT trigger these (VoiceOver does not fire fake keyboard events that would rattle those event handlers, and I believe it shouldn't either - though it may have to come to this if no better solution is available). The naive solution here would be to also layer touch/gesture events on top of the keyboard listener events, but this will not work because VO will intercept touch events/gestures (similar to role="document" on desktop/keyboard/AT scenarios, and there's no sensible way to force touch/AT like VO to go into role="application" mode). The only way, if there WERE touch/gesture listeners, for a user would be to perform an override/pass-through gesture in VO (double-tap and hold, then swipe) to pass actual swipe gestures to the underlying page/JS, bypassing VO...but this, in my mind, is the philosophical equivalent of saying "keyboard users can just use mouse keys to interact with this" - it's a crutch, not a solution (and not all platforms have this sort of pass-through gesture...for instance, I don't believe Android/Talkback offers anything like this).

In short, even with such a simple widget as a slider, following the current advice/design pattern (which tacitly assumes a false dychotomy of mouse/keyboard only) results in a construct that simply will not work for touch/AT users.

[ZoeBijl]

From @patrickhlauke on May 25, 2015 14:1

Another example of a construct that follows ARIA design patterns and fails for touch/AT: http://hanshillen.github.io/jqtest/#goto_menubar which follows http://www.w3.org/WAI/PF/aria-practices/#menu - specifically, if a menu item has a submenu, the pattern currently expects

If the menu item with focus has a submenu, pressing Enter, Space, or the right arrow key opens the submenu and puts focus on the first submenu item.

With iOS/VO, I can open the first level of each menu, set focus to any item, but if the item also has a submenu, i cannot trigger it (in Hans' implementation), as this will NOT send Enter/Space/right arrow, but simply a click event

[ZoeBijl]

From @patrickhlauke on May 25, 2015 14:6

A third one, for good measure: http://hanshillen.github.io/jqtest/#goto_tree which implements the TreeView http://www.w3.org/WAI/PF/aria-practices/#TreeView

With iOS/VO, I can set focus to individual visible nodes (and also hidden nodes, but this seems a shortcoming in Hans' implementation, rather than intention), but I cannot open/expand a tree branch, as I cannot fire

Right arrow key expands a closed node, moves to the first child of an open node, or does nothing on an end node.

I can only fire a click event at the actual focused node, which only triggers what I assume is the mouse-based equivalent coded for

Enter key performs the default action on end nodes.

[ZoeBijl]

From @patrickhlauke on May 25, 2015 19:38

Additionally, when I originally said that for touch/AT situation we can only currently rely on focus/blur/click ... the situation is actually even murkier, as some OS/Browser combos don't send focus/blur when you'd expect it. see http://patrickhlauke.github.io/touch/tests/results/#mobile-tablet-touchscreen-assistive-technology-events

[ZoeBijl]

From @patrickhlauke on May 26, 2015 8:38

Quick additional note, from testing: iOS devices with a paired bluetooth keyboard do not fire key events (whether VO is enabled or not) unless focus is in a text input/textarea (so patterns reliant on key events will fail even for "keyboard" users on iOS); Android with a paired keyboard does act mostly like a traditional "desktop" device with keyboard - no keys are intercepted, fired as normal (in standard Browser or Chrome; some browsers, like Firefox/Android, do provide additional shortcut keys when running Talkback, and these are intercepted)

[ZoeBijl]

From @mraccess77 on June 10, 2015 15:47

In my experience custom ARIA combo boxes are also problematic with mobile devices such as iOS and Android. Listboxes faired better in my testing. My presentation at CSUN indicated the same challenges with sliders, menus, and trees as discussed above. Spinboxes also were not supported.

[ZoeBijl]

From @DavidMacDonald on June 15, 2015 14:53

I'm totally on board with this discussion and we'll be looking at it carefully with the Mobile Task Force

[ZoeBijl]

From @hshepherd on January 25, 2016 21:2

It seems like interactions with role="slider" elements are still an issue in iOS Voiceover even when using the recommended aria attributes (aria-valuemin, aria-valuemax, aria-valuenow). Are there any plans to revisit this?

[ZoeBijl]

From @patrickhlauke on January 25, 2016 21:6

interactions with ARIA menus that have submenus are still an issue in iOS/VO too. generally, over the last few months, i've actually started advising clients NOT to implement certain ARIA patterns as they fall apart on mobile.

[ZoeBijl]

• @MichielBijl to test all code examples with touch.

[mfairchild365]

From @hshepherd on January 25, 2016 21:2

It seems like interactions with role="slider" elements are still an issue in iOS Voiceover even when using the recommended aria attributes (aria-valuemin, aria-valuemax, aria-valuenow). Are there any plans to revisit this?

I just tried the slider examples on iOS/Safari/VO and they don't work. Are there any plans to revisit this?

[patrickhlauke]

there is no way to fix/revisit this - ARIA "faked" sliders rely on keyboard and mouse event listeners, but once VO encounters a slider (faked with role="slider") it expects it to behave like a regular native slider and respond directly to swipes up/down to change value. related, on Android, the native behavior when using TalkBack is to use volume keys up/down to change the value (see https://patrickhlauke.github.io/aria/presentation/?full#157). in both cases, JS can't "listen" for those events.

[mfairchild365]

Understood, thank you for the explanation. I will leave a link to a related WebKit bug https://bugs.webkit.org/show_bug.cgi?id=174898 in case anyone else is curious. It sounds like the proposed Accessibility Object Model might be able to fix this. I'll have to look into that further.

Are there any plans to provide a more explicit warning within APG that specific patterns/examples will not work on iOS? I think a common assumption for people referencing these patterns/examples is that they will work everywhere. Section 2.3 "Mobile and Touch Support" does exist, but it is easy to miss as a separate section.

[patrickhlauke]

Are there any plans to provide a more explicit warning within APG that specific patterns/examples will not work on iOS

reminder: not just iOS - any situation where AT+touchscreen is used, in general.

and yes, the aim of AOM events is to fill this gap. but we're a long way away from seeing them, even in initial stages of support.

[accdc]

It is always a good idea when implementing ARIA drag and drop sliders to add Decrement and Increment buttons for touch device support, even if implemented as ARIA mapped solutions specifically for screen reader users, since the underlying mappings for iOS will never be able to determine JS drag and drop interactions and how these relate to native swipe interactions in the future.

You can see an example of this here:
http://whatsock.com/tsg/Coding%20Arena/ARIA%20Sliders/Horizontal/demo.htm

[mraccess77]

Sliders may also have issues meeting SC 2.5.1 Point Gestures under WCAG 2.1 without buttons to increase or decrease the value -- there is an active discussion on this on the WCAG 2.1 github issues page.

[mfairchild365]

Link to the WCAG issue for those that are curious: w3c/wcag#522

[julesongithub]

From @hshepherd on January 25, 2016 21:2

It seems like interactions with role="slider" elements are still an issue in iOS Voiceover even when using the recommended aria attributes (aria-valuemin, aria-valuemax, aria-valuenow). Are there any plans to revisit this?

I've found the only solution is to employ an actual range control, that's hidden, with a fake slider overlaid.
I first tried this by putting the range control off-screen but, for whatever reason, that didn't work with VoiceOver.
A great cross-browser solution is to hide the range control by making all the elements transparent with input::-webkit-slider-thumb {} etc. You then overlay the fake slider, set pointer-events: none, and then listen for changes and set the fake thumb position by percentage. Just make sure to listen to 'input' and 'change' since iOS+VO only fires the 'change' event.

This even works in IE11.

[patrickhlauke]

had a stab at the "real range input with faked slider on top" approach https://codepen.io/patrickhlauke/pen/byWPMX

(noting that in that case, the whole ARIA stuff on the slider is unnecessary, since it's not what users will interact with)

is it worth adding an explict note to the slider pattern itself https://www.w3.org/TR/wai-aria-practices-1.1/#slider ? rather than just relying on the very high-level generic "mobile and touch support" bit https://www.w3.org/TR/wai-aria-practices-1.1/#mobile_and_touch_support ? /cc @scottaohara

[scottaohara]

I do think that the examples should better call out gaps, or implications for mobile (re: #420).

Per Patrick's comment,

rather than just relying on the very high-level generic "mobile and touch support" bit

Anecdotally speaking, i find that section 2 at large is ignored. Developers come here looking for the pattern they need, jump to the example(s) to compare against or replicate, and then get frustrated when issues (like some pattern's lack of mobile support) are pointed out.

I know that future work will address these mobile issues in more detail, but in the meantime, as this particular issue thread is nearly 2 years old, it's likely worth adding an editorial stop gap to better call attention to sections 2.2 and 2.3.

2.2 already has a link at the top of the examples... giving that more prominence and mentioning mobile in the link text would even be an improvement, imo.

[alastc]

+1 for adding a warning to the examples / patterns, I just did this:

Developers come here looking for the pattern they need, jump to the example(s) to compare against or replicate, and then get frustrated when issues (like some pattern's lack of mobile support) are pointed out.

More broadly, @patrickhlauke mentioned AOM will enable scripting to detect gestures and support ARIA patterns. However, in the meantime could the AT (like VO/TB) do a little reverse feature detection and plug the gap?

For example, if the user lands on a slider thumb which has keyboard events but no AOM slider events, then trigger the keyboard events. Swipe up = arrow up etc.

[ZoeBijl]

Would appreciate some help with the wording:

Warning: this example does not work with touch based assistive technologies such as VoiceOver or TalkBack. The required events aren’t yet supported. Please use a native input with its type set to range.

Example styling:

There are plans to add support tables to the examples. But I think this needs some attention before we reach that point. This issue has been open for four years…

There’s an open pull request to add a warning similar to this to the slider example pages.

---

Can we, together, compile a list of examples that don’t work on touch devices?

Have issues

• Slider, all examples: the events needed to make this work for touch based AT aren’t available.
• Slider (multi-thumb), same as slider.
• Grid
• Menu
• Tree View
• Treegrid

Need to look at

• Combobox
• Menu Button
• Radiobutton, will need to check support when it’s in a toolbar.
• Window Splitter

Not optimised but should work

• Accordion, has optional up|down arrow keys (can those be emulated in touch)
• Alert and Message Dialogs (same as Dialog)
• Button
• Carousel, don’t see any keyboard interactions that look problematic. But would like to test
• Dialog, not sure if this is an issue but there’s no Esc key in touch(?). Should that be noted? (two finger scrub in voiceover)
• Feed
• Link, is there a way to trigger the context menu (Shift + F10) on touch devices?
• Listbox
• Spinbutton
• Tabs, is there a way to trigger the context menu (Shift + F10) on touch devices?
• Toolbar, is there a way to trigger Home and End?
• Tooltip, is there a way to trigger Esc?

No issues

• Alert (no keyboard interaction)
• Breadcrumb (no keyboard interaction)
• Checkbox
• Disclosure (Show/Hide)
• Table

[patrickhlauke]

lately coming back to this (and the list of patterns the @ZoeBijl has in the comment above, but that I can't actually tick) to say that basically, every time a pattern expects/relies on listening to a particular character/key (rather than just relying on a humble click event), it will have issues in touch+AT scenarios. in many cases, there will also be similar issues for mouse-only users (e.g. data grids, example 3: scrollable data grid ... unless there are additional controls to scroll, mouse/touch/touch+AT users won't be able to trigger the PgUp/PgDown functionality.

From some brief testing, a few observations of the current state of some of the patterns with touch+AT:

• combobox: cumbersome to use; interacting with the text input opens the on-screen keyboard; suggestions that appear are currently not announced; only way to interact with suggestions is to close the on-screen keyboard, and then navigate forward with VO to reach the list (which the user may not even know is there).
• dialog: works, but can't rely on just having Esc to close. needs an explicit close button. same for mouse users.
• grid: works for the most part, though some of the examples are a bit rough to operate (e.g. for data grids, Example 2 allows you to edit the editable cell, but once "done" with editing it remains a text input field; Example 3: Scrollable Data Grid With Column Hiding doesn't work, as it's not possible to trigger the PgUp/PgDown scrolling); anything involving special things like selecting rows/columns/cells will be difficult to get to work. same for mouse users.
• link: i don't think there's a way to trigger contextmenu events (which is really what any script should be listening to for supporting the context menu stuff) even without AT running iOS at least (it may be possible with a long-press or similar on Android, Windows desktop/laptop with touchscreen, though); certainly no way to trigger explicitly Shift+F10 though.
• listbox: collapsible dropdown listbox example doesn't close once user selects an option from the dropdown, user can't activate Esc to close it
• menu: anything reliant on cursor keys won't work (e.g. opening a submenu). must listen for actual click event only
• spinbutton: works insofar as the user can focus and activate the various up/down arrow buttons (it seems to ignore the tabindex="-1"). spinbutton element itself is not announced as anything specific/interactive though. see the toolbar case later on here for a spinbutton that doesn't work with touch+AT
• tabs: can't operate the Delete to remove a tab functionality, as user can't trigger that key. the alternative of using a context menu to offer this as functionality also won't work (see above)
• toolbar: works, but in the example, the spinbutton there for fontsize doesn't work for touch+AT users (it's just announced as "Font size in points, 14 points" with no hint of interactivity, and no way to trigger a change)
• tooltip: needs something other than Esc to dismiss the widget. also, by default, navigating with touch+AT doesn't set focus or fire mouse event either, so need to check if it can even be triggered in the first place
• tree view: navigates like a series of disclosure widgets; likely not easy/possible to have any multi-select scenarios though that work with touch+AT
• treegrid: can't currently expand/collapse any of the rows; it will get problematic too if a row/cell has functionality in addition to acting as an expand/collapse row - user will only be able to double-tap to activate the cell's functionality, not the expand/collapse functionality. needs at least one explicit cell with an explicit expand/collapse button, can't rely on cursor keys to expand/collapse.
• window splitter: can likely only toggle it to collapse, since it can't respond to cursor keys or similar. also can't trigger F6 to cycle window panes

So, while some patterns kind of work, others are quite fundamentally borked. particularly for the really broken ones, can I suggest that in addition to having the note (like there is now on the slider pattern) in the main document, examples that are broken also have the same warning right there in the example page itself?

[patrickhlauke]

x-ref #928

[a11ydoer]

@mcking65 @jongund

Can we update this issue or other connected ones since APG has a slider example which was tested with different mobile devices? I will also try to triage all the slider issues.

[a11ydoer]

@ZoeBijl
APG group updated the warning as below.

"WARNING! Some users of touch-based assistive technologies may experience difficulty utilizing widgets that implement this slider pattern because the gestures their assistive technology provides for operating sliders may not yet generate the necessary output. To change the slider value, touch-based assistive technologies need to respond to user gestures for incrementing and decrementing the value by synthesizing key events. This is a new convention that may not be fully implemented by some assistive technologies. Authors should fully test slider widgets using assistive technologies on devices where touch is a primary input mechanism before considering incorporation into production systems."

[patrickhlauke]

As mentioned above as part of the slider update, VoiceOver/iOS now fires fakes/synthetic keyboard events (as per the AOM User action events from Assistive Technology model) - see https://www.youtube.com/watch?v=x__Rf4foatU / https://codepen.io/patrickhlauke/pen/MMgBjZ

Needs more testing, but likely that VO/iOS now addresses a few other cases that before - due to their reliance on keyboard event handling - were failing miserably in touch+AT scenarios (though personally, rather than having AT fire fake keyboard events, I'd have rather liked to see the opposite, with browsers/user agents firing input-agnostic events regardless of whether they came from a real keyboard or from an AT gesture...but that idea - first floated as IndieUI Events many moons ago - just doesn't seem to get any traction; so we're now left with a weird "AT have to pretend to fire keyboard events so that ARIA Practices patterns don't break", which is a bit dirty...)

[ruckc]

I just want a role="swipe"... i have other buttons for non-swipe interfaces.... but when i do an on:click for a swipe action... and then get nagged about setting a aria role... I don't know what to do.