Conflicting VoiceOver results for test 5 "Open a menu to the last item" (Navigation Menu Button, V24.10.12)

[IsaDC]

Description of Behavior

This behavior can also be observed in Test 7: Navigate to the first item in a menu, Test 8: Navigate to the last item in a menu, and Test 9: Navigate to an item in a menu by typing a character.

Please note that, in this case, the role "menu item" appears as part of a VoiceOver hint, which should not be tested.

Test Setup

• Test File: test-14-openMenuToLastItem-voiceover_macos.collected.html
• AT: VoiceOver for macOS (Version 11.6 and above)
• Browser: Safari

Review Conflicts for "Open a menu to the last item"

1. Assertion Results for "Up Arrow (quick nav off)" Command and "Role of the focused item, 'menu item', is conveyed" Assertion

• Tester IsaDC recorded output "
  menu 6 items Accessible Name and Description
  You are currently on a menu item. To choose this menu item, press Control-Option-Space. To close this menu, press Escape." and marked assertion as failing.
• Tester tactics2 recorded output "menu, 6 items, accessible name and description, you are currently on a menu item" and marked assertion as passing.

[mcking65]

The question raised by this issue:

Is the VoiceOver response sufficient for conveying role menuitem given that the role is conveyed in a hint?

[css-meeting-bot]

The ARIA-AT Community Group just discussed Testing navigation menu button.

The full IRC log of that discussion

<jugglinmike> Topic: Testing navigation menu button
<jugglinmike> github: https://github.com//issues/1137
<jugglinmike> Matt_King: Is it okay if the role is conveyed in the hint?
<jugglinmike> Matt_King: I think it would be very difficult to tell Apple that the role isn't conveyed at all
<jugglinmike> James: To be honest, I would find a menu infuriating if it said "menu" all the time
<jugglinmike> IsaDC: It's just a "may" assertion
<jugglinmike> James: I'm glad for that!
<jugglinmike> James: If we were talking about a button, and VoiceOver just said the name and only said the word "button" in the hint, I think we'd all feel a bit different
<jugglinmike> IsaDC: But in this case, you know that you're in a menu and that you're moving through items
<jugglinmike> IsaDC: Do we pass it now? Because we failed it elsewhere when it was in a hint
<jugglinmike> IsaDC: Recognizing that it's a "may" assertion
<jugglinmike> Matt_King: If it was a "must" and it was in a hint, then there's a problem when the hint goes away
<jugglinmike> Matt_King: But they're on by default
<jugglinmike> James: If you have to wait for the hint to tell you something, then you'd never get anything done
<jugglinmike> IsaDC: For consistency, if we pass it here, we have to revert that other test and pass it there, as well
<jugglinmike> James: So in the activedescendent plan, the text is present in the hint, and we failed it. You're saying that if we passed it in this one, it wouldn't feel good because we'd have to go back to the other and change it (even though it's already been removed from the test queue)
<jugglinmike> Dean: We shouldn't do it the wrong way now just because we did it the wrong way before
<jugglinmike> Matt_King: Let's decide what's correct right now and then do whatever work that entails
<jugglinmike> James: I'm going on three things. The first is Apple's assertion of how many people turn off hints. Secondly, the fact that it's a MAY assertion, so it does not reduce the support level (in order to notice that this is even a fail, someone would have to look very closely at the report--it's not a headline reduction in the support level).
<jugglinmike> James: I forget the third thing
<jugglinmike> s/I forget the third thing/Third, if we were saying that this is an important assertion and then it failed, we would feel much more strongly that this was an issue/
<jugglinmike> Matt_King: The fact that it appears in the hint, though. The words are there. There's some cognitive dissonance there. We just need consistency
<jugglinmike> Matt_King: When we record the output, we definitely need to record all the output no matter what. If we are going to say that hints cannot convey role or state or name or value (which we could), then there's questions about properties
<jugglinmike> Matt_King: I'm not sure whether hints should be allowed to convey certain properties
<jugglinmike> Jame: I think hints are hints and should not be required to use anything
<jugglinmike> s/Jame:/James:/
<jugglinmike> q
<jugglinmike> Matt_King: The thing for us to write down someplace, maybe the testing manual or in the glossary, is that hints are part of the output, but we won't process information in the hints to determine whether or not role, name, state, value, or properties are conveyed
<jugglinmike> James: I think that's fair. I disagree with Apple on many things. I fully agree on their stance, here, though.
<jugglinmike> James: They recognize that many people opt out of hints, and that among the 10-15% of users who have them enabled, that many of those people are developers or accessibility testers
<jugglinmike> James: And so that's why Apple views them as slightly redundant
<jugglinmike> Dean: I think we need to make a solid decision here. Either hint text or no hint text. If it's "no hint text", then turn off the feature and don't include it in the output
<jugglinmike> Dean: If Apple is saying that a low percentage of people use it, then we're creating problems by observing it
<jugglinmike> James: If we say that hints don't contribute to results but we include them in the AT responses, then we open ourselves to the kind of feedback that isn't helpful
<jugglinmike> James: With JAWS, hints are spoken immediately. With VoiceOver, there is a significant pause
<jugglinmike> Matt_King: Going back on our principle of testing out of the box...
<jugglinmike> Dean: In that case, we should do the opposite of what I described earlier
<jugglinmike> Matt_King: I'm trying to have my cake and eat it, too!
<jugglinmike> Matt_King: VoiceOver and JAWS gets hints wrong sometimes, and I would like to be able to call that out
<jugglinmike> Dean: It doesn't make sense to say that "We use out-of-the-box settings" when 80% of users don't use out-of-the-box settings
<jugglinmike> Dean: If 15% of people hear additional information, that's fine, but we need to decide what's mandatory in the output that everyone is hearing
<jugglinmike> Matt_King: The problem is that we don't have those numbers. Vispero tells us that beyond changing the speech rate, most users use out-of-the-box settings. Apple is telling us that most people turn off hints. I believe that's true on iOS
<jugglinmike> Matt_King: I would say that 100% of developers have the hints on.
<jugglinmike> Dean: If they're getting their 10-15% based on developers, then that's not relevant
<jugglinmike> Dean: My vote is that everyone turns off hints when testing with VoiceOver
<jugglinmike> Matt_King: We don't want different rules for different screen readers
<jugglinmike> James: I felt that James Craig was quite firm in his assertion that hints should be discarded
<jugglinmike> James: I know many advanced users that make many customizations yet they still have the hints turned on
<jugglinmike> Matt_King: The first thing I do with VoiceOver is turn down the verbosity
<jugglinmike> Matt_King: I'm feeling stuck between what Dean is saying and things that we've agreed to earlier. This concept of testing out-of-the-box is pretty important
<jugglinmike> James: If we write a test plan for aria-description, and a VoiceOver user navigates to the control that has aria-description, and VoiceOver says "more content available" without reading more. Would you, at that point, say the failure has occurred, or would you say the user should open the item and find the description, and assign a verdict based on what the screen reader says then
<jugglinmike> Matt_King: We haven't discussed how to interpret responses in that case
<jugglinmike> IsaDC: What should we do?
<jugglinmike> Matt_King: Dean believes it's unacceptable to include hints in the output but then ignore that information when assigning verdicts
<jugglinmike> Dean: That will complicate things far too much in my opinion
<jugglinmike> Matt_King: I guess we need to review the pros and cons. Maybe this is a slightly more complicated decision than we would like it to be
<jugglinmike> James: The question is, are we going to be able to find more information that makes this decision easier?
<jugglinmike> James: The only things I can think is: speaking to more community group members and speaking to Apple
<jugglinmike> Matt_King: I would like to create a table of pros and cons and presenting that to others. I think getting alignment on that information is important
<jugglinmike> Matt_King: Even though this is an optional assertion right here, it's going to come up in a "must" or "should" at some point. That's why I don't want to decide based on the fact that the issue right now is for a "may" assertion
<jugglinmike> Matt_King: I don't think we need to make a decision based on the suggestion of one screen reader developer. I want some consensus
<jugglinmike> Matt_King: I'll try to help us map out a decision path
<jugglinmike> IsaDC: So we won't move anything right now
<jugglinmike> Zakim, end the meeting

[mcking65]

Depends on #365

[mcking65]

@IsaDC

Can this be closed now?

[IsaDC]

Solved by @mcking65 comments on #365