Create tests for APG example: Editable Combobox With Both List and Inline Autocomplete

[zcorpan]

APG design pattern: https://w3c.github.io/aria-practices/#combobox

• Test committed: https://w3c.github.io/aria-at/review/combobox-autocomplete-both.html
• Test reviewed:
  • Review of test description
  • JAWS commands
  • NVDA commands
  • VoiceOver commands
• Test results
  • JAWS/Chrome
  • JAWS/Firefox
  • NVDA/Chrome
  • NVDA/Firefox
  • Mac VoiceOver/Chrome
  • Mac VoiceOver/Safari

[mcking65]

Attaching a complete draft of tests in a spreadsheet. The tests in this spreadsheet replace all the tests that are already in wpt format.
Combobox-autocomplete-both.xlsx

[mcking65]

While there is one combobox design pattern, we currently have 4 examples with different features. We will soon have a 5th. We probably want a separate issue for each example. I'm going to change the title of this issue to match the name of the combobox example we are working on now for the production system design phase. We will be testing only one combobox example during the design phase.

[mfairchild365]

I see this at https://w3c.github.io/aria-at/review/combobox-autocomplete-both.html - is it ready for a review?

[mcking65]

@mfairchild365

I see this at https://w3c.github.io/aria-at/review/combobox-autocomplete-both.html - is it ready for a review?

Nop, not updated from the spreadsheet yet.

Related thought ...

@spectranaut, do we have a way of specifying the sequence of tests in the plan for a pattern? At least for me, when writing and reading test plans, and even when reading results, the sequence really matters. In my spreadsheet, I tried to follow a logical sequence of operations and states that could roughly map to a flow for using combobox.

[spectranaut]

I had to modify the spreadsheet you uploaded to work with Jon's script, here is the modified file:
tests.xlsx

Right now, we don't have a way to sequence tests. We should make a backlog test to consider it!

[spectranaut]

Here is the test plan for review: https://w3c.github.io/aria-at/review/combobox-autocomplete-both.html

Things that still need to be done for these tests:

• setup scripts written
• some key commands aren't found, i think this is a data format problem

[ghost]

Thanks spectranaut & Matt!

Made a high-level review for combobox, mainly from JAWS.

Review from test settings and consistency

I observed several concerns by comparing tests over multiple SR and the other widgets (menubar and checkbox). These are not specifically for combobox, but leave notes here for the record.

1-1.Test setting

• NVDA key used in Tester instruction is not clear, such as "Put NVDA into Focus Mode using NVDA+Space" > I think we should use expression such as Insert + Space

• Even though filtered the test radio button to VoiceOver , Test 1 shows the test applies to JAWS and NVDA.

• The "Mode" used in the first list may be confusing for tasks involve mode switching > Use the term "Initial mode" or "Default mode"?

1-2.Inconsistency

• "Applies to" information seems always 'Desktop Screen readers" in checkbox and menubar, while varies in combobox.

• The name of tests were "Test number + Test name + in reading/interaction mode" in checkbox and menubar, while the test name didn't contain " in reading/interaction mode" in combobox.

Review from individual tests

The majority of concerns came from missing data probably from data formatting to me. Aside from missing data, I observed 3 concerns..

2-1.Command concerns from Individual test review

• For the tasks to read, Insert + UP (Say from cursor) could also be an option for key comamnds? (Test 21 - 26)

2-2.Language concerns from Individual test review

• The scripted instruction reads "Set focus on 'States' button; Ensure combobox is empty and collapsed". But 'State' seems like a name/label to me, not button. Do we call 'button' even if it's not clickable? (Test 5)

• I think we need Scripted Instruction to differentiate 10 & 11 (filled & empty), because mode and Tester Instructions of Test 10 and 11 are the same. (Test 10- 11)

2-3.Data format concerns from Individual test review

Test No.	Detail
Test 1	No key commands are available.
Test 2	No assertions are available.
Test 4	No instructions, commands, assertions available.
Test 9	No key commands are available.
Test 14 - 17	First test instruction is empty/No key commands are available.
Test 19	First test instruction is empty/No key commands are available.
Test 20	Priority for second and third assertions were missing.
Test 29	First test instruction is empty/No key commands are available.

As Michael mentioned in his post for menubar](#54 (comment)), having common matrix or grid would be preferable to ensure the quality in the long run.

[mcking65]

@Yohta89 wrote:

• NVDA key used in Tester instruction is not clear, such as "Put NVDA into Focus Mode using NVDA+Space" > I think we should use expression such as Insert + Space

Was wondering about this. I am fine with saying insert+space. Testers will be trained and can make their own translation as needed, e.g., use capslock if they have that enabled.

We will need to specify which defaults are OK to change, e.g., voice rate, keyboard, etc. But, it is acceptable that we provide only desktop layout key assignments in the instructions, since desktop is the default.

Kind of funny that we have that legacy hang over ... most people have laptops but desktop is the default.

• The "Mode" used in the first list may be confusing for tasks involve mode switching > Use the term "Initial mode" or "Default mode"?

I don't understand what you are talking about here.

• "Applies to" information seems always 'Desktop Screen readers" in checkbox and menubar, while varies in combobox.

Correct, and we will need to update checkbox and menubar accordingly. This has been done for checkbox now, I think. Each test needs the correct applies to value.

• The name of tests were "Test number + Test name + in reading/interaction mode" in checkbox and menubar, while the test name didn't contain " in reading/interaction mode" in combobox.

Seems redundant to specify it in two places. I wonder if this part of the name should be auto-generated.

• For the tasks to read, Insert + UP (Say from cursor) could also be an option for key comamnds? (Test 21 - 26)

No, insert+up does not apply in text inputs; it only reads the current line, not the current control.

• The scripted instruction reads "Set focus on 'States' button; Ensure combobox is empty and collapsed". But 'State' seems like a name/label to me, not button. Do we call 'button' even if it's not clickable? (Test 5)

Sorry, this assumes a later version of the combobox example that has not been merged yet. In that version, the "open" button is named "States" and it has aria-expanded on it.

[ghost]

Thanks for the response Matt!

With regard to the following bullet point, I meant to address the lists of bullets points under the test title, such as "Mode: reading" or "Mode: interaction". Currently, this information shows the mode when testers started the task.
In the case of the mode-switching task, however, a mode can be switched from one mode to the other thorough a task. So I was wondering if we want to use the terminology such as "Initial mode" or "Default mode" instead of saying just "Mode", so that it's more self-describing.

• The "Mode" used in the first list may be confusing for tasks involve mode switching > Use the term "Initial mode" or "Default mode"?

[mcking65]

@Yohta89

With regard to the following bullet point, I meant to address the lists of bullets points under the test title, such as "Mode: reading" or "Mode: interaction". Currently, this information shows the mode when testers started the task.
In the case of the mode-switching task, however, a mode can be switched from one mode to the other thorough a task. So I was wondering if we want to use the terminology such as "Initial mode" or "Default mode" instead of saying just "Mode", so that it's more self-describing.

Oh, I understand your point now.

Since the test and assertions describe when a mode is expected to switch and how it is expected to switch, I don't think the extra words would necessarily add clarity.

There are very few tests where mode is expected to switch. In general, mode switching only applies to JAWS and NVDA, and only to a limited set of widgets. These are probably fewer than 0.5% of all desktop screen reader tests, and an even lower percentage of screen reader tests when we expand to cover mobile.

[mcking65]

@spectranaut, Attaching updated tests.xls with changes we discussed yesterday + a couple more tests.
tests.xlsx

[spectranaut]

I'm going to try not to be involved in test writing unless it's really necessary (like when Jon was out last week!) and I haven't gotten to re-creating the tests.

@jongund, I just copied over your scripts in order to create these tests the first time. The csv files in the data/ directory are old, you will have to re-export the appropriate CSV files from the spreadsheet Matt uploaded in the previous comment. It will be easier to re-run these tests after the changes I outlined in this issue

Also, the setup scripts haven't been written yet for these tests!

[mfairchild365]

I had a chance to review some of the tests. I didn't have enough time to go through every test in the plan. Overall, this is a great set of tests! The notes/questions that I have are mostly about formatting and consistency than anything else.

Notes:

• Test names
  • Should we use the present tense for test tiles ("Navigate to" instead of "Navigating to") (other plans do this)?
  • should we include the mode in test tiles? Otherwise, we will see duplicate test titles in listings, and it will be hard to differentiate tests without looking at the review or starting a test run. This can be done either manually or automatically.
  • These test names appear to include assertion-like language ("conveys combobox role; name; editability; value; and state"). Test names in other plans are centered on the task rather than the expectations. Should we do this same here?
  • The words 'popup' and 'popup item' seem to be used in place of 'listbox' and 'option' roles. In some test names, the word 'option' is used instead of 'popup item'. Is there a reason for this, or would it be better to refer to the ARIA roles?
• Setup scripts
  • test 'Navigating by line to filled; editable; expanded combobox conveys role; editability; value; and state' expects the target to be filled but is missing a setup script
• Re assertion 'Caret position is conveyed' - are you expecting numeric information about the position of the cursor, or would the character that the caret just moved in front of being announced pass? (I noticed that JAWS always announces 'blank' in the APG example)
• Carrot instructions for JAWS include MAC commands. Is that expected?
• ARIA roles, states, and properties
  • I didn't see any assertions for aria-controls - is that expected?
  • I didn't see any assertions related to aria-activedescendant - is that expected?

[jscholes]

PAC are working on updates to this test plan to increase its coverage and consistency. As such, and given that a lot of information in this thread is now out-of-date, I'm closing this issue in favour of #340.

[jscholes]

Updates from the April 1, 2021 CG meeting: we should remove the files related to the older iteration of the test plan, and then add a prefix to the title of this issue to indicate that it is now "deprecated". In this case, deleting the files and entry in support.json should suffice, because no data was gathered from testers that we want to keep (or at all).

[jscholes]

I've submitted PR #417 to remove the old files. The deprecated prefix can be added to this issue once it has been merged.

[jscholes]

Note: deprecated prefix should include the date.