APG Redesign - Usability Study Report

[isaacdurazo]

Introduction

Over the week of September 13 of 2021, we conducted the first Usability Study for the Aria Authoring Practices Guideline (APG) website redesign project. The goal of this study was to determine the effectiveness of the new information architecture of the site in reducing the time and effort it takes users to find the resources they need as well as identify design inconsistencies and usability problems.

The majority of the participants of the study were successful at completing their tasks in a reasonable amount of time. It was clear that having a central navigation and the content split up sped up the process of finding resources. In addition, we identified several areas of improvement where we can simplify and continue improving the experience for users.

Participants

We recruited participants via https://web-a11y.slack.com and Twitter. Their profiles included people with the following roles: Accessibility Specialist, Front End Developer, Designer, UI/UX Engineer, QA Engineer, and Accessibility Tester. All of them with at least 2 years of experience in Accessibility and the majority with more than 6 years in the field. All but one participant were somewhat proficient at using a screen-reader and two of them declared to be very proficient.

Tasks and Scenarios

We designed tasks and scenarios for the study to cover and test different areas of the site. Participants were able to navigate through different resources on the Fundamentals page, explore different design patterns and dive into some of their examples.

For more details about the tasks and scenarios, please refer to the APG Redesign Usability Study Plan.

What went well

For the most part, participants were able to complete their tasks without much difficulty.

• People with the most experience in Accessibility and those who use APG often, found resources quickly, while those with less experience and familiarity with APG required a little bit more time.
• The new central navigation sped things up when finding resources and it received several positive comments.
• People unanimously agreed that the current content categories made sense.
• People liked the cards’ layout of the Patterns and Widgets and Fundamentals pages. Many mentioned how clean and easy it is to look at all these from a high-level view.
• Several people commented they loved the usage of illustrations for each pattern, especially how those would become recognizable after visiting the page often.
• When searching for different resources under specific patterns, such as implementation techniques, keyboard support, best practices, and examples, the majority of participants had no major issues.
• A couple of participants appreciated the usage of a side navigation in pages like “Landmark Regions”, which gives them a clear outline of the content of the page.

Positive comments from participants

Easy and comfortable. Easy to navigate and to find examples. Not mentally heavy.

It’s really nice to have horizontal navigation instead of the old one.

With regard to the Patterns and Widgets page

As someone who goes there very often, I love that I’m gonna start recognizing these icons and don’t even have to read to select what I need.

This is lovely! This is actually super great!

I like that there are other ways to navigate to examples, either through the patterns page or using the Examples page.

I appreciate having a standalone section for Fundamentals. That’s gonna help a lot of people

It is a simple interface and it’s really great to be able to share urls more easily.

Areas of improvement

• When participants were asked to search for Keyboard support, several preferred searching for a pattern example and learn about keyboard support from there rather than looking under Fundamentals.
• When searching for examples, people took different approaches. Some went through the Examples page and the tables we have there, but it was difficult, others went through the Patterns and Widgets page.
• When asked about finding the best practices to implement a specific ARIA Widget, everyone was able to find the Widget, but a couple of participants didn’t notice the links to its examples.
• When asked to find some examples of Landmark roles, people would generally look under fundamentals last. A couple of participants looked under the Examples page without finding any. Only one participant was able to find an example of a landmark role from the Landmark Regions page.
• A couple of participants didn’t notice the cards under fundamentals had a clickable heading.
• When participants were asked to find information about the ARIA attributes of a particular Widget, it generally took them some time. Some went through the single pattern page of said widget or the example of that widget, and others went through the Examples page’s tables.

Feedback from participants

• Almost half of the participants considered we should re-think the order of the navigation elements and have Fundamentals come first. Right after Home.
• The home page is pretty content-heavy and could be re-worked, especially to make it friendlier. Consider adding information about when ARIA came into existence and why it did. It would be nice to promote other types of content as well, like a few relevant patterns.
• In some places, it would be nice to link to WCAG.
• For the Patterns and Widgets, and the Fundamentals pages consider cards to be clickable or have a “read more about x” link.
• Incorporate search functionality
• It would be nice to include a content outline or side navigation on lengthy pages that don’t have it
• Comparisons of how something looks with and without ARIA would be great.
• Add breadcrumbs when possible
• In the single Pattern page, have a tab panel for JS, CSS, and HTML

Next Steps

Now that we have concluded the first usability study and we have identified the successes and areas for improvement, we will

• Think about how we can approach each of the challenges usability test participants faced.
• Create a list of recommendations for each of the usability issues
• Document bugs and accessibility problems found during the study
• Estimate and prioritize the improvements we want to make