Infrastructure: Fix failing `npm run regression-report` script #2291

howard-e · 2022-04-13T20:50:09Z

Address comments in #959.

howard-e · 2023-11-01T14:04:03Z

@mcking65 this has been updated following #2837.

The result of npm run regression-report now gives me:

#### Regression test coverage:


#### Examples without any regression tests:

None found.


#### Examples missing some regression tests:

- dialog-modal/examples/datepicker-dialog.html:
   - textbox-aria-describedby
- listbox/examples/listbox-collapsible.html:
   - key-tab
- listbox/examples/listbox-grouped.html:
   - key-tab
   - key-character
- listbox/examples/listbox-rearrangeable.html:
   - key-tab
   - key-character
- listbox/examples/listbox-scrollable.html:
   - key-tab
   - key-character
- menu-button/examples/menu-button-actions-active-descendant.html:
   - menu-up-arrow
   - menu-down-arrow
   - menu-home
   - menu-character
- menu-button/examples/menu-button-links.html:
   - menu-enter
- menubar/examples/menubar-navigation.html:
   - menubar-menuitem-aria-current
- slider/examples/slider-color-viewer.html:
   - group-aria-labelledby
- slider/examples/slider-temperature.html:
   - aria-valuetext
- switch/examples/switch.html:
   - key-tab
- toolbar/examples/toolbar.html:
   - toolbar-spinbutton-role


#### Example pages with Keyboard or Attribute table rows that do not have data-test-ids:

- alertdialog/examples/alertdialog.html
   - "Keyboard Support" table(s):
      - Shift + Tab
      - Command + S
   - "Attributes" table(s):
      - alert
      - aria-disabled="true"

#### SUMMARY:

  58 example pages found.
  0 example pages have no regression tests.
  12 example pages are missing approximately 18 out of approximately 870 tests.

ERROR - missing tests:

  Please write missing tests for this report to pass.

a11ydoer · 2023-11-02T18:29:21Z

@howard-e it may be the separate issue but I have error of missing script:"regression-report" when I did npm run regression-report. When I check the available list of script using "npm run", fmt, lint, and boostrap were only available. I am mentioning @spectranaut in case she can help.

howard-e · 2023-11-02T19:05:41Z

@a11ydoer when I do npm run, I get the following:

npm run result

❯ npm run
Lifecycle scripts included in [email protected]:
  test
    npm run lint && npm run regression

available via `npm run-script`:
  fix
    npm run lint:es -- --fix && stylelint --fix  "**/*.css" && prettier --write "**/*.css"
  htmlhint
    htmlhint "**/*.html" "**/*.template" --ignore "common/**/*.html" --format unix
  lint
    npm run lint:es && npm run lint:css && npm run lint:html && npm run lint:spelling
  lint:css
    stylelint "**/*.css" && prettier --check "**/*.css"
  lint:es
    eslint . --report-unused-disable-directives
  lint:js
    npm run lint:es
  lint:html
    npm run htmlhint && npm run vnu-jar
  lint:spelling
    cspell "**/*.*"
  link-checker
    node ./scripts/link-checker.js
  reference-tables
    node scripts/reference-tables.js
  regression
    ava --timeout=1m
  regression-report
    node test/util/report
  vnu-jar
    java -jar node_modules/vnu-jar/build/dist/vnu.jar --filterfile .vnurc --no-langdetect --skip-non-html content/
  create-gh-project
    node ./scripts/create-gh-project.js
  prepare
    husky install

I'm not familiar with the fmt and bootstrap commands in this repo. Is it possible you may have ran that command in a separate repo's project?

a11ydoer · 2023-11-02T19:17:20Z

@howard-e sorry. you are right. this is downside of multi-tasking, not focusing on one thing. I was using old repo not updated one.

a11ydoer

"npm run regression-report" is working due to directory path update.

a11ydoer · 2023-11-02T18:23:28Z

test/util/report.js

@@ -7,7 +7,7 @@ const fs = require('fs');
 const htmlparser2 = require('htmlparser2');
 const spawn = require('cross-spawn');

-const examplePath = path.resolve(__dirname, '..', '..', 'examples');
+const examplePath = path.resolve(__dirname, '..', '..', 'content', 'patterns');


@howard-e yes, updated path info was what I needed to run regression test. Thanks. I will also check other changes.

@howard-e I confirmed that the path update here works. Thanks again!

mcking65 · 2023-12-11T03:56:43Z

@howard-e

I am getting a runtime error as follows.

npm run regression-report

[email protected] regression-report
node test/util/report
C:\Users\mck\git\APGRepo\main\test\util\report.js:208
    exampleCoverage[example].missingTests.delete(matchResults[2]);
                             ^
TypeError: Cannot read properties of undefined (reading 'missingTests')
    at getRegressionTestCoverage (C:\Users\mck\git\APGRepo\main\test\util\report.js:208:30)
    at Object.<anonymous> (C:\Users\mck\git\APGRepo\main\test\util\report.js:217:1)
    at Module._compile (node:internal/modules/cjs/loader:1159:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1213:10)
    at Module.load (node:internal/modules/cjs/loader:1037:32)
    at Module._load (node:internal/modules/cjs/loader:878:12)
    at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:81:12)
    at node:internal/main/run_main_module:23:47
Node.js v18.12.0

howard-e · 2023-12-11T15:34:32Z

@howard-e

I am getting a runtime error as follows.

@mcking65 Oops! Some cross-platform support (for Windows) was missed there. 0423ad7 should address this.

mcking65

Thank you @howard-e! This is now working.

howard-e added the regression-testing Related to AVA regression tests of example pages or AVA framework implementation within repo label Apr 13, 2022

howard-e requested a review from mcking65 April 13, 2022 20:50

howard-e force-pushed the bocoup/fix-issue-959 branch from b27c712 to dd13ac1 Compare April 13, 2022 20:59

howard-e mentioned this pull request Jul 7, 2022

Infrastructure: Bump htmlparser2 from 7.2.0 to 8.0.1 #2299

Merged

howard-e force-pushed the bocoup/fix-issue-959 branch from dd13ac1 to 373ef16 Compare July 7, 2022 14:29

mcking65 mentioned this pull request Oct 15, 2023

Update regression test report to work with new repo structure #2837

Closed

howard-e force-pushed the bocoup/fix-issue-959 branch from 373ef16 to 6c119fc Compare November 1, 2023 13:49

howard-e added the Infrastructure Related to maintaining task force and repo operations, processes, systems, documentation label Nov 1, 2023

This was linked to issues Nov 1, 2023

Update regression test report to work with new repo structure #2837

Closed

Regression test coverage report fials when run locally #959

Closed

howard-e requested review from mcking65 and removed request for mcking65 November 1, 2023 16:01

howard-e added 7 commits November 2, 2023 09:31

proposed fix for running regression-report on windows

be99936

replaced spawnSync with cross-spawn's '.sync'

f5109fa

replaced spawnSync with cross-spawn's '.sync'

84562f1

Account for carriage return from Windows when ignoring files

b4c1d08

Support for new content/patterns structure

cfbddcc

Update cspell.json

b701f22

Fix package-lock.json after rebase

6a83aac

howard-e force-pushed the bocoup/fix-issue-959 branch from e2789ed to 6a83aac Compare November 2, 2023 13:33

howard-e requested a review from a11ydoer November 2, 2023 13:55

a11ydoer approved these changes Nov 3, 2023

View reviewed changes

Additional path support for Windows

0423ad7

mcking65 approved these changes Dec 12, 2023

View reviewed changes

mcking65 merged commit aa6307a into main Dec 12, 2023
20 checks passed

mcking65 deleted the bocoup/fix-issue-959 branch December 12, 2023 22:25

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Infrastructure: Fix failing `npm run regression-report` script #2291

Infrastructure: Fix failing `npm run regression-report` script #2291

howard-e commented Apr 13, 2022 •

edited by michael-n-cooper-bot

Loading

howard-e commented Nov 1, 2023

a11ydoer commented Nov 2, 2023

howard-e commented Nov 2, 2023

a11ydoer commented Nov 2, 2023 •

edited

Loading

a11ydoer left a comment

a11ydoer Nov 2, 2023

a11ydoer Nov 3, 2023

mcking65 commented Dec 11, 2023

howard-e commented Dec 11, 2023

mcking65 left a comment

Infrastructure: Fix failing npm run regression-report script #2291

Infrastructure: Fix failing npm run regression-report script #2291

Conversation

howard-e commented Apr 13, 2022 • edited by michael-n-cooper-bot Loading

howard-e commented Nov 1, 2023

a11ydoer commented Nov 2, 2023

howard-e commented Nov 2, 2023

a11ydoer commented Nov 2, 2023 • edited Loading

a11ydoer left a comment

Choose a reason for hiding this comment

a11ydoer Nov 2, 2023

Choose a reason for hiding this comment

a11ydoer Nov 3, 2023

Choose a reason for hiding this comment

mcking65 commented Dec 11, 2023

howard-e commented Dec 11, 2023

mcking65 left a comment

Choose a reason for hiding this comment

Infrastructure: Fix failing `npm run regression-report` script #2291

Infrastructure: Fix failing `npm run regression-report` script #2291

howard-e commented Apr 13, 2022 •

edited by michael-n-cooper-bot

Loading

a11ydoer commented Nov 2, 2023 •

edited

Loading