Add JavaScript functional test harness

[biancadanforth]

As discussed here, there is no test coverage for functions exported by Fathom's JavaScript library that require rendering a page in a browser.

Given that we've already had issues with some of these error-prone functions (like isVisible, see #118 ), it would be wise to set up some automated testing for them in our CI.

As Erik mentioned, we may be able to achieve this using Firefox's Headless mode.

As part of closing this issue, a test for isVisible should be written.

If we prioritize this issue, I am very interested in working on it. For now, I will leave it unassigned.

[biancadanforth]

I spoke with jlast who is on the DevTools team and in part works on their CI.

For a web development test harness (no compatibility with landing code in Firefox), he recommends Cypress, which is Electron based. DevTools used this themselves until they switched to Mochitest (Firefox only).

There is ongoing work to make significant improvements to Puppeteer for Firefox, which he thinks will be complete around December 2019. At that point, he would recommend Puppeteer.

[erikrose]

I watched Cypress's intro movie, and my takeaways (mostly for my own reference) were…

1. It looks amazing UI-wise for building tests of complicated, multi-step interactions with JS apps. Our use case, however, is going to be dominated by unit-like tests of utilsForFrontEnd.js helpers for the forseeable future.
2. They're pushing a subscription-based payment plan for their own Travis-like web-based frontend. It's not immediately obvious how to use it on our existing infra, but I don't know that it's impossible.
3. It's monolithic, including (and requiring?) its own assertion lib and test runner.

[erikrose]

A couple more links to follow (sorry, I got interested and couldn't help it):

• https://docs.travis-ci.com/user/gui-and-headless-browsers/
• https://developer.mozilla.org/en-US/docs/Mozilla/Firefox/Headless_mode (has some webdriver stuff to evaluate)

[biancadanforth]

2. They're pushing a subscription-based payment plan for their own Travis-like web-based frontend. It's not immediately obvious how to use it on our existing infra, but I don't know that it's impossible.

It looks like they have a free plan (limit: 3 users, 500 test recordings) an OSS plan (limit: 5 users, unlimited test recordings), but one of the requirements for the OSS plan is that the team or their organization is a non-commercial entity; that sounds like it would disqualify us.

A couple more links to follow (sorry, I got interested and couldn't help it):

• https://docs.travis-ci.com/user/gui-and-headless-browsers/
• https://developer.mozilla.org/en-US/docs/Mozilla/Firefox/Headless_mode (has some webdriver stuff to evaluate)

I was reading the TravisCI stuff, and they don't support Docker on macOS, and all of the instructions either require having your own web server to serve the pages or making an account (possibly free?) through SauceLabs. I have used Taskcluster before (Mozilla's CI tool), so another option is that we could use Taskcluster and then have access to their docker workers.

That second link has some good guidance on how we might use Selenium Webdriver. It has a good article on doing so using NodeJS. Perhaps that's a good first route to go?

In general, do we want cross-browser testing? Perhaps that can be a follow-up issue?

[erikrose]

I was reading the TravisCI stuff, and they don't support Docker on macOS,

I think what they mean is that they don't support Docker on their virtualized Mac CI boxes.

[biancadanforth]

I think what they mean is that they don't support Docker on their virtualized Mac CI boxes.

D'oh. Yeah. That makes sense. TravisCI runs in a VM.

Since part of this issue is to add a test for isVisible, here are test cases that I manually checked in PR #116 which should return false:

• display: none
• Ancestor is display: none
• Visibility: hidden
• Ancestor is visibility: hidden
• Opacity: 0
• Ancestor opacity: 0
• getBoundingClientRect().width: 0 && overflow:hidden
• getBoundingClientRect().height: 0 && overflow: hidden
• Ancestor getBoundingClientRect().width(): 0 and itself overflow: hidden
• Ancestor getBoundingClientRect().height(): 0 and itself overflow: hidden
• Element is off-screen above page
• Element is off-screen below page
• Element is off-screen to the left of the page
• Element is off-screen to the right of the page

[pdehaan]

Ah cool. I've used Cypress and Puppeteer [locally, not in any CI] in the past for various website testings and screenshot generation and they're both pretty neat; although I think both are still mostly Chrome based (although I understand Puppeteer has an experimental puppeteer-firefox project[1]). Cypress.io still has cross-browser on their roadmap: cypress-io/cypress#310

I also watched the "Complete Code Coverage with Cypress" webcast the other week and it was very interesting [if you have ~60 minutes].

[1] https://aslushnikov.github.io/ispuppeteerfirefoxready/

[biancadanforth]

Thanks @pdehaan for the information. Cypress looks really polished, but as Erik mentioned, they are monolithic (we'd have to throw out our current CI, Travis).

What's more, I was able to quickly get a working solution for Travis on Monday. It spins up the latest release version of Firefox in headless mode and tests isVisible on MDN with Selenium Webdriver using Mocha.

What's left to do there is for PR #116 to merge, so that we have the right isVisible to add a test for, add our own custom test page and update the test cases accordingly.