End to end tests?

[crhistianramirez]

The pull request template defines various checks to complete to ensure you didn't break functionality. I was wondering if there was any interest in adding automated testing to run through those scenarios for us? I've used testcafe in the past and would be happy to add them myself.

[jasonthibeault]

I think that would be an extremely welcome and good improvement to make -- we've a few times now had bugs pop up in console, and data go missing when there's garbage in the source spreadsheet, so anything that can edge-case-test automatically would be great

[caseyhelbling]

I'm 100% here for this. I think we should quickly discuss what testing framework to use and would like a few other friends to weigh in. I don't have a preference. Also necessary to get it hooked into Netlify (assuming netlify can build and run tests like jenkins or other CI platforms.)

[jdalt]

I'm also in favor of this. We're able to crowd source things fairly well, so I would definitely start light.

Just knowing that clicking on an item in the location bar goes to the thing on the map would be awesome and probably weed out some dumb javascript errors.

I wonder if we'd want a fixed QA spread sheet behind this to avoid data errors failing a build...of course data model changes would then require changes in two places...classic e2e problem.

[andyinabox]

There's a Cypress plugin for Netlify here: https://github.com/cypress-io/netlify-plugin-cypress. Seems like it would be a pretty safe option since its featured prominently on their marketing website.

[crhistianramirez]

There's a Cypress plugin for Netlify here

Cypress seems neat. When we were evaluating it for my day job we passed on it because of its lack of support for Safari or Internet explorer. Then again, might not be a huge deal - Cypress' CEO defends testing against a single browser

I wonder if we'd want a fixed QA spread sheet behind this to avoid data errors failing a build...of course data model changes would then require changes in two places...classic e2e problem.

I think we would definitely want a fixed data set, otherwise our assertions would be pretty weak (check that any value exists vs check that this specific value exists). Yep, data model changes would break tests but I'd argue they kind of should, it is a bit annoying though I agree.

[andyinabox]

@crhistianramirez true. They also have a zillion open issues right now: https://github.com/cypress-io/cypress/issues

[caseyhelbling]

Are there any other recommendations? We might be able to get pretty far with something simple like Jest. I also saw a reference to Karma on the netlify docs.

[wbushey]

Hey all, I'm just catching up on this thread and this project.

From what I can tell, Netlify doesn't really provide support for testing in CI. But that's alright; we can add Gitlab CI or CircleCI and get test runs in PRs.

I'm kind of out of the loop on JS test runners, but Karma looks pretty nice.

Just to clarify, do we want to be running automated tests in all four browsers that are in the PR template?

[crhistianramirez]

@caseyhelbling Jest is more of a unit testing framework isn't it? Not that we shouldn't be writing unit tests as well but they serve slightly different functions. In my experience end to end tests seem to catch a lot more bugs as they simulate real user interactions.

All that said unit tests are far simpler both to write and maintain/automate since they don't need a UI to interact with so maybe we decide to start there and add the more complex unit tests later on?

[wbushey]

Hey all, I spent some time looking at JS end-to-end testing tools, and have a bunch of notes below. Based on all of this, I'm inclined to go with https://webdriver.io/. It's Webdriver based, so it can run tests across a whole bunch of browsers and works with cloud testing services like Sauce Labs/BrowserStack (leaving the door open for testing on mobile). But it also seems to be easier to use and setup then something like Selenium.

Gotta say, the ability to walk through the steps of a test in Cypress is pretty nice. If Cypress supported more browsers, or cloud testing services, I'd probably recommend that.

Useful Resources

• An Overview of Javascript Testing in 2020

Criteria

• Can run tests on Chrome, Edge, Firefox, Safari
• Can run tests browser based tests against a running server
• Can run on local dev machine
• Can run in a CI runner

Possibilities

Cypress

• Can run tests on Chrome, Edge, Firefox, Safari

  • Supports Chrome, Edge, Firefox
  • Does not support Safari (Github Issue)

• Can run tests against a running server

  • Yes

• Can run on local dev machine

  • Yes. Provides a nice GUI for watching and running tests, and for stepping through tests

• Can run in a CI runner

  • Cypress docs on setting up CI
  • Cypress appears to profie it's own CI service that integrations with others (like CircleCI)
  • Cypress also provides Docker containers that be used to run Cypress tests
  • Supports headless test running

• Pros:

  • Seems to be pretty easy to setup
  • Records tests runs; can walk through steps of a test for debugging

• Cons:

  • No Safari support
  • No Cloud test provider support

• Notes:

  • Tests are run within browsers, and the Cypress test runner supports specific browsers

Jest

• Can run tests on Chrome, Edge, Firefox, Safari

  • Yes, would need to setup a Webdriver library

• Can run tests against a running server

  • Yes

• Can run on local dev machine

  • Yes

• Can run in a CI runner

  • Yes, as long as required browers are installed

• Pros:

  • Very popular and well supported
  • Fast

• Cons:

  • Primarily designed for unit testing; browser based feature test needs to be added
  • Isn't directly supported by Webdriver.Io; setting up Selenium is complicated

TestCafe

• Can run tests on Chrome, Edge, Firefox, Safari

  • Yes

• Can run tests against a running server

  • Yes

• Can run on local dev machine

  • Yes

• Can run in a CI runner

  • Yes, as long as required browers are installed
  • Site includes documentation on setting up with CircleCI and Gitlab CI

• Pros:

  • Support for Saucelabs and BrowserStack via plugins
  • Nice API for interacting with pages

• Cons

  • Seems to be trying to recreate Webdrivers
  • Got to async everything

• Notes:

  • Doesn't use Webdrivers, but from an node process manages tests running within a browser

Webdriver.io

• Can run tests on Chrome, Edge, Firefox, Safari

  • Yes, runs tests using the Webdriver protocol

• Can run tests against a running server

  • Yes

• Can run on local dev machine

  • Yes

• Can run in a CI runner

  • Yes, as long as required browers are installed

• Pros:

  • Webdriver based, so it can test a wide variety of browsers. Could also be used with services like Sauce Labs and BrowserStack to test various mobile devices.
  • Nice API hides a lot of the complexity of using Webdriver, including asyncronous Webdriver commands
  • debug can be used to pause a test and start a REPL in the test in the browser

• Cons:

  • Docs are not clear sometimes

• Notes:

  • Comes with support for Mocha, Jasmine, or Cucumber for tests
  • Comes with it's own test runner, or can be used with other test runners

[wbushey]

OK, now that I've actually tried to use Webdrivers.io, I'm less inclined to go with that. It runs Java Selenium under the hood. Between that and some of the obtuse documentation, it's kind of a pain to get going.

I tried out TestCafe, and that was soooo much easier to get started.

[wbushey]

I've started PR #140, which adds TestCafe and a test around opening and closing the Help/Info window. It would be nice to add tests that cover everything else in the template. It would also be nice to run tests in CI.

My capacity to help out is going to be spotty over the week, and I don't want that to hold folks up from making progress on this. So I'm open to whatever folks want to do with PR #140 .

[crhistianramirez]

@wbushey Thank you for taking the time to analyze those libraries! 🎉

I'll close this issue now that we've decided on a library and I'll create issues for completing the remaining test cases as well as figuring out how to wire this up with our CI/CD pipeline