Skip to content
This repository has been archived by the owner on Dec 3, 2020. It is now read-only.

Add automated test suite. #32

Merged
merged 2 commits into from
Jul 30, 2018
+877 −5
Merged

Add automated test suite. #32

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
6519787
Fix #28: Add Firefox/Marionette/Tape-based test suite.
Jul 23, 2018
5e80640
Improve error message when Firefox binary path is misconfigured.
Jul 24, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 10 additions & 1 deletion .circleci/config.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,13 +2,16 @@ version: 2
jobs:
build:
docker:
- image: circleci/node:10-browsers
- image: circleci/python:2.7-node-browsers
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

2.7? 😢

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blame mozbase and friends.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This comment recommends using Geckodriver or similar (at least for non-Mozilla projects). Why wouldn't we want to do that here? (granted this IS a Mozilla project).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Geckodriver translates the Webdriver protocol into Marionette commands. While Webdriver does include methods for executing JS in the browser, it does not have support for switching to a chrome context and executing privileged code. Since that is one of the benefits of Marionette/Firefox based testing (testing privileged chrome code), we don't want to use Geckodriver since we'd lose that ability.

steps:
- checkout
- run:
name: Install dependencies
command: |
pipenv install
npm install
pipenv run mozdownload --version latest --destination $HOME/firefox.tar.bz2
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there any reason why you chose pipenv run instead of pipenv shell?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's less stateful, as pipenv shell would leave you in a pipenv-enabled shell afterwards. Easier to not have to remember that if we add more commands to the script later.

pipenv run mozinstall --destination $HOME $HOME/firefox.tar.bz2
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CircleCI browser images have a ludicrously old version of Firefox (like, 48). Maybe there's more recent versions elsewhere in the image that I didn't look, but this ensures we test using the latest version.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Edit: this was answered in slack, but I'll leave it here for posterity.

Looks like we are downloading and installing the latest Firefox for each run; could we add an "update" step or make our own image?

I see that the Shield Studies Addon Template uses "latest-browsers" for their docker image, though I haven't found a similar one that also has Python.

Looks like we may also be able to cache the Firefox binary, so we don't have to install/download all the time?

Osmose's reply in slack:

Downloading the latest firefox kinda sucks, but it also avoids having to maintain a docker image with the latest Firefox, and specifically one that also has Python and node installed
And it's really fast, like the download and install per test run takes like 10 seconds

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We know that the docker image's Firefox version was old, because Marionette outputs the Firefox version to the terminal right (on client.start_session())?

mozversion INFO | application_version: 61.0.1

- run:
name: Build extension
command: |
Expand All @@ -17,9 +20,15 @@ jobs:
name: Run linting checks
command: |
npm run lint
- run:
name: Run automated tests
command: |
pipenv run test -- --firefox_bin $HOME/firefox/firefox
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is the -- meant to pass the options to the script run by pipenv run test? (i.e. these are not options for npm to handle)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correct, -- is a separator that tells Pipenv that all following options are to be passed to the underlying script instead of to Pipenv itself.

- run:
name: Package extension as XPI
command: |
npm run package
- store_artifacts:
path: web-ext-artifacts
- store_artifacts:
path: gecko.log
1 change: 1 addition & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
node_modules
web-ext-artifacts
build
gecko.log
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is generated by Firefox during the Marionette run. It has useful info if Firefox fails in some way during the run.

18 changes: 18 additions & 0 deletions Pipfile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[dev-packages]

[packages]
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why are these listed under packages and not dev-packages?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since Python is only necessary for development, I figured we could save having to add --dev to pipenv install by just making them non-dev dependencies.

click = "*"
marionette-client = "*"
mozdownload = "*"
mozinstall = "*"

[requires]
python_version = "2.7"

[scripts]
test = "npm test"
251 changes: 251 additions & 0 deletions Pipfile.lock
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

31 changes: 30 additions & 1 deletion README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,35 @@ Prerequisites:
npm start
```

## NPM Scripts
## Running Tests

Automated tests are run in a Firefox browser instance using [Marionette][]. We use the Python client for Marionette since there is no up-to-date JavaScript client.

To set up your environment for running the tests, you must have:

- A Firefox binary. On MacOS, this can be found within the `.app` folder at `Firefox.app/Contents/MacOS/firefox`.
- Python 2.7
- [Pipenv][]

With these installed, you can set up the test suite:

1. Install Python dependencies:

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I needed to pip install pipenv first, so maybe add a check for that?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The "Prerequisites" section above lists Pipenv as required, and links to the Pipenv website with instructions on how to download. I figured that should be enough to indicate that you need Pipenv installed.

```sh
pipenv install
```
2. Save the path to your Firefox binary with `npm`:

```sh
npm config set webext-commerce:firefox_bin <PATH_TO_FIREFOX_BINARY>
Copy link
Collaborator

@biancadanforth biancadanforth Jul 24, 2018 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I might add an example path,since on Mac, it's a pain to find the path to Firefox:
Example: "/Applications/Firefox.app/Contents/MacOS/firefox-bin"

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is called out earlier in the README.

```

After this, you can run `pipenv run test` to run the automated test suite.

[Marionette]: https://firefox-source-docs.mozilla.org/testing/marionette/marionette/index.html
[Pipenv]: https://docs.pipenv.org/
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TIL: You can make inline links in this way.


## Scripts

| Command | Description |
| --- | --- |
Expand All @@ -39,6 +67,7 @@ Prerequisites:
| `npm run build` | Compile source files with Webpack |
| `npm run watch` | Watch for changes and rebuild |
| `npm run package` | Package the extension into an XPI file |
| `pipenv run test` | Run test suite (See "Running Tests" for setup) |

## License

Expand Down
Loading