Skip to content

Latest commit

 

History

History
123 lines (82 loc) · 7.9 KB

Configuration.md

File metadata and controls

123 lines (82 loc) · 7.9 KB

Configuration

Configuration from code

All the necessary aspects of the Toolbox can be configured from code. Look for such parameters on various methods. The root configuration class that can be set up for every test individually is OrchardCoreUITestExecutorConfiguration.

External configuration

Note that since the tests are xUnit tests you can configure general parameters of test execution, including the level or parallelization, with an xUnit configuration file (xunit.runner.json). A default suitable one is included in the UI Testing Toolbox and will be loaded into your test projects; if you want to override that then:

  1. Add a suitable xunit.runner.json file to your project's folder.
  2. In the csproj configure its "Build Action" as "Content", and "Copy to Output Directory" as "Copy if newer" to ensure it'll be used by the tests. This is how it looks like in the project file:
<ItemGroup>
  <None Remove="xunit.runner.json" />
  <Content Include="xunit.runner.json">
    <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
  </Content>
</ItemGroup>

Certain test execution parameters can be configured externally too, the ones retrieved via the TestConfigurationManager class. All configuration options are basic key-value pairs and can be provided in one of the two ways:

  • Key-value pairs in a TestConfiguration.json file. Note that this file needs to be in the folder where the UI tests execute. By default this is the build output folder of the given test project, i.e. where the projects's DLL is generated (e.g. bin/Debug/net6.0).
  • Environment variables: Their names should be prefixed with Lombiq_Tests_UI, followed by the config with a __ as it is with (ASP).NET configuration, e.g. Lombiq_Tests_UI__OrchardCoreUITestExecutorConfiguration__MaxRetryCount (instead of the double underscore you can also use a : on certain platforms like Windows). Keep in mind that you can set these just for the current session too. Configuration in environment variables will take precedence over the TestConfiguration.json file. When you're setting environment variables while trying out test execution keep in mind that you'll have to restart the app after changing any environment variable.

Here's a full TestConfiguration.json file example, something appropriate during development when you have a fast machine (probably faster then the one used to execute these tests) and want tests to fail fast instead of being reliable:

{
  "Lombiq_Tests_UI": {
    "AgentIndex": 3,
    "TimeoutConfiguration": {
      "RetryTimeoutSeconds": 5,
      "RetryIntervalMillisecondSeconds": 300,
      "PageLoadTimeoutSeconds": 120
    },
    "OrchardCoreUITestExecutorConfiguration": {
      "MaxRetryCount": 0,
      "RetryIntervalSeconds": 0,
      "MaxParallelTests": 0
    },
    "BrowserConfiguration": {
      "Headless": true
    }
  }
}

Note that this will execute tests in headless mode, so no browser windows will be opened (for browsers that support it). If you want to troubleshoot a failing test then disable headless mode.

We encourage you to experiment with a RetryTimeoutSeconds value suitable for your hardware. Higher, paradoxically, is usually less safe.

If you have several UI test projects it can be cumbersome to maintain a TestConfiguration.json file for each. Instead you can set the value of the LOMBIQ_UI_TESTING_TOOLBOX_SHARED_TEST_CONFIGURATION environment variable to the absolute path of a central configuration file and then each project will look it up. If you place an individual TestConfiguration.json into a test directory it will still take precedence in case you need special configuration for just that one.

MaxParallelTests sets how many UI tests should run at the same time. It is an important property if you want to run your UI tests in parallel, check out the inline documentation in OrchardCoreUITestExecutorConfiguration.

Multi-process test execution

UI tests are executed in parallel by default for the given test execution process (see the xUnit documentation). However, if you'd like multiple processes to execute tests like when multiple build agents run tests for separate branches on the same build machine then you'll need to tell each process which build agent they are on. This is so clashes on e.g. network port numbers can be prevented.

Supply the agent index in the AgentIndex configuration. It doesn't need to but is highly recommended to be zero-indexed (see the docs on limits) and it must be unique to each process. You can also use this to find a port interval where on your machine there are no other processes listening.

If you have multiple UI test projects in a single solution and you're executing them with a single dotnet test command then disable them being executed in parallel with the xUnit "parallelizeAssembly": false configuration (i.e. while tests within a project will be executed in parallel, the two test projects won't, not to have port and other clashes due to the same AgentIndex). This is provided by the xunit.runner.json file of the UI Testing Toolbox by default.

Using SQL Server from a Docker container

Setup

You can learn more about the microsoft-mssql-server container here. You have to mount a local volume that can be shared between the host and the container. Update the values of device and SA_PASSWORD in the code below and execute it.

On Windows

docker pull mcr.microsoft.com/mssql/server
docker run --name sql2019 -e "ACCEPT_EULA=Y" -e "SA_PASSWORD=Password1!" -p 1433:1433 -d mcr.microsoft.com/mssql/server:2019-latest
docker exec -u 0 sql2019 bash -c "mkdir /data; chmod 777 /data -R; chown mssql:root /data"

On Linux

You need to put the shared directory inside your $HOME, in this example ~/.local/docker/mssql/data:

docker pull mcr.microsoft.com/mssql/server
docker run --name sql2019 -e 'ACCEPT_EULA=Y' -e 'SA_PASSWORD=Password1!' -p 1433:1433 -d 'mcr.microsoft.com/mssql/server:2019-latest'
docker exec -u 0 sql2019 bash -c 'mkdir /data; chmod 777 /data -R; chown mssql:root /data'

If you haven't yet done it yet, add your user to the docker group.

If you get PlatformNotSupportedException, that's a known problem with Microsoft.Data.SqlClient on .Net 5 and above. As a workaround temporarily set the project's runtime identifier to linux-x64 either on the terminal or by adding <RuntimeIdentifier>linux-x64</RuntimeIdentifier> to the project file.

On Both

If you want to test it out, type docker exec -u 0 -it sql2019 /opt/mssql-tools/bin/sqlcmd -S localhost -U sa -P "Password1!" to access the SQL console.

You can use Docker Desktop or Portainer to stop or start the container going forward.

Extending TestConfiguration.json

SQL Server on Linux only has SQL Authentication and you still have to tell the toolbox about your backup paths. Add the following properties to your Lombiq_Tests_UI. Adjust the Password field of the connection string and HostSnapshotPath property as needed.

{
  "SqlServerDatabaseConfiguration": {
    "ConnectionStringTemplate": "Server=.;Database=LombiqUITestingToolbox_{{id}};User Id=sa;Password=Password1!;MultipleActiveResultSets=True;Connection Timeout=60;ConnectRetryCount=15;ConnectRetryInterval=5"
  },
  "DockerConfiguration": {
    "ContainerSnapshotPath": "/data/Snapshots",
    "ContainerName": "sql2019"
  }
}

The default value of ContainerSnapshotPath is "/data/Snapshots" so you can omit that.