Support for running tests without a browser (OSOE-365)

[Piedone]

In certain cases, we need a running app but not a browser (or at least not a full one), like for testing web APIs. For such cases, we could start tests without a browser and then use something like RestEase for APIs or ScrapySharp as a barebone browser. The browser is launched when AtataFactory.CreateDriver() runs, this is the only thing necessary to switch off. However, a browser would still sometimes be needed for the initial setup (though we could add a feature to only open a browser when the setup is run but not for the test itself, or utilize Auto Setup).

Jira issue

[sarahelsaig]

ScrapySharp is built on HTML Agility Pack (see dependencies tab here), which is excellent. I have used HAP for years so you will hear no complaints about it from me. But OC already references AngleSharp which is its competitor, so aren't we supposed to use that?

However, a browser would still sometimes be needed for the initial setup

We could make CLI based setup an option?

[0liver]

Afair, we used ScrapySharp for scraping back in discoverize. Really liked it.

[Piedone]

CLI-based setup can work too. We allow something like that with Setup Extensions since you only have to access a RESTful API there (i.e. no browser needed). Or, we could do some magic in ConfigureUITesting() and run the setup, much like Auto Setup, from IConfiguration data (and that way, we can supply setup parameters in the CLI, meaning we can also open them up in OrchardCoreUITestExecutorConfiguration).

This might actually be preferable for setting up the site in every case, unless we specifically want to test the setup screen.

No opinion on HAP vs AngleSharp, though having a different dependency than OC wouldn't really matter.

[Piedone]

With BrowserUITestContextExtensions added by Dávid here we now have the basic facilities for this, except for not starting a browser. And CreateAndSwitchToTenantAsync() can be used to set up a tenant without a browser.

[sarahelsaig]

The security scanning that was implemented in #322 would greatly benefit from this. It's already using its own internal crawler for the scan so the browser just takes up RAM and does nothing after setup. So now even a very stripped-down version of this feature would be beneficial.

As for the setup, consider adding a configuration to make it automatic/browserless even if the rest of the test uses a browser. I mean if the site uses auto-setup or stock unaltered OC setup, then we are UI testing the same stock features again and again. (the same situation as login was before the SignInDirectlyAsync extensions)
We could enable browser setup just in one test (e.g. BasicOrchardFeaturesShouldWork). The rest should be set up in programmatically to save time and power.

[Piedone]

With ExecuteTestAfterSetup the setup is run only once for each set of tests (that use the same setup operations, which is commonly the whole test suite), and then cached, so we're not wasting time on the setup repeatedly (and that single setup serves as the test of the setup itself).

However, for security scanning tests yes, not running a browser at all would be beneficial.

[sarahelsaig]

I know. But spinning up a Chrome process for the setup and using Selenium to type in the values to each field one by one still takes some time. I understand that it's not a huge cost, but on large projects it still adds up to machine time that could've been spent more productively.

[Piedone]

Sure.

[Piedone]

Yevgeniy told that in AtataFactory we could use UseDriverInitializationStage(AtataContextDriverInitializationStage.None) to not start a WebDriver but still have an Atata context.

[Piedone]

@wAsnk @sarahelsaig if you need this, it should be really easy now, with just a config bool and see my previous comment.

[sarahelsaig]

That sounds exciting! It will be sufficient for testing a REST API and for reducing overhead during security scanning.

Although for anything more complicated I'd still like a custom IWebDriver implementation that uses AngleSharp and HttpClient, so we can keep using all the Atata features and the accumulated extensions in UITT. (This could be a huge performance boost for testing features that don't require JS.)

[Piedone]

Yep! If you'd utilize this in the project whose PR this was linked from above, then please take on it.

[Piedone]

ScrapySharp, with no release since 2018 and no code changes since 2020 seems to be dead, unfortunately.

I didn't find an IWebDriver implementation that uses HtmlAgilityPack, but there's one for AngleSharp: https://github.com/TheJayMann/Selenium.AngleSharp.WebDriver It seems to be more of an abandoned experiment, though. It does provide a glimpse into the LOE to build such a driver.

JavasScript can supposedly be disabled in all the browsers we use by changing their configs, what's possible from consumer code today via BrowserOptionsConfigurator. This would only be a niche speed-up for certain tests, since the browser is still there, but may be useful. I was curious if this makes a differnece, and tried to disable JS like this in CreateDriverInnerAsync:

var prefs = new Dictionary<string, object>
{
    { "webkit.webprefs.javascript_enabled", false },
    { "profile.content_settings.exceptions.javascript.*.setting", 2 },
    { "profile.default_content_setting_values.javascript", 2 },
    { "profile.managed_default_content_settings.javascript", 2 },
};

chromeConfig.Options.AddUserProfilePreference("prefs", prefs);

// Add the disable-javascript argument
chromeConfig.Options.AddArgument("--disable-javascript");

if (configuration.Headless) chromeConfig.Options.AddArgument("headless=new");

But this didn't disable JS, window.alert("hello"); in the page still worked. (I only added headless from the linked SO answer but didn't use headless mode.)

But more to the point of running tests without a browser:

It turns out that what's even better than AtataContextDriverInitializationStage.None is AtataContextDriverInitializationStage.OnDemand, what only builds the driver if AtataContext.Driver is accessed. This sounds great, though we have a couple of places where the driver is accessed without any explicit consumer request (like to assert the browser logs), so for that we need a config. And even before those, strangely, Chrome is opening for me on ln 39, so even before the context is built...

I guess we'd need to use the UseDriver(Func<IWebDriver> driverFactory) overload, but we need async. So, we need a custom config.

BTW a browser-less setup can be simply utilizing Auto Setup too.

[Piedone]

After some refactoring, we can have on-demand driver creation and thus browser launch. However, these are still using it even for security scans: 3da1e43 I'll need to think about this.

BTW on my machine, not opening a browser brings about a two-second gain for a test, which is actually significant (and even more if it's done a lot more, on a slower machine).

[Piedone]

Well, this was a bit more than a bool config but I think it's now done properly.

[sarahelsaig]

there's one for AngleSharp: https://github.com/TheJayMann/Selenium.AngleSharp.WebDriver It seems to be more of an abandoned experiment, though. It does provide a glimpse into the LOE to build such a driver.

This doesn't look so bad.

BTW on my machine, not opening a browser brings about a two-second gain for a test, which is actually significant (and even more if it's done a lot more, on a slower machine).

The difference is also impactful in terms of RAM use, which can become a bottleneck for parallel testing. In a previous gig where we tested remote servers (on not very powerful machines), this was a major problem. We always groaned when a site couldn't run without JS, because it meant interacting with it would need a real browser and it would max out the machine while running.

[Piedone]

Yeah, memory is perhaps a bigger concern for the runtime of a workflow overall.

I've opened #399 about this.