Auto-generating the entry point

[caaavik-msft]

This PR implements an opt-in feature for auto-generating the entry point for BenchmarkDotNet applications. Today, all BDN programs need to implement their own entry point method, and it is quite common for them to be a simple one-liner that invokes the BenchmarkRunner or BenchmarkSwitcher over the benchmarks in the current assembly while passing in a default configuration to use. Other test suite libraries such as MSTest and xUnit all have this functionality, meaning that developers only need to worry about writing tests.

In addition, there is a PR out for adding VSTest support to BenchmarkDotNet. This feature allows for using the VSTest discovery and execution protocols to invoke BDN, as opposed to running it like a normal console application. For people who wish to primarily interact with BDN through the VSTest protocol, or don’t need any of the customisations in the entry point apart from the default behaviour, this feature will be convenient and lead to a cleaner project structure.

Implementation

The entry point source code is stored inside the NuGet package, along with a props file which ensures that consumers of the library include the entry point when compiling. To support multiple languages, an entry point file is written for C#, F#, and VB, with the props file able to select the correct one to load for the project type. This approach to auto-generating the entry point is also used by MSTest and xUnit.

Since the majority of BDN projects today have an entry point already, this functionality would be opt-in rather than assuming that an automated entry point should be generated so updating BDN does not cause a breaking change. Opting in is as simple as setting GenerateBDNEntryPoint to true in the csproj or equivalent.

Since this feature is implemented through the NuGet package only, it is not possible to enable this feature for the samples project that lives inside this repo as it does not consume it via a PackageReference, but instead a direct ProjectReference.

Default behaviour of entry-point

In my implementation, the auto-generated C# entry point looks as follows:

public static void Main(string[] args) =>
    BenchmarkSwitcher.FromAssembly(Assembly.GetEntryAssembly()).Run(args); 

This invokes the BenchmarkSwitcher which when run from the command line will display an interactive experience to select which benchmarks to run, but also supports several command line arguments to customize which benchmarks get run and the configurations they are run under. Another option for the default behaviour would be to use BenchmarkRunner.Run which is what is used in the official project templates. I opted for BenchmarkSwitcher as it is much more customizable as a default experience, but if people think BenchmarkRunner makes more sense as a default then I wouldn't be opposed to changing it.

Default configuration for assembly

Today, it is common for people to use the entry point as the place to specify a default configuration such as how many iterations should be done, setting up loggers, exporters, analysers etc. If a developer is using the auto-generated entry point, they would not be able to do this. Since developers often use the entry point for setting this up, it means that anything done in the entry point is not available if you you wish to invoke benchmarks using the VSTest protocol.

BDN today has support for setting a config that applies to a whole assembly, but the feature is not well used to my knowledge. The way to set a config for the whole assembly is to create an attribute type that extends Attribute and implements IConfigSource. Then you just need to set up an assembly attribute in your AssemblyInfo.cs which uses this attribute. An example of this can be seen in the following project:

https://github.com/dotnet/BenchmarkDotNet/tree/master/tests/BenchmarkDotNet.IntegrationTests.ConfigPerAssembly

By doing this instead, you can get the benefits of the auto-generated entry point, while also having the configuration available to use with VSTest.

Despite this feature being available, this will not fully replace the need for a custom entry point for many consumers if the BDN program needs access to the command line arguments that get passed in. For these projects, it would still be beneficial to make use of the assembly configuration for anything that does not have a dependency on command line arguments so that it can be used when invoked by VSTest. BDN will merge together the configurations passed in using the entry point, set on the assembly, and set on each benchmark type.

Testing

To test this PR, I built and packed BDN locally, I generated the BDN template applications for C#, F# and VB, deleted the entry point files, set GenerateBDNEntryPoint to true in the proj file, and verified that everything worked with .NET Framework and .NET 8.0.

[adamsitnik]

I need to get a better understanding of how it’s going to affect some of the scenarios.

So far, we’ve supported only Console Apps. Will this change make it possible to support libraries?

• If so, what command will be needed to run the benchmarks? dotnet test?
• If so, what will be the behavior for libraries that target .NET standard rather than a specific runtime (example: netstandard2.0)?

Edge case: will it work with the "old" project files?

BTW I am not convinced that this is the right direction. Currently all BDN users are used to the fact that only Console Apps are supported (providing an entry point is a must-have). With this change, they won’t need to provide the entry point, but they will still have to do something to get it working (setting GenerateBDNEntryPoint to true). From my perspective it’s less intuitive (something new) and it does not solve the problem of passing the config.

[adamsitnik]

Since this type will be part of the produced .exe, we can use typeof(X).Assembly here:

Suggested change

	public static void Main(string[] args) => BenchmarkSwitcher.FromAssembly(Assembly.GetEntryAssembly()).Run(args);
	public static void Main(string[] args) => BenchmarkSwitcher.FromAssembly(typeof(__AutoGeneratedEntryPointClass).Assembly).Run(args);

[timcassell]

BTW I am not convinced that this is the right direction. Currently all BDN users are used to the fact that only Console Apps are supported (providing an entry point is a must-have). With this change, they won’t need to provide the entry point, but they will still have to do something to get it working (setting GenerateBDNEntryPoint to true). From my perspective it’s less intuitive (something new) and it does not solve the problem of passing the config.

I am of a similar mind. Can this be done as part of the BenchmarkDotNet.TestAdapter only?

[adamsitnik]

Can this be done as part of the BenchmarkDotNet.TestAdapter only?

This is great question! 👍

[caaavik-msft]

Good idea, I have moved the stuff from this PR into the VSTest Adapter PR then instead rather than have them separate, and to make this only under the TestAdapter project