RunStartupMethodsSequentially

This RunStartupMethodsSequentially .NET library (shortened to RunSequentially) to as is designed to updates to single resources, e.g. a database, on startup of an application that has multiple instances running (e.g. a ASP.NET Core web site with Azure Scale Out is turned on). This library runs when the application starts up and run your startup services. Typical startup services are: migrating to a database, seeding a database and/or ensuring that an admin user has been added. It can also be used with non-database resources as Azure blob, common files, etc.

Scale Out: A scale out operation is the equivalent of creating multiple copies of your web site and adding a load balancer to distribute the demand between them (taken from Microsoft Azure docs).

This open-source library available on NuGet as Net.RunMethodsSequentially. The documentation can be found in the README file, and the article How to safely apply an EF Core migrate on ASP.NET Core startup and see the Release notes for details of changes.

How it works, and what to watch out for

This library allows you create services known as startup services which are run sequentially on startup within a DistributedLock global lock. This global lock stops the problem of multiple instances of the application trying to update one common resource at the same time. For instance EF Core's Migrate feature doesn't work if multiple migrations are applied at the same time.

But be aware every startup service will be run on every application's instances, for example if your application is running four instances then your startup service will be run four times. This means your startup services should check if the database has already been updated, e.g. if your service adds an admin user to the the authentication database it should first check that that admin user isn't already been added (NOTE: EF Core's Migrate method checks if the database needs to be updated, which stops your database being migrated multiple times).

Secondly, you should test your use of this library because in ASP.NET Core it is run as a hosted service and if there is an exception the application won't give you any feedback on what went wrong. I have added a class called RegisterRunMethodsSequentiallyTester which makes it easier to test your use of the the RunSequentially library - see the first test in the TestRegisterRunMethodsSequentiallyTester xUnit Test.

Another warning is that applying a database migration is that the migration must not contain what are known as breaking changes - that is a change that will cause a the currently running application from working (see the five-stage app update in this article for more info). This issue isn't specific to this library, but applies whenever you are updating an application without any break in the service, known as 24/7 or continuous application.

What you have to do to use this library?

1. Add the Net.RunMethodsSequentially NuGet package to your main application.
2. Create your startup services that will update a common resource, e.g. Migrating a database. Each startup service must implement the RunMethodsSequentially's IStartupServiceToRunSequentially interface.
3. Register the RegisterRunMethodsSequentially extension method to your dependency injection provider, and select:
   • What global resource(s) you want to lock on.
   • Register your startup service(s) you want run on startup.

If you are working with ASP.NET Core, then that is all you need to do. If you are working with another approach, like a Generic Host you have to manually call the IGetLockAndThenRunServices service (see RegisterAsHostedService in the Extra options section).

1. Add NuGet Net.RunMethodsSequentially to your app

Simply add the Net.RunMethodsSequentially NuGet package to your application. This NuGet library is needed in the project holding your startup services, and the project containing the an ASP.NET Core Program class for registering the library.

2. Creating your startup services

To create a startup service that will be run while in a lock you need to create a class that inherits the IStartupServiceToRunSequentially interface. This has a method defined has a ValueTask ApplyYourChangeAsync(IServiceProvider scopedServices). This method is where you put your code to update a shared resource, such as a database.

Here is an example of a startup service, in this case its a EF Core Migrate of a database.

public class MigrateDbContextService : 
    IStartupServiceToRunSequentially
{
    public int OrderNum { get; }

    public async ValueTask ApplyYourChangeAsync(
        IServiceProvider scopedServices)
    {
        var context = scopedServices
            .GetRequiredService<TestDbContext>();

        //NOTE: The Migrate method will only update 
        //the database if there any new migrations to add
        await context.Database.MigrateAsync();
    }
}

The ApplyYourChangeAsync is given a scoped service provider, which means its copy of the normal services used in the main application. From this you can get the services you need to apply your changes to the shared resource, in this case a database.

NOTE: You can use the normal constructor DI injection, but as a scoped service provider is already available you can use that instead.

The OrderNum is a way to define the order you want your startup services are run. If services have the same OrderNum value, like zero, they will be run in the order they were registered (see next section).

I want to emphasize that your startup services are going to be run multiple times, once for each instance of your application. Therefore your startup service must check if the update it wants to do hasn't already been applies. EF Core's MigrateAsync automatically checks if the database needs to be updated, but if you are seeding a database then you check that the seeded hasn't already been done.

Here is a example of adding a admin user to the individual account authentication database, which checks if the admin user hasn't already been added - see test at the bottom of the code.

public class IndividualAccountsAddSuperUser : IStartupServiceToRunSequentially
{
    public int OrderNum { get; }

    public async ValueTask ApplyYourChangeAsync(IServiceProvider scopedServices)
    {
        var context = scopedServices
            .GetRequiredService<UserManager<TIdentityUser>>();
        var config = scopedServices
            .GetRequiredService<IConfiguration>();
        var superSection = config.GetSection("SuperAdmin");

        var user = await userManager
            .FindByEmailAsync(superSection["Email"]);

        if (user != null)
            return; //ALREADY SO DON'T TRY TO ADD AGAIN

        //... code to add a new user left out
    }
}

3. Register RunMethodsSequentially your application

A typical configuration setup for the RunMethodsSequentially feature depends on what your application is:

For ASP.NET Core

You most likely want to get the database connection string held in the appsettings.json file using the IConfiguration service and an known folder, either the IWebHostEnvironment's ContentRootPath or WebRootPath. The code below shows how to do this in the net6.0 Program class

using RunMethodsSequentially;

var builder = WebApplication.CreateBuilder(args);

var connectionString = builder.Configuration
    .GetConnectionString("DefaultConnection");
var lockFolder = builder.Environment.WebRootPath;

builder.Services.RegisterRunMethodsSequentially(options =>
    {
        options.AddSqlServerLockAndRunMethods(connectionString));
        options.AddFileSystemLockAndRunMethods(lockFolder);
    })
    .RegisterServiceToRunInJob<MigrateDatabaseOnStartup>()
    .RegisterServiceToRunInJob<SeedDatabaseOnStartup>();

//... other setup code left out

NOTE: The RunStartupMethodsSequentially repo contains a ASP.NET Core application called WebSiteRunSequentially that I used to check that it worked on Azure with scale out. The Program class.

For non-ASP.NET Core applications

For applications where you don't have access to the you will need to rely on a constant for your connectionString and static methods such as AppContext.BaseDirectory or Environment.CurrentDirectory to get a folder.

Also, you need to set the RegisterAsHostedService, which will register as a IGetLockAndThenRunServices transient service instead of the the ASP.NET Core specific HostedService.

services.RegisterRunMethodsSequentially(options =>
    {
        options.RegisterAsHostedService = false;
        options.AddSqlServerLockAndRunMethods(connectionString));
        options.AddFileSystemLockAndRunMethods(
            Environment.CurrentDirectory);
    })
.RegisterServiceToRunInJob<MigrateDatabaseOnStartup>()
.RegisterServiceToRunInJob<SeedDatabaseOnStartup>();

You then need to run the IGetLockAndThenRunServices transient service and call its LockAndLoadAsync method.

Explanation of the various methods

The following subsections describe each part:

3.a RegisterRunMethodsSequentially extension method

This method allows you to register one or more LockAndRun methods. Each LockAndRun method first checks that the provided resource type exists. If the resource isn't found, then it exits and allows later LockAndRun methods to try to get a lock on a resource. If no resource can be found to create a global lock, the it will throw an exception.

There are four LockAndRun methods (and its not hard to create others):

• AddSqlServerLockAndRunMethods(connectionString), which works with a SQL Server database
• AddPostgreSqlLockAndRunMethods(connectionString), which is for a PostGreSQL database
• AddFileSystemLockAndRunMethods(- path to global directory -), which uses a FileSystem directory shared across all of the application's instances.
• AddRunMethodsWithoutLock() which doesn't lock anything and just runs your startup services. This gives you a way to turn off the lock if you don't have multiple instances (locking a local SQL Server database takes 1 ms, but no lock only takes 50 us.)

NOTE: It is fairly easy to add another LockAndRun methods as long as the DistributedLock has a lock version for a global lock available to your application. So if you want to lock on Redis, Azure blob etc. then you can write the extra lock code yourself - just follow the pattern of the existing LockAndRun code.

The reason for having a series of LockAndRun methods is to allow for resources that might not created yet. For instance, if the database doesn't currently exist, then it can't obtain a lock on the database. In this case the second FileSystem LockAndRun method can obtain a lock on a FileSystem directory that all the applications can access.

The first LockAndRun method that obtains a global lock will then run all your startup services you have registered to RunMethodsSequentially (see RegisterServiceToRunInJob<T> section later). Once the your startup services have run on one application instance it is on, then it releases the global lock, which allows another instance of the application to run until all the instances have been run.

Of course, once one instance has successfully applied your startup services to the global resource, then other instances are still going to run. That is why your startup services must check if the update they are planning to add haven't already been applied.

Extra options

The options also has default settings for some of the code, but you can override these. They are:

• options.RegisterAsHostedService: By default this is true, and the the IGetLockAndThenRunServices service isn't registered, but the GetLockAndThenRunHostedService is registered as a HostedService. In ASP.NET Core that means the IGetLockAndThenRunServices code is run on startup.
• options.DefaultLockTimeoutInSeconds: By default this is set to 100 seconds, and defines the time it will wait for a lock can be acquired. _NOTE: When you have NNN multiple instances the time for the ALL of your startup services must be less that DefaultLockTimeoutInSeconds / NNN.
• There are other settings - see RunSequentiallyOptions for full details.

3.b Registering your start services

The code below just shows just the registering of the RunMethodsSequentially code, with the last two lines using the RegisterServiceToRunInJob<T> extension method to register your startup services. In this case its two startup services, but it can be many more (NOTE: see the 2. Creating your startup services section about the order in which your startup services are run).

services.RegisterRunMethodsSequentially(options =>
    {
        options.AddSqlServerLockAndRunMethods(connectionString));
        options.AddFileSystemLockAndRunMethods(lockFolder);
    })
    .RegisterServiceToRunInJob<MigrateDatabaseOnStartup>()
    .RegisterServiceToRunInJob<SeedDatabaseOnStartup>();

There are two rules about your startup services:

1. They must inherit the IStartupServiceToRunSequentially interface.
2. Your startup services most likely have to be run in a certain order, for instance you should Migrate/Create your database startup service before any startup services that access that database. You have two options:
   1. The simplest approach is you register your services in the correct order using the RegisterServiceToRunInJob<T> method.
   2. If you have a complex situation (like the one in my AuthPermissions.AspNetCore library) then you can use the OrderNum in your startup service to define the order.

[End]