Sanic Server WorkerManager refactor

[ahopkins]

NOTE: When I say "Sanic" I mean Sanic Server. This PR is inapplicable to ASGI mode.

Background

The long overdue refactor of Sanic multiprocessing...

This PR intends to redo how Sanic creates processes and sockets. As a side effect, it will also change how auto-reload works, and perhaps most importantly finally fix serving multiple workers from Windows. It will also expose a new API for manually triggering restarts, accessing the current worker process context, and passing objects to the processes like multiprocessing.Queue that will enable sync between single-node, multi-worker instances.

Design concept

As discussed in #2364, there will always* be a main process when you start Sanic. The main process will create a WorkerManger that is responsible for the lifecycle of one or more server processes, and optionally a reload process. This pattern will also be extensible by API to allow Sanic Extension and other plugins to piggyback off the manager for example to create a health-check process.

* When I say always, I really mean "sometimes". We will also create a secondary Sanic.single_serve() method that creates and runs the server in one process. No auto-reload or multi-worker support. You will need to go out of your way to use this for those people that need it. OOTB, we want to make a consistent experience between DEBUG and PROD.

Once this is complete, we can make further enhancements to Sanic.prepare. Currently if you prepare multiple HTTP versions and/or socket bindings, they will run in each process on the same loop. We can make this better to allow HTTP servers instances to have their own loops inside individual processes. This is not a part of this PR and will likely be an addition in v22.12.

One or more sockets will be created by the Main process and passed to each worker process (as needed). Because most of this implementation will be all new, it should be pretty easy to maintain backwards compatibility by taking the existing Sanic.serve and moving it to Sanic.serve_legacy.

New API

• Sanic.serve_legacy - Use the old version of Sanic.serve
• Sanic.serve_single - Run Sanic w/o multiprocessing (no auto-reload allowed)
• app.run(..., single_process=True)
• app.run(..., legacy=True)
• app.m.restart() - Manually restart an application
• app.shared_ctx - see below
• $ sanic path.to:app --inspect - CLI command to inspect running application
• app.manager.manage(...) - see below

Shared state

For sharing safe values between workers (on the same Sanic runtime), there is a new app.shared_ctx. This will be an object explicitly for things like multiprocessing sync and shared state objects (Queue, Pipe, Value, Array, etc). They should ONLY be added in the main process like this:

@app.main_process_start
async def test(app, _):
    pipe = Pipe()
    app.shared_ctx.value = Value("i", 0, lock=False)  # OK
    app.shared_ctx.pipe = pipe                        # OK
    app.shared_ctx.pipe_side = pipe[0]                # OK
    app.shared_ctx.queue = Queue()                    # OK
    app.shared_ctx.unsafe = {}                        # raises warning
    app.shared_ctx.also_unsafe = (0,)                 # raises warning

Process management

A user can hook into the Sanic worker manager to allow it to manage any additional subprocess. All it needs is a callable. If that subprocess will be blocking, then it should also handle some common signals.

from signal import SIGINT, SIGTERM
from signal import signal as signal_func

def my_process(*args, **kwargs):
    run = True

    def stop(*_):
        nonlocal run
        run = False

    signal_func(SIGINT, stop)
    signal_func(SIGTERM, stop)

    while run:
        print(".")
        sleep(1)

@app.main_process_ready
async def ready(app: Sanic, _):
    app.manager.manage("MyProcess", my_process, {"foo": "bar"})

Sanic will now startup the process and manage its lifecycle.

Worker state

There is an object available on the application multiplexer called "state":

app.m.state

It is a dictionary-like object containing basic details about the current state of the current worker process:

{
    'server': True,
    'state': 'ACKED',
    'pid': 1386076,
    'start_at': datetime.datetime(2022, 9, 14, 8, 9, 44, 651607, tzinfo=datetime.timezone.utc),
    'starts': 1
}

This object has some basic operations to interact with it like a dictionary:

request.app.m.state["arbitrary"] = 123
request.app.m.state.update({"foo": "bar"})

It should be noted that the multiplexer object is only available inside of a server worker process.

Inspector

There is a special worker process called the "Inspector". You must opt-in to use it:

app.config.INSPECTOR = True

This will open a local port that is exposed to allow an outside process to get information and interact with the running application. In the below screenshot, we execute a CLI command to get the state of the current instance:

Other commands:

--inspect              Inspect the state of a running instance, human readable
--inspect-raw          Inspect the state of a running instance, JSON output
--trigger-reload       Trigger worker processes to reload
--trigger-shutdown     Trigger all processes to shutdown

Breaking Changes

• Setting app.config values in main_process_start will have no impact. (This is a sort of unintended side effect. Probably more a bug that should not have been allowed. Nonetheless, it is no longer possible and may be a breaking change for some).
• app.run() or Sanic.serve() must be inside of a if __name__ == "__main__" block

Impacted Issues

If you know of some other Issues that this touches not listed below, please LMK.

Closes #2534
Closes #2494
Closes #2467
Closes #2429
Closes #2364
Closes #2312
Closes #1471
Closes #1346

TODO

• Windows socket binding pattern
• Do not pre-bind socket in legacy mode
• Unit tests
• Move loop setup to global scope
• Expose API for passing args to processes
• Signal for reloader
• Worker introspection
• Move health module to Sanic Extensions

Technical discussion and overview: https://youtu.be/m8HCO8NK7HE

[ahopkins]

FYI - If you are looking at this PR an see the sanic.worker.health module, I plan to pull this out and move it to Sanic Extensions as an optional add-on. If you are looking at the PR and do not see it, then I already did.

[ahopkins]

See sanic-org/sanic-ext#111 for health monitoring implementation

[ahopkins]

I was thinking before! Kinda like peer-reviewing, so I (and hopefully others too) can add our pair of eyes to it :)

https://youtu.be/m8HCO8NK7HE

[sjsadowski]

I'm putting my chop on this. I was satisfied with the walkthrough and the Q&A, and I think with tests passing as best as can be expected, I'm ready for this to be merged.

[ahopkins]

@sjsadowski Nice. I am going to merge, which will unblock a few other items I want to get done this week.

[cnicodeme]

Do you have a date for this release? I'm curious and eager to move it to prod :)

[ahopkins]

Do you have a date for this release? I'm curious and eager to move it to prod :)

Likely Sunday (2022-09-26).

[cnicodeme]

Great, thank you :)