[Epic] Allow for extensions to add custom RPC endpoints for out-of-proc workers

[jviau]

Request

As our focus on out-of-proc workers grows, there is an increasing need for extensions to have their own RPC communication from worker back to the host. This can range from simple sending of notifications from worker to host, to full-on call-response pattern from gRPC, HTTP, or other methods.

Examples:

1. Durable Functions: the various durable clients used in each language communicate back to the host over either HTTP (JS, python) or gRPC (dotnet isolated, java). Today durable manages creating local server itself and sends the server details as part of input binding. The limitation here is that server details are not known at startup, so the clients can only be used as part of input binding (and not DI'd like in-proc).
   • Change Grpc port microsoft/durabletask-dotnet#145
2. Service Bus - @JoshLove-msft can you expand on your needs here?

Proposal

The proposal is to allow for extensions to register their own AspNetCore Endpoints onto the existing server we use to facilitate host to worker RPC. This will give extensions full control over their RPC implementation details - be it HTTP, gRPC, or whatever they desire. On the worker side, the server URL is known, so the extension can supply a corresponding client however they need - either add it to DI services (for dotnet), or resolve it as part of input binding.

For dotnet isolated worker specifically, we will provide some method for retrieving a Grpc.Core.Api.ChannelBase from the IServiceProvider (either through a factory or options pattern). This will allow extenders to construct their client with a channel already built to communicate back to the host.

Tasks

Preview Give feedback

1. Map WebJobs RPC extensions onto the gRPC host #9291
   +0
   
2. Add RPC extension packages and abstractions azure-webjobs-sdk#2985
   +0
   
3. API for extensions to access a gRPC ChannelBase to communicate back to host with azure-functions-dotnet-worker#1563
   +0
   

[daviburg]

Adding an example for our built-in SAP capabilities:

• We are using out-of-proc isolated worker as our main proc is .NET Core while SAP client library is .NET Framework only.
• SAP client library has a callback pattern; we register for callback at app start-up. We receive the callback on an arbitrary thread in the isolated worker, and need to pass the callback payload to the main .NET Core proc (where it will be an Azure Functions trigger).
• Later an asynchronous response will come to the .NET Core for the request from SAP. We will pass this response for the callback to the isolated worker. The isolated worker needs to pass that response on the original thread where SAP called (the SAP client library does not support .NET async-await pattern).

[jviau]

@daviburg it sounds like your SAP feature has a third NetFx process which your extension manages. Are you hoping to leverage this feature for commnication with that 3rd process? If so, this feature is not designed with that in mind - it is designed for communication between the host process and worker process. And in out-of-proc workers the host process is .NET6.0 currently.

Now, this doesn't stop you from having that third process and managing your own RPC mechanism with it. This feature just won't help with that, unless you do have a need for custom functions host <--> functions worker RPC.

[daviburg]

@fabiocav could you help us here for isolated language worker (.NET Framework)? The isolated language worker is a Functions feature, we thought the RPC communication was meant to help with that.

[jviau]

@daviburg we can sync on this. I may need more context to understand what your scenario is.

[fabiocav]

@jviau netx is supported as a worker as well, and that's what LA uses. This wouldn't impact the WebJobs/host extensions targets, but the worker would need the ability to target netfx.

Let's indeed have a sync on this.

[jviau]

We have synced - the netfx-only specifics threw me off and I thought there was an extension-owned independent netfx process involved here. But @daviburg has clarified this is just the dotnet isolated worker targeting netfx. This feature will work for that scenario.