feat(azure-functions): support Azure Functions programming model v4

[trentm]

See https://learn.microsoft.com/en-ca/azure/azure-functions/functions-node-upgrade-v4
This builds on the work in #4178
by @qzxlkj (type that 5 times fast), which provides most of the runtime code fix.

The rest of this is adding testing and updated examples and docs.

Refs: #4178
Closes: #3185

checklist

• example update
• docs update
• changelog

[trentm]

tl;dr: Distributed tracing for incoming requests to Azure Functions using their Node.js programming model v4 does not work. This is due to a bug in Azure Functions that results in InvocationContext.traceContext not being usable.
(This instrumentation is choosing Option B, described below.)

distributed tracing issues with Azure Functions

Azure Functions currently has behaviour that breaks distributed tracing. The following describes the behaviour with no telemetryMode setting in the Azure Function application's "host.json" file.

Instrumentation of Azure Functions is done via the invocation hooks mechanism. The InvocationContext passed to the preInvocation hook includes a traceContext with W3C trace-context data. Its behaviour is:

1. If the incoming request (e.g. an HTTP request for an HTTP-triggered function) includes trace-context, e.g.:

   traceparent: 00-12345678901234567890123456789012-1234567890123456-01
   

   then Azure internally creates a child span resulting in

   traceContext.traceParent = '00-12345678901234567890123456789012-48909501584ddaed-01'
   

   Notice that the parent-id has changed. The internal span (id=48909501584ddaed) cannot be collected, so the result is a broken trace.

2. If the incoming request does not have trace-context, then Azure internally creates a span in a new trace, something like:

   traceContext.traceParent = '00-deed8e23efe326408bca77616654d445-ea4f834079f1596f-00'
   

   Notice that the sampled flag is false. This results in the whole trace being sampled out.

See the following for this issue having been reported:
Azure/azure-functions-dotnet-worker#2733 (comment)
Azure/azure-functions-host#10641

options

To cope with this at least partly-unusable trace-context:

First, this instrumentation prefers to directly use the trace-context from the incoming HTTP headers, if any.
HTTP-triggered Azure Functions have a request object with the headers.
However the issue is whether those headers are available to the preInvocation hook.
In the now-old v3 Node.js programming model the InvocationContext includes the HTTP request object, so the headers are accessible.
In the v4 Node.js programming model this is no longer the case.

(Dev Note: Future work could possibly monkey-patch app.http from @azure/functions to access the incoming request headers. This would likely mean changing away from using the hooks mechanism at all for instrumentation.)

Option A: use invocationContext.traceContext as a fallback

If we use invocationContext.traceContext used as a fallback, then we need to contend with the two issues above.

There is no coping with issue 1, AFAIK. (It is possible with Azure Functions' current and coming OTel integration that this internal span would be accessible for ingest.)

We could cope with issue 2 by strongly encouraging the usage of the traceContinuationStrategy: 'restart_external' APM agent config (doc link). The result is a still a broken distributed trace (because of issue 1: the missing internal span), but at least a sane sampling decision will be made.

The Elastic APM UI trace view shows:

• a span link for the ignore internal span context; and
• shows a Incomplete trace warning about the distributed trace being incomplete.

Option B: don't use invocationContext.traceContext

The experience with Option A is still poor for the user. It is clearer to just not use the often completely- and always partial-broken traceContext and document for the user that distributed tracing incoming to Azure Functions just doesn't work because of the issues linked to above.

[trentm]

For the record, though we are not using it, this is how Option A above is implemented:

% diff -u azure-functions.js azure-functions.js.option-a
--- azure-functions.js	2025-01-16 13:31:47
+++ azure-functions.js.option-a	2025-01-16 13:31:34
@@ -423,6 +423,14 @@
       log.trace(
         { traceparent, tracestate },
         'azure-functions: get trace-context from HTTP trigger request headers',
+      );
+    }
+    if (!traceparent && context?.traceContext?.traceParent) {
+      traceparent = context.traceContext.traceParent;
+      tracestate = context.traceContext.traceState;
+      log.trace(
+        { traceparent, tracestate },
+        'azure-functions: get trace-context from invocationContext.traceContext',
       );
     }

[trentm]

Also for the posterity, this is what the azfunc example looked like when the "Hi" function called the "Bye" function before returning:

const { app } = require('@azure/functions');

app.http('Hi', {
  methods: ['GET'],
  authLevel: 'anonymous',
  handler: async (request, _context) => {
    // Call `GET /api/Bye` on this same Azure Function.
    const url = new URL(request.url);
    url.pathname = '/api/Bye';
    const byeRes = await fetch(url);
    const byeBody = await byeRes.json();

    const body = JSON.stringify({
      hi: 'there',
      'from /api/Bye call': byeBody
    });
    return {
      status: 200,
      headers: {
        'Content-Type': 'application/json',
        'Content-Length': Buffer.byteLength(body)
      },
      body
    };
  },
});

[trentm]

Some screenshots of tracing data from requests to the example app deployed to Azure:

% curl -i https://trentm-example-azure-function-app.azurewebsites.net/api/hello
HTTP/2 200
content-type: application/json
date: Thu, 16 Jan 2025 21:56:07 GMT
content-length: 80
request-context: appId=cid-v1:333aa370-882c-4ebf-b6fb-a2e8cc533807

{"hello":"world","current time in Vancouver":"2025-01-16T13:56:07.819933-08:00"}

% curl -i https://trentm-example-azure-function-app.azurewebsites.net/api/hello
HTTP/2 500
date: Thu, 16 Jan 2025 21:56:13 GMT
content-length: 0
request-context: appId=cid-v1:333aa370-882c-4ebf-b6fb-a2e8cc533807

Here the HTTP request made inside the Azure Function timed out.

[trentm]

Reviewer note: I'm also dropping support for the now long-ancient 'elastic-apm-traceparent' header. This is obsolete.

[trentm]

Credit: this block is directly from #4178
This is the bulk of the fix for supporting Azure Functions v4

[trentm]

Reviewer note: This is a direct copy of some of the functions that were in azure-functions.test.js before it was broken into two separate test files for v3 and v4.

[trentm]

Reviewer note: Testing azure functions involves npm installing the ridiculously large azure-functions-core-tools dep. To limit the insanity I've limited testing of v3 to Node.js 18 and v4 to Node.js 20.

[trentm]

Reviewer note: This and all the remaining diffs in this file are from renaming the fixture: s/AJsAzureFnApp/azfunc3/.

[trentm]

Reviewer note: This file is an almost exact copy of azure-functions-v3.test.js, except it tests the separate fixtures/azfunc4, and it doesn't test as many scenarios/edge cases because I didn't think that is worth a day or two of added test writing at this point.

There could likely be more code sharing between these two files, but again, I didn't want to blow too much time perfecting this.