Datadog plugin integration errors

[judehunter]

I tried using this plugin with the serverless-datadog-plugin, but I get errors when EventBridge tries to invoke the function.
Presumably because of the fact that the datadog plugin installs and attaches a Lambda Layer to the function, which the warmup function has troubles with? Not sure though.
The error is:

{
    "errorType": "Runtime.ImportModuleError",
    "errorMessage": "Error: Cannot find module 'handler'\nRequire stack:\n- /var/runtime/UserFunction.js\n- /var/runtime/index.js",
    "stack": [
        "Runtime.ImportModuleError: Error: Cannot find module 'handler'",
        "Require stack:",
        "- /var/runtime/UserFunction.js",
        "- /var/runtime/index.js",
        "    at _loadUserApp (/var/runtime/UserFunction.js:100:13)",
        "    at Object.module.exports.load (/var/runtime/UserFunction.js:140:17)",
        "    at Object.<anonymous> (/var/runtime/index.js:43:30)",
        "    at Module._compile (internal/modules/cjs/loader.js:1015:30)",
        "    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1035:10)",
        "    at Module.load (internal/modules/cjs/loader.js:879:32)",
        "    at Function.Module._load (internal/modules/cjs/loader.js:724:14)",
        "    at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:60:12)",
        "    at internal/main/run_main_module.js:17:47"
    ]
}

Removing the datadog plugin from the plugin list in serverless.yml fixes the problem entirely.
I'm curious to hear if others had problems with this as well, or if maybe there's something wrong with my configuration.

My warmup setup is pretty barebones, with enabled: true, prewarm: true and schedule set to every 5 minutes.

"serverless": "^2.11.0",
"serverless-plugin-warmup": "^4.9.0",
"serverless-plugin-datadog": "^2.6.0"

[juanjoDiaz]

Hi @judehunter ,

In which order are you setting up the plugins?
serverless-plugin-warmup adds an extra function to your service, the warmer. This is a standard function configured to be triggered by a scheduled CloudWatch event.
serverless-plugin-datadog seems to be adding a layer to each handler and wrapping each handler in their own.

So serverless-plugin-warmup should be declared before serverless-plugin-datadog.
The fact that the warmer function is erroring, seems to point that you are already using the right order.

I took a quick look at the serverless-plugin-datadog codebase and everything else seems fine.

So other than that, I'd need to get an example project to be able to debug and see what's going on.
And considering that the lambdas don't seem OSS, I'm not sure of how to do that.
You might want to try asking for support on the serverless-plugin-datadog end.

[nholbrook]

@judehunter The serverless-datadog-plugin by default overrides the handlers of all functions if you have the addLayers flag enabled with their plugin. A solution to this has been merged into the serverless-plugin-datadog repo, which allows you to manually exclude functions from this default behavior, and should be included in it's next release. @juanjoDiaz You can feel free to close this issue because it isn't relevant to this plugin (DataDog/serverless-plugin-datadog#87).

[d-corler]

ℹ To help those who need an immediate response

If you use the default configuration, add this in your Datadog configuration :

custom:
  datadog:
    exclude: 
      - warmUpPluginDefault

[astuyve]

Hey folks! I maintain the Datadog Serverless Plugin and can help provide some insight here.

For Datadog's instrumentation to work, we configure a function with a few environment variables, apply our layers, and modify the handler path.

This library creates a warmer function, which the Datadog plugin can modify, however it seems those settings are being reverted back. The end result is an unusable warmer function, which keeps the handler configuration, but removes the added environment variables and removes the layers:

"WarmUpPluginOfficeHoursWarmerLambdaFunction": {
      "Type": "AWS::Lambda::Function",
      "Properties": {
        "Code": {
          "S3Bucket": {
            "Ref": "ServerlessDeploymentBucket"
          },
          "S3Key": "serverless/warmup-test/dev/1682606318668-2023-04-27T14:38:38.668Z/warmUpPluginOfficeHoursWarmer.zip"
        },
        "Handler": "/opt/nodejs/node_modules/datadog-lambda-js/handler.handler",
        "Runtime": "nodejs18.x",
        "FunctionName": "warmup-test-dev-warmup-plugin-officeHoursWarmer",
        "MemorySize": 128,
        "Timeout": 10,
        "Description": "Serverless WarmUp Plugin (warmer \"officeHoursWarmer\")",
        "Tags": [
          {
            "Key": "dd_sls_plugin",
            "Value": "v5.28.0"
          }
        ],
        "Environment": {
          "Variables": {
            "DD_LAMBDA_HANDLER": ".warmup/officeHoursWarmer/index.warmUp"
          }
        },
        "Role": {
          "Fn::GetAtt": [
            "WarmUpPluginOfficeHoursWarmerRole",
            "Arn"
          ]
        },
        "Layers": []
      },

This leaves the warmer function in a bad state.

A user reached out via our support and wanted to specifically monitor logs from the warmer function, so I dug into this library and realized we could modify this behavior a few ways - however I am unsure of the side effects.

The fastest fix is to return early if this library detects the warmer function already exists: https://github.com/juanjoDiaz/serverless-plugin-warmup/blob/main/src/warmer.js#L216

function addWarmUpFunctionToService(service, warmerName, warmerConfig) {
 if (service.functions[`warmUpPlugin${capitalize(warmerName)}`] !== undefined) {
   return;
 }

Alternatively, the addWarmUpFunctionToService could merge/splat existing args for layers, environment variables, and handler.

I tested this change and confirmed Datadog can receive logs, metrics, traces, and profile data from the warmer:

@juanjoDiaz - is there a preference that you have? I'd be happy to investigate/open a PR

[sfirrin]

Hi @juanjoDiaz can you take a look at @astuyve's comment and help us figure out a direction for a fix? We have been contacted by another customer with a similar issue, we would happy to open a PR to get it resolved for them.

[juanjoDiaz]

Hi @astuyve,

Thanks for the good work.

This is a tricky one...
There is a reason why the warmers are set twice.
If you browser through this repo you can find quite a few issues related to other plugins (webpack, rollup, etc..).
The problem is always the same: not all of us use the same lifecycle events.
I asked long time ago the serverless guidance to document the lifecycle events and provide better guidance for us the plugin authors.

The real issue comes from here:

serverless-plugin-warmup/src/index.js

Line 75 in c9ee9fa

'before:package:createDeploymentArtifacts': this.resetWarmerConfigs.bind(this),

The warmup lambdas are added on the after:package:initialize event. Which is the same event used by the datadog plugin.
However, the webpack, bundle and similar plugins use the before:package:createDeploymentArtifacts event and break things.
For that, I need to reset the warmers during the before:package:createDeploymentArtifacts event.
I wish I could get rid of that hack but then I'd break the library for all people bundling their JS.

Ideally, we all should use the same event...
Sadly, that seems unlikely.

I am happy to hear suggestions.
And I am happy to accept any PR as long as it still works with the webpack, bundle and similar plugins :sli

[juanjoDiaz]

#338 is exactly about that.
Let's discuss there.

Another option is that you implement double events in the datadog plugin like I do here to avoid issues.

[astuyve]

Yup, sounds good - we're happy to implement whatever is easiest and have similar challenges with bundlers.