@opentelemetry/instrumentation-aws-lambda/AwsLambdaInstrumentation requestHook not working as expected.

[MutazAshhab]

What happened?

Here's the updated bug report with your actual tracing configuration:

Steps to Reproduce

1. Create a Node.js AWS Lambda function using the lambda-api framework
2. Configure OpenTelemetry with AWS Lambda auto-instrumentation using the following setup:

// config/tracing.ts
import { diag, DiagConsoleLogger, DiagLogLevel } from "@opentelemetry/api";
import { getNodeAutoInstrumentations } from "@opentelemetry/auto-instrumentations-node";
import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-http";
// import { AwsLambdaInstrumentation } from "@opentelemetry/instrumentation-aws-lambda"; dependency installed when needed and removed later
import { NodeSDK } from "@opentelemetry/sdk-node";

diag.setLogger(new DiagConsoleLogger(), DiagLogLevel.ERROR);

export const tracingSdk = new NodeSDK({
  traceExporter: new OTLPTraceExporter(),
  instrumentations: [
    getNodeAutoInstrumentations({
      "@opentelemetry/instrumentation-fs": {
        enabled: false,
      },
      "@opentelemetry/instrumentation-aws-lambda": {
        enabled: true,
        requestHook: (span, hookInfo) => {
          console.log("Lambda instrumentation created span:", span.spanContext());
          span.setAttribute("handler", hookInfo.context.functionName);
          console.dir({ span, hookInfo }, { depth: null });
        },
      },
      "@opentelemetry/instrumentation-mongodb": {
        enabled: true,
        enhancedDatabaseReporting: true,
      },
    }),
     // new AwsLambdaInstrumentation(), // Ive also tried this, when i tried 
     // it i had commented out the "@opentelemetry/instrumentation-aws-lambda" section in getNodeAutoInstrumentations
  ],
});

tracingSdk.start();

3. Import tracing first in main handler:

// CRITICAL: Import tracing FIRST before any other application imports
import { tracingSdk } from "./config/tracing";

import {
  APIGatewayProxyEvent,
  APIGatewayProxyResult,
  Context,
} from "aws-lambda";
import { api } from "./config/application";
import { MongodbClientService } from "./services/database/mongodb-client-service";
import { trace } from "@opentelemetry/api";

type RouterEvent = APIGatewayProxyEvent;

export const handler = async (
  event: RouterEvent,
  context: Context
): Promise<APIGatewayProxyResult> => {
  console.log(
    "🔍 STEP 1 - Handler entry:",
    trace.getActiveSpan() ? "Span exists" : "No span"
  );

  await MongodbClientService.connect();

  console.log(
    "🔍 STEP 2 - Before api.run():",
    trace.getActiveSpan() ? "Span exists" : "No span"
  );

  const result = await api.run(event, context);

  console.log(
    "🔍 STEP 3 - After api.run():",
    trace.getActiveSpan() ? "Span exists" : "No span"
  );

  tracingSdk.shutdown(); // Have tried with and without this line of code.
  return result;
};

4. Deploy to AWS Lambda and invoke the function

Expected Result

• AWS Lambda auto-instrumentation should automatically create a root span when the Lambda function is invoked
• trace.getActiveSpan() should return a valid span object both in the main handler
• The requestHook callback should be triggered and log span creation details
• Traces should be exported to the configured OTLP endpoint

Actual Result

• No root span is created by the AWS Lambda auto-instrumentation
• trace.getActiveSpan() returns undefined throughout the entire execution flow
• The requestHook callback is never triggered (no logs appear in CloudWatch)
• No traces are exported
• Both the auto-instrumentation configuration and explicit AwsLambdaInstrumentation instance fail to create a root span

CloudWatch logs show:

🔍 STEP 1 - Handler entry: No span
🔍 STEP 2 - Before api.run(): No span
🔍 STEP 3 - After api.run(): No span

Additional Details

Framework and Dependencies:

• lambda-api framework for handling HTTP routing in Lambda
• Node.js 20.x runtime in AWS Lambda
• TypeScript compilation
• Module type: CommonJS

Configuration Notes:

• Using both getNodeAutoInstrumentations with AWS Lambda instrumentation enabled AND an explicit AwsLambdaInstrumentation instance
• DiagLogLevel set to ERROR - no OpenTelemetry diagnostic logs appear
• MongoDB instrumentation works correctly when I manually create an active span

Environment:

• AWS Lambda with Node.js 20.x runtime
• Function invoked via API Gateway
• OpenTelemetry environment variables configured:
  • OTEL_SERVICE_NAME='application'
  • OTEL_EXPORTER_OTLP_PROTOCOL='http/protobuf'
  • OTEL_EXPORTER_OTLP_TRACES_ENDPOINT='https://api.honeycomb.io/v1/traces'
  • OTEL_EXPORTER_OTLP_HEADERS='x-honeycomb-team=API_KEY'

☝ data in honeycomb is present when i manually create an active span, im not facing a honey comb issue.

Behavior observations:

• The tracing module loads successfully
• OpenTelemetry SDK starts without errors
• Neither the auto-instrumentation nor explicit instrumentation instances trigger
• Manual span creation using tracer.startActiveSpan() works correctly and creates proper trace context
• Other auto-instrumentations (MongoDB) work as expected when manual root spans are present

Code execution flow:

1. AWS Lambda runtime invokes handler function
2. Handler performs setup operations (database connection, etc.)
3. Handler delegates to lambda-api via api.run(event, context)
4. lambda-api processes routing and calls controller methods
5. Throughout this entire flow, no automatic spans are created despite dual Lambda instrumentation configuration

Successful workaround:
Manual root span creation using tracer.startActiveSpan() in the handler successfully creates trace context that propagates correctly to all downstream operations including controller methods and MongoDB operations.

OpenTelemetry Setup Code

import { diag, DiagConsoleLogger, DiagLogLevel } from "@opentelemetry/api";
import { getNodeAutoInstrumentations } from "@opentelemetry/auto-instrumentations-node";
import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-http";
import { AwsLambdaInstrumentation } from "@opentelemetry/instrumentation-aws-lambda";
import { NodeSDK } from "@opentelemetry/sdk-node";

diag.setLogger(new DiagConsoleLogger(), DiagLogLevel.ERROR);

export const tracingSdk = new NodeSDK({
  traceExporter: new OTLPTraceExporter(),
  instrumentations: [
    getNodeAutoInstrumentations({
      "@opentelemetry/instrumentation-fs": {
        enabled: false,
      },
      "@opentelemetry/instrumentation-aws-lambda": {
        enabled: true,
        requestHook: (span, hookInfo) => {
          console.log("Lambda instrumentation created span:", span.spanContext());
          span.setAttribute("handler", hookInfo.context.functionName);
          console.dir({ span, hookInfo }, { depth: null });
        },
      },
      "@opentelemetry/instrumentation-mongodb": {
        enabled: true,
        enhancedDatabaseReporting: true,
      },
    }),
    new AwsLambdaInstrumentation(), // Explicit additional instance
  ],
});

tracingSdk.start();

package.json

{
  "name": "demo-lambda",
  "version": "0.1.0",
  "private": true,
  "sideEffects": false,
  "scripts": {
    "test": "jest --passWithNoTests",
    "build": "node esbuild-builder.js",
    "local": "node ./node_modules/dotenv-cli/cli.js -e ../../.env.overrides -e ../../.env -- ts-node ./src/local.ts",
    "lint:fix": "node ./node_modules/prettier/bin/prettier.cjs --config ../../.prettierrc --write ./src/"
  },
  "unbundledModules": [],
  "engines": {
    "node": ">=20.0.0"
  },
  "dependencies": {
    "aws-lambda": "~1.0.7",
    "lambda-api": "~1.1.0",
    "mongodb": "~6.16.0",
    "@aws-sdk/client-s3": "~3.821.0",
    "@aws-sdk/s3-request-presigner": "~3.821.0",
    "@aws-sdk/client-cognito-identity-provider": "~3.821.0",
    "openai": "~5.3.0",
    "joi": "~17.13.3",
    "@opentelemetry/auto-instrumentations-node": "~0.60.1",
    "@opentelemetry/exporter-trace-otlp-http": "~0.202.0",
    "@opentelemetry/sdk-node": "~0.202.0",
    "@opentelemetry/api": "~1.9.0"
  },
  "devDependencies": {
    "@types/jest": "^29.5.12",
    "@types/node": "18.15.3",
    "jest": "^29.7.0",
    "ts-jest": "^29.1.2",
    "ts-node": "^10.9.2",
    "typescript": "^5.4.2",
    "esbuild": "0.24.0",
    "rimraf": "~6.0.1",
    "@types/aws-lambda": "~8.10.146",
    "dotenv-cli": "~7.2.1",
    "prettier": "~3.5.3"
  }
}

Relevant log output

In cloudwatch when I run with otel debug level diagnostics i get this suspicious log

2025-06-25T02:51:46.740Z	undefined	WARN	@opentelemetry/instrumentation-aws-lambda Module /var/task/main.js has been loaded before @opentelemetry/instrumentation-aws-lambda so it might not work, please initialize it before requiring /var/task/main.js

Operating System and Version

nodejs:20.v65 lambda runtime

Runtime and Version

nodejs:20.v65 lambda runtime

[dyladan]

Transferred to contrib because this seems to be a bug with the AWS lambda instrumentation.

@jj22ee can you please take a look at this?

[dyladan]

@jj22ee I assigned this P2 as it does seem to be a bug. Please re-triage if necessary

[jj22ee]

It may help if you also provide esbuild-builder.js. However, I did a quick test with my own:

import * as esbuild from 'esbuild'

await esbuild.build({
  entryPoints: ['ts-index.ts'],
  outfile: 'index.js',
  format: 'cjs'
})

await esbuild.build({
  entryPoints: ['ts-tracing.ts'],
  outfile: 'tracing.js',
  format: 'cjs'
})

I also had to update handler method to avoid some esbuild issue (TypeError - "Cannot redefine property: handler"):

const handler = async (event) => {
  // Handler Code
}
module.exports = { handler }

I also commented out await MongodbClientService.connect(); and const result = await api.run(event, context);

It seems that having the import at the top of the file is not enough for specifically AWS Lambda Instrumentation to work:

// CRITICAL: Import tracing FIRST before any other application imports
import { tracingSdk } from "./config/tracing";

Workaround

The workaround is to follow a similar approach to the Node.js OTel Lambda Layer.

I did a quick test, what you can do is "require" the "tracingSdk" even before the index.js and the associated Handler are run (this seems necessary specifically for Aws Lambda Instrumentation).

Just follow these steps:

1. First, create a wrapper shell script wrapper.sh:

#!/bin/bash
export NODE_OPTIONS="--require ${LAMBDA_TASK_ROOT}/tracing.js"
exec "$@"

(Make the wrapper script executable: chmod +x wrapper.sh)

2. Set the AWS_LAMBDA_EXEC_WRAPPER environment variable in your Lambda configuration to run wrapper.sh:

AWS_LAMBDA_EXEC_WRAPPER=/var/task/wrapper.sh

My output is then:

2025-07-08T21:20:05.671Z	undefined	WARN	AWS Lambda plans to remove support for callback-based function handlers starting with Node.js 24. You will need to update this function to use an async handler to use Node.js 24 or later. For more information and to provide feedback on this change, see https://github.com/aws/aws-lambda-nodejs-runtime-interface-client/issues/137. To disable this warning, set the AWS_LAMBDA_NODEJS_DISABLE_CALLBACK_WARNING environment variable.
START RequestId: 1beb9b96-4d10-47e3-a17e-950c5016f7fa Version: $LATEST
2025-07-08T21:20:05.710Z	1beb9b96-4d10-47e3-a17e-950c5016f7fa	INFO	🔍 STEP 1 - Handler entry: Span exists
2025-07-08T21:20:05.711Z	1beb9b96-4d10-47e3-a17e-950c5016f7fa	INFO	🔍 STEP 2 - Before api.run(): Span exists
2025-07-08T21:20:05.711Z	1beb9b96-4d10-47e3-a17e-950c5016f7fa	INFO	🔍 STEP 3 - After api.run(): Span exists
END RequestId: 1beb9b96-4d10-47e3-a17e-950c5016f7fa
REPORT RequestId: 1beb9b96-4d10-47e3-a17e-950c5016f7fa	Duration: 70.32 ms	Billed Duration: 71 ms	Memory Size: 128 MB	Max Memory Used: 101 MB	Init Duration: 787.71 ms	

My reproduction of your project looked like:

├── index.js
├── tracing.js
└── wrapper.sh

I am guessing the issue is that "AWS Lambda Instrumentation" is for some reason unable to patch the "index.js" if the "index.js" file is initializing the tracing and contains the Function Handler.

[MutazAshhab]

Thanks @jj22ee I had gotten the same TypeError - "Cannot redefine property: handler" error and did the same workaround, I had forgotten to include that in the issue.

As for my esbuild config, here it is

const esbuild = require("esbuild");
const { join } = require("path");
const { exit } = require("process");
const rimraf = require("rimraf");

async function build() {
  const bundlePath = join(__dirname, "dist");
  rimraf.sync(bundlePath);

  await esbuild
    .build({
      entryPoints: ["src/main.ts"],
      bundle: true,
      platform: "node",
      target: "node20",
      outfile: join(bundlePath, "main.js"),
      sourcemap: true,
      minify: true,
      minifyWhitespace: true,
      minifySyntax: true,
    })
    .catch(() => exit());
}

build();

I gave your workaround a shot and it works for me. As you can see @opentelemetry/instrumentation-aws-lambda applies its instrumentation with no warning.

2025-07-13T03:36:39.307Z	undefined	DEBUG	@opentelemetry/instrumentation-aws-lambda Applying instrumentation patch for nodejs module file on require hook {
  module: '/var/task/main.js',
  version: '0.1.0',
  fileName: 'main',
  baseDir: '/var/task'
}

For reference this is the wrapper.sh that I've added

#!/bin/bash
export NODE_OPTIONS="--require ${LAMBDA_TASK_ROOT}/tracing.js"

if [[ $OTEL_RESOURCE_ATTRIBUTES != *"service.name="* ]]; then
  export OTEL_RESOURCE_ATTRIBUTES="service.name=${AWS_LAMBDA_FUNCTION_NAME},${OTEL_RESOURCE_ATTRIBUTES}"
fi

if [[ -z "$OTEL_PROPAGATORS" ]]; then
  export OTEL_PROPAGATORS="tracecontext,baggage,xray"
fi

exec "$@"

With the following output:

dist
├── main.js
├── main.js.map
├── package.json
├── tracing.js
└── wrapper.sh

---

However, I now have other problems that are not directly related to the aws lambda instrumentation.

There are now 2 new error logs:

error reading machine id: Error: ENOENT: no such file or directory, open '/etc/machine-id'
error reading machine id: Error: ENOENT: no such file or directory, open '/var/lib/dbus/machine-id'

Spans/metrics not being sent.

I did some digging on the flushing log and found this issue @dyladan I saw this comment of yours but it doesnt seem to have fixed this?

I've also added to the end of the tracing.ts file but that has not worked as well.

process.on("SIGTERM", async () => {
  await tracingSdk.shutdown();
  process.exit(0);
});

Any thoughts?