@opentelemetry/instrumentation-aws-lambda should read API gateway parameters by default

[lizthegrey]

Is your feature request related to a problem? Please describe

Invokes create a root span, but populate only details about the lambda itself (e.g. /usr/bin/node, lambda runtime version, function name) rather than about the request being performed. When the lambda is being called directly from API gateway, these details should automatically be populated, rather than relying upon telemetry from API gateway to have been also plumbed up and amazon trace header used for correlation.

Describe the solution you'd like to see

the following event.input (but not event.body, to avoid PII leaks) values should be automatically extracted and put into attributes (essentially as a default requestHook); example from https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-create-api-as-simple-proxy-for-lambda.html

  "input":{
    "resource":"/helloworld",
    "path":"/helloworld",
    "httpMethod":"POST",
    "headers": {
      "Accept":"*/*",
      "content-type":"application/json",
      "Host":"r275xc9bmd.execute-api.us-east-1.amazonaws.com",
      "User-Agent":"curl/7.64.0",
      "X-Amzn-Trace-Id":"Root=1-1a2b3c4d-a1b2c3d4e5f6a1b2c3d4e5f6",
      "X-Forwarded-For":"72.21.198.64",
      "X-Forwarded-Port":"443",
      "X-Forwarded-Proto":"https"
    },
    "queryStringParameters":{
      "city":"Seattle",
      "name":"John"
    },
  },

Describe alternatives you've considered

These attributes could be left on an API gateway span, except there's no easy way to produce OTel formatted spans out of API gateway logs, and even then, API gateway only speaks amazon trace header, not w3c tracecontext, so you'd need to use amazon propagator.

Additional context

tagging component owners @willarmiros @NathanielRN and tagging Honeycomb PM @cartermp
tagging users who encountered this problem and showed me: @longility @nikordaris @meleksomai

[github-actions]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 14 days.

[lizthegrey]

still is a problem/footgun.

[NathanielRN]

Tagging @vasireddy99 and @bryan-aguilar to triage this for investigation.

[dyladan]

This is not stale. Tagging component owners @willarmiros @NathanielRN

[github-actions]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 14 days.

[cartermp]

Still not stale. Happy to contribute a fix myself, just unsure where I'd access the kind of information that we should be attaching to the created span.

[willarmiros]

Hi @cartermp,

It seems these additional attributes for APIGW-invoked Lambdas would have to be added in the Lambda instrumentation around here:

opentelemetry-js-contrib/plugins/node/opentelemetry-instrumentation-aws-lambda/src/instrumentation.ts

Line 166 in 4a9442c

const span = plugin.tracer.startSpan(

You would need to parse the event used to invoke the Lambda to determine if it is an APIGW event, then record the desired parameters if so.

[NPellet]

@willarmiros

In case the lambda is triggered from the Gateway API, should the span status be set to Status.Error if the lambda is returning a statusCode != 2XX ? (See https://docs.aws.amazon.com/apigateway/latest/developerguide/handle-errors-in-lambda-integration.html)

We have Sentry that's catching our thrown exceptions and returning a proper 500, but in that case, the lambda itself doesn't really fail, hence the span is not errored, while we receive a Sentry alert.

Just wondering