[BUG]: dd-trace-go/v2 internal/telemetry when used in AWS Lambda environment posts repeatedly to non-exist agent sidecar

[magnetikonline]

Tracer Version(s)

v2.1.0

Go Version(s)

go version go1.24.5 darwin/arm64

Bug Report

Hello team,

we've noted that since upgrading to dd-trace-go/v2 within our AWS Lambda functions, we're now getting a flood of log lines in the following form:

2025/07/25 01:04:59 Datadog Tracer v1.74.3 WARN: appsec: error while flushing SCA Security Data: Post "http://localhost:8126/telemetry/proxy/api/v2/apmtelemetry": dial tcp 127.0.0.1:8126: connect: connection refused

Digging into the code, it seems the internal/telemetry package, via the ddtrace/tracer package is setting up this telemetry client (via a startTelemetry() function) that's posting/flushing data to an endpoint of /telemetry/proxy/api/v2/apmtelemetry every 60 seconds at 127.0.0.1:8126.

It seems we could disable this regression by setting a DD_INSTRUMENTATION_TELEMETRY_ENABLED environment variable in our Lambda functions - but ideally it would be good if the internal/telemetry package and it's func Disabled() bool function could determine if we're running as a Lambda function (e.g. check for existence of the AWS_LAMBDA_FUNCTION_NAME env var) and silently disable the telemetry gathering.

Thansks.

Reproduction Code

No response

Error Logs

2025/07/25 01:04:59 Datadog Tracer v1.74.3 WARN: appsec: error while flushing SCA Security Data: Post "http://localhost:8126/telemetry/proxy/api/v2/apmtelemetry": dial tcp 127.0.0.1:8126: connect: connection refused

Go Env Output

No response

[eliottness]

👋 Thanks for reporting this. This seems like a symptom of a bigger issue in lambda because this warning suggest that no telemetry data is sent at all even though it should be. I agree that the warning is does not really reflect the issue at hand so we will rework it and maybe make it a debug log when necessary.

[eliottness]

My current theory is that you are using DD_API_KEY_SECRET_ARN (sorry if I misspelled) which we cannot transform into an plain text API key on our own and the datadog-lambda-go does not give it does us after it obtained it from the arn. In a sense the error is even more missleading because since we don't have an API key we hit an endpoint that does not exist because we assume to be in the standard setup where the host agent is available.

[magnetikonline]

Thanks @eliottness for getting back to me 👍

This seems like a symptom of a bigger issue in lambda because this warning suggest that no telemetry data is sent at all even though it should be. I agree that the warning is does not really reflect the issue at hand so we will rework it and maybe make it a debug log when necessary.

Agreed - making it a debug log would be a nice start (reduce noise in our logs) - but I still think this is suboptimal, as each Lambda function is still trying to HTTP post to an endpoint that does not exist - wasting time/IO.

My current theory is that you are using DD_API_KEY_SECRET_ARN (sorry if I misspelled) which we cannot transform into an plain text API key on our own and the datadog-lambda-go

We are very much not doing that. All our Lambda stdout/logging is pushed out to CloudWatch logs (as per the default with AWS Lambda) - we then have your Datadog log forwarder as a subscription to push logs out of CloudWatch logs -> Datadog logs API ingress.

As part of this - all the tracing data is currently magically converted into coded JSON blobs and pushed into CloudWatch logs over stdout and moved along by the forwarder.

[magnetikonline]

Looks like DataDog/datadog-lambda-go#206 (comment) is now going to address this.

[magnetikonline]

... or possibly now: #3943?