Issue with @opentelemetry/instrumentation-aws-sdk when tracing sqs and tracestate is empty

[bradleyquinn140]

What version of OpenTelemetry are you using?

As far as I can see I'm using the latest version(s)

"@opentelemetry/auto-instrumentations-node": "^0.37.0",
 "@opentelemetry/instrumentation": "^0.39.1",
 "@opentelemetry/sdk-node": "^0.39.1",

What version of Node are you using?

v16.20.0

What did you do?

1. Have a trace where there is no tracestate (ie the internal representation is an empty map).
2. Make a request to SQS with @opentelemetry/instrumentation-aws-sdk enabled.

What did you expect to see?

No error messages from the aws sdk

What did you see instead?

Message (user) attribute 'tracestate' must contain a non-empty value of type 'String'.

InvalidParameterValue: Message (user) attribute 'tracestate' must contain a non-empty value of type 'String'.
    at Request.extractError (/opt/app/node_modules/aws-sdk/lib/protocol/query.js:50:29)
    at Request.callListeners (/opt/app/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
    at Request.emit (/opt/app/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
    at Request.emit (/opt/app/node_modules/aws-sdk/lib/request.js:686:14)
    at Request.transition (/opt/app/node_modules/aws-sdk/lib/request.js:22:10)
    at AcceptorStateMachine.runTo (/opt/app/node_modules/aws-sdk/lib/state_machine.js:14:12)
    at /opt/app/node_modules/aws-sdk/lib/state_machine.js:26:10
    at Request.<anonymous> (/opt/app/node_modules/aws-sdk/lib/request.js:38:9)
    at Request.<anonymous> (/opt/app/node_modules/aws-sdk/lib/request.js:688:12)
    at Request.callListeners (/opt/app/node_modules/aws-sdk/lib/sequential_executor.js:116:18)
    at Request.emit (/opt/app/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
    at Request.emit (/opt/app/node_modules/aws-sdk/lib/request.js:686:14)
    at Request.transition (/opt/app/node_modules/aws-sdk/lib/request.js:22:10)
    at AcceptorStateMachine.runTo (/opt/app/node_modules/aws-sdk/lib/state_machine.js:14:12)
    at /opt/app/node_modules/aws-sdk/lib/state_machine.js:26:10
    at Request.<anonymous> (/opt/app/node_modules/aws-sdk/lib/request.js:38:9)
    at Request.<anonymous> (/opt/app/node_modules/aws-sdk/lib/request.js:688:12)
    at Request.callListeners (/opt/app/node_modules/aws-sdk/lib/sequential_executor.js:116:18)
    at callNextListener (/opt/app/node_modules/aws-sdk/lib/sequential_executor.js:96:12)
    at IncomingMessage.onEnd (/opt/app/node_modules/aws-sdk/lib/event_listeners.js:417:13)
    at /opt/app/node_modules/@opentelemetry/context-async-hooks/build/src/AbstractAsyncHooksContextManager.js:50:55
    at AsyncLocalStorage.run (node:async_hooks:319:14)

Additional context

I'm not sure if the issue is in this package or in the tracestate implementation (if I'm wrong sorry, I'll raise it in open-telemetry/opentelemetry-js)

It seems like sqs does not allow empty attributes ... from here

Important
All components of a message attribute are included in the 256 KB message size restriction.
The Name, Type, Value, and the message body must not be empty or null.

Which doesn't seem like it will work with the traceState implementation eg

createTraceState().serialize() // == ""

[pichlermarc]

@carolabadeer would you mind looking into this one?

[carolabadeer]

Hi @bradleyquinn140, thanks for raising this issue. I will work on reproducing it on my end and post my findings here

[bradleyquinn140]

Thanks @carolabadeer have you had any luck with this?

[carolabadeer]

@bradleyquinn140 I tried reproducing the issue but couldn't get to a point where I see this error on my end, can you please provide a simple reproduction? Also curious, what's the use case for having an empty trace state? Does this only happen with SQS?

[yvd]

After trying quite a few workarounds, I was able to fix this issue by using preRequestHook option in the aws sdk instrumentation. For all AWS SQS requests that are being traced, have a non empty trace state by default:

/* instrumentation.ts */
import { NodeSDK } from '@opentelemetry/sdk-node'
import { Resource } from '@opentelemetry/resources'
import { SemanticResourceAttributes } from '@opentelemetry/semantic-conventions'
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-proto'
import { getNodeAutoInstrumentations } from '@opentelemetry/auto-instrumentations-node'

import { AwsInstrumentation, AwsSdkRequestHookInformation } from '@opentelemetry/instrumentation-aws-sdk'
import { TraceState } from '@opentelemetry/core'

const awsInstrumentationConfig = {
	preRequestHook: (span: Span, requestInfo: AwsSdkRequestHookInformation) => {
		if (requestInfo.request.serviceName == 'SQS') {
			span.spanContext().traceState = new TraceState('c=d')
		}
	},
}

const sdk = new NodeSDK({
	resource: new Resource({
		[SemanticResourceAttributes.SERVICE_NAME]: process.env.OTEL_SERVICE_NAME,
		[SemanticResourceAttributes.SERVICE_VERSION]: '1.0',
	}),
	traceExporter: new OTLPTraceExporter({
		url: process.env.OTEL_EXPORTER_OTLP_TRACES_ENDPOINT,
		headers: {},
	}),
	instrumentations: [getNodeAutoInstrumentations(), new AwsInstrumentation(awsInstrumentationConfig)],
})
sdk.start()

Hope this helps someone :)

[ray58750034]

Hello experts,

We've encountered this issue in our project. We use both New Relic and OpenTelemetry together. When New Relic receives an invalid tracestate, it modifies the tracestate and sets the value to an empty string, which leads to this error.

From my understanding, I think it would be safer if the instrumentation-aws-sdk could check for an empty value before setting it to MessageAttribute here.

@carolabadeer @pichlermarc