Skip to content

[Playground] OTLP-Mapper - instana to otlp span mapper#2534

Draft
abhilash-sivan wants to merge 129 commits into
mainfrom
chore-otel-span-mapper
Draft

[Playground] OTLP-Mapper - instana to otlp span mapper#2534
abhilash-sivan wants to merge 129 commits into
mainfrom
chore-otel-span-mapper

Conversation

@abhilash-sivan

@abhilash-sivan abhilash-sivan commented May 4, 2026

Copy link
Copy Markdown
Contributor

Summary

Introduces the Instana-to-OTLP Converter Playground.

  • Adds support for exporting traces via OTLP (4318/v1/traces).
  • Supports OpenTelemetry Semantic Conventions (SemConv) v1.23.

Out of Scope

  • Metrics export
  • Logs export

TODO

Core Implementation

  • Handle agent port usage (data port vs. connection port) (chore: make agent data port configurable independently from the agent connection port #2559)
  • Support multiple span data keys (e.g., span.data.peer, span.data.mongo)
  • Finalize overall design
  • Implement SemConv lookup handling
  • Implement span name mapping
  • Complete metadata attribute mapping:
  • Direct mappings
  • Computed mappings
  • Research span status mapping
  • Research span name mapping (@abhilash-sivan)
  • Implement OpenTelemetry instrumentation-specific handling
  • Implement log-specific handling
  • Implement resource mapping
  • Implement required metrics mappings
  • Implement agent connection logic
  • Error handling @abhilash-sivan
  • Resource-service name mapping
  • Dual transformation flow in span data @abhilash-sivan
  • Implement configuration option for enabling OTLP export (INSTANA_OTLP_ENABLED?) - NOT PART OF THIS

Span Data Mapping @abhilash-sivan

  • HTTP
  • Kafka
  • Databases
  • Cloud services
  • OpenTelemetry instrumentation

Testing

Unit Tests

  • Single-span scenarios
  • Multi-span scenarios

End-to-End Testing (@abhilash-sivan)

  • SemConv v1.23 validation with the current design
  • SemConv v1.41 compatibility validation
  • OpenTelemetry wrapper instrumentation testing
  • Metadata validation with SemConv v1.23 and span data validation with v1.43
  • without metrics - corelation issue

Performance Testing

  • Performance testing

Special Cases

  • Fix type definitions

Issue Tracking

Validation Checklist

Before merging, verify the following:

  • Traces are successfully exported through the OTLP endpoint (4318/v1/traces).
  • Metrics continue to be exported through the existing endpoint.
  • The UI correctly identifies and displays applications.

Future Enhancements/TODO

  • Support the agent announcement cycle disablement use case.
  • Evaluate exporting logs to the OTLP /v1/logs endpoint.
  • Implement metrics mapping.
  • Refactor the current dual transformation flow in span data (internal mapping and OTLP mapping).

References

@abhilash-sivan abhilash-sivan force-pushed the chore-otel-span-mapper branch from 4490ab0 to f1c0d6f Compare May 4, 2026 18:07
@abhilash-sivan abhilash-sivan changed the base branch from main to chore-otlp-playground May 7, 2026 06:42
@abhilash-sivan abhilash-sivan force-pushed the chore-otel-span-mapper branch from 53e1fcc to d261bc5 Compare May 7, 2026 12:45
Comment thread packages/collector/src/agentConnection.js
Comment thread packages/core/src/tracing/otlp_mapper/mapper.js Outdated
Comment thread packages/core/src/tracing/otlpTransformer.js Outdated
Comment thread packages/core/src/tracing/otlpTransformer.js Outdated
Comment thread packages/core/src/tracing/otlpTransformer.js Outdated
Comment thread packages/core/src/tracing/spanBuffer.js Outdated
@abhilash-sivan abhilash-sivan changed the title Chore otel span mapper chore: instana-otlp span mapper May 7, 2026
@abhilash-sivan abhilash-sivan changed the title chore: instana-otlp span mapper [Playground] OTLP-Mapper - instana to otlp span mapper May 8, 2026
Comment thread packages/collector/src/agentConnection.js Outdated
Comment thread packages/core/src/tracing/backend_mappers/index.js Outdated
Comment thread packages/core/src/tracing/otlp_mapper/mapper.js Outdated
@kirrg001

Copy link
Copy Markdown
Contributor

test: send traces via otlp port and metrics via standard port - Idenfies 2 diff apps 1. Otel and 2. Node.js

This needs to be communicated in the next WG. Traces & Metrics mapping is required.

@abhilash-sivan abhilash-sivan force-pushed the chore-otel-span-mapper branch 2 times, most recently from e827e9c to d6c01a6 Compare May 15, 2026 05:58
@abhilash-sivan abhilash-sivan changed the base branch from chore-otlp-playground to main May 18, 2026 05:18
@abhilash-sivan abhilash-sivan force-pushed the chore-otel-span-mapper branch 2 times, most recently from 74c5459 to b5a51c8 Compare May 18, 2026 07:25
Comment thread packages/core/src/tracing/OTLP_TRANSFORMATION_README.md Outdated
Comment thread packages/core/src/tracing/OTLP_TRANSFORMATION_README.md Outdated
Comment thread packages/core/src/tracing/OTLP_TRANSFORMATION_README.md Outdated
Comment thread packages/core/src/tracing/OTLP_TRANSFORMATION_README.md Outdated
Comment thread scripts/instana-to-otel-converter.js Outdated
Comment thread scripts/instana-to-otel-converter.js Outdated
Comment thread scripts/instana-to-otel-converter.js Outdated
Comment thread scripts/instana-to-otel-converter.js Outdated
* RabbitMQ-specific mappings
* Extends base messaging with RabbitMQ-specific fields
*/
rabbitmq: {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not verified rabbitmq per se. It's just to show how we differentiate children under messaging or any other group.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But for RabbitMQ, there’s nothing really common with BASE_MAPPINGS.messaging. I don’t think any of the others share much with it either, apart from Kafka.

Comment thread packages/core/src/tracing/converters/transformers.js Outdated
mappings: SPAN_ATTRIBUTE_MAPPINGS.kafka,
prefix: 'messaging.kafka',
additionalAttributes: {
'messaging.system': 'kafka'

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we need that here? If we have already a custom kafka mapping object?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

messaging.system is an additional attribute that we append to the span. For Kafka, the system name also happens to be kafka, so it may seem redundant at first glance.

However, additionalAttributes is intended for constant or computed attributes that do not directly map from a single field inside span.data.kafka.

messaging.system falls into that category since it is effectively a constant defined per transformer, rather than something extracted from the incoming span payload itself.

I have changed this to: 'messaging.system': this.systemName

which will fetch the appropriate value in this case

Comment thread packages/core/src/tracing/converters/instana-to-otel-converter-utils.js Outdated
Comment thread packages/core/src/tracing/converters/instana-to-otel-converter.js Outdated
Comment thread packages/core/src/tracing/converters/instana-to-otel-converter.js Outdated
Comment thread packages/core/src/tracing/converters/transformers.js Outdated
Comment thread packages/core/src/otlp/traces/mappers/helper.js Outdated
@aryamohanan aryamohanan force-pushed the chore-otel-span-mapper branch from e6f4c61 to 0b054c3 Compare June 16, 2026 15:17
Comment thread packages/core/src/otlp/common/transformers/resource.js Outdated
util.init(config);
util.hasThePackageBeenInitializedTooLate.activate();
secrets.init(config);
otlp.init(config);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we call the module exporters/otlpExporter?

otlp is a bit too short and semantically not easy to read.
WDYT. Choose

Comment thread packages/core/src/otlp/README.md Outdated
Comment thread packages/collector/src/agentConnection.js Outdated
Comment thread packages/collector/src/agentConnection.js Outdated

// Use dataPort for sending all telemetry data (metrics, spans, profiles, events, etc.)
// dataPort defaults to agentPort but can be configured separately for future use cases (e.g., OTel format)
const isOtlpEndpoint = path.startsWith('/v1/');

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is multiple checks for isOtlpX in this file. Lets have one place and just provide the data for the call.


downstreamConnection.sendMetrics(
payload,
{ otlpEnabled },

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

downstreamConnection knows that we use otlp, right?

@@ -0,0 +1,106 @@
/*
* (c) Copyright IBM Corp. 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need this app still?
Wondering if we can just have a separate npm script in the example-apps/collector app?

With config otlp being on.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can remove this after finishing the playground work, since the example app also includes Kafka and database tests that are only required for playground testing.

let options = {};
let callback;

if (typeof optionsOrCallback === 'function') {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this logic new? 🤔
Maybe lets revert this to its bare minimum for otlp

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah maybe it got added because of the provided config option. We can read the config from the init config object

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants