Document existing OTel support integrated into the Java agent

[jasonjkeller]

Document how all of the following works and provide some example projects:

• [WIP] OTel Bridge API support #1886
• Auto-configure the opentelemetry SDK #1798
• NR-134815 Treat OTel WithSpan annotation as Trace #1841

[workato-integration]

https://new-relic.atlassian.net/browse/NR-329806

[jasonjkeller]

A readme document has been added to this PR: c308a60

Below is a copy for reference.

---

OpenTelemetry Instrumentation

This instrumentation module weaves parts of the OpenTelemetry SDK to incorporate bits of OpenTelemetry functionality into the New Relic Java agent.

Specifically it can:

• Detect OpenTelemetry Spans and include them in New Relic Java agent traces.
• Detect OpenTelemetry dimensional metrics and report them to the APM entity being monitored by the Java agent.
• Autoconfigure the OpenTelemetry SDK so that OpenTelemetry data is sent to New Relic and properly associated with an APM entity guid.

New Relic Java Agent Configuration

To use the OpenTelemetry Span and dimensional metric functionality incorporated into the New Relic Java agent you must enable the following config options:

Configuration via yaml:

opentelemetry:
  sdk: 
    autoconfigure:
      enabled: true
    spans:
      enabled: true

Configuration via system property:

-Dopentelemetry.sdk.autoconfigure.enabled=true
-Dopentelemetry.sdk.spans.enabled=true

Configuration via environment variable:

NEW_RELIC_OPENTELEMETRY_SDK_AUTOCONFIGURE_ENABLED=true
NEW_RELIC_OPENTELEMETRY_SDK_SPANS_ENABLED=true

OpenTelemetry Dimensional Metrics

OpenTelemetry APIs can be used to create dimensional metrics which will be detected by the New Relic Java agent and reported to the APM entity being monitored by the New Relic Java agent.

To use this functionality, enable the feature as documented above, add the required opentelemetry dependencies to your application:

    implementation(platform("io.opentelemetry:opentelemetry-bom:1.44.1"))
    implementation("io.opentelemetry:opentelemetry-sdk-extension-autoconfigure")
    implementation("io.opentelemetry:opentelemetry-exporter-otlp")

Then utilize the OpenTelemetry APIs to record dimensional metrics:

LongCounter longCounter = GlobalOpenTelemetry.get().getMeterProvider().get("my-application").counterBuilder("my.application.counter").build();
longCounter.add(1, Attributes.of(AttributeKey.stringKey("foo"), "bar"));

Any recorded dimensional metrics can be found in the Metrics Explorer for the associated APM entity and can be used to build custom dashboards.

OpenTelemetry Spans

Documented below are several approaches for incorporating OpenTelemetry Spans into New Relic Java agent traces.

@WithSpan Annotation

The New Relic Java agent will detect usage of the OpenTelemetry @WithSpan annotation. The @WithSpan annotation can be used as an alternative to the @Trace annotation.

This does not currently support the following config options:

• Suppressing @WithSpan instrumentation
• Creating spans around methods with otel.instrumentation.methods.include

Note that OpenTelemetry config properties can be set through environment or system properties, like our agent, and eventually through a config file. We can use our existing OpenTelemetry instrumentation model to get access to the normalized version of the instrumentation settings to include and exclude methods and pass those to the core agent through the bridge.

See ClassTransformerConfigImpl.java for implementation details of the @WithSpan annotation.

Spans Emitted From OpenTelemetry Instrumentation

The New Relic Java agent will detect Spans emitted by OpenTelemetry instrumentation. It does this by weaving the io.opentelemetry.sdk.trace.SdkTracerProvider so that it will create a New Relic Tracer each time an OpenTelemetry Span is started and weaving the io.opentelemetry.context.Context to propagate context between New Relic and OpenTelemetry Spans.

Currently, the New Relic Java agent does not load any OpenTelemetry instrumentation it simply detects Spans emitted by OpenTelemetry manual instrumentation, native instrumentation, library instrumentation, or zero code instrumentation (i.e. bytecode instrumentation that would also require running the OpenTelemetry Java agent).

Depending on the OpenTelemetry Span SpanKind, it may result in the New Relic Java agent starting a transaction (when one doesn't already exist).

• SpanKind.INTERNAL
  • Creating a span with no SpanKind, which defaults to SpanKind.INTERNAL, will not start a transaction
  • If SpanKind.INTERNAL spans occur within an already existing New Relic transaction they will be included in the trace
• SpanKind.CLIENT
  • Creating a span with SpanKind.CLIENT will not start a transaction. If a CLIENT span has certain db attributes it will be treated as a DB span, and other specific attributes will cause it to be treated as an external span
  • If SpanKind.CLIENT spans occur within an already existing New Relic transaction they will be included in the trace
• SpanKind.SERVER
  • Creating a span with SpanKind.SERVER will start a WebTransaction/Uri/* transaction.
  • If SpanKind.SERVER spans occur within an already existing New Relic transaction they will be included in the trace
• SpanKind.CONSUMER
  • Creating a span with SpanKind.CONSUMER will start a OtherTransaction/* transaction.
  • If SpanKind.CONSUMER spans occur within an already existing New Relic transaction they will be included in the trace
• SpanKind.PRODUCER
  • Creating a span with SpanKind.PRODUCER will not start a transaction. There is no explicit processing for PRODUCER spans currently.
  • If SpanKind.PRODUCER spans occur within an already existing New Relic transaction they will be included in the trace (though it's effectively no different from a SpanKind.INTERNAL span)