Incorrect span relationships between instrumentation-express, instrumentation-undici and instrumentation-net, instrumentation-dns

[obs-gh-abhishekrao]

OpenTelemetry Versions

  "@opentelemetry/api": "^1.9.0",
  "@opentelemetry/auto-instrumentations-node": "^0.59.0",

What version of Node are you using?

18.16.1

What did you do?

server.js

require('dotenv').config()

const express = require('express')
const undici = require('undici')
const net = require('net');
const dns = require('dns');
const https = require('https');
const app = express()
const port = 3001


app.get('/', async (req, res) => {
  const {
    statusCode,
    headers,
    trailers,
    body
  } = await undici.request('https://www.google.com/')

  for await (const data of body) { console.log('data', data) }
  console.log('response received', statusCode)
  console.log('headers', headers)

  res.send('Hello World!')
})

app.listen(port, () => {
  console.log(`Example app listening on port ${port}`)
})

What did you expect to see?

The DNS and Connect spans should have the request span from Undici as parent:

• request handler - / (instrumentation-express) -> GET (instrumentation-undici)
• GET (instrumentation-undici) -> tls.connect (instrumentation-net)
• GET (instrumentation-undici) -> tcp.connect (instrumentation-net)
• GET (instrumentation-undici) -> dns.lookup (instrumentation-dns)

What did you see instead?

The DNS and Connect spans have the span from Express as parent::

• request handler - / (instrumentation-express) -> GET (instrumentation-undici)
• request handler - / (instrumentation-express) -> tls.connect (instrumentation-net)
• request handler - / (instrumentation-express) -> tcp.connect (instrumentation-net)
• request handler - / (instrumentation-express) -> dns.lookup (instrumentation-dns)

Additional context

Steps to run the above example:

• export OTEL_TRACES_EXPORTER="console"
• export OTEL_NODE_RESOURCE_DETECTORS="env,host,os,serviceinstance"
• export OTEL_SERVICE_NAME="express-undici-example"
• export NODE_OPTIONS="--require @opentelemetry/auto-instrumentations-node/register"
• node server.js

[dyladan]

This is happening because undici doesn't set its client span as the active span in context when doing the export. When dns and connect instrumentations look into the context, they're getting the active span that was set by express. Because the undici instrumentation is implemented using tracing channels, I'm not sure how easy this would be to fix. There should be a way to solve this with async context, but I don't know it off the top of my head. I'll look into this.