feat: Improve OpenTelemetry trace instrumentation #6417

cedricziel · 2024-06-22T12:53:54Z

This PR enhances the opentelemetry instrumentation for caddy by:

creating client spans for reverse proxy request to fix servicegraph calculations
injecting, conditionally, a server-timing header to propagate the traceparent down to the caller

modules/caddyhttp/tracing/module.go

Co-authored-by: Mohammed Al Sahaf <mohammed@caffeinatedwonders.com>

modules/caddyhttp/reverseproxy/reverseproxy.go

modules/caddyhttp/tracing/tracer.go

mholt · 2024-10-02T14:24:13Z

Thanks @hairyhenderson !

@cedricziel Would you be able to fix up the lint errors and make sure the tests pass, then we can get this in the next beta release?

mholt · 2024-10-02T14:48:05Z

Thanks for the commit!

Looks like the order of the modified imports needs to be adjusted (for lint) and for tests, there's:

    module_test.go:107: Invalid traceparent: 
--- FAIL: TestTracing_ServeHTTP_Propagation_Without_Initial_Headers (0.00s)

mohammed90 · 2024-10-02T14:50:23Z

For lint, try golangci-lint run --fix. I think that takes care of most of them.

caddyserver#6531) * Add per host config * Pass host label when option is enabled * Test per host enabled * metrics: scope metrics per loaded config * doc and linter Signed-off-by: Mohammed Al Sahaf <msaa1990@gmail.com> * inject the custom registry into the admin handler Co-Authored-By: Dave Henderson <dhenderson@gmail.com> * remove `TODO` comment * fixes Signed-off-by: Mohammed Al Sahaf <msaa1990@gmail.com> * refactor to delay metrics admin handler provision Signed-off-by: Mohammed Al Sahaf <msaa1990@gmail.com> --------- Signed-off-by: Mohammed Al Sahaf <msaa1990@gmail.com> Co-authored-by: Hussam Almarzooq <me@hussam.io> Co-authored-by: Dave Henderson <dhenderson@gmail.com>

…7142) Bumps the actions-deps group with 6 updates: | Package | From | To | | --- | --- | --- | | [step-security/harden-runner](https://github.com/step-security/harden-runner) | `2.12.1` | `2.13.0` | | [actions/upload-artifact](https://github.com/actions/upload-artifact) | `4.6.1` | `4.6.2` | | [sigstore/cosign-installer](https://github.com/sigstore/cosign-installer) | `e9a05e6d32d7ed22b5656cd874ef31af58d05bfa` | `d58896d6a1865668819e1d91763c7751a165e159` | | [anchore/sbom-action](https://github.com/anchore/sbom-action) | `0.20.1` | `0.20.4` | | [ossf/scorecard-action](https://github.com/ossf/scorecard-action) | `2.4.1` | `2.4.2` | | [github/codeql-action](https://github.com/github/codeql-action) | `3.29.0` | `3.29.4` | Updates `step-security/harden-runner` from 2.12.1 to 2.13.0 - [Release notes](https://github.com/step-security/harden-runner/releases) - [Commits](step-security/harden-runner@002fdce...ec9f2d5) Updates `actions/upload-artifact` from 4.6.1 to 4.6.2 - [Release notes](https://github.com/actions/upload-artifact/releases) - [Commits](actions/upload-artifact@v4.6.1...ea165f8) Updates `sigstore/cosign-installer` from e9a05e6d32d7ed22b5656cd874ef31af58d05bfa to d58896d6a1865668819e1d91763c7751a165e159 - [Release notes](https://github.com/sigstore/cosign-installer/releases) - [Commits](sigstore/cosign-installer@e9a05e6...d58896d) Updates `anchore/sbom-action` from 0.20.1 to 0.20.4 - [Release notes](https://github.com/anchore/sbom-action/releases) - [Changelog](https://github.com/anchore/sbom-action/blob/main/RELEASE.md) - [Commits](anchore/sbom-action@9246b90...7b36ad6) Updates `ossf/scorecard-action` from 2.4.1 to 2.4.2 - [Release notes](https://github.com/ossf/scorecard-action/releases) - [Changelog](https://github.com/ossf/scorecard-action/blob/main/RELEASE.md) - [Commits](ossf/scorecard-action@f49aabe...05b42c6) Updates `github/codeql-action` from 3.29.0 to 3.29.4 - [Release notes](https://github.com/github/codeql-action/releases) - [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md) - [Commits](github/codeql-action@ce28f5b...4e828ff) --- updated-dependencies: - dependency-name: step-security/harden-runner dependency-version: 2.13.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: actions-deps - dependency-name: actions/upload-artifact dependency-version: 4.6.2 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: actions-deps - dependency-name: sigstore/cosign-installer dependency-version: d58896d6a1865668819e1d91763c7751a165e159 dependency-type: direct:production dependency-group: actions-deps - dependency-name: anchore/sbom-action dependency-version: 0.20.4 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: actions-deps - dependency-name: ossf/scorecard-action dependency-version: 2.4.2 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: actions-deps - dependency-name: github/codeql-action dependency-version: 3.29.4 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: actions-deps ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…cepted sockets (caddyserver#7151) Fix caddyserver#7144

If global DNS provider is configured, it does not need to be repeated in the JSON. If acme_* options are used, base automation policies should populate their issuers accordingly. Global issuer settings like acme_* options don't need to specify subjects in the automation policy since they should apply as a global default.

…dates (caddyserver#7155) Bumps the all-updates group with 10 updates in the / directory: | Package | From | To | | --- | --- | --- | | [github.com/KimMachineGun/automemlimit](https://github.com/KimMachineGun/automemlimit) | `0.7.1` | `0.7.4` | | [github.com/alecthomas/chroma/v2](https://github.com/alecthomas/chroma) | `2.19.0` | `2.20.0` | | [github.com/google/cel-go](https://github.com/google/cel-go) | `0.24.1` | `0.26.0` | | [github.com/prometheus/client_golang](https://github.com/prometheus/client_golang) | `1.19.1` | `1.23.0` | | [github.com/smallstep/certificates](https://github.com/smallstep/certificates) | `0.26.1` | `0.28.4` | | [github.com/yuin/goldmark](https://github.com/yuin/goldmark) | `1.7.8` | `1.7.13` | | [go.opentelemetry.io/contrib/propagators/autoprop](https://github.com/open-telemetry/opentelemetry-go-contrib) | `0.42.0` | `0.62.0` | | [go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc](https://github.com/open-telemetry/opentelemetry-go) | `1.31.0` | `1.37.0` | | [github.com/libdns/libdns](https://github.com/libdns/libdns) | `1.0.0-beta.1` | `1.1.0` | | [github.com/pires/go-proxyproto](https://github.com/pires/go-proxyproto) | `0.7.1-0.20240628150027-b718e7ce4964` | `0.8.1` | Updates `github.com/KimMachineGun/automemlimit` from 0.7.1 to 0.7.4 - [Release notes](https://github.com/KimMachineGun/automemlimit/releases) - [Commits](KimMachineGun/automemlimit@v0.7.1...v0.7.4) Updates `github.com/alecthomas/chroma/v2` from 2.19.0 to 2.20.0 - [Release notes](https://github.com/alecthomas/chroma/releases) - [Changelog](https://github.com/alecthomas/chroma/blob/master/.goreleaser.yml) - [Commits](alecthomas/chroma@v2.19.0...v2.20.0) Updates `github.com/google/cel-go` from 0.24.1 to 0.26.0 - [Release notes](https://github.com/google/cel-go/releases) - [Commits](google/cel-go@v0.24.1...v0.26.0) Updates `github.com/prometheus/client_golang` from 1.19.1 to 1.23.0 - [Release notes](https://github.com/prometheus/client_golang/releases) - [Changelog](https://github.com/prometheus/client_golang/blob/main/CHANGELOG.md) - [Commits](prometheus/client_golang@v1.19.1...v1.23.0) Updates `github.com/smallstep/certificates` from 0.26.1 to 0.28.4 - [Release notes](https://github.com/smallstep/certificates/releases) - [Changelog](https://github.com/smallstep/certificates/blob/master/CHANGELOG.md) - [Commits](smallstep/certificates@v0.26.1...v0.28.4) Updates `github.com/smallstep/nosql` from 0.6.1 to 0.7.0 - [Commits](smallstep/nosql@v0.6.1...v0.7.0) Updates `github.com/yuin/goldmark` from 1.7.8 to 1.7.13 - [Release notes](https://github.com/yuin/goldmark/releases) - [Commits](yuin/goldmark@v1.7.8...v1.7.13) Updates `go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp` from 0.56.0 to 0.61.0 - [Release notes](https://github.com/open-telemetry/opentelemetry-go-contrib/releases) - [Changelog](https://github.com/open-telemetry/opentelemetry-go-contrib/blob/main/CHANGELOG.md) - [Commits](open-telemetry/opentelemetry-go-contrib@zpages/v0.56.0...zpages/v0.61.0) Updates `go.opentelemetry.io/contrib/propagators/autoprop` from 0.42.0 to 0.62.0 - [Release notes](https://github.com/open-telemetry/opentelemetry-go-contrib/releases) - [Changelog](https://github.com/open-telemetry/opentelemetry-go-contrib/blob/main/CHANGELOG.md) - [Commits](open-telemetry/opentelemetry-go-contrib@zpages/v0.42.0...zpages/v0.62.0) Updates `go.opentelemetry.io/otel` from 1.31.0 to 1.37.0 - [Release notes](https://github.com/open-telemetry/opentelemetry-go/releases) - [Changelog](https://github.com/open-telemetry/opentelemetry-go/blob/main/CHANGELOG.md) - [Commits](open-telemetry/opentelemetry-go@v1.31.0...v1.37.0) Updates `go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc` from 1.31.0 to 1.37.0 - [Release notes](https://github.com/open-telemetry/opentelemetry-go/releases) - [Changelog](https://github.com/open-telemetry/opentelemetry-go/blob/main/CHANGELOG.md) - [Commits](open-telemetry/opentelemetry-go@v1.31.0...v1.37.0) Updates `go.opentelemetry.io/otel/sdk` from 1.31.0 to 1.37.0 - [Release notes](https://github.com/open-telemetry/opentelemetry-go/releases) - [Changelog](https://github.com/open-telemetry/opentelemetry-go/blob/main/CHANGELOG.md) - [Commits](open-telemetry/opentelemetry-go@v1.31.0...v1.37.0) Updates `github.com/libdns/libdns` from 1.0.0-beta.1 to 1.1.0 - [Release notes](https://github.com/libdns/libdns/releases) - [Commits](libdns/libdns@v1.0.0-beta.1...v1.1.0) Updates `github.com/pires/go-proxyproto` from 0.7.1-0.20240628150027-b718e7ce4964 to 0.8.1 - [Release notes](https://github.com/pires/go-proxyproto/releases) - [Commits](https://github.com/pires/go-proxyproto/commits/v0.8.1) Updates `github.com/prometheus/client_model` from 0.5.0 to 0.6.2 - [Release notes](https://github.com/prometheus/client_model/releases) - [Commits](prometheus/client_model@v0.5.0...v0.6.2) Updates `go.opentelemetry.io/otel/trace` from 1.31.0 to 1.37.0 - [Release notes](https://github.com/open-telemetry/opentelemetry-go/releases) - [Changelog](https://github.com/open-telemetry/opentelemetry-go/blob/main/CHANGELOG.md) - [Commits](open-telemetry/opentelemetry-go@v1.31.0...v1.37.0) Updates `go.step.sm/crypto` from 0.45.0 to 0.67.0 - [Release notes](https://github.com/smallstep/crypto/releases) - [Commits](smallstep/crypto@v0.45.0...v0.67.0) --- updated-dependencies: - dependency-name: github.com/KimMachineGun/automemlimit dependency-version: 0.7.4 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: all-updates - dependency-name: github.com/alecthomas/chroma/v2 dependency-version: 2.20.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: all-updates - dependency-name: github.com/google/cel-go dependency-version: 0.26.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: all-updates - dependency-name: github.com/prometheus/client_golang dependency-version: 1.23.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: all-updates - dependency-name: github.com/smallstep/certificates dependency-version: 0.28.4 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: all-updates - dependency-name: github.com/smallstep/nosql dependency-version: 0.7.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: all-updates - dependency-name: github.com/yuin/goldmark dependency-version: 1.7.13 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: all-updates - dependency-name: go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp dependency-version: 0.61.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: all-updates - dependency-name: go.opentelemetry.io/contrib/propagators/autoprop dependency-version: 0.62.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: all-updates - dependency-name: go.opentelemetry.io/otel dependency-version: 1.37.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: all-updates - dependency-name: go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc dependency-version: 1.37.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: all-updates - dependency-name: go.opentelemetry.io/otel/sdk dependency-version: 1.37.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: all-updates - dependency-name: github.com/libdns/libdns dependency-version: 1.1.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: all-updates - dependency-name: github.com/pires/go-proxyproto dependency-version: 0.8.1 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: all-updates - dependency-name: github.com/prometheus/client_model dependency-version: 0.6.2 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: all-updates - dependency-name: go.opentelemetry.io/otel/trace dependency-version: 1.37.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: all-updates - dependency-name: go.step.sm/crypto dependency-version: 0.67.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: all-updates ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* feat: add bcrypt cost parameter to hash-password * revert: typos * refactor: take the cost out of interface * fix: default bcrypt cost to 14 * fix: follow bcrypt library for min and max cost * doc: mention defaultBcryptCost in cost parameter description * chore: gci format * fix: more specific bcrypt cost algorithm flag * feat: bcrypt cost provisioning * Revert "feat: bcrypt cost provisioning" This reverts commit e09d4bd. * chore: gci format * chore: gci format * chore: gci format * chore: golangcilint fmted --------- Co-authored-by: Mohammed Al Sahaf <msaa1990@gmail.com>

Signed-off-by: youzichuan <youzichuan6@outlook.com>

…dyserver#7179) revert changes to automation.go from e276994

Signed-off-by: joemicky <joemickychang@outlook.com> Co-authored-by: Matt Holt <mholt@users.noreply.github.com>

…er#7158) Co-authored-by: Matt Holt <mholt@users.noreply.github.com>

…server#7176) * http: clean up listeners if some of the listeners fail to bind * check for nil server due to failure to start --------- Co-authored-by: Matt Holt <mholt@users.noreply.github.com>

…dyserver#7157) Co-authored-by: Matt Holt <mholt@users.noreply.github.com>

…7183) * ci: set proper build tags in golangci and minor cleanup * clean * review --------- Co-authored-by: Matt Holt <mholt@users.noreply.github.com>

…7173) Co-authored-by: Matt Holt <mholt@users.noreply.github.com>

…Content Length header to appear (caddyserver#7042) Co-authored-by: Matt Holt <mholt@users.noreply.github.com>

…ts (caddyserver#6961) * use the new http.Protocols to handle h1, h2 and h2c requests * fix lint * keep ConnCtxKey for now * fix handling for h2c * check http version while reading the connection * check if connection implements connectionStater when it should * add comments about either h1 or h2 must be used in the listener * fix if check * return a net.Conn that implements connectionStater if applicable * remove http/1.1 from alpn if h1 is disabled * fix matching if only h1 is enabled --------- Co-authored-by: Matt Holt <mholt@users.noreply.github.com>

Signed-off-by: Mohammed Al Sahaf <msaa1990@gmail.com>

cedricziel added 5 commits June 22, 2024 13:21

feat: inject span_id as var and log context

e35ce3d

fix: correct to span_id

bb450ed

fix: inject server-timing when configured

6c05741

feat: record client span for reverseproxy scenarios

5f9398c

fix: switch to use roundtripper from otel

8f40754

mohammed90 reviewed Jun 22, 2024

View reviewed changes

modules/caddyhttp/tracing/module.go Outdated Show resolved Hide resolved

cedricziel and others added 6 commits June 22, 2024 15:29

fix: amend test to capture new default

9f9d9c5

fix: fix config

b6966f4

Update modules/caddyhttp/tracing/module.go

023999e

Co-authored-by: Mohammed Al Sahaf <mohammed@caffeinatedwonders.com>

fix: clean fixture

f704d0b

fix: gofumpt

3d93ce4

fix: shorten config

f58b6fb

mohammed90 reviewed Jun 22, 2024

View reviewed changes

modules/caddyhttp/reverseproxy/reverseproxy.go Show resolved Hide resolved

cedricziel added 5 commits June 22, 2024 15:53

fix: lint

72ab14f

fix: lint

558e28d

fix: lint

ed087ec

fix: dont inject twice

ae1d4ef

fix: set global propagator

ced6ba4

caddyserver deleted a comment from ChenYujunjks Jul 23, 2024

Merge branch 'master' into trace-servicegraph

a0b1a8d

hairyhenderson reviewed Oct 2, 2024

View reviewed changes

modules/caddyhttp/tracing/tracer.go Show resolved Hide resolved

hairyhenderson approved these changes Oct 2, 2024

View reviewed changes

fix: gofumpt

5ccad83

fahrradflucht mentioned this pull request Aug 29, 2025

Reverse proxy does not emit OpenTelemetry client spans for upstream requests #7223

Open

mohammed90 and others added 3 commits August 29, 2025 17:22

ci: install xcaddy to fix release flow (caddyserver#6602)

870372a

go.mod: Upgrade some dependencies

4446d95

mholt and others added 29 commits August 29, 2025 17:27

go.mod: Upgrade dependencies

19697b6

Add test for 5b727bd

fa05e86

core: Use KeepAliveConfig to pass keepalive_interval to listener's ac…

9356811

…cepted sockets (caddyserver#7151) Fix caddyserver#7144

caddytls: Improve ECH error logging (close caddyserver#7152)

36f5b9b

Ignore irrelevant linter warning

b0862c4

cmd: Allow caddy adapt to read from stdin (caddyserver#7163)

629961e

bcrypt: wrong cost flag name (caddyserver#7168)

7c96cb9

chore: fix inconsistent function name in comment (caddyserver#7174)

99e5b71

Signed-off-by: youzichuan <youzichuan6@outlook.com>

caddytls: fix regression in external certificate manager support (cad…

f86abf9

…dyserver#7179) revert changes to automation.go from e276994

caddyhttp: Free up quic listener when stopping (caddyserver#7177)

0a430d0

chore: bump Go to v1.25 (caddyserver#7184)

2041625

caddyhttp: refactor to use reflect.TypeFor (caddyserver#7187)

0b57367

use a more modern writing style to simplify code (caddyserver#7182)

7bcfa5c

Signed-off-by: joemicky <joemickychang@outlook.com> Co-authored-by: Matt Holt <mholt@users.noreply.github.com>

http: disable keepalive when KeepAliveInterval is negative (caddyserv…

664726a

…er#7158) Co-authored-by: Matt Holt <mholt@users.noreply.github.com>

http: clean up listeners if some of the listeners fail to bind (caddy…

bf24310

…server#7176) * http: clean up listeners if some of the listeners fail to bind * check for nil server due to failure to start --------- Co-authored-by: Matt Holt <mholt@users.noreply.github.com>

reverse_proxy: use the new KeepAliveConfig to set probe interval (cad…

1488832

…dyserver#7157) Co-authored-by: Matt Holt <mholt@users.noreply.github.com>

ci: set proper build tags in golangci and minor cleanup (caddyserver#…

dd7304f

…7183) * ci: set proper build tags in golangci and minor cleanup * clean * review --------- Co-authored-by: Matt Holt <mholt@users.noreply.github.com>

fileserver: Add a few doc lines about Etag file content (caddyserver#…

aa89afb

…7173) Co-authored-by: Matt Holt <mholt@users.noreply.github.com>

fileserver: set Range header for precompressed static files to force …

b79b2e8

…Content Length header to appear (caddyserver#7042) Co-authored-by: Matt Holt <mholt@users.noreply.github.com>

go.mod: Upgrade CertMagic to v0.24.0

c193c79

http: Make logger first, before TLS provisioning (caddyserver#7198)

75f20bb

httpcaddyfile: Fix acme_dns regression (caddyserver#7199)

b4337e7

caddyfile: Fix importing nested tokens for {block} (caddyserver#7189)

9d00c6e

lint

5bd3204

Signed-off-by: Mohammed Al Sahaf <msaa1990@gmail.com>

mohammed90 mentioned this pull request Aug 29, 2025

otel: Improve OpenTelemetry trace instrumentation #7224

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: Improve OpenTelemetry trace instrumentation #6417

feat: Improve OpenTelemetry trace instrumentation #6417

cedricziel commented Jun 22, 2024 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mholt commented Oct 2, 2024

Uh oh!

mholt commented Oct 2, 2024

Uh oh!

mohammed90 commented Oct 2, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants

Uh oh!

feat: Improve OpenTelemetry trace instrumentation #6417

Are you sure you want to change the base?

feat: Improve OpenTelemetry trace instrumentation #6417

Conversation

cedricziel commented Jun 22, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mholt commented Oct 2, 2024

Uh oh!

mholt commented Oct 2, 2024

Uh oh!

mohammed90 commented Oct 2, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants

cedricziel commented Jun 22, 2024 •

edited

Loading