Evaluating how to support tracing and OpenTelemetry

[stebet]

This is something that I have been wondering about for some time on what would be the best approach for observability.

Since debugging a library like this can be hard, I wonder what we'd want to do to make this easier.

I'd really like to suggest using System.Diagnostics.DiagnosticSource to provide parity with a lot of other .NET libraries and frameworks (both Core and Framework).

Using this library it becomes a lot easier to plug in logging frameworks and profiling tools to monitor and measure operations within the library. It also has the added benefit to have next to no overhead when no-one is listening for the events, since it is exposing ETW events (and in .NET Core, can expose event counters).

[vhatsura]

I also suggest looking into the new Tracing API on Activity which should be part of .NET 5 release.

[stebet]

Yeah, that looks good. A good first step would be to add basic Activity support, ETW and event counters. I'm planning to look at that soon-ish. This would be a logical next step after that.

[jaycdave88]

Hi @stebet, just wanted to follow-up on your previous note and see if you had any sorta update or a product roadmap you could share around Activity support?

[michaelklishin]

@jaycdave88 right now the plan is to investigate System.Diagnostics.DiagnosticSource. This may or may not happen by 7.0 but can go into a 7.x release. We need to spike it first. If there are popular alternatives in the community, we would consider them. Let us know ;)

[CodeBlanch]

@michaelklishin Depending on the timeline, definitely check out the new ActivitySource API. Very similar to DiagnosticSource, but it supports sampling and you can write tags/attributes on the events instead of writing raw objects (which people usually need to consume reflectively). It releases with .NET 5 (~November), but you don't need .NET 5 to use it. The NuGet package targets all actively supported .NET versions.

The main benefit to using ActivitySource is it means instrumentation libraries like OpenTelemetry will work without needing to build an adapter or anything.

Also happy to help with this if you are open to it.

[michaelklishin]

If ActivitySource will support all then/currently supported versions of .NET, we should consider it. A proof-of-concept PR would be appreciated (I assume there are preview releases of some kind that developers can try?)

[stebet]

ActivitySource should be the correct one to look at. I spiked this myself a few months back, time to revisit :)

What would people consider good activities/counters to start with?

Here are a few rough ideas:

• Bytes sent
• Bytes received
• Frames deserialized
• Frames serialized
• Total Connections opened
• Connections currently open
• Consumers active
• Channels Open
• Total channels opened
• ???

[cijothomas]

If ActivitySource will support all then/currently supported versions of .NET, we should consider it. A proof-of-concept PR would be appreciated (I assume there are preview releases of some kind that developers can try?)

ActivitySource/Activity are part of the nuget package https://www.nuget.org/packages/System.Diagnostics.DiagnosticSource/5.0.0-rc.1.20451.14. It can be used right now with preview releases.
An example on using ActivitySource is available here in OpenTelemetry repo.
https://github.com/open-telemetry/opentelemetry-dotnet/blob/master/examples/Console/TestConsoleExporter.cs#L41-L94

[snakorse]

ActivitySource should be the correct one to look at. I spiked this myself a few months back, time to revisit :)

What would people consider good activities/counters to start with?

Here are a few rough ideas:

• Bytes sent
• Bytes received
• Frames deserialized
• Frames serialized
• Total Connections opened
• Connections currently open
• Consumers active
• Channels Open
• Total channels opened
• ???

I think something likeFrames serializing should be considered, the telemetry instruments may want to inject tracing header into the frame before it serialized to send.

[cijothomas]

Just to clarify - ActivitySource/Activity API is for tracing , and not for metrics. Things like BytesSent, etc are metric, which require a metric API. .NET has EventCounter API to support this already, but it is a very limited API. There is a new Metric API (Opentelemetry compatible), coming with .NET 6 (Nov 2021), which will support all the old versions of the .NET itself. (open-telemetry/opentelemetry-dotnet#1501)

[stebet]

@cijothomas Yep. I'm aware of the differences. We're looking to integrate both the new dotnet counters (for metrics) as well as support for Activity Tracing. Tracing will come later since the OpenTelemetry is currently being worked on but I'm watching it closely :)

[cijothomas]

@stebet Thanks!
To instrument, you won't need OpenTelemetry dependency, right? You can use 5.0.0 of DiagnosticSource.
(Also OpenTelemetry will release 1.0.0 in <2 weeks)

Please also check the Metric plans in .NET 6 as well.

[stebet]

@stebet Thanks!
To instrument, you won't need OpenTelemetry dependency, right? You can use 5.0.0 of DiagnosticSource.
(Also OpenTelemetry will release 1.0.0 in <2 weeks)

Please also check the Metric plans in .NET 6 as well.

Thanks for the info :)