Skip to content

Decrease in trace metrics after updating to >v68 #688

@HiDAl

Description

@HiDAl

Hello,

some days ago my team tried to update the currently used layer to the latest version from v65. First, we tried with v68 and noticed a huge decrease in the hits metrics generated through the tracing tool: for example when using tracer.trace('name', ...), we saw a decrease in the metrics trace.name.hits.
Checking documentation and such, we noticed that the env var DD_TRACE_SAMPLE_RATE was deprecated so we migrated to use the new one DD_TRACE_SAMPLING_RULES (moving from DD_TRACE_SAMPLE_RATE=0.4 to DD_TRACE_SAMPLING_RULES='[{"sample_rate": 0.4}]'. Sadly, the issue was still there. We decided to give a try to v78, but the problem persisted.
When enabling the debugging option, I see the following output (shortened for simplicity):

// v65 using DD_TRACE_SAMPLE_RATE
{
  "os_name": "Linux",
  "os_version": "5.10.236-249.934.amzn2.aarch64",
  "architecture": "arm64",
  "version": "4.44.0",
  "lang": "nodejs",
  "lang_version": "20.19.1",
  "sample_rate": 0.4,
  "sampling_rules": [
    {
      "matchers": [],
      "_sampler": {
        "_rate": 0.4
      },
      "_limiter": {
        "_rateLimit": 100,
        "_limiter": {
          "tokenBucket": {
            "bucketSize": 100,
            "tokensPerInterval": 100,
            "interval": 1000,
            "parentBucket": null,
            "content": 99,
            "lastDrip": 1748592474968
          },
          "curIntervalStart": 1558141,
          "tokensThisInterval": 2
        },
        "_tokensRequested": 2,
        "_prevIntervalTokens": 0,
        "_prevTokensRequested": 0
      }
    }
  ],
  ...
}

// v78 using DD_TRACE_SAMPLING_RULES
{
  "os_name": "Linux",
  "os_version": "5.10.236-249.934.amzn2.aarch64",
  "architecture": "arm64",
  "version": "4.44.0",
  "lang": "nodejs",
  "lang_version": "20.19.1",
  "sampling_rules": [
    {
      "matchers": [],
      "_sampler": {
        "_rate": 0.4
      }
    }
  ],
  ...
}

One thing that I noticed, but not sure at all, is that even though the sample rate was configured, I saw a 100% of the traces —no sampling?—, so not sure whether it's normal after the update.

thank you

Metadata

Metadata

Assignees

Labels

bugSomething isn't workingenhancementNew feature or requestversion/nextIssues related to DD_EXTENSION_VERSION: next

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions