Skip to content

chore(profiling): [4/n] lock optimization - remove 'is_internal' concept from wrappers#17859

Merged
vlad-scherbich merged 0 commit into
vlad/lockprof-fix-threading-mod-cloning-bugfrom
vlad/lock-remove-is-internal-concept
May 5, 2026
Merged

chore(profiling): [4/n] lock optimization - remove 'is_internal' concept from wrappers#17859
vlad-scherbich merged 0 commit into
vlad/lockprof-fix-threading-mod-cloning-bugfrom
vlad/lock-remove-is-internal-concept

Conversation

@vlad-scherbich
Copy link
Copy Markdown
Contributor

@vlad-scherbich vlad-scherbich commented May 2, 2026

Description

Remove the is_internal per-lock flag from _ProfiledLock wrapper, since we will now have the concept of "excluded modules" (#16922). We can replaces it with _ALWAYS_EXCLUDED_MODULES, a list of stdlib modules implementing sync and async locks, which will never be profiled. This will effectively be the same behavior: no locks backing higher-level locks (Condition, Semaphore) will be profiled; no sample double-counting; and a perf win by removing now-unnecessary bookeeping.

What changed

Before: When a stdlib primitive (e.g. threading.Semaphore) created an internal lock, _profiled_allocate_lock would still wrap it as a _ProfiledLock, and set is_internal=True. The wrapper would exit early on each call to _acquire, _release, and _flush_sample.

After: Internal stdlib locks are never wrapped at all thanks to _ALWAYS_EXCLUDED_MODULES. _profiled_allocate_lock checks the caller's __name__ against the excluded modules and returns the native lock directly on a match.

Why this is better

Before (is_internal) After (_ALWAYS_EXCLUDED_MODULES)
Wrapper created for internal lock? Yes No
Runtime overhead per acquire/release self.is_internal check on every call None
Sample suppression sites _acquire, _release, _flush_sample
Internal lock detection Filename comparison (realpath per lock create) Module name set lookup (O(1), done once at patch)

Testing

  • CI
  • 2 new unit tests

Risks

n/a

@cit-pr-commenter-54b7da
Copy link
Copy Markdown

cit-pr-commenter-54b7da Bot commented May 2, 2026

Codeowners resolved as

ddtrace/internal/settings/profiling.py                                  @DataDog/profiling-python

@vlad-scherbich vlad-scherbich force-pushed the vlad/lock-remove-is-internal-concept branch from 11a6631 to b110b2b Compare May 3, 2026 01:40
@vlad-scherbich vlad-scherbich force-pushed the vlad/lockprof-fix-threading-mod-cloning-bug branch 7 times, most recently from 9f05d4f to 7532d4a Compare May 3, 2026 16:50
@vlad-scherbich vlad-scherbich force-pushed the vlad/lock-remove-is-internal-concept branch from b110b2b to 865670c Compare May 3, 2026 16:57
@vlad-scherbich vlad-scherbich force-pushed the vlad/lockprof-fix-threading-mod-cloning-bug branch from 316dab1 to de23ba2 Compare May 3, 2026 17:27
@vlad-scherbich vlad-scherbich force-pushed the vlad/lock-remove-is-internal-concept branch from 865670c to 11b9d24 Compare May 3, 2026 17:28
@vlad-scherbich vlad-scherbich force-pushed the vlad/lockprof-fix-threading-mod-cloning-bug branch from de23ba2 to 348989c Compare May 3, 2026 18:47
@vlad-scherbich vlad-scherbich force-pushed the vlad/lock-remove-is-internal-concept branch 3 times, most recently from ef2ec3f to be269aa Compare May 4, 2026 00:49
@vlad-scherbich vlad-scherbich force-pushed the vlad/lockprof-fix-threading-mod-cloning-bug branch 2 times, most recently from 86dfe5c to 0f71d26 Compare May 4, 2026 00:59
@vlad-scherbich vlad-scherbich changed the title Vlad/lock remove is internal concept perf(profiling): Lock optimization: remove 'is_internal' concept from wrappers May 5, 2026
@vlad-scherbich vlad-scherbich changed the title perf(profiling): Lock optimization: remove 'is_internal' concept from wrappers chore(profiling): Lock optimization: remove 'is_internal' concept from wrappers May 5, 2026
@vlad-scherbich vlad-scherbich added changelog/no-changelog A changelog entry is not required for this PR. Profiling Continous Profling labels May 5, 2026
@vlad-scherbich vlad-scherbich force-pushed the vlad/lock-remove-is-internal-concept branch from be269aa to e6762a0 Compare May 5, 2026 14:31
@vlad-scherbich vlad-scherbich changed the title chore(profiling): Lock optimization: remove 'is_internal' concept from wrappers chore(profiling): lock optimization - remove 'is_internal' concept from wrappers May 5, 2026
@datadog-prod-us1-3

This comment has been minimized.

@vlad-scherbich vlad-scherbich force-pushed the vlad/lockprof-fix-threading-mod-cloning-bug branch from 9ace0ba to 2a801fe Compare May 5, 2026 15:17
@vlad-scherbich vlad-scherbich force-pushed the vlad/lock-remove-is-internal-concept branch from bbb95e9 to 6ce13fe Compare May 5, 2026 15:17
@vlad-scherbich vlad-scherbich force-pushed the vlad/lockprof-fix-threading-mod-cloning-bug branch from 2a801fe to 96b9239 Compare May 5, 2026 15:49
@vlad-scherbich vlad-scherbich force-pushed the vlad/lock-remove-is-internal-concept branch from 7f7d912 to abbacc1 Compare May 5, 2026 15:49
@vlad-scherbich
Copy link
Copy Markdown
Contributor Author

@chatgpt-codex-connector please review

@vlad-scherbich vlad-scherbich requested a review from Copilot May 5, 2026 15:56
@vlad-scherbich vlad-scherbich marked this pull request as ready for review May 5, 2026 15:58
@vlad-scherbich vlad-scherbich requested a review from a team as a code owner May 5, 2026 15:58
Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: abbacc1836

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread tests/profiling/collector/test_threading.py Outdated
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR simplifies lock profiling internals by removing _ProfiledLock.is_internal and moving stdlib internal-lock suppression to allocation time via _ALWAYS_EXCLUDED_MODULES. In the profiling codebase, that shifts stdlib lock handling from “wrap then early-exit” to “don’t wrap excluded-module locks at all”.

Changes:

  • Add _ALWAYS_EXCLUDED_MODULES handling in the lock allocator and remove is_internal bookkeeping from _ProfiledLock.
  • Remove asyncio collector-specific internal-module-file configuration that is no longer needed.
  • Update typing stubs and add/adjust tests around always-excluded modules and wrapper shape.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 4 comments.

Show a summary per file
File Description
ddtrace/profiling/collector/_lock.pyx Reworks lock allocation to skip wrapping for always-excluded modules and removes internal-lock wrapper state.
ddtrace/profiling/collector/_lock.pyi Syncs stubs with the wrapper/API changes.
ddtrace/profiling/collector/asyncio.py Drops obsolete asyncio internal-file detection fields.
tests/profiling/collector/test_threading.py Updates threading lock tests for the new native-internal-lock behavior and wrapper slots.
tests/profiling/test_profiling_config.py Adds config-level regression tests for always-excluded stdlib modules.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread tests/profiling/collector/test_threading.py Outdated
Comment thread tests/profiling/collector/test_threading.py Outdated
Comment thread ddtrace/profiling/collector/_lock.pyx Outdated
Comment thread ddtrace/profiling/collector/asyncio.py
@vlad-scherbich vlad-scherbich force-pushed the vlad/lockprof-fix-threading-mod-cloning-bug branch from 96b9239 to 63032bd Compare May 5, 2026 17:11
@vlad-scherbich vlad-scherbich requested a review from a team as a code owner May 5, 2026 17:11
@vlad-scherbich vlad-scherbich requested review from juanjux and removed request for a team May 5, 2026 17:11
@vlad-scherbich vlad-scherbich force-pushed the vlad/lock-remove-is-internal-concept branch from abbacc1 to fd852dc Compare May 5, 2026 17:11
@vlad-scherbich vlad-scherbich changed the title chore(profiling): lock optimization - remove 'is_internal' concept from wrappers chore(profiling): [4/n] lock optimization - remove 'is_internal' concept from wrappers May 5, 2026
@vlad-scherbich vlad-scherbich force-pushed the vlad/lock-remove-is-internal-concept branch from fd852dc to c145e41 Compare May 5, 2026 18:35
@vlad-scherbich vlad-scherbich force-pushed the vlad/lockprof-fix-threading-mod-cloning-bug branch from 63032bd to bed8587 Compare May 5, 2026 18:45
@vlad-scherbich vlad-scherbich requested a review from a team as a code owner May 5, 2026 18:45
@vlad-scherbich vlad-scherbich force-pushed the vlad/lock-remove-is-internal-concept branch from 92fa9ff to bb3207a Compare May 5, 2026 18:45
@vlad-scherbich vlad-scherbich force-pushed the vlad/lock-remove-is-internal-concept branch from ca10e69 to 08ee721 Compare May 5, 2026 19:01
@vlad-scherbich vlad-scherbich merged commit 08ee721 into vlad/lockprof-fix-threading-mod-cloning-bug May 5, 2026
3 of 37 checks passed
@vlad-scherbich vlad-scherbich force-pushed the vlad/lockprof-fix-threading-mod-cloning-bug branch from bed8587 to 08ee721 Compare May 5, 2026 19:01
@vlad-scherbich vlad-scherbich deleted the vlad/lock-remove-is-internal-concept branch May 5, 2026 19:01
gh-worker-dd-mergequeue-cf854d Bot pushed a commit that referenced this pull request May 7, 2026
…rs (#17902)

[< Prev PR](#16775) |

## Description

Remove the `is_internal` per-lock flag from `_ProfiledLock` wrapper, since we will now have the concept of "excluded modules" (#16922). We can replaces it with `_ALWAYS_EXCLUDED_MODULES`, a list of stdlib modules implementing sync and async locks, which will never be profiled. This will effectively be the same behavior: no locks backing higher-level locks (Condition, Semaphore) will be profiled; no sample double-counting; and a perf win by removing now-unnecessary bookeeping.

### What changed

**Before:** When a stdlib primitive (e.g. `threading.Semaphore`) created an internal lock, `_profiled_allocate_lock` would still wrap it as a `_ProfiledLock`, and set `is_internal=True`. The wrapper would exit early on each call to `_acquire`, `_release`, and `_flush_sample`.

**After:** Internal stdlib locks are never wrapped at all thanks to `_ALWAYS_EXCLUDED_MODULES`. `_profiled_allocate_lock` checks the caller's `__name__` against the excluded modules and returns the **native** lock directly on a match.

### Why this is better

| | Before (`is_internal`) | After (`_ALWAYS_EXCLUDED_MODULES`) |
|---|---|---|
| Wrapper created for internal lock? | Yes | **No** |
| Runtime overhead per acquire/release | `self.is_internal` check on every call | **None** |
| Sample suppression sites | `_acquire`, `_release`, `_flush_sample` | — |
| Internal lock detection | Filename comparison (`realpath` per lock create) | Module name set lookup (O(1), done once at patch) |

## Testing
* CI
* 2 new unit tests

## Risks

n/a

> Replaces #17859 which was accidentally auto-merged with 0 commits.


Co-authored-by: vlad.scherbich <vlad.scherbich@datadoghq.com>
dubloom pushed a commit that referenced this pull request May 11, 2026
…rs (#17902)

[< Prev PR](#16775) |

## Description

Remove the `is_internal` per-lock flag from `_ProfiledLock` wrapper, since we will now have the concept of "excluded modules" (#16922). We can replaces it with `_ALWAYS_EXCLUDED_MODULES`, a list of stdlib modules implementing sync and async locks, which will never be profiled. This will effectively be the same behavior: no locks backing higher-level locks (Condition, Semaphore) will be profiled; no sample double-counting; and a perf win by removing now-unnecessary bookeeping.

### What changed

**Before:** When a stdlib primitive (e.g. `threading.Semaphore`) created an internal lock, `_profiled_allocate_lock` would still wrap it as a `_ProfiledLock`, and set `is_internal=True`. The wrapper would exit early on each call to `_acquire`, `_release`, and `_flush_sample`.

**After:** Internal stdlib locks are never wrapped at all thanks to `_ALWAYS_EXCLUDED_MODULES`. `_profiled_allocate_lock` checks the caller's `__name__` against the excluded modules and returns the **native** lock directly on a match.

### Why this is better

| | Before (`is_internal`) | After (`_ALWAYS_EXCLUDED_MODULES`) |
|---|---|---|
| Wrapper created for internal lock? | Yes | **No** |
| Runtime overhead per acquire/release | `self.is_internal` check on every call | **None** |
| Sample suppression sites | `_acquire`, `_release`, `_flush_sample` | — |
| Internal lock detection | Filename comparison (`realpath` per lock create) | Module name set lookup (O(1), done once at patch) |

## Testing
* CI
* 2 new unit tests

## Risks

n/a

> Replaces #17859 which was accidentally auto-merged with 0 commits.


Co-authored-by: vlad.scherbich <vlad.scherbich@datadoghq.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/no-changelog A changelog entry is not required for this PR. Profiling Continous Profling

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants