From e0981beb426f0e567e9f5e0c5e07567a51359a16 Mon Sep 17 00:00:00 2001 From: Emmett Butler Date: Tue, 10 Jun 2025 09:17:26 -0700 Subject: [PATCH 01/11] init contributing-design document --- docs/contributing-design.rst | 38 ++++++++++++++++++++++++++++++++++++ docs/contributing.rst | 29 --------------------------- 2 files changed, 38 insertions(+), 29 deletions(-) create mode 100644 docs/contributing-design.rst diff --git a/docs/contributing-design.rst b/docs/contributing-design.rst new file mode 100644 index 00000000000..839f210db47 --- /dev/null +++ b/docs/contributing-design.rst @@ -0,0 +1,38 @@ +The Design of dd-trace-py +========================= + +Parts of the Library +-------------------- + +When designing a change, one of the first decisions to make is where it should be made. This is an overview +of the main functional areas of the library. + +A **product** is a unit of code within the library that implements functionality specific to a small set of +customer-facing Datadog products. Examples include the `appsec module `_ +implementing functionality for `Application Security Management `_ +and the `profiling `_ module implementing +functionality for `Continuous Profiling `_. Ideally it only contains code +that is specific to the Datadog product being supported, and no code related to Integrations. + +An **integration** is one of the modules in the `contrib `_ +directory, hooking our code into the internal logic of a given Python library. Ideally it only contains code +that is specific to the library being integrated with, and no code related to Products. + +The **core** of the library is the abstraction layer that allows Products and Integrations to keep their concerns +separate. It is implemented in the Python files in the `top level of ddtracepy `_ +and in the `internal` module. As an implementation detail, the core logic also happens to directly support +`Application Performance Monitoring `_. + +Be mindful and intentional about which of these categories your change fits into, and avoid mixing concerns between +categories. If doing so requires more foundational refactoring or additional layers of abstraction, consider +opening an issue describing the limitations of the current design. + + +How Autoinstrumentation Works +----------------------------- + +Autoinstrumentation is the feature of dd-trace-py that hooks into application code to facilitate application-level +product functionality, for example the detailed traces visible in Datadog Application Observability. This is an +overview of the technical implementation of this hooking functionality. + + diff --git a/docs/contributing.rst b/docs/contributing.rst index 13e607a76d3..e2e1538e862 100644 --- a/docs/contributing.rst +++ b/docs/contributing.rst @@ -69,35 +69,6 @@ Each minor version has its own branch. If your pull request is a ``fix`` or ``ci`` change, apply the backport labels corresponding to the minor versions that need the change. -Implementation Guidelines -========================= - -Parts of the Library --------------------- - -When designing a change, one of the first decisions to make is where it should be made. This is an overview -of the main functional areas of the library. - -A **product** is a unit of code within the library that implements functionality specific to a small set of -customer-facing Datadog products. Examples include the `appsec module `_ -implementing functionality for `Application Security Management `_ -and the `profiling `_ module implementing -functionality for `Continuous Profiling `_. Ideally it only contains code -that is specific to the Datadog product being supported, and no code related to Integrations. - -An **integration** is one of the modules in the `contrib `_ -directory, hooking our code into the internal logic of a given Python library. Ideally it only contains code -that is specific to the library being integrated with, and no code related to Products. - -The **core** of the library is the abstraction layer that allows Products and Integrations to keep their concerns -separate. It is implemented in the Python files in the `top level of ddtracepy `_ -and in the `internal` module. As an implementation detail, the core logic also happens to directly support -`Application Performance Monitoring `_. - -Be mindful and intentional about which of these categories your change fits into, and avoid mixing concerns between -categories. If doing so requires more foundational refactoring or additional layers of abstraction, consider -opening an issue describing the limitations of the current design. - Tests ----- From 32989002e4bcacb7a743f8db9560787de7a3268a Mon Sep 17 00:00:00 2001 From: Emmett Butler Date: Tue, 10 Jun 2025 10:13:22 -0700 Subject: [PATCH 02/11] step one --- docs/contributing-design.rst | 36 ++++++++++++++++++++++++++++++++++++ 1 file changed, 36 insertions(+) diff --git a/docs/contributing-design.rst b/docs/contributing-design.rst index 839f210db47..b4b91d21ac9 100644 --- a/docs/contributing-design.rst +++ b/docs/contributing-design.rst @@ -35,4 +35,40 @@ Autoinstrumentation is the feature of dd-trace-py that hooks into application co product functionality, for example the detailed traces visible in Datadog Application Observability. This is an overview of the technical implementation of this hooking functionality. +This is the series of steps involved in autoinstrumentation: +1. import bootstrap.sitecustomize, preload, clean up loaded modules, execute preexisting sitecustomizes +2. start products +3. set up an import hook for each integration +4. when an integrated-with module is imported, the import hook calls the contrib's patch() +5. patch() replaces important functions in the module with transparent wrappers. These wrappers use the core API to create a + tree of ExecutionContext objects. The context tree is traversed to generate the data to send to product intake. + +Step 1: Bootstrap +----------------- + +The autoinstrumentation entrypoint is `import ddtrace.bootstrap.sitecustomize`. This can be done in user code, either directly or via +`import ddtrace.auto`, or behind the scenes by `ddtrace-run`. + +ddtrace's sitecustomize script's basic goal is to start the Products that the library implements. +These are subpackages like `_trace`, `profiling`, `debugging`, `appsec`, et cetera. Before starting these Products, +some setup has to happen, especially the execution of preexisting PEP648 `sitecustomize` scripts to maintain +user-facing guarantees about the runtime environment in which the application will execute. ddtrace's sitecustomize +also attempts to "leave no trace" on the runtime environment, especially by unloading all of the modules it has +used in the course of its setup phases. This helps reduce the possibility that ddtrace will use the same copy of +an integrated-with module that the application uses, which can lead to undefined behavior. + +Step 2: Start Products +---------------------- + + +Step 3: Set Up Import Hooks +--------------------------- + + +Step 4: Import Occurs +--------------------- + + +Step 5: Application Logic Runs +------------------------------ From 1d39868ddd54ee14a95600678be952677c446b57 Mon Sep 17 00:00:00 2001 From: Emmett Butler Date: Tue, 10 Jun 2025 10:45:47 -0700 Subject: [PATCH 03/11] other steps --- docs/contributing-design.rst | 37 +++++++++++++++++++++++++++++++++--- 1 file changed, 34 insertions(+), 3 deletions(-) diff --git a/docs/contributing-design.rst b/docs/contributing-design.rst index b4b91d21ac9..48c4b674cb6 100644 --- a/docs/contributing-design.rst +++ b/docs/contributing-design.rst @@ -40,9 +40,10 @@ This is the series of steps involved in autoinstrumentation: 1. import bootstrap.sitecustomize, preload, clean up loaded modules, execute preexisting sitecustomizes 2. start products 3. set up an import hook for each integration -4. when an integrated-with module is imported, the import hook calls the contrib's patch() -5. patch() replaces important functions in the module with transparent wrappers. These wrappers use the core API to create a - tree of ExecutionContext objects. The context tree is traversed to generate the data to send to product intake. +4. when an integrated-with module is imported, the import hook calls the contrib's patch(). patch() replaces important + functions in the module with transparent wrappers. +5. These wrappers use the core API to create a tree of ExecutionContext objects. The context tree is traversed + to generate the data to send to product intake. Step 1: Bootstrap ----------------- @@ -61,14 +62,44 @@ an integrated-with module that the application uses, which can lead to undefined Step 2: Start Products ---------------------- +The Products implemented by the library conform to a Product Manager Protocol, which controls common functionality +like setup, start, and stop. Each enabled Product module is loaded into the Manager instance on import, which happens in +sitecustomize. In turn, sitecustomize runs the Protocol with `manager.run_protocol()` and later runs additional Product +setup steps via `preload.post_preload`. Together, these calls comprise the setup of Products that will operate during the +application's lifetime. Step 3: Set Up Import Hooks --------------------------- +The core functionality set up during step 2 is autoinstrumentation, which is a prerequisite for many of the other Products. + +Autoinstrumentation setup involves registering hooks that will execute when a particular module is imported. The list of +modules whose imports trigger this registration is defined in `_monkey.py`. During this step, each of these modules has an +import hook registered for it. In the rest of this document, these modules are called "Instrumented Modules". + +Note that as of this writing, autoinstrumentation is implemented as a side effect of the `_trace` Product's setup phase. +In a more abstract sense, autoinstrumentation can function as its own Product, and in the future may be refactored as such. Step 4: Import Occurs --------------------- +The next step of autoinstrumentation happens when the application imports an Instrumented Module. The import triggers the +import hook that was registered in step 3. The function that the hook executes has the primary goal of calling the `patch()` +function of the integration module located at `ddtrace.contrib..patch`. + +The goal of an integration's `patch()` function is to invisibly wrap the Instrumented Module with logic that generates the +data about the module that's necessary for any relevant Products. The most commonly used wrapping approach is based on the +`wrapt` library, which is included as a vendored dependency in `ddtrace`. Inside of the wrappers, it's common for +integrations to build trees of `core.ExecutionContext` objects that store information about the running application's +call stack. + +Whatever approach is taken, this step only sets up logic that will run later. Step 5: Application Logic Runs ------------------------------ + +When the application uses the Instrumented Module after importing it, the wrappers created in step 4 are executed. This causes +data about the running application to be collected in memory, often as a tree of `ExecutionContext` objects. Any Product +that was started in step 2 may access these data and use them to build a payload to send to the relevant intake endpoints. +The classic example is the `_trace` Product, which periodically traverses the context tree for the purpose of creating a `Trace` +object that is subsequently serialized and sent to Datadog to power a flamegraph in the Application Observability product. From 6e23554a1ed4ae30473ca0bf67a9ddf85741bd0b Mon Sep 17 00:00:00 2001 From: Emmett Butler Date: Tue, 10 Jun 2025 10:47:32 -0700 Subject: [PATCH 04/11] link to new doc --- docs/contributing.rst | 3 +++ 1 file changed, 3 insertions(+) diff --git a/docs/contributing.rst b/docs/contributing.rst index e2e1538e862..b958d8e1ddb 100644 --- a/docs/contributing.rst +++ b/docs/contributing.rst @@ -14,6 +14,8 @@ Before working on the library, install `docker `_. +`Library design documentation for contributors `_. + Thanks for working with us! .. _change_process: @@ -108,6 +110,7 @@ Keep the following in mind when writing logging code: .. toctree:: :hidden: + contributing-design contributing-integrations contributing-testing contributing-tracing From 25444b995db273c84be17706eac6ab00bd77e49b Mon Sep 17 00:00:00 2001 From: Emmett Butler <723615+emmettbutler@users.noreply.github.com> Date: Tue, 10 Jun 2025 12:52:59 -0700 Subject: [PATCH 05/11] Update docs/contributing-design.rst Co-authored-by: wantsui --- docs/contributing-design.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/contributing-design.rst b/docs/contributing-design.rst index 48c4b674cb6..1913a01ca9b 100644 --- a/docs/contributing-design.rst +++ b/docs/contributing-design.rst @@ -32,7 +32,7 @@ How Autoinstrumentation Works ----------------------------- Autoinstrumentation is the feature of dd-trace-py that hooks into application code to facilitate application-level -product functionality, for example the detailed traces visible in Datadog Application Observability. This is an +product functionality, for example the detailed traces visible in Datadog Application Monitoring. This is an overview of the technical implementation of this hooking functionality. This is the series of steps involved in autoinstrumentation: From e5b6a10e6dacec9cf84498b66b1c112f75f1fde9 Mon Sep 17 00:00:00 2001 From: Emmett Butler <723615+emmettbutler@users.noreply.github.com> Date: Tue, 10 Jun 2025 12:53:11 -0700 Subject: [PATCH 06/11] Update docs/contributing-design.rst Co-authored-by: wantsui --- docs/contributing-design.rst | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/docs/contributing-design.rst b/docs/contributing-design.rst index 1913a01ca9b..af099c8801f 100644 --- a/docs/contributing-design.rst +++ b/docs/contributing-design.rst @@ -40,8 +40,7 @@ This is the series of steps involved in autoinstrumentation: 1. import bootstrap.sitecustomize, preload, clean up loaded modules, execute preexisting sitecustomizes 2. start products 3. set up an import hook for each integration -4. when an integrated-with module is imported, the import hook calls the contrib's patch(). patch() replaces important - functions in the module with transparent wrappers. +4. when an integrated-with module is imported, the import hook calls the contrib's patch(). patch() replaces important functions in the module with transparent wrappers. 5. These wrappers use the core API to create a tree of ExecutionContext objects. The context tree is traversed to generate the data to send to product intake. From cc3dee78ab565e0433289aaeda317e06aa78c32a Mon Sep 17 00:00:00 2001 From: Emmett Butler Date: Tue, 10 Jun 2025 12:54:08 -0700 Subject: [PATCH 07/11] new name --- docs/contributing-design.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/contributing-design.rst b/docs/contributing-design.rst index af099c8801f..8751b949866 100644 --- a/docs/contributing-design.rst +++ b/docs/contributing-design.rst @@ -9,7 +9,7 @@ of the main functional areas of the library. A **product** is a unit of code within the library that implements functionality specific to a small set of customer-facing Datadog products. Examples include the `appsec module `_ -implementing functionality for `Application Security Management `_ +implementing functionality for `App and API Protection`_ and the `profiling `_ module implementing functionality for `Continuous Profiling `_. Ideally it only contains code that is specific to the Datadog product being supported, and no code related to Integrations. From 8f01350ef5eef135d164ece0132399f2b5874a9f Mon Sep 17 00:00:00 2001 From: Emmett Butler <723615+emmettbutler@users.noreply.github.com> Date: Wed, 11 Jun 2025 06:15:03 -0700 Subject: [PATCH 08/11] Update docs/contributing-design.rst Co-authored-by: Juanjo Alvarez Martinez --- docs/contributing-design.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/contributing-design.rst b/docs/contributing-design.rst index 8751b949866..35360839538 100644 --- a/docs/contributing-design.rst +++ b/docs/contributing-design.rst @@ -88,7 +88,7 @@ function of the integration module located at `ddtrace.contrib. Date: Wed, 11 Jun 2025 06:40:18 -0700 Subject: [PATCH 09/11] formatting --- docs/contributing-design.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/contributing-design.rst b/docs/contributing-design.rst index 35360839538..e3f3748f043 100644 --- a/docs/contributing-design.rst +++ b/docs/contributing-design.rst @@ -9,7 +9,7 @@ of the main functional areas of the library. A **product** is a unit of code within the library that implements functionality specific to a small set of customer-facing Datadog products. Examples include the `appsec module `_ -implementing functionality for `App and API Protection`_ +implementing functionality for `App and API Protection `_ and the `profiling `_ module implementing functionality for `Continuous Profiling `_. Ideally it only contains code that is specific to the Datadog product being supported, and no code related to Integrations. From d45b893799357962fb48cf552d8a58351af93162 Mon Sep 17 00:00:00 2001 From: Emmett Butler Date: Wed, 11 Jun 2025 07:00:15 -0700 Subject: [PATCH 10/11] spellcheck --- docs/spelling_wordlist.txt | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/docs/spelling_wordlist.txt b/docs/spelling_wordlist.txt index 105a916478f..1eb40f058c4 100644 --- a/docs/spelling_wordlist.txt +++ b/docs/spelling_wordlist.txt @@ -27,6 +27,8 @@ asyncio asyncpg attrs autodetected +autoinstrumentation +Autoinstrumentation autopatching autoreload autoreloading @@ -48,6 +50,7 @@ boto botocore cassandra cattrs +cetera CGroup cgroups cherrypy @@ -76,6 +79,7 @@ datastores dbapi ddtrace deallocating +decompiling deprecations DES deserializing @@ -106,6 +110,7 @@ exec Fargate fastapi Firehose +flamegraph fnmatch formatter freezegun @@ -213,6 +218,7 @@ posix postgres pre preconfigured +preload prepend prepended profiler @@ -259,6 +265,8 @@ serializable serverless Serverless sha +sitecustomize +sitecustomizes sns SpanContext sql @@ -326,4 +334,4 @@ Wrapt wsgi xfail yaaredis -openai-agents \ No newline at end of file +openai-agents From b7df5a75670a186706191d6ebd2b7ba196742821 Mon Sep 17 00:00:00 2001 From: Emmett Butler Date: Wed, 11 Jun 2025 07:34:25 -0700 Subject: [PATCH 11/11] spellcheck --- docs/spelling_wordlist.txt | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/spelling_wordlist.txt b/docs/spelling_wordlist.txt index 1eb40f058c4..82b36fac795 100644 --- a/docs/spelling_wordlist.txt +++ b/docs/spelling_wordlist.txt @@ -286,6 +286,7 @@ subdirectory subdomains submodule submodules +subpackages Subprocess substring suitespec