test(llmobs): migrate llama_index tests to assert on LLMObsSpanData by Yun-Kim · Pull Request #17797 · DataDog/dd-trace-py

Yun-Kim · 2026-04-30T18:28:57Z

Migrates llama_index LLMObs tests from inspecting projected wire LLMObsSpanEvents (via mock_llmobs_writer.enqueue.* / llmobs_events) to reading the canonical LLMObsSpanData directly off span.meta_struct["_llmobs"] and asserting via assert_llmobs_span_data(_get_llmobs_data_metastruct(spans[i]), ...).

Conftest: replaces the test_spans override / mock_llmobs_writer plumbing with a llama_index_llmobs(tracer, monkeypatch) fixture that sets _DD_LLMOBS_TEST_KEEP_META_STRUCT=1, enables LLMObs against the test tracer, and mocks the span writer.

Stacked on #17789.

cit-pr-commenter-54b7da · 2026-04-30T18:28:58Z

Codeowners resolved as

tests/contrib/llama_index/conftest.py                                   @DataDog/python-guild
tests/contrib/llama_index/test_llama_index_llmobs.py                    @DataDog/apm-core-python @DataDog/apm-idm-python

pr-commenter · 2026-04-30T18:58:56Z

Benchmarks

Benchmark execution time: 2026-04-30 18:58:54

Comparing candidate commit 6445ccc in PR branch yunkim/llmobs-test-meta-struct-llama_index with baseline commit 9daa831 in branch yunkim/llmobs-test-meta-struct-openai.

Found 0 performance improvements and 3 performance regressions! Performance is the same for 584 metrics, 4 unstable metrics.

scenario:iastaspects-lstrip_noaspect

🟥 execution_time [+18.916µs; +29.281µs] or [+10.789%; +16.701%]

scenario:iastaspectsospath-ospathbasename_aspect

🟥 execution_time [+83.917µs; +94.112µs] or [+19.539%; +21.913%]

scenario:telemetryaddmetric-1-count-metric-1-times

🟥 execution_time [+149.766ns; +180.301ns] or [+7.087%; +8.532%]

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

…x_llmobs fixture Mirrors the pydantic_ai pattern: a named opt-in fixture per integration enables LLMObs, sets _DD_LLMOBS_TEST_KEEP_META_STRUCT=1, mocks the writer, and injects _dd_api_key + _llmobs_ml_app + service via override_global_config. Tests request the fixture explicitly instead of activating LLMObs through the implicit ddtrace_global_config parametrize, and the class-level parametrize decorator (and the now-unused LLMOBS_GLOBAL_CONFIG dict) are gone. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

…_llama_index fixtures The integration registers no config keys and no test parametrizes either fixture, so the override_config wrapper was a no-op. Inline the single-use default_global_config() helper as well. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

…ixture The base fixture's _dd_api_key override only takes effect for APM-only tests in test_llama_index.py — none of which read it (APM tracing routes through the trace agent, no api_key validation). The llama_index_llmobs fixture sets _dd_api_key independently for LLMObs tests, where the inner override shadows this one. Vestigial from before the llmobs fixture split. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Yun-Kim · 2026-05-04T17:06:54Z

/merge

gh-worker-devflow-routing-ef8351 · 2026-05-04T17:06:57Z

View all feedbacks in Devflow UI.

2026-05-04 17:06:57 UTC ℹ️ Start processing command /merge

2026-05-04 17:07:04 UTC ℹ️ MergeQueue: waiting for PR to be ready

This pull request is not mergeable according to GitHub. Common reasons include pending required checks, missing approvals, or merge conflicts — but it could also be blocked by other repository rules or settings.
It will be added to the queue as soon as checks pass and/or get approvals. View in MergeQueue UI.
Note: if you pushed new commits since the last approval, you may need additional approval.
You can remove it from the waiting list with /remove command.

2026-05-04 20:28:10 UTC ℹ️ MergeQueue: merge request added to the queue

The expected merge time in main is approximately 52m (p90).

2026-05-04 22:28:01 UTC 🚨 MergeQueue: This merge request is in error

error while getting head build completion result

Details

Error: There was an error while retrieving the result for pipeline 111357059

FullStacktrace:
child workflow execution error (type: mergequeue_private.MergeQueue_WaitForChecksOrUntilIsFinal, workflowID: 019df4ad-48b8-78d8-a685-bbc7cadc67a7_72, runID: 019df4ad-aa15-7d2b-9070-221ba4e193b3, initiatedEventID: 72, startedEventID: 73): child workflow execution error (type: mergequeue.MergeQueue_WaitForCompletionOfRef, workflowID: 019df4ad-aa15-7d2b-9070-221ba4e193b3_8, runID: 019df4ad-aa91-721b-8650-008e82f82d53, initiatedEventID: 8, startedEventID: 10): There was an error while retrieving the result for pipeline 111357059 (type: FlowError, retryable: false): There was an error while retrieving the result for pipeline 111357059

wantsui

Approving from an integrations perspective since this change is limited to LLM.

Yun-Kim · 2026-05-05T14:47:59Z

/merge

gh-worker-devflow-routing-ef8351 · 2026-05-05T14:48:04Z

View all feedbacks in Devflow UI.

2026-05-05 14:48:04 UTC ℹ️ Start processing command /merge

2026-05-05 14:48:09 UTC ℹ️ MergeQueue: pull request added to the queue

The expected merge time in main is approximately 51m (p90).

2026-05-05 15:36:12 UTC ❌ MergeQueue: The checks failed on this merge request

Tests failed on this commit 56ea6d7:

serverless-system-tests / Get parameters / Get parameters

What to do next?

Investigate the failures and when ready, re-add your pull request to the queue!
If your PR checks are green, try to rebase/merge. It might be because the CI run is a bit old.
Any question, go check the FAQ.

Yun-Kim · 2026-05-05T15:43:40Z

/merge

gh-worker-devflow-routing-ef8351 · 2026-05-05T15:43:45Z

View all feedbacks in Devflow UI.

2026-05-05 15:43:44 UTC ℹ️ Start processing command /merge

2026-05-05 15:43:49 UTC ℹ️ MergeQueue: pull request added to the queue

The expected merge time in main is approximately 52m (p90).

2026-05-05 16:36:24 UTC ℹ️ MergeQueue: This merge request was merged

…17797) Migrates llama_index LLMObs tests from inspecting projected wire `LLMObsSpanEvent`s (via `mock_llmobs_writer.enqueue.*` / `llmobs_events`) to reading the canonical `LLMObsSpanData` directly off `span.meta_struct["_llmobs"]` and asserting via `assert_llmobs_span_data(_get_llmobs_data_metastruct(spans[i]), ...)`. Conftest: replaces the `test_spans` override / `mock_llmobs_writer` plumbing with a `llama_index_llmobs(tracer, monkeypatch)` fixture that sets `_DD_LLMOBS_TEST_KEEP_META_STRUCT=1`, enables LLMObs against the test tracer, and mocks the span writer. Stacked on #17789. Co-authored-by: yun.kim <yun.kim@datadoghq.com>

Yun-Kim requested review from a team as code owners April 30, 2026 18:28

Yun-Kim requested review from emmettbutler, rithikanarayan and wconti27 and removed request for a team April 30, 2026 18:28

Yun-Kim added the changelog/no-changelog A changelog entry is not required for this PR. label Apr 30, 2026

Yun-Kim removed request for emmettbutler, rithikanarayan and wconti27 April 30, 2026 19:29

Yun-Kim force-pushed the yunkim/llmobs-test-meta-struct-openai branch from 57b1e17 to d6f7311 Compare May 2, 2026 20:05

Yun-Kim requested a review from a team as a code owner May 2, 2026 20:05

Yun-Kim force-pushed the yunkim/llmobs-test-meta-struct-llama_index branch from 26b464d to 334ce20 Compare May 2, 2026 20:38

Base automatically changed from yunkim/llmobs-test-meta-struct-openai to main May 4, 2026 15:23

Yun-Kim and others added 4 commits May 4, 2026 11:31

test(llmobs): migrate llama_index tests to assert on LLMObsSpanData

36b80e4

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Yun-Kim force-pushed the yunkim/llmobs-test-meta-struct-llama_index branch from 334ce20 to 7269cc6 Compare May 4, 2026 15:31

ZStriker19 approved these changes May 4, 2026

View reviewed changes

wantsui approved these changes May 4, 2026

View reviewed changes

Yun-Kim mentioned this pull request May 4, 2026

test(llmobs): migrate tests/llmobs SDK tests to assert on LLMObsSpanData #17876

Merged

Merge branch 'main' into yunkim/llmobs-test-meta-struct-llama_index

48c31d9

gh-worker-dd-mergequeue-cf854d Bot merged commit 831157c into main May 5, 2026
282 checks passed

gh-worker-dd-mergequeue-cf854d Bot deleted the yunkim/llmobs-test-meta-struct-llama_index branch May 5, 2026 16:36

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

test(llmobs): migrate llama_index tests to assert on LLMObsSpanData#17797

test(llmobs): migrate llama_index tests to assert on LLMObsSpanData#17797
gh-worker-dd-mergequeue-cf854d[bot] merged 5 commits into
mainfrom
yunkim/llmobs-test-meta-struct-llama_index

Yun-Kim commented Apr 30, 2026 •

edited

Loading

Uh oh!

cit-pr-commenter-54b7da Bot commented Apr 30, 2026 •

edited

Loading

Uh oh!

pr-commenter Bot commented Apr 30, 2026

Uh oh!

Yun-Kim commented May 4, 2026

Uh oh!

gh-worker-devflow-routing-ef8351 Bot commented May 4, 2026 •

edited

Loading

Uh oh!

wantsui left a comment

Uh oh!

Yun-Kim commented May 5, 2026

Uh oh!

gh-worker-devflow-routing-ef8351 Bot commented May 5, 2026 •

edited

Loading

Uh oh!

Yun-Kim commented May 5, 2026

Uh oh!

gh-worker-devflow-routing-ef8351 Bot commented May 5, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

Yun-Kim commented Apr 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cit-pr-commenter-54b7da Bot commented Apr 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codeowners resolved as

Uh oh!

pr-commenter Bot commented Apr 30, 2026

Benchmarks

scenario:iastaspects-lstrip_noaspect

scenario:iastaspectsospath-ospathbasename_aspect

scenario:telemetryaddmetric-1-count-metric-1-times

Uh oh!

Yun-Kim commented May 4, 2026

Uh oh!

gh-worker-devflow-routing-ef8351 Bot commented May 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

wantsui left a comment

Choose a reason for hiding this comment

Uh oh!

Yun-Kim commented May 5, 2026

Uh oh!

gh-worker-devflow-routing-ef8351 Bot commented May 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What to do next?

Uh oh!

Yun-Kim commented May 5, 2026

Uh oh!

gh-worker-devflow-routing-ef8351 Bot commented May 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Yun-Kim commented Apr 30, 2026 •

edited

Loading

cit-pr-commenter-54b7da Bot commented Apr 30, 2026 •

edited

Loading

gh-worker-devflow-routing-ef8351 Bot commented May 4, 2026 •

edited

Loading

gh-worker-devflow-routing-ef8351 Bot commented May 5, 2026 •

edited

Loading

gh-worker-devflow-routing-ef8351 Bot commented May 5, 2026 •

edited

Loading