test(llmobs): migrate claude_agent_sdk tests to assert on LLMObsSpanData#17800
Merged
gh-worker-dd-mergequeue-cf854d[bot] merged 6 commits intoMay 5, 2026
Conversation
Codeowners resolved as |
Yun-Kim
added
the
changelog/no-changelog
Yun-Kim
force-pushed
the
yunkim/llmobs-test-meta-struct-openai
branch
from
57b1e17 to
d6f7311
Compare
Yun-Kim
force-pushed
the
yunkim/llmobs-test-meta-struct-claude_agent_sdk
branch
from
26cd090 to
bf11c70
Compare
Migrates contrib claude_agent_sdk LLMObs tests from llmobs_events-based wire-event assertions to assert_llmobs_span_data(_get_llmobs_data_metastruct(spans[i]), ...), asserting against the canonical LLMObsSpanData payload on the span's meta_struct["_llmobs"]. Stacked on #17789 (openai PR — adds the matcher's role normalization and mock.ANY whole-value support). What changes: - conftest.py: new test_spans override sets _DD_LLMOBS_TEST_KEEP_META_STRUCT=1 via monkeypatch so the meta_struct scrub is skipped while LLMObs is enabled. Mocks out _llmobs_span_writer so no background flush thread is left running. Wraps override_global_config(ddtrace_global_config) because the existing claude_agent_sdk fixture only sets per-integration config, not global config. - Drops the now-unused llmobs, llmobs_span_writer, llmobs_events fixtures from conftest, plus the default_global_config helper. Full migration, no surviving fixtures. - Drops the _expected_llmobs_llm_span_event / _expected_llmobs_non_llm_span_event imports. - token_metrics= -> metrics=; per-kwarg error fields (error=, error_message=, error_stack=) collapse into a single error={"type", "message", "stack"} dict. The unittest.mock.ANY sentinel is used for stack on every error case (was previously only used for the ValueError stack). - Module-level LLMOBS_GLOBAL_CONFIG constant; the test class is decorated with @pytest.mark.parametrize("ddtrace_global_config", [LLMOBS_GLOBAL_CONFIG]) to trigger the override path for every test. - pop_traces is drained across all traces (spans = [s for trace in pop_traces() for s in trace]) and len(spans) is asserted up-front, replacing the previous len(llmobs_events) checks. - The multi-turn test's final agent-span check, which was previously a one-line ["meta"]["span"]["kind"] poke, is upgraded to assert_llmobs_span_data(..., span_kind="agent") for parity with the rest of the suite. Fixtures left for review: none — fully migrated. The two _llmobs_span_writer references that remain in conftest.py are the standard mock-and-stop pattern from the test_spans override (mirrored from openai/google_adk/mcp), not surviving fixtures. The claude_agent_sdk venv build fails locally with a Rust native-ext compilation error in the riot docker container (sccache cannot find a temp file path under /home/bits/project/src/native/target3.11/release/deps/); this reproduces unchanged on the parent branch and is environmental, not introduced here. Migrated tests will exercise correctly in CI. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…onfig_claude_agent_sdk fixtures The integration registers no config keys and no test parametrizes either fixture, so the override_config wrapper was a no-op. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Yun-Kim
force-pushed
the
yunkim/llmobs-test-meta-struct-claude_agent_sdk
branch
from
bf11c70 to
e6991a2
Compare
ZStriker19
approved these changes
May 4, 2026
Contributor
Author
|
/merge |
|
View all feedbacks in Devflow UI.
The expected merge time in
The merge request has been interrupted because the build 0 took longer than expected. The current limit for the base branch 'main' is 120 minutes. |
Contributor
Author
|
/merge |
|
View all feedbacks in Devflow UI.
The expected merge time in
|
gh-worker-dd-mergequeue-cf854d
Bot
deleted the
yunkim/llmobs-test-meta-struct-claude_agent_sdk
branch
emmettbutler
pushed a commit
that referenced
this pull request
May 6, 2026
…ata (#17800) Migrates claude_agent_sdk LLMObs tests from inspecting projected wire `LLMObsSpanEvent`s (via `mock_llmobs_writer.enqueue.*` / `llmobs_events`) to reading the canonical `LLMObsSpanData` directly off `span.meta_struct["_llmobs"]` and asserting via `assert_llmobs_span_data(_get_llmobs_data_metastruct(spans[i]), ...)`. Conftest: replaces the `test_spans` override / `mock_llmobs_writer` plumbing with a `claude_agent_sdk_llmobs(tracer, monkeypatch)` fixture that sets `_DD_LLMOBS_TEST_KEEP_META_STRUCT=1`, enables LLMObs against the test tracer, and mocks the span writer. Stacked on #17789. Co-authored-by: yun.kim <yun.kim@datadoghq.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Migrates claude_agent_sdk LLMObs tests from inspecting projected wire
LLMObsSpanEvents (viamock_llmobs_writer.enqueue.*/llmobs_events) to reading the canonicalLLMObsSpanDatadirectly offspan.meta_struct["_llmobs"]and asserting viaassert_llmobs_span_data(_get_llmobs_data_metastruct(spans[i]), ...).Conftest: replaces the
test_spansoverride /mock_llmobs_writerplumbing with aclaude_agent_sdk_llmobs(tracer, monkeypatch)fixture that sets_DD_LLMOBS_TEST_KEEP_META_STRUCT=1, enables LLMObs against the test tracer, and mocks the span writer.Stacked on #17789.