feat: add embedding hiding configuration and align spec with instrumentation

[codefromthecrypt]

Impact

This PR enhances data privacy for embedding operations and brings consistency across all embedding instrumentation providers, including BeeAI and Haystack.

Key Features

🔒 Privacy Controls for Embeddings

• New configuration options to protect sensitive embedding data:
  • OPENINFERENCE_HIDE_EMBEDDINGS_VECTORS: Redacts embedding vectors with "__REDACTED__"
  • OPENINFERENCE_HIDE_EMBEDDINGS_TEXT: Redacts embedding text content
• Consistent redaction behavior across all instrumentation packages

🎯 Standardized Embedding Instrumentation

All providers now use:

• Consistent span name: "CreateEmbeddings"
• Standardized attribute structure: embedding.embeddings.N.embedding.{text|vector}
• Unified invocation parameter tracking: embedding.invocation_parameters
• Proper llm.system attribute for provider identification

Provider-specific improvements:

BeeAI:

• Added embedding.invocation_parameters extraction from input events
• Ensured embedding vectors are always lists (not tuples)
• Added proper llm.system: "beeai" identification
• Removed unnecessary child spans for cleaner traces

Haystack:

• Added dynamic llm.system based on component class name
• Improved embedding vector decoding (supports base64 and raw arrays)
• Added invocation parameter extraction (excludes input data)
• Enhanced numpy array compatibility

OpenAI/LiteLLM:

• Full batch embedding support with indexed attributes
• Separated invocation parameters from input data
• Improved handling of token IDs vs text inputs

📋 Specification Alignment

• Updated spec to clarify when text attributes are populated (only for string inputs, not token IDs)
• Documented why token IDs aren't decoded (cross-provider incompatibility, runtime constraints)
• Clarified redaction behavior for each configuration option

[codefromthecrypt]

I will pull out of draft once I unbreak the build

[codefromthecrypt]

@mikeldking this should be ready to review because the code that does embedding is up to date. There are a couple red packages unrelated to this change and need some work to get green.

Agno: It essentially doesn't function because it's only partially ported to version 2.0. We need to bump the minimum version requirement to 2.0, as 1.5 is no longer compatible. Additionally, the tests aren't correct because they are only catching the LLM spans not the http requests made by the agent. This results in inconsistent runs.If you'd like, I can share my partially completed branch, but I won't be able to dedicate time to iterating on it further.

smolagents : Fixing the mypy errors in smolagents might require less effort overall.

[codefromthecrypt]

here's the branch but I can't complete but got pretty far https://github.com/Arize-ai/openinference/compare/main...codefromthecrypt:openinference:refactor-agno-tests?expand=1

[codefromthecrypt]

added the smolagents fix. crewai is new but I can't reproduce that locally. agno is the biggie, but not related to this change either

[mikeldking]

we need this to be ddgs as it got renamed for agno

[codefromthecrypt]

cool.

[codefromthecrypt]

FYI I plan to break apart this PR into pieces, to make it easier to review. so any comments here I'll carry over to the partitioned PRs and mark this draft until only the spec/substrate remains.

[codefromthecrypt]

part 1: openai #2210

once in I'll do the same changes for litellm and pull both off this PR