Logs data model canonical OTLP encoding for null any values

[aabmass]

What are you trying to achieve?

Updating the Python OTLP exporter to support None values nested within the Logs SDK any value open-telemetry/opentelemetry-python#4400. It is unclear how to convert null values into OTLP protobuf AnyValue messages. I may have missed it in the spec already but there are some unclear cases:

1. KeyValue with null value: {"foo": None}. Should KeyValue.value be left unset or get an empty AnyValue?

2. ArrayValue containing null: {"foo": [1, 2, None]}. The common spec is pretty clear that exporters should preserve the ordering:

   If exporters do not support exporting null values, they MAY replace those values by 0, false, or empty strings. This is required for map/dictionary structures represented as two arrays with indices that are kept in sync

   I think the empty AnyValue would be appropriate here.

3. Body is null. Should LogRecord.body be left unset or be set to an explicit empty AnyValue?

For languages with undefined vs null distinction, this all gets more complicated so we should consider OTel JS too.

Additional context.

Note that 2 affects tracing and metrics as well since "null values within arrays MUST be preserved as-is (i.e., passed on to span processors / exporters as null)".

[aabmass]

Protobuf's standard Struct message uses a separate oneof variant for NullValue.

[danielgblanco]

@open-telemetry/javascript-maintainers tagging you here to get your perspective on this, as @aabmass mentioned JS above.

[pellared]

@aabmass, is it not a duplicate of #3835 which has been already addressed via #3853?

For languages with undefined vs null distinction.

Both should be encoded as empty empty AnyValue?

Should LogRecord.body be left unset or be set to an explicit empty AnyValue?

What is the difference?

Notice that "log attributes" have currently different definition from "standard attributes".
There is an ongoing work to extend the "standard attributes" so that they also contain the types from Logs Data Model.
See:

• Allow adding new attribute types #4602
• all: drop attribute values restrictions opentelemetry-proto#707
• [EndGoal] [DoNotMerge] Extend the set of attribute value types #4636

Can we close this issue as a duplicate of #3835?

[aabmass]

I don't think so. #3853 is for the data model. It only says that values may be null but not how to serialize them into OTLP. I'm looking for clarity on how to serialize null values into OTLP (JSON and Protobuf).

[pellared]

The answer for OTLP Protobuf is:

Empty values MUST be converted to AnyValue with no value field being set.

As for OTLP JSON https://github.com/open-telemetry/opentelemetry-proto/blob/main/docs/specification.md#json-protobuf-encoding it should be JSON Protobuf encoded empty AnyValue. @open-telemetry/collector-approvers, is this what otlphttpexporter is doing?

[jade-guiton-dd]

@pellared Did some tests with the pdata JSON/protobuf serializer used by otlphttpexporter, and confirmed the results are as expected:

• nil converts to an empty AnyValue when using the FromRaw method
• an empty AnyValue is encoded as a zero byte AnyValue message in protobuf, and {} in JSON
• a KeyValue with an empty value does not omit the KeyValue.value field
• an ArrayValue containing an empty value does not omit it, reorder it, or replace it by something else
• a LogRecord with an empty body does not omit the LogRecord.body field