Source not being mapped in Json return from minimal API

[werto165]

Elastic.Clients.Elasticsearch version:
8.14.6
Elasticsearch version:
8.14.0
.NET runtime version:
.NET8
Operating system version:
Windows 11
Description of the problem including expected versus actual behavior:

I first noticed this issue when trying to return SearchResponse from a minimal api endpoint. There is something going wrong with the serialization at some point in the serialization chain.

When using minimal APIs there is a silent error (no exceptions or anything untoward) when using System.Text.Json.JsonSerializer the default serializer used for response bodies, this results in the message "can't parse JSON" (shows up in swagger only, so potentially malformed json?) when running a search returning the type Results<Ok<SearchResponse>>.

group.MapGet("testEP",
        async Task<Results<Ok<SearchResponse<Individual>>, UnauthorizedHttpResult, ProblemHttpResult, NotFound>> (ElasticsearchClient esc, string query, HttpResponse context) =>
        {
            //await Microsoft.AspNetCore.Http.HttpResponseJsonExtensions.WriteAsJsonAsync(context, response);
            var options = new JsonSerializerOptions
            {
                WriteIndented = true,
                AllowTrailingCommas = true,
                IncludeFields = true,

            };
            var request1 = new SearchRequest("individual")
            {
                From = 0,
                Size = 10,
                Query = new MatchQuery("firstName") { Query = query },

            };
            var response1 = await esc.SearchAsync<Individual>(request1);
            var jsonResponse = JsonConvert.SerializeObject(response1.Hits.First());
            var output = System.Text.Json.JsonSerializer.Serialize<Hit<Individual>>(response1.Hits.First(), options);
            return TypedResults.Ok(response1);
});

JsonResponse

{
  "Explanation": null,
  "Fields": null,
  "Highlight": null,
  "Id": "XRlRE5EBezDyKn2DnhaG",
  "Ignored": null,
  "IgnoredFieldValues": null,
  "Index": "individual",
  "InnerHits": null,
  "MatchedQueries": null,
  "Nested": null,
  "Node": null,
  "PrimaryTerm": null,
  "Routing": null,
  "Score": 0.2876821,
  "SeqNo": null,
  "Shard": null,
  "Sort": null,
  "Source": { "firstName": "bill", "lastName": "johnson" },
  "Version": null
}

Output

{
  "_explanation": null,
  "fields": null,
  "highlight": null,
  "_id": "XRlRE5EBezDyKn2DnhaG",
  "_ignored": null,
  "ignored_field_values": null,
  "_index": "individual",
  "inner_hits": null,
  "matched_queries": null,
  "_nested": null,
  "_node": null,
  "_primary_term": null,
  "_routing": null,
  "_score": 0.2876821,
  "_seq_no": null,
  "_shard": null,
  "sort": null,
  "_source": 
  "_version": null
}

return from endpoint

{
  "aggregations": null,
  "_clusters": null,
  "fields": null,
  "hits": {
    "hits": [
      {
        "_explanation": null,
        "fields": null,
        "highlight": null,
        "_id": "XRlRE5EBezDyKn2DnhaG",
        "_ignored": null,
        "ignored_field_values": null,
        "_index": "individual",
        "inner_hits": null,
        "matched_queries": null,
        "_nested": null,
        "_node": null,
        "_primary_term": null,
        "_routing": null,
        "_score": 0.2876821,
        "_seq_no": null,
        "_shard": null,
        "sort": null,
        "_source": 
        "_version": null
      }
    ],
    "max_score": 0.2876821,
    "total": {}
  },
  "max_score": null,
  "num_reduce_phases": null,
  "pit_id": null,
  "profile": null,
  "_scroll_id": null,
  "_shards": {
    "failed": 0,
    "failures": null,
    "skipped": 0,
    "successful": 1,
    "total": 1
  },
  "suggest": null,
  "terminated_early": null,
  "timed_out": false,
  "took": 1
}

Steps to reproduce:

1. create a minimal api project
2. DI the elastic search client.
3. try and view the return type of SearchResult resulting in an error.

Expected behavior

The source value should be mapped but isn't, may be erroring because there is a missing comma from the return type, that is why the json is malformed. As there should be a value which is created but isn't be created?

JsonResponse returns source just fine.
output doesn't return the source value. This is the same or similar to what is being used for serialization by ASPNET.

I have a feeling it is related to the following code:

public override void Write(Utf8JsonWriter writer, T value, JsonSerializerOptions options)
	{
		var converter = options.GetConverter(typeof(SourceMarker<>).MakeGenericType(typeof(T)));

		if (converter is SourceConverter<T> sourceConverter)
		{
			sourceConverter.Write(writer, new SourceMarker<T> { Source = value }, options); //does not hit breakpoint set here.
		}
	}

in IntermediateSourceConverter.cs

When it works it hits only the Read code, when it doesn't it doesn't hit the Write because the type check always fails.

{Elastic.Clients.Elasticsearch.Serialization.SourceConverter} (READ function)
{System.Text.Json.Serialization.Converters.ObjectDefaultConverter<Elastic.Clients.Elasticsearch.Serialization.SourceMarker>} (WRITE function)

[flobernd]

Hi @werto165,

Manually (de-)serializing any of the Elasticsearch client types is generally unsupported. You might be lucky when using client.RequestResponseSerializer instead of using JsonConvert/JsonSerializer, but there is no guarantee.

One reason for this is, that internal converters are not registered for the default JsonSerializer, but only for the internally used instance. This can be worked around using the client.RequestResponseSerializer.

Another reason is, that types do not neccessarily support round-tripping. E.g. request-only types can only be serialized while response-only types can only be deserialized. This can not be worked around. If the response uses one of these types, you are out of luck.

We usually suggest to create a custom POCO class / model to which you assign all the required properties from the Elasticsearch client response. This could for example look like this:

var response = new ResponseModel<Individual>
{
  Took = clientResponse.Took,
  // ...
  Hits = new HitsModel<Individual>[]
  {
    new HitsModel<Individual>
    {
        Source = clientResponse.Source
        // ...
    }
  }
}

Btw.: Re-serializing the actual document (type: Individual) always works - even with the default JsonSerializer.

Please let me know if I answered your question.

[werto165]

Hi there,

Thanks for the response. I am quite new to elastic search in general, so apologies for the trivial question again, but it seems strange that I can't have a wrapper around the normal REST API when that is the response you get from the actual API i.e SearchResponse, I have cross checked most of the fields on this response type and it seems to marry up with what is being returned by the actual REST API. I will give the custom POCO a go.

With regards to your comment about (de-)serialization are you saying that the _source returned from the response may also be used elsewhere in requests? Therefore this field is not serialized properly?

If you could answer another query I have it would be greatly appreciated, I have types that define my mappings, I am using the old client to create my mapping for the index. With this mapping I have set dynamic to "strict" which has allowed me to enforce that when passing an update for example:

POST myindex/_update
{
doc: { "fieldwhichdoesntexist" : "value"}
}

it throws an error. However, I am having trouble enforcing types, so if in my mapping I set the type to "text" for example, in the update i can set this field to whatever data type such as integer, this creates a problem in C# because the class is expecting a string but with the update it has allowed me to change it to an int. Therefore when deserializing the response into the object it throws an error. Are there any mechanisms that can prevent changing types with the API?

[flobernd]

Hi @werto165,

it seems strange that I can't have a wrapper around the normal REST API when that is the response you get from the actual API i.e SearchResponse

This is absolutely no problem. Returning, using, wrapping the SearchResponse and other response types is 100% valid. You just can't manually (re-)serialize them.

With regards to your comment about (de-)serialization are you saying that the _source returned from the response may also be used elsewhere in requests? Therefore this field is not serialized properly?

_source is your custom POCO type (Individual in this case). This type you can manually (re-)serialize without any constraints. The Elasticsearch Client types are the problematic ones. It's for example not trivially possible to (re-)serialize Hit<Individual>.

If you could answer another query I have it would be greatly appreciated

I'm not quite sure I can completely follow that question, but it seems like server side behavior. It would be the best to ask this in our Community Forums.

I'm going to close this issue for now, but feel free to open another one if you have problems related to the .NET client.