Cosmos Data Extension GetPropertyValue Fails with Nested Partition Key Path

[cf-ehakin]

When transferring from Azure Blob Json source to Cosmos DB sink, the following error occurs if partition key path on destination DB is:

/partitionkeyvalue1/partitionkeyvalue2

Sink Settings are

 "SinkSettings": {
                "Container": "CosmosContainer",
                "IncludeMetaDataFields": true,
                "PartitionKeyPath": "/partitionkeyvalue1/partitionkeyvalue2"
            }

Source Json is:

[
  {
    "_segments": [],
    "_executionLogs": [],
    "partitionkeyvalue1": {
		"partitionkeyvalue2": "<GUID>",
		"somevaule4": "<GUID>",
		"UserName": "<username>",
		"othervalue": "<data>",
		"othervalue3": "<data_2>"
    }
  }
]

This produces an error:

  Data transfer failed
    System.AggregateException: One or more errors occurred. (The specified key 'partitionkeyvalue1/partitionkeyvalue2' does not exist in the ExpandoObject.)
     ---> System.Collections.Generic.KeyNotFoundException: The specified key 'partitionkeyvalue1/partitionkeyvalue2' does not exist in the ExpandoObject.
       at System.Dynamic.ExpandoObject.System.Collections.Generic.IDictionary<System.String,System.Object>.get_Item(String key)
       at Cosmos.DataTransfer.CosmosExtension.CosmosDataSinkExtension.GetPropertyValue(ExpandoObject item, String propertyName) in /__w/data-migration-desktop-tool/data-migration-desktop-tool/Extensions/Cosmos/Cosmos.DataTransfer.CosmosExtension/CosmosDataSinkExtension.cs:line 211

This seems to be directly related to the GetPropertyValue method:

private static string? GetPropertyValue(ExpandoObject item, string propertyName)
{
    return ((IDictionary<string, object?>)item)[propertyName]?.ToString();
}

Is this something that can be addressed, or are we defining the partition key in a not best practice manner?

[markjbrown]

Am I reading this right? You are defining a partition key that is an object?

This sort of thing is not allowed.

"partitionkeyvalue1": {
		"partitionkeyvalue2": "<GUID>",
		"somevaule4": "<GUID>",
		"UserName": "<username>",
		"othervalue": "<data>",
		"othervalue3": "<data_2>"
    }

Partition keys must be strings. Partition key definitions for a container are an array with 1-3 properties that form a hierarchy of values. Ideally this would be a hierarchy innate to the data itself, (e.g. /companyId, /departmentId, /employeeId) but technically it can be any string values.

[cf-ehakin]

Thanks for getting back to me. No, the container defines PartitionKeyPath of nestedProperty/someValue - where someValue is a string value.

So partitionkeypath = /partitiionkeyvalue1/partitionkeyvalue2

The JSON within Cosmos looks like:

{
  "_segments": [],
  "_executionLogs": [],
  "partitionkeyvalue1": {
      "partitionkeyvalue2": "stringvalue"
      "idvalue_2": "idstring"
   }
...
}

[markjbrown]

Can you send me the json output for the partition key definition from az cosmosdb sql container show for this container?

You can't create a container with a partition key like what I'm seeing here. I want to be sure I'm clear on your partition key definition.

Thanks.

[cf-ehakin]

Here's the output:

  "partitionKey": {
      "kind": "Hash",
      "paths": [
        "/partitionkeyvalue1/partitionkeyvalue2"
      ],
      "systemKey": null,
      "version": null
    },

Thanks for your help.

[markjbrown]

Ok got it :) You can't define a hierarchical partition key like this. This would only result in a single partition key of partitionkeyvalue2 with this path, "/partitionkeyvalue1/partitionkeyvalue2". Also the kind is incorrect for hierarchical and you need to specify version number as well. 2 is preferred for either type of partition key.

If you were defining a hierarchical partition key it would need to look like this.

"partitionKey": {
"kind": "MultiHash",
"paths": [
"/partitionkeyvalue1",
"/partitionkeyvalue2"
],
"version": 2
},

[davidames]

Hi @markjbrown , I work with @cf-ehakin.

I can provide some more context.

We are not trying to define a hierarchical partition key; we are defining our partition key as pointing to a value that is nested inside the document, as per

REF: https://learn.microsoft.com/en-us/azure/cosmos-db/partitioning-overview#choose-a-partition-key

[markjbrown]

Ok, now I got it!!! Your property names were confusing me :)

My spidy senses make me think this is an escaping issue.

@bowencode do you have thoughts on this? Any chance you can try to repro?