`polymorphicDefaultDeserializer` won't work with `useArrayPolymorphism=true`

[Komdosh]

Describe the bug

polymorphicDefaultDeserializer won't work with useArrayPolymorphism=true

To Reproduce

public fun main() {
    val arrayPolymorphism = true
    val json = Json {
        serializersModule = SerializersModule {
            useArrayPolymorphism = arrayPolymorphism
            polymorphicDefaultDeserializer(ChildWithPoly::class) { ChildWithPoly.ChildLegacy.serializer() }
        }
    }
    val obj = Parent(ChildWithPoly.ChildLegacy("Hello World!"))

    println(json.encodeToString(Parent.serializer(), obj))

    if (arrayPolymorphism) {
        val parentWithType = json.decodeFromString<Parent>("""{"child":["ChildLegacy",{"s":"Hello World!"}]}""")
        println(obj == parentWithType)
    } else {
        val parentWithType = json.decodeFromString<Parent>("""{"child":{"type":"ChildLegacy", "s":"Hello World!"}}""")
        println(obj == parentWithType)
    }

    val parentWithoutType = json.decodeFromString<Parent>("""{"child":{"s":"Hello World!"}}""")
    println(obj == parentWithoutType) // fail if arrayPolymorphism == true
}

@Serializable
public data class Parent(
    val child: ChildWithPoly,
)

@Serializable
public sealed interface ChildWithPoly {

    @Serializable
    @SerialName("ChildLegacy")
    public data class ChildLegacy(
        val s: String,
    ) : ChildWithPoly

    @Serializable
    @SerialName("ChildNew")
    public data class ChildNew(
        val s: String,
    ) : ChildWithPoly
}

Expected behavior

polymorphicDefaultDeserializer works with useArrayPolymorphism

Environment

• Kotlin version: 2.1.21
• Library version: 1.8.1
• Kotlin platforms: JVM, JS
• Gradle version: 8.14.1

[pdvrieze]

val parentWithoutType = json.decodeFromString<Parent>("""{"child":{"s":"Hello World!"}}""")

You seem to have misunderstood what the defaultPolymorphicDeserializer does. What it does is allows you to provide a fallback deserializizer if the type is not known. In other words it still expects an array with a type: {"child:["something",{"s":"Hello World!"}]}

This is not visible in the case of type parameter polymorphism - which merges the polymorphic "type+data" into the data, as that handles the absence of a type attribute and uses the default in such case.

[Komdosh]

@pdvrieze Thanks for your reply. This is not an obvious behavior. I understand how it works, I understand that "type" just a null in a such case and we fall in defaultPolymorphicDeserializer. But it also sounds like a won't fix bug, why?

I need to migrate a legacy non-polymorphic field to a polymorphic one. Since I already have many stored JSON objects using useArrayPolymorphism, disabling it isn’t an option. Additionally, I’m using a StreamingJsonDecoder, which complicates implementing a fallback deserializer—I can’t simply wrap it in a try-catch block because tokens would be consumed prematurely.

How can I seamlessly transition from concrete classes to polymorphic ones? What would you recommend for this migration scenario?

defaultPolymorphicDeserializer seems like a potential solution. Why not to try to deserialize object and use default deserializer if no [] provided?

[pdvrieze]

@Komdosh Technically it is very hard to support your case as the way it works is:

1. read begin array → this fails
2. read type → missing, but as it is an array not visible
3. read data → would only work if we can skip the first 2 steps
4. read end array

In other words, it never gets to the point to try to read the type, it fails because it expects to read an array (consider that the data could be an array instead of a struct....). Note that the system does not "look ahead" for efficiency reasons, but parses as it deserializes.

The way forward for you would be a custom serializer (perhaps using the content-polymorphic base class). This would read the entire JsonElement content and then allows you to parse that content as desired. As you know the content you don't need to worry about the content being very large.

[Komdosh]

Data schema migration with kotlinx.serialization is full of pain :(

[pdvrieze]

Data schema migration with kotlinx.serialization is full of pain :(

Data schema migration is always full of pain. The best approach is to have a version marker early on so you only need a custom serializer for that bit. A good approach then is to have (potentially private) serialization delegate objects that read the old version, and in the final deserializer make them an implementation of the actual object you want (of course if you have optionals that also works).

If instead of rigid formats you want more flexible parsing, you could use custom serializers (with type aliases) for things like "string or list", defaultOrPolymorphicStruct etc.

[Komdosh]

Providing versions for each serializable class seems unrealistic and would bloat the data with many symbols that may never be used.

Yes, we currently use custom deserializers, but this seems like a hacky solution.

[pdvrieze]

It would be a version for the entire file, not for individual types (indeed that makes little sense, unless an incompatible change forces you). Custom deserializers are not a hack per se, but are a customization point. They are often needed for more complex functionality (including multiple representations). For example in I use an InstantAsEpochSeconds type alias that (de)serializes an instant as seconds from the epoch (as required by oauth in many places). This is still concise and clear.

I find that using (private) serial delegate classes is often a good solution where the class and its serialization don't match well. The custom serializer is then fairly simple:

• the descriptor is that of the delegate (best with a different name)
• encoding involves creating a delegate from the value, and encoding that
• decoding involves decoding the delegate, and then calling a "toOuter" to create the outer with the delegate.
  • In many cases you can also have a delegate that is more lax (for compatibility) where the conversion will actually enforce correctness
    The delegate and its serializer can be private so this is transparent to users.

[Komdosh]

Could you provide an example of a private serialization delegate? I want to ensure I understand your approach correctly.

[pdvrieze]

You may look at: pdvrieze/xmlutil/xmlschema/src/commonMain/kotlin/io/github/pdvrieze/formats/xmlschema/datatypes/serialization/XSLocalComplexType.kt but note that this example is not private. In this case it deals with the fact that the correct type to create depends on the underlying data - something a generic serialization algorithm can never do on its own.

[sandwwraith]

Sorry, it is not clear from your original request what exactly is not working. Are you complaining that decodeFromString<Parent>("""{"child":{"s":"Hello World!"}}""") throws Expected start of the array '[', but had '{' instead at path when useArrayPolymorphism is enabled? I.e. you expect json to use a default deserializer in the case polymorphic input is not an array at all?

[pdvrieze]

Sorry, it is not clear from your original request what exactly is not working. Are you complaining that decodeFromString<Parent>("""{"child":{"s":"Hello World!"}}""") throws Expected start of the array '[', but had '{' instead at path when useArrayPolymorphism is enabled? I.e. you expect json to use a default deserializer in the case polymorphic input is not an array at all?

That is how I understand it. And of course that isn't supported out of the box.

[Komdosh]

I.e. you expect json to use a default deserializer in the case polymorphic input is not an array at all?

Yes, I expect that, because val child: ChildWithPoly is not an array in Parent class, it is an object, but useArrayPolymorphism = true treats it as an array during serialization.

@Serializable
public data class Parent(
    val child: ChildWithPoly,
)

So, I expect that a child could be deserialized as an object with default serializer (because it is an object, not an array) if no type is provided.

[sandwwraith]

I suppose we could fix this eventually, but it will indeed require a lot of effort, compared to the niche use of the feature that we discourage.
If your only purpose is to migrate old data, I can recommend writing you a custom deserializer like

fun deserialize(decoder: Decoder): ChildWithPoly {
  val element = (decoder as JsonDecoder).decodeJsonElement()
  return when(element) {
    is JsonArray -> getSerializerByName(element[0]).let { decoder.json.decodeFromJsonElement(it, element[1]) }
    else -> decoder.json.decodeFromJsonElement(ChildDefaultDeserializer, element)
 }
}

see https://github.com/Kotlin/kotlinx.serialization/blob/master/docs/json.md#under-the-hood-experimental for details