CBOR serialization into an `OutputStream` or `WritableByteChannel` or externally allocated `ByteBuffer`

[0x6675636b796f75676974687562]

What is your use-case and why do you need this feature?

When encoding data as JSON, it's possible to write serialized data directly into an OutputStream, via Json.encodeToStream(SerializationStrategy<T>, T, OutputStream).

Yet, with CBOR, only writing to or reading from a ByteArray (byte[] JVM type) seems possible (Cbor.encodeToByteArray(SerializationStrategy<T>, T): ByteArray), and the byte array returned gets allocated by the library itself, can't be supplied externally, nor can it be reused.

This leads to extra allocations and frequent OutOfMemoryError's when encoding large data hierarchies as CBOR, which never happens with JSON.

Describe the solution you'd like

We'd appreciate if any of the following extensions, with the corresponding decodeXyz() counterparts, are eventually implemented:

• Cbor.encodeToStream(SerializationStrategy<T>, T, OutputStream)
• Cbor.encodeToChannel(SerializationStrategy<T>, T, WritableByteChannel)
• Cbor.encodeToBuffer(SerializationStrategy<T>, T, ByteBuffer)

[xiaozhikang0916]

This feature may be commonized as Supporting streaming decoding and encoding with multiplatform.

Personally I like this idea. My team has also faced OOM issue using FastJSON and got solved by using streaming decoding. With this feature in serialization lib, we can have more flexiable control.

We can introduce a new stream-ext lib to support it with help of kotlinx.io, but it would need some refactoring in the core lib and all the formats.

[corlaez]

I don't really care for multiplatform. Would love to at least see the Stream encode/decode in JVM. Or at least more info in the readme of what are my options if I really need stream support, instead of trial and error of each library

[Aphexus]

So nada?

[Jean-Daniel]

The ironic part is that the internal Cbor encoder / decoder only support streams.

But instead of exposing the stream API, Cbor creates bytes array input/output stream.