# Alternative and custom formats (experimental) This is the sixth chapter of the [Kotlin Serialization Guide](serialization-guide.md). It goes beyond JSON, covering alternative and custom formats. Unlike JSON, which is stable, these are currently experimental features of Kotlin Serialization. **Table of contents** * [CBOR (experimental)](#cbor-experimental) * [Ignoring unknown keys](#ignoring-unknown-keys) * [Byte arrays and CBOR data types](#byte-arrays-and-cbor-data-types) * [ProtoBuf (experimental)](#protobuf-experimental) * [Field numbers](#field-numbers) * [Integer types](#integer-types) * [Lists as repeated fields](#lists-as-repeated-fields) * [Packed fields](#packed-fields) * [ProtoBuf schema generator (experimental)](#protobuf-schema-generator-experimental) * [Properties (experimental)](#properties-experimental) * [Custom formats (experimental)](#custom-formats-experimental) * [Basic encoder](#basic-encoder) * [Basic decoder](#basic-decoder) * [Sequential decoding](#sequential-decoding) * [Adding collection support](#adding-collection-support) * [Adding null support](#adding-null-support) * [Efficient binary format](#efficient-binary-format) * [Format-specific types](#format-specific-types) ## CBOR (experimental) [CBOR][RFC 7049] is one of the standard compact binary encodings for JSON, so it supports a subset of [JSON features](json.md) and is generally very similar to JSON in use, but produces binary data. > CBOR support is (experimentally) available in a separate > `org.jetbrains.kotlinx:kotlinx-serialization-cbor:` module. [Cbor] class has [Cbor.encodeToByteArray] and [Cbor.decodeFromByteArray] functions. Let us take the basic example from the [JSON encoding](basic-serialization.md#json-encoding), but encode it using CBOR. ```kotlin @Serializable data class Project(val name: String, val language: String) fun main() { val data = Project("kotlinx.serialization", "Kotlin") val bytes = Cbor.encodeToByteArray(data) println(bytes.toAsciiHexString()) val obj = Cbor.decodeFromByteArray(bytes) println(obj) } ``` > You can get the full code [here](../guide/example/example-formats-01.kt). We print a filtered ASCII representation of the output, writing non-ASCII data in hex, so we see how all the original strings are directly represented in CBOR, but the format delimiters themselves are binary. ```text {BF}dnameukotlinx.serializationhlanguagefKotlin{FF} Project(name=kotlinx.serialization, language=Kotlin) ``` In [CBOR hex notation](http://cbor.me/), the output is equivalent to the following: ``` BF # map(*) 64 # text(4) 6E616D65 # "name" 75 # text(21) 6B6F746C696E782E73657269616C697A6174696F6E # "kotlinx.serialization" 68 # text(8) 6C616E6775616765 # "language" 66 # text(6) 4B6F746C696E # "Kotlin" FF # primitive(*) ``` > Note, CBOR as a format, unlike JSON, supports maps with non-trivial keys > (see the [Allowing structured map keys](json.md#allowing-structured-map-keys) section for JSON workarounds), > and Kotlin maps are serialized as CBOR maps, but some parsers (like `jackson-dataformat-cbor`) don't support this. ### Ignoring unknown keys CBOR format is often used to communicate with [IoT] devices where new properties could be added as a part of a device's API evolution. By default, unknown keys encountered during deserialization produce an error. This behavior can be configured with the [ignoreUnknownKeys][CborBuilder.ignoreUnknownKeys] property. ```kotlin val format = Cbor { ignoreUnknownKeys = true } @Serializable data class Project(val name: String) fun main() { val data = format.decodeFromHexString( "bf646e616d65756b6f746c696e782e73657269616c697a6174696f6e686c616e6775616765664b6f746c696eff" ) println(data) } ``` > You can get the full code [here](../guide/example/example-formats-02.kt). It decodes the object, despite the fact that `Project` is missing the `language` property. ```text Project(name=kotlinx.serialization) ``` In [CBOR hex notation](http://cbor.me/), the input is equivalent to the following: ``` BF # map(*) 64 # text(4) 6E616D65 # "name" 75 # text(21) 6B6F746C696E782E73657269616C697A6174696F6E # "kotlinx.serialization" 68 # text(8) 6C616E6775616765 # "language" 66 # text(6) 4B6F746C696E # "Kotlin" FF # primitive(*) ``` ### Byte arrays and CBOR data types Per the [RFC 7049 Major Types] section, CBOR supports the following data types: - Major type 0: an unsigned integer - Major type 1: a negative integer - **Major type 2: a byte string** - Major type 3: a text string - **Major type 4: an array of data items** - Major type 5: a map of pairs of data items - Major type 6: optional semantic tagging of other major types - Major type 7: floating-point numbers and simple data types that need no content, as well as the "break" stop code By default, Kotlin `ByteArray` instances are encoded as **major type 4**. When **major type 2** is desired, then the [`@ByteString`][ByteString] annotation can be used. ```kotlin @Serializable data class Data( @ByteString val type2: ByteArray, // CBOR Major type 2 val type4: ByteArray // CBOR Major type 4 ) fun main() { val data = Data(byteArrayOf(1, 2, 3, 4), byteArrayOf(5, 6, 7, 8)) val bytes = Cbor.encodeToByteArray(data) println(bytes.toAsciiHexString()) val obj = Cbor.decodeFromByteArray(bytes) println(obj) } ``` > You can get the full code [here](../guide/example/example-formats-03.kt). As we see, the CBOR byte that precedes the data is different for different types of encoding. ```text {BF}etype2D{01}{02}{03}{04}etype4{9F}{05}{06}{07}{08}{FF}{FF} Data(type2=[1, 2, 3, 4], type4=[5, 6, 7, 8]) ``` In [CBOR hex notation](http://cbor.me/), the output is equivalent to the following: ``` BF # map(*) 65 # text(5) 7479706532 # "type2" 44 # bytes(4) 01020304 # "\x01\x02\x03\x04" 65 # text(5) 7479706534 # "type4" 9F # array(*) 05 # unsigned(5) 06 # unsigned(6) 07 # unsigned(7) 08 # unsigned(8) FF # primitive(*) FF # primitive(*) ``` ## ProtoBuf (experimental) [Protocol Buffers](https://developers.google.com/protocol-buffers) is a language-neutral binary format that normally relies on a separate ".proto" file that defines the protocol schema. It is more compact than CBOR, because it assigns integer numbers to fields instead of names. > Protocol buffers support is (experimentally) available in a separate > `org.jetbrains.kotlinx:kotlinx-serialization-protobuf:` module. Kotlin Serialization is using proto2 semantics, where all fields are explicitly required or optional. For a basic example we change our example to use the [ProtoBuf] class with [ProtoBuf.encodeToByteArray] and [ProtoBuf.decodeFromByteArray] functions. ```kotlin @Serializable data class Project(val name: String, val language: String) fun main() { val data = Project("kotlinx.serialization", "Kotlin") val bytes = ProtoBuf.encodeToByteArray(data) println(bytes.toAsciiHexString()) val obj = ProtoBuf.decodeFromByteArray(bytes) println(obj) } ``` > You can get the full code [here](../guide/example/example-formats-04.kt). ```text {0A}{15}kotlinx.serialization{12}{06}Kotlin Project(name=kotlinx.serialization, language=Kotlin) ``` In [ProtoBuf hex notation](https://protogen.marcgravell.com/decode), the output is equivalent to the following: ``` Field #1: 0A String Length = 21, Hex = 15, UTF8 = "kotlinx.serialization" Field #2: 12 String Length = 6, Hex = 06, UTF8 = "Kotlin" ``` ### Field numbers By default, field numbers in the Kotlin Serialization [ProtoBuf] implementation are automatically assigned, which does not provide the ability to define a stable data schema that evolves over time. That is normally achieved by writing a separate ".proto" file. However, with Kotlin Serialization we can get this ability without a separate schema file, instead using the [ProtoNumber] annotation. ```kotlin @Serializable data class Project( @ProtoNumber(1) val name: String, @ProtoNumber(3) val language: String ) fun main() { val data = Project("kotlinx.serialization", "Kotlin") val bytes = ProtoBuf.encodeToByteArray(data) println(bytes.toAsciiHexString()) val obj = ProtoBuf.decodeFromByteArray(bytes) println(obj) } ``` > You can get the full code [here](../guide/example/example-formats-05.kt). We see in the output that the number for the first property `name` did not change (as it is numbered from one by default), but it did change for the `language` property. ```text {0A}{15}kotlinx.serialization{1A}{06}Kotlin Project(name=kotlinx.serialization, language=Kotlin) ``` In [ProtoBuf hex notation](https://protogen.marcgravell.com/decode), the output is equivalent to the following: ``` Field #1: 0A String Length = 21, Hex = 15, UTF8 = "kotlinx.serialization" (total 21 chars) Field #3: 1A String Length = 6, Hex = 06, UTF8 = "Kotlin" ``` ### Integer types Protocol buffers support various integer encodings optimized for different ranges of integers. They are specified using the [ProtoType] annotation and the [ProtoIntegerType] enum. The following example shows all three supported options. ```kotlin @Serializable class Data( @ProtoType(ProtoIntegerType.DEFAULT) val a: Int, @ProtoType(ProtoIntegerType.SIGNED) val b: Int, @ProtoType(ProtoIntegerType.FIXED) val c: Int ) fun main() { val data = Data(1, -2, 3) println(ProtoBuf.encodeToByteArray(data).toAsciiHexString()) } ``` > You can get the full code [here](../guide/example/example-formats-06.kt). * The [default][ProtoIntegerType.DEFAULT] is a varint encoding (`intXX`) that is optimized for small non-negative numbers. The value of `1` is encoded in one byte `01`. * The [signed][ProtoIntegerType.SIGNED] is a signed ZigZag encoding (`sintXX`) that is optimized for small signed integers. The value of `-2` is encoded in one byte `03`. * The [fixed][ProtoIntegerType.FIXED] encoding (`fixedXX`) always uses a fixed number of bytes. The value of `3` is encoded as four bytes `03 00 00 00`. > `uintXX` and `sfixedXX` protocol buffer types are not supported. ```text {08}{01}{10}{03}{1D}{03}{00}{00}{00} ``` In [ProtoBuf hex notation](https://protogen.marcgravell.com/decode) the output is equivalent to the following: ``` Field #1: 08 Varint Value = 1, Hex = 01 Field #2: 10 Varint Value = 3, Hex = 03 Field #3: 1D Fixed32 Value = 3, Hex = 03-00-00-00 ``` ### Lists as repeated fields By default, kotlin lists and other collections are representend as repeated fields. In the protocol buffers when the list is empty there are no elements in the stream with the corresponding number. For Kotlin Serialization you must explicitly specify a default of `emptyList()` for any property of a collection or map type. Otherwise you will not be able deserialize an empty list, which is indistinguishable in protocol buffers from a missing field. ```kotlin @Serializable data class Data( val a: List = emptyList(), val b: List = emptyList() ) fun main() { val data = Data(listOf(1, 2, 3), listOf()) val bytes = ProtoBuf.encodeToByteArray(data) println(bytes.toAsciiHexString()) println(ProtoBuf.decodeFromByteArray(bytes)) } ``` > You can get the full code [here](../guide/example/example-formats-07.kt). ```text {08}{01}{08}{02}{08}{03} Data(a=[1, 2, 3], b=[]) ``` In [ProtoBuf diagnostic mode](https://protogen.marcgravell.com/decode) the output is equivalent to the following: ``` Field #1: 08 Varint Value = 1, Hex = 01 Field #1: 08 Varint Value = 2, Hex = 02 Field #1: 08 Varint Value = 3, Hex = 03 ``` ### Packed fields Collection types (not maps) can be **written** as packed fields when annotated with the `@ProtoPacked` annotation. Per the standard packed fields can only be used on primitive numeric types. The annotation is ignored on other types. Per the [format description](https://developers.google.com/protocol-buffers/docs/encoding#packed) the parser ignores the annotation, but rather reads list in either packed or repeated format. ### ProtoBuf schema generator (experimental) As mentioned above, when working with protocol buffers you usually use a ".proto" file and a code generator for your language. This includes the code to serialize your message to an output stream and deserialize it from an input stream. When using Kotlin Serialization this step is not necessary because your `@Serializable` Kotlin data types are used as the source for the schema. This is very convenient for Kotlin-to-Kotlin communication, but makes interoperability between languages complicated. Fortunately, you can use the ProtoBuf schema generator to output the ".proto" representation of your messages. You can keep your Kotlin classes as a source of truth and use traditional protoc compilers for other languages at the same time. As an example, we can display the following data class's ".proto" schema as follows. ```kotlin @Serializable data class SampleData( val amount: Long, val description: String?, val department: String = "QA" ) fun main() { val descriptors = listOf(SampleData.serializer().descriptor) val schemas = ProtoBufSchemaGenerator.generateSchemaText(descriptors) println(schemas) } ``` > You can get the full code [here](../guide/example/example-formats-08.kt). Which would output as follows. ```text syntax = "proto2"; // serial name 'example.exampleFormats08.SampleData' message SampleData { required int64 amount = 1; optional string description = 2; // WARNING: a default value decoded when value is missing optional string department = 3; } ``` Note that since default values are not represented in ".proto" files, a warning is generated when one appears in the schema. See the documentation for [ProtoBufSchemaGenerator] for more information. ## Properties (experimental) Kotlin Serialization can serialize a class into a flat map with `String` keys via the [Properties][kotlinx.serialization.properties.Properties] format implementation. > Properties support is (experimentally) available in a separate > `org.jetbrains.kotlinx:kotlinx-serialization-properties:` module. ```kotlin @Serializable class Project(val name: String, val owner: User) @Serializable class User(val name: String) fun main() { val data = Project("kotlinx.serialization", User("kotlin")) val map = Properties.encodeToMap(data) map.forEach { (k, v) -> println("$k = $v") } } ``` > You can get the full code [here](../guide/example/example-formats-09.kt). The resulting map has dot-separated keys representing keys of the nested objects. ```text name = kotlinx.serialization owner.name = kotlin ``` ## Custom formats (experimental) A custom format for Kotlin Serialization must provide an implementation for the [Encoder] and [Decoder] interfaces that we saw used in the [Serializers](serializers.md) chapter. These are pretty large interfaces. For convenience the [AbstractEncoder] and [AbstractDecoder] skeleton implementations are provided to simplify the task. In [AbstractEncoder] most of the `encodeXxx` methods have a default implementation that delegates to [`encodeValue(value: Any)`][AbstractEncoder.encodeValue] — the only method that must be implemented to get a basic working format. ### Basic encoder Let us start with a trivial format implementation that encodes the data into a single list of primitive constituent objects in the order they were written in the source code. To start, we implement a simple [Encoder] by overriding `encodeValue` in [AbstractEncoder]. ```kotlin class ListEncoder : AbstractEncoder() { val list = mutableListOf() override val serializersModule: SerializersModule = EmptySerializersModule() override fun encodeValue(value: Any) { list.add(value) } } ``` Now we write a convenience top-level function that creates an encoder that encodes an object and returns a list. ```kotlin fun encodeToList(serializer: SerializationStrategy, value: T): List { val encoder = ListEncoder() encoder.encodeSerializableValue(serializer, value) return encoder.list } ``` For even more convenience, to avoid the need to explicitly pass a serializer, we write an `inline` overload of the `encodeToList` function with a `reified` type parameter using the [serializer] function to retrieve the appropriate [KSerializer] instance for the actual type. ```kotlin inline fun encodeToList(value: T) = encodeToList(serializer(), value) ``` Now we can test it. ```kotlin @Serializable data class Project(val name: String, val owner: User, val votes: Int) @Serializable data class User(val name: String) fun main() { val data = Project("kotlinx.serialization", User("kotlin"), 9000) println(encodeToList(data)) } ``` > You can get the full code [here](../guide/example/example-formats-10.kt). As a result, we got all the primitive values in our object graph visited and put into a list in _serial_ order. ```text [kotlinx.serialization, kotlin, 9000] ``` > By itself, that's a useful feature if we need compute some kind of hashcode or digest for all the data > that is contained in a serializable object tree. ### Basic decoder A decoder needs to implement more substance. * [decodeValue][AbstractDecoder.decodeValue] — returns the next value from the list. * [decodeElementIndex][CompositeDecoder.decodeElementIndex] — returns the next index of a deserialized value. In this primitive format deserialization always happens in order, so we keep track of the index in the `elementIndex` variable. See the [Hand-written composite serializer](serializers.md#hand-written-composite-serializer) section on how it ends up being used. * [beginStructure][Decoder.beginStructure] — returns a new instance of the `ListDecoder`, so that each structure that is being recursively decoded keeps track of its own `elementIndex` state separately. ```kotlin class ListDecoder(val list: ArrayDeque) : AbstractDecoder() { private var elementIndex = 0 override val serializersModule: SerializersModule = EmptySerializersModule() override fun decodeValue(): Any = list.removeFirst() override fun decodeElementIndex(descriptor: SerialDescriptor): Int { if (elementIndex == descriptor.elementsCount) return CompositeDecoder.DECODE_DONE return elementIndex++ } override fun beginStructure(descriptor: SerialDescriptor): CompositeDecoder = ListDecoder(list) } ``` A couple of convenience functions for decoding. ```kotlin fun decodeFromList(list: List, deserializer: DeserializationStrategy): T { val decoder = ListDecoder(ArrayDeque(list)) return decoder.decodeSerializableValue(deserializer) } inline fun decodeFromList(list: List): T = decodeFromList(list, serializer()) ``` That is enough to start encoding and decoding basic serializable classes. ```kotlin fun main() { val data = Project("kotlinx.serialization", User("kotlin"), 9000) val list = encodeToList(data) println(list) val obj = decodeFromList(list) println(obj) } ``` > You can get the full code [here](../guide/example/example-formats-11.kt). Now we can convert a list of primitives back to an object tree. ```text [kotlinx.serialization, kotlin, 9000] Project(name=kotlinx.serialization, owner=User(name=kotlin), votes=9000) ``` ### Sequential decoding The decoder we have implemented keeps track of the `elementIndex` in its state and implements `decodeElementIndex`. This means that it is going to work with an arbitrary serializer, even the simple one we wrote in the [Hand-written composite serializer](serializers.md#hand-written-composite-serializer) section. However, this format always stores elements in order, so this bookkeeping is not needed and undermines decoding performance. All auto-generated serializers on the JVM support the [Sequential decoding protocol (experimental)](serializers.md#sequential-decoding-protocol-experimental), and the decoder can indicate its support by returning `true` from the [CompositeDecoder.decodeSequentially] function. ```kotlin class ListDecoder(val list: ArrayDeque) : AbstractDecoder() { private var elementIndex = 0 override val serializersModule: SerializersModule = EmptySerializersModule() override fun decodeValue(): Any = list.removeFirst() override fun decodeElementIndex(descriptor: SerialDescriptor): Int { if (elementIndex == descriptor.elementsCount) return CompositeDecoder.DECODE_DONE return elementIndex++ } override fun beginStructure(descriptor: SerialDescriptor): CompositeDecoder = ListDecoder(list) override fun decodeSequentially(): Boolean = true } ``` > You can get the full code [here](../guide/example/example-formats-12.kt). ### Adding collection support This basic format, so far, cannot properly represent collections. In encodes them, but it does not keep track of how many elements there are in the collection or where it ends, so it cannot properly decode them. First, let us add proper support for collections to the encoder by implementing the [Encoder.beginCollection] function. The `beginCollection` function takes a collection size as a parameter, so we encode it to add it to the result. Our encoder implementation does not keep any state, so it just returns `this` from the `beginCollection` function. ```kotlin class ListEncoder : AbstractEncoder() { val list = mutableListOf() override val serializersModule: SerializersModule = EmptySerializersModule() override fun encodeValue(value: Any) { list.add(value) } override fun beginCollection(descriptor: SerialDescriptor, collectionSize: Int): CompositeEncoder { encodeInt(collectionSize) return this } } ``` The decoder, for our case, needs to only implement the [CompositeDecoder.decodeCollectionSize] function in addition to the previous code. > The formats that store collection size in advance have to return `true` from `decodeSequentially`. ```kotlin class ListDecoder(val list: ArrayDeque, var elementsCount: Int = 0) : AbstractDecoder() { private var elementIndex = 0 override val serializersModule: SerializersModule = EmptySerializersModule() override fun decodeValue(): Any = list.removeFirst() override fun decodeElementIndex(descriptor: SerialDescriptor): Int { if (elementIndex == elementsCount) return CompositeDecoder.DECODE_DONE return elementIndex++ } override fun beginStructure(descriptor: SerialDescriptor): CompositeDecoder = ListDecoder(list, descriptor.elementsCount) override fun decodeSequentially(): Boolean = true override fun decodeCollectionSize(descriptor: SerialDescriptor): Int = decodeInt().also { elementsCount = it } } ``` That is all that is needed to support collections and maps. ```kotlin @Serializable data class Project(val name: String, val owners: List, val votes: Int) @Serializable data class User(val name: String) fun main() { val data = Project("kotlinx.serialization", listOf(User("kotlin"), User("jetbrains")), 9000) val list = encodeToList(data) println(list) val obj = decodeFromList(list) println(obj) } ``` > You can get the full code [here](../guide/example/example-formats-13.kt). We see the size of the list added to the result, letting the decoder know where to stop. ```text [kotlinx.serialization, 2, kotlin, jetbrains, 9000] Project(name=kotlinx.serialization, owners=[User(name=kotlin), User(name=jetbrains)], votes=9000) ``` ### Adding null support Our trivial format does not support `null` values so far. For nullable types we need to add some kind of "null indicator", telling whether the upcoming value is null or not. In the encoder implementation we override [Encoder.encodeNull] and [Encoder.encodeNotNullMark]. ```kotlin override fun encodeNull() = encodeValue("NULL") override fun encodeNotNullMark() = encodeValue("!!") ``` In the decoder implementation we override [Decoder.decodeNotNullMark]. ```kotlin override fun decodeNotNullMark(): Boolean = decodeString() != "NULL" ``` Let us test nullable properties both with not-null and null values. ```kotlin @Serializable data class Project(val name: String, val owner: User?, val votes: Int?) @Serializable data class User(val name: String) fun main() { val data = Project("kotlinx.serialization", User("kotlin") , null) val list = encodeToList(data) println(list) val obj = decodeFromList(list) println(obj) } ``` > You can get the full code [here](../guide/example/example-formats-14.kt). In the output we see how not-null`!!` and `NULL` marks are used. ```text [kotlinx.serialization, !!, kotlin, NULL] Project(name=kotlinx.serialization, owner=User(name=kotlin), votes=null) ``` ### Efficient binary format Now we are ready for an example of an efficient binary format. We are going to write data to the [java.io.DataOutput] implementation. Instead of `encodeValue` we must override the individual `encodeXxx` functions for each of ten [primitives](builtin-classes.md#primitives) in the encoder. ```kotlin class DataOutputEncoder(val output: DataOutput) : AbstractEncoder() { override val serializersModule: SerializersModule = EmptySerializersModule() override fun encodeBoolean(value: Boolean) = output.writeByte(if (value) 1 else 0) override fun encodeByte(value: Byte) = output.writeByte(value.toInt()) override fun encodeShort(value: Short) = output.writeShort(value.toInt()) override fun encodeInt(value: Int) = output.writeInt(value) override fun encodeLong(value: Long) = output.writeLong(value) override fun encodeFloat(value: Float) = output.writeFloat(value) override fun encodeDouble(value: Double) = output.writeDouble(value) override fun encodeChar(value: Char) = output.writeChar(value.code) override fun encodeString(value: String) = output.writeUTF(value) override fun encodeEnum(enumDescriptor: SerialDescriptor, index: Int) = output.writeInt(index) override fun beginCollection(descriptor: SerialDescriptor, collectionSize: Int): CompositeEncoder { encodeInt(collectionSize) return this } override fun encodeNull() = encodeBoolean(false) override fun encodeNotNullMark() = encodeBoolean(true) } ``` The decoder implementation mirrors encoder's implementation overriding all the primitive `decodeXxx` functions. ```kotlin class DataInputDecoder(val input: DataInput, var elementsCount: Int = 0) : AbstractDecoder() { private var elementIndex = 0 override val serializersModule: SerializersModule = EmptySerializersModule() override fun decodeBoolean(): Boolean = input.readByte().toInt() != 0 override fun decodeByte(): Byte = input.readByte() override fun decodeShort(): Short = input.readShort() override fun decodeInt(): Int = input.readInt() override fun decodeLong(): Long = input.readLong() override fun decodeFloat(): Float = input.readFloat() override fun decodeDouble(): Double = input.readDouble() override fun decodeChar(): Char = input.readChar() override fun decodeString(): String = input.readUTF() override fun decodeEnum(enumDescriptor: SerialDescriptor): Int = input.readInt() override fun decodeElementIndex(descriptor: SerialDescriptor): Int { if (elementIndex == elementsCount) return CompositeDecoder.DECODE_DONE return elementIndex++ } override fun beginStructure(descriptor: SerialDescriptor): CompositeDecoder = DataInputDecoder(input, descriptor.elementsCount) override fun decodeSequentially(): Boolean = true override fun decodeCollectionSize(descriptor: SerialDescriptor): Int = decodeInt().also { elementsCount = it } override fun decodeNotNullMark(): Boolean = decodeBoolean() } ``` We can now serialize and deserialize arbitrary data. For example, the same classes as were used in the [CBOR (experimental)](#cbor-experimental) and [ProtoBuf (experimental)](#protobuf-experimental) sections. ```kotlin @Serializable data class Project(val name: String, val language: String) fun main() { val data = Project("kotlinx.serialization", "Kotlin") val output = ByteArrayOutputStream() encodeTo(DataOutputStream(output), data) val bytes = output.toByteArray() println(bytes.toAsciiHexString()) val input = ByteArrayInputStream(bytes) val obj = decodeFrom(DataInputStream(input)) println(obj) } ``` > You can get the full code [here](../guide/example/example-formats-15.kt). As we can see, the result is a dense binary format that only contains the data that is being serialized. It can be easily tweaked for any kind of domain-specific compact encoding. ```text {00}{15}kotlinx.serialization{00}{06}Kotlin Project(name=kotlinx.serialization, language=Kotlin) ``` ### Format-specific types A format implementation might provide special support for data types that are not among the list of primitive types in Kotlin Serialization, and do not have a corresponding `encodeXxx`/`decodeXxx` function. In the encoder this is achieved by overriding the [`encodeSerializableValue(serializer, value)`][Encoder.encodeSerializableValue] function. In our `DataOutput` format example we might want to provide a specialized efficient data path for serializing an array of bytes since [DataOutput][java.io.DataOutput] has a special method for this purpose. Detection of the type is performed by looking at the `serializer.descriptor`, not by checking the type of the `value` being serialized, so we fetch the builtin [KSerializer] instance for `ByteArray` type. > This an important difference. This way our format implementation properly supports > [Custom serializers](serializers.md#custom-serializers) that a user might specify for a type that just happens > to be internally represented as a byte array, but need a different serial representation. ```kotlin private val byteArraySerializer = serializer() ``` > Specifically for byte arrays, we could have also used the builtin > [ByteArraySerializer][kotlinx.serialization.builtins.ByteArraySerializer()] function. We add the corresponding code to the [Encoder] implementation of our [Efficient binary format](#efficient-binary-format). To make our `ByteArray` encoding even more efficient, we add a trivial implementation of `encodeCompactSize` function that uses only one byte to represent a size of up to 254 bytes. ```kotlin override fun encodeSerializableValue(serializer: SerializationStrategy, value: T) { if (serializer.descriptor == byteArraySerializer.descriptor) encodeByteArray(value as ByteArray) else super.encodeSerializableValue(serializer, value) } private fun encodeByteArray(bytes: ByteArray) { encodeCompactSize(bytes.size) output.write(bytes) } private fun encodeCompactSize(value: Int) { if (value < 0xff) { output.writeByte(value) } else { output.writeByte(0xff) output.writeInt(value) } } ``` A similar code is added to the [Decoder] implementation. Here we override the [decodeSerializableValue][Decoder.decodeSerializableValue] function. ```kotlin @Suppress("UNCHECKED_CAST") override fun decodeSerializableValue(deserializer: DeserializationStrategy, previousValue: T?): T = if (deserializer.descriptor == byteArraySerializer.descriptor) decodeByteArray() as T else super.decodeSerializableValue(deserializer, previousValue) private fun decodeByteArray(): ByteArray { val bytes = ByteArray(decodeCompactSize()) input.readFully(bytes) return bytes } private fun decodeCompactSize(): Int { val byte = input.readByte().toInt() and 0xff if (byte < 0xff) return byte return input.readInt() } ``` Now everything is ready to perform serialization of some byte arrays. ```kotlin @Serializable data class Project(val name: String, val attachment: ByteArray) fun main() { val data = Project("kotlinx.serialization", byteArrayOf(0x0A, 0x0B, 0x0C, 0x0D)) val output = ByteArrayOutputStream() encodeTo(DataOutputStream(output), data) val bytes = output.toByteArray() println(bytes.toAsciiHexString()) val input = ByteArrayInputStream(bytes) val obj = decodeFrom(DataInputStream(input)) println(obj) } ``` > You can get the full code [here](../guide/example/example-formats-16.kt). As we can see, our custom byte array format is being used, with the compact encoding of its size in one byte. ```text {00}{15}kotlinx.serialization{04}{0A}{0B}{0C}{0D} Project(name=kotlinx.serialization, attachment=[10, 11, 12, 13]) ``` --- This chapter concludes [Kotlin Serialization Guide](serialization-guide.md). [RFC 7049]: https://tools.ietf.org/html/rfc7049 [IoT]: https://en.wikipedia.org/wiki/Internet_of_things [RFC 7049 Major Types]: https://tools.ietf.org/html/rfc7049#section-2.1 [java.io.DataOutput]: https://docs.oracle.com/javase/8/docs/api/java/io/DataOutput.html [serializer]: https://kotlinlang.org/api/kotlinx.serialization/kotlinx-serialization-core/kotlinx.serialization/serializer.html [KSerializer]: https://kotlinlang.org/api/kotlinx.serialization/kotlinx-serialization-core/kotlinx.serialization/-k-serializer/index.html [kotlinx.serialization.builtins.ByteArraySerializer()]: https://kotlinlang.org/api/kotlinx.serialization/kotlinx-serialization-core/kotlinx.serialization.builtins/-byte-array-serializer.html [Encoder]: https://kotlinlang.org/api/kotlinx.serialization/kotlinx-serialization-core/kotlinx.serialization.encoding/-encoder/index.html [Decoder]: https://kotlinlang.org/api/kotlinx.serialization/kotlinx-serialization-core/kotlinx.serialization.encoding/-decoder/index.html [AbstractEncoder]: https://kotlinlang.org/api/kotlinx.serialization/kotlinx-serialization-core/kotlinx.serialization.encoding/-abstract-encoder/index.html [AbstractDecoder]: https://kotlinlang.org/api/kotlinx.serialization/kotlinx-serialization-core/kotlinx.serialization.encoding/-abstract-decoder/index.html [AbstractEncoder.encodeValue]: https://kotlinlang.org/api/kotlinx.serialization/kotlinx-serialization-core/kotlinx.serialization.encoding/-abstract-encoder/encode-value.html [AbstractDecoder.decodeValue]: https://kotlinlang.org/api/kotlinx.serialization/kotlinx-serialization-core/kotlinx.serialization.encoding/-abstract-decoder/decode-value.html [CompositeDecoder.decodeElementIndex]: https://kotlinlang.org/api/kotlinx.serialization/kotlinx-serialization-core/kotlinx.serialization.encoding/-composite-decoder/decode-element-index.html [Decoder.beginStructure]: https://kotlinlang.org/api/kotlinx.serialization/kotlinx-serialization-core/kotlinx.serialization.encoding/-decoder/begin-structure.html [CompositeDecoder.decodeSequentially]: https://kotlinlang.org/api/kotlinx.serialization/kotlinx-serialization-core/kotlinx.serialization.encoding/-composite-decoder/decode-sequentially.html [Encoder.beginCollection]: https://kotlinlang.org/api/kotlinx.serialization/kotlinx-serialization-core/kotlinx.serialization.encoding/-encoder/begin-collection.html [CompositeDecoder.decodeCollectionSize]: https://kotlinlang.org/api/kotlinx.serialization/kotlinx-serialization-core/kotlinx.serialization.encoding/-composite-decoder/decode-collection-size.html [Encoder.encodeNull]: https://kotlinlang.org/api/kotlinx.serialization/kotlinx-serialization-core/kotlinx.serialization.encoding/-encoder/encode-null.html [Encoder.encodeNotNullMark]: https://kotlinlang.org/api/kotlinx.serialization/kotlinx-serialization-core/kotlinx.serialization.encoding/-encoder/encode-not-null-mark.html [Decoder.decodeNotNullMark]: https://kotlinlang.org/api/kotlinx.serialization/kotlinx-serialization-core/kotlinx.serialization.encoding/-decoder/decode-not-null-mark.html [Encoder.encodeSerializableValue]: https://kotlinlang.org/api/kotlinx.serialization/kotlinx-serialization-core/kotlinx.serialization.encoding/-encoder/encode-serializable-value.html [Decoder.decodeSerializableValue]: https://kotlinlang.org/api/kotlinx.serialization/kotlinx-serialization-core/kotlinx.serialization.encoding/-decoder/decode-serializable-value.html [kotlinx.serialization.properties.Properties]: https://kotlinlang.org/api/kotlinx.serialization/kotlinx-serialization-properties/kotlinx.serialization.properties/-properties/index.html [ProtoBuf]: https://kotlinlang.org/api/kotlinx.serialization/kotlinx-serialization-protobuf/kotlinx.serialization.protobuf/-proto-buf/index.html [ProtoBuf.encodeToByteArray]: https://kotlinlang.org/api/kotlinx.serialization/kotlinx-serialization-protobuf/kotlinx.serialization.protobuf/-proto-buf/encode-to-byte-array.html [ProtoBuf.decodeFromByteArray]: https://kotlinlang.org/api/kotlinx.serialization/kotlinx-serialization-protobuf/kotlinx.serialization.protobuf/-proto-buf/decode-from-byte-array.html [ProtoNumber]: https://kotlinlang.org/api/kotlinx.serialization/kotlinx-serialization-protobuf/kotlinx.serialization.protobuf/-proto-number/index.html [ProtoType]: https://kotlinlang.org/api/kotlinx.serialization/kotlinx-serialization-protobuf/kotlinx.serialization.protobuf/-proto-type/index.html [ProtoIntegerType]: https://kotlinlang.org/api/kotlinx.serialization/kotlinx-serialization-protobuf/kotlinx.serialization.protobuf/-proto-integer-type/index.html [ProtoIntegerType.DEFAULT]: https://kotlinlang.org/api/kotlinx.serialization/kotlinx-serialization-protobuf/kotlinx.serialization.protobuf/-proto-integer-type/-d-e-f-a-u-l-t/index.html [ProtoIntegerType.SIGNED]: https://kotlinlang.org/api/kotlinx.serialization/kotlinx-serialization-protobuf/kotlinx.serialization.protobuf/-proto-integer-type/-s-i-g-n-e-d/index.html [ProtoIntegerType.FIXED]: https://kotlinlang.org/api/kotlinx.serialization/kotlinx-serialization-protobuf/kotlinx.serialization.protobuf/-proto-integer-type/-f-i-x-e-d/index.html [ProtoBufSchemaGenerator]: https://kotlinlang.org/api/kotlinx.serialization/kotlinx-serialization-protobuf/kotlinx.serialization.protobuf.schema/-proto-buf-schema-generator/index.html [Cbor]: https://kotlinlang.org/api/kotlinx.serialization/kotlinx-serialization-cbor/kotlinx.serialization.cbor/-cbor/index.html [Cbor.encodeToByteArray]: https://kotlinlang.org/api/kotlinx.serialization/kotlinx-serialization-cbor/kotlinx.serialization.cbor/-cbor/encode-to-byte-array.html [Cbor.decodeFromByteArray]: https://kotlinlang.org/api/kotlinx.serialization/kotlinx-serialization-cbor/kotlinx.serialization.cbor/-cbor/decode-from-byte-array.html [CborBuilder.ignoreUnknownKeys]: https://kotlinlang.org/api/kotlinx.serialization/kotlinx-serialization-cbor/kotlinx.serialization.cbor/-cbor-builder/ignore-unknown-keys.html [ByteString]: https://kotlinlang.org/api/kotlinx.serialization/kotlinx-serialization-cbor/kotlinx.serialization.cbor/-byte-string/index.html