| IOLink Python 1.11.0

Loading...
Searching...
No Matches
Serialization

IOLink provides a mechanism of serialization for most of its objects. It allows to encode any object into a StreamAccess, and then, decode them to re-build similar objects in memory.

For each object, an encoding format has been defined, but a header is common to all objects.

Encoding format is versioned to check compatibility at decoding. A major and a minor value are encoded in the header.

Only minor versions are compatible.

To avoid any endianness issue, all data are encoded in big-endian and decoded in native-endian.

# Common Header

Starts every IOLink object serialization.

Offset | Size | Type | Information
--- | --- | --- | ---
0 | 4 | uint32_t | Magic number "IOL " (ASCII encoding)
4 | 2 | uint16_t | Serialization major version
6 | 2 | uint16_t | Serialization minor version
8 | 4 | uint32_t | Object identifier (see each object formatting)

Header is only written once for a whole object, and is not repeated for each internal object encoding.

e.g.: Specifically encoding a Vector triggers the writing of a header. But to encode an ImageView (which contains many Vectors), headers for Vector objects are skipped.

# Version compatibility

In the header, a `serialization version` field indicates which version was used for `encoding`. When `decoding`, this version is checked to ensure compatibility:

encoding version | decoding version | compatibility
--- | --- | ---
X.Y | X.Y | compatible
X.Y | X.Z | compatible if Z > Y
X.Y | W.Z | not compatible

Internally, in a newer minor version, only optional fields can be added to an existing object (already defined `object identifier`). A new major version is defined if new object are created or for any other modifications in existing ones.

# Data section

Generic data are always encoded the same and as follows:

Offset | Size | Type | Information
--- | --- | --- | ---
0 | 8 | uint64_t | Size of data (in bytes)
8 | see previous field | uint8_t | data

# String section

String are encoded as follows:

Offset | Size | Type | Information
--- | --- | --- | ---
0 | 8 | uint64_t | Length of string in bytes - no null-character at termination
8 | Data section size | Data section | string content

# Vector and VectorX

`Vector` and `VectorX`, despite they are 2 different objects in IOLink, have the same encoding format so as to ease the conversion from one to the other during decoding.

Object identifier
"VECT" (ASCII encoding)

Vector and VectorX are encoded as follow:

Offset (from header) | Size | Type | Information
--- | --- | --- | ---
0 | 8 | uint64_t | Scalar type identifier (see DataType)
8 | 8 | uint64_t | Element count in the Vector(X) (is redundant with Type id for Vector objects)

At offset 16, vector components are encoded into a data section (See Data section).

For a `Vector4u64` containing values `[5, 15, 25, 35]`, elements are encoded in following order:

- first 64-bit-word: `5`
- second 64-bit-word : `15`
- third 64-bit-word: `25`
- fourth 64-bit-word: `35`

Here is an example of serialization of a Vector:

# we use an in-memory stream, but it can be any StreamAccess
stream = iolink.StreamAccessFactory.allocate()
vect = iolink.Vector2u64(50, 125)
iolink.Serialization.encode_vector_2u_64(vect, stream)

And then the decoding (in the meanwhile, stream object was reset to its initial position):

# decode the expected Vector2u64 from the stream
vect = iolink.Serialization.decode_vector_2u_64(stream);
# or decode the Vector2u64 as a VectorXu64
# vect = iolink.Serialization.decodeVector_xu_64(stream);

# ImageView

ImageViews can be fully serialized. At decoding, an in-memory copy of original ImageView will be created.

A header with following OID precedes the encoding:

Object IDentifier
"IMAG" (ASCII encoding)

ImageView is encoded as follow:

Offset (from header) | Size | Type | Information
--- | --- | --- | ---
0 | depends on dimension count | VectorXu64 | shape of the ImageView (see VectorX)
undefined | 8 | uint64_t | datatype identifier (see DataType)
undefined | size of ImageProperties section | ImageProperties | ImageProperties of ImageView (see ImageProperties)
undefined | size of Metadata section | Metadata | Metadata tree of ImageView (see Metadata)
undefined | image Size x DataType size | uint8_t | image data (See Data)

Here is an example of encoding and decoding of an ImageView:

# we use an in-memory stream, but it can be any StreamAccess
stream = iolink.StreamAccessFactory.allocate()
img = iolink.ImageViewFactory.allocate((20, 20), iolink.DataTypeId.UINT8)
iolink.Serialization.encode_image_view(img, stream)
# reset stream
stream.seek(0, iolink.SeekOrigin_BEGIN)
# decode ImageView
imgDecoded = iolink.Serialization.decode_image_view(stream)

# ImageProperties

Offset | Size | Type | Information
--- | --- | --- | ---
0 | size of SpatialCalibration section | SpatialCalibration | SpatialCalibration of image (see SpatialCalibration)
undefined | size of ImageInfoProperties section | ImageInfoProperties | ImageInfoProperties of image (see ImageInfoProperties)

# SpatialCalibration

Offset | Size | Type | Information
--- | --- | --- | ---
0 | 24 | Vector3d | origin of image (see VectorX)
24 | 24 | Vector3d | spacing of image (see VectorX)
48 | 24 | Vector3d | direction column of image (see VectorX)
72 | 24 | Vector3d | direction row of image (see VectorX)
96 | size of string section | String | unit (see String)

# ImageInfoProperties

Offset | Size | Type | Information
--- | --- | --- | ---
0 | 8 | uint64_t | AxesInterpretation
8 | 8 | uint64_t | ImageInterpretation
16 | 8 | uint64_t | hasAlpha
24 | 8 | uint64_t | bitDepth
32 | 16 | Vector2d | Value Range (see VectorX)

# Metadata

Metadata is an optional field of ImageView (can be null). Thus, each Metadata node section starts with a 64-bit-word indicating the presence of a node or not.

Tag value | Description
--- | ---
"NMET" (ASCII encoding) | No node
"META" (ASCII encoding) | a node is encoded

And then, node is encoded as follows:

Offset after Tag | Size | Type | Information
--- | --- | --- | ---
0 | Size of string section | String | Key of node
undefined | Size of VariantDataValue section | VariantDataValue | Value of node (see VariantDataValue)
undefined | 8 | uint64_t | Child count
undefined | Size of Metadata section | Metadata | Child 1 (if exists)
undefined | Size of Metadata section | Metadata | Child 2 (if exists)
undefined | Size of Metadata section | Metadata | Child ...(if exists)

For example, if we have the following Metadata tree:

Example of Metadata tree

it will be stored in this following order:

Metadata
root
MetadataNode1
MetadataNode4
MetadataNode2
MetadataNode5
MetadataNode8
MetadataNode9
MetadataNode6
MetadataNode7
MetadataNode3

# VariantDataValue

Value of a node is an optional field (can be null). Thus, each VariantDataValue section starts with 64-bit-word indicating the presence of a value or not.

Tag value | Description
--- | ---
"NVAR" (ASCII encoding) | No value
"VARI" (ASCII encoding) | a value is encoded

And then, VariantDataValue is encoded as follows:

Offset after Tag | Size | Type | Information
--- | --- | --- | ---
0 | 8 | uint64_t | DataType ident of value (see DataType)
8 | 8 | uint64_t | value count
16 | Data section size | DataSection | list of values (See Data section)

# ArrayX

A header with the following OID precedes the encoding:

Object IDentifier
"ARRY" (ASCII encoding)

ArrayX is encoded as follows:

Offset (from header) | Size | Type | Information
--- | --- | --- | ---
0 | variable | VectorXu64 | shape of the ArrayX (see VectorX)
undefined | 8 | uint64_t | datatype identifier (see DataType)
undefined | array size x data type size | uint8_t | image data (See Data)

# DataFrameView

A header with the following OID precedes the encoding:

Object IDentifier
"DATF" (ASCII encoding)

DataFrameViews are encoded as follows:

Offset (from header) | Size | Type | Information
--- | --- | --- | ---
0 | 16 | Vector2u64 | shape of the DataFrameView (see Vector)
16 | size of string section | String | name of column #0 (see String)
undefined | 8 | uint64_t | datatype identifier for column #0 (see DataType)
undefined | size of string section | String | name of column #1 (see String)
undefined | 8 | uint64_t | datatype identifier for column #1 (see DataType)
undefined | size of string section | String | name of column #... (see String)
undefined | 8 | uint64_t | datatype identifier for column #... (see DataType)
undefined | size of string section | String | name of column #N (see String)
undefined | 8 | uint64_t | datatype identifier for column #N (see DataType)
undefined | DataFrame data section size | uint8_t | column #0 data (See Data)
undefined | DataFrame data section size | uint8_t | column #1 data (See Data)
undefined | DataFrame data section size | uint8_t | column #... data (See Data)
undefined | DataFrame data section size | uint8_t | column #N data (See Data)

## DataFrame data

DataFrameView column data can be either a list of strings or numerical values. Consequently column data can be encoded in a classical data section or in a String section, depending on DataType of each column.

# Matrix

A header with the following OID precedes the encoding:

Object IDentifier
"MATR" (ASCII encoding)

Matrix is encoded as follows:

Offset (from header) | Size | Type | Information
--- | --- | --- | ---
0 | 8 | uint64_t | dimension N of Matrix (3 or 4)
8 | 8 | uint64_t | datatype identifier (see DataType)
16 | N x N x data type size | uint8_t | matrix data (See Data)