| IOLink Python 1.11.0

Loading...
Searching...
No Matches
DataAccess and DataStorage

# Introduction

In IOLink we need interfaces to handle raw data, or byte arrays, in order to use them as underlying data of `ImageView` instances for example. We want to access that raw data in a way fitting the caller use, also a way to know where this data came from, and sometimes to access data stored relative to it. There are two concepts here:

- `DataAccess`: defines a way to access to a resource raw data. There are different families, that will be detailed after.
- `DataStorage`: acts like a factory of `DataAccess` instances, that also bear some information on the source of the data.

# DataAccess: an Interface to Access Bytes

`DataAccess` instances provide various services in order to read or write their underlying raw data. There are two families of `DataAccess`:

- `StreamAccess`: behaves like a classical stream, with a cursor that will change everytime you read or write, so reading the same number of bytes in a row with the same function call will return different result.
- `RandomAccess`: works like an enhanced buffer, each call to read or write data will give the same result if the arguments are the same.

# StreamAccess: Stream-like Data Access

This family of accessors mainly mimics the classic streams. It uses an internal cursor that will be the starting position of read and write operations, with the cursor moving during each of these. One of the properties of a stream is that it usually allows to extend its size when data is written at the end. The cursor is important, as it induces a state based behaviour: doing multiple read operations using the same arguments, will give different results. So a stream can be dangerous in a multithreaded environment.

`StreamAccess` defines various capabilities:

- `READ`: used to read data from the stream, usually reading the desired bytes from the current cursor position.
- `WRITE`: used to write data to the stream, usually writing the desired bytes at the current cursor position.
- `SEEK`: enable to move the stream's cursor position.
- `MEMORY_ACCESS`: enable to access the internal CPU memory buffer.

The `StreamAccessFactory` factory provides diverse ways to create `StreamAccess` instances. For example, you can create an in-memory extensible stream this way:

stream = StreamAccessFactory.allocate()

For a more complete example:

# open a stream access on a file
file_access = StreamAccessFactory.open_file("/home/user/folder/file.txt")
assert file_access.support(StreamAccessCapability.READ)
# if the stream cursor is not at the beginning, we reset it.
if file_access.tell() != 0:
file_access.seek(0, SeekOrigin_BEGIN)
# all data is written in one call
buffer = bytes([1, 2, 3, 4, 5])
file_access.write(buffer)
# as we reached the stream end by writing in it, we must reset the cursor.
file_access.seek(0, SeekOrigin_BEGIN)
# we read all data
extracted = file_access.read(len(buffer))
# extracted now contains {1, 2, 3, 4, 5}

# RandomAccess: Array-like Data Access

This family of interfaces behaves like a virtual buffer with a fixed size, from wich you can read or write some slice of data. Operations are stateless in this kind of objects and can be used in a multithreaded environment if the implementation allows it. By nature, a `RandomAccess` cannot be expanded automatically, and trying to write data over its end will cause an error. You can see these interfaces as fancy arrays.

`RandomAccess` support the following capabilities:

- `READ`: used to read data, usually reading a given number of bytes from a given offet.
- `WRITE`: used to write data, usually reading a given number of bytes from a given offet.
- `RESIZE`: enable to change the size of the accessor, as it can't be expanded by another way.
- `MEMORY_ACCESS`: enable to access the internal CPU memory buffer.

The `RandomAccessFactory` factory provides several ways to create `RandomAccess` instances. For example, you can create an in-memory `RandomAccess` instance this way:

accessor = RandomAccessFactory.allocate(1000)

For a more complete example:

# create an in-memory random access
memoryAccess = RandomAccessFactory.allocate(5)
assert memoryAccess.support(RandomAccessCapability.READ_WRITE)
buffer = bytes([1, 2, 3, 4, 5])
memoryAccess.write(0, buffer)
extracted = memoryAccess.read(0, len(buffer))
# extracted now contains {1, 2, 3, 4, 5}

# DataStorage: DataAccess Factory

`DataStorage` should be considered as a `DataAccess` factory/manager that will provide access to resources that are under that storage. It is an interface that will enable users to create `StreamAccess` or `RandomAccess` instances, that are relative to the storage instance. There is also a way to identify the storage's source, using the `source` method that will return a string describing the origin of the data, usually in an URI form. For example a file system storage will return a string of the form "file:/home/user/folder".

When requesting access to resources, you must give the resource id, and the capabilities you want: read, write, or both. If the storage can't return an accessor corresponding to these properties, it will raise an exception.

Current supported `DataStorage` variants:

- FileSystemStorage: usually having a source with a "file:" scheme, these storages only support opening of `StreamAccess`, but with all the access capacities.
- HttpStorage: described by a source using the "http:" or "https" schemes, supports `RandomAccess` opening only in read-only, and `StreamAccess` opening only in write-only.

An example, using LDM data:

headerStream = StreamAccessFactory.open_file("/home/user/folder/header.ldm")
storage = headerStream.storage
dataStream = storage.open_stream_access("data.dat", AccessCapabilities_READ_WRITE)