| IOLink C# 1.11.0

Loading...
Searching...
No Matches
DataFrames

# What is a DataFrame ?

A dataframe is simply a model for tabular data, organised in columns and rows. More specifically, a dataframe is column-oriented, meaning that data is stored in columns, rows just being an aggregate of columns' data. Because of this construction, type of data is homogeneous along columns, but can be heterogeneous along rows.

# DataFrameView API

The API for dataframe defines a view that has various capabilities and methods. `DataFrameView` has three capabilities: `READ` for reading columns' data, `WRITE` for editing it, and `RESHAPE` for adding and removing columns or rows.

There are also structure methods that will be available for all instances, especially for information about the shape of the dataset, and column informations. Columns are usually accessed by indices in this interface, ranging from 0 to the shape's width. But more information can be accessed using the following methods:

Vector2u64 shape = frame.Shape;
// Here we display the name and data type of each column
for(uint i = 0; i < shape[0]; ++i)
{
string name = frame.ColumnName(i);
DataType dtype = frame.ColumnDataType(i);
Console.Out.WriteLine("{0}, {1}", name, dtype.ToString());
}

# Creating a memory data frame

The factory `DataFrameViewFactory` has a method `allocate` that can be used to create a `DataFrameView` with its data stored in memory:

var columnNames = new string[]{"age", "name"};
var columnDataTypes = new DataType[]{DataTypeId.UINT16, DataTypeId.UTF8_STRING};
DataFrameView dframe = DataFrameViewFactory.Allocate(new Vector2u64(2, 5), columnNames, columnDataTypes);

# Reading and writing data

The reading interface of `DataFrameView` is quite simple and low level. You can only read data from one column at a time, and you must pass a buffer to the reading method with a size corresponding to the amount of elements you want to read.

In this example, we want to read data from the first five rows of the data frame, that has a column with an integer type, and another storing strings.

int[] buffer0 = new int[5];
string[] buffer1 = new string[5];
// Read first column data
frame.Read<int>(0, 0, 5, buffer0);
// Read second column data
frame.Read<string>(1, 0, 5, buffer1);
// You can also directly return the read data by removing the last argument.
int[] data = frame.Read<int>(0, 0, 5);

The interface to write data is quite similar, with the difference that you must fill the given buffer with the data to write. Similary to previous example, we can write the top five rows of our data frame like this:

int[] buffer0 = new int[]{1, 2, 3, 4, 5};
string[] buffer1 = new string[]{"monday", "tuesday", "wednesday", "thursday", "friday"};
// Write first column data
frame.Write<int>(0, 0, 5, buffer0);
// Write second column data
frame.Write<string>(1, 0, 5, buffer1);

For getting one element of a column, there are some more user-friendly methods to use:

// get the second element of the first column
int value1 = frame.At<int>(0, 1);
// get the third element of the second column
string value2 = frame.At<string>(1, 2);
// get the second element of the "order" column
int value3 = frame.At<int>("order", 1);
// get the third element of the "name" column
string value4 = frame.At<string>("name", 2);

# Changing the dataframe shape

The `RESHAPE` capability offers method to change the shape of a `DataFrameView` instance by adding and removing columns and rows. Because of the column-oriented structure, columns can only be affected one by one, but rows can be processed in continuous chunks.

Columns operations examples:

// Append a new column, its index being the previous width of the shape
frame.AddColumn("age", DataTypeId.UINT32);
// We remove the first column
frame.RemoveColumn(0);

Row operations example:

// Add two lines after the first one
frame.AddRows(1, 2);
// Removing the first two lines
frame.RemoveRows(0, 2);
// Add five lines at the end
frame.AddRows(frame.Shape[1], 5);

It is also possible to entirely reshape the data frame. Existing Data are lost.

var columnNames = new string[]{"col1", "col2"};
var columnDataTypes = new DataType[]{DataTypeId.UINT32, DataTypeId.UINT64};
dframe.Reshape(new Vector2u64(2, 10), columnNames, columnDataTypes);