In many experimental sciences, we acquire data from one or more instruments (camera, photodetectors, other sensors, etc…). Oftentimes data is encoded in binary numeric types, structures of numeric types and bit fields.
Here I assume we are dealing with “medium-data”, which is, using Wes McKinney words:
Medium Data (n): Not too big for a single machine, but too big to be dumb about.
We can easily and efficiently read this kind of data using Numpy, the foundational library for any numeric computing in Python. In this post I show the powerful tools numpy offers to “interpret” (or decode) binary data.
An example of binary-data¶
Recently, I had to read data from a new instrument that sends to the PC “words” of 48 bits.
The bit layout of the word is:
[placeholder]
Within each word, the byte order goes from byte1 (the first) to byte6 (the last).
The word is comprised of three numeric fields MT, mT, CH (unsigned integers of bit-width 29, 14 and 3 bits respectively), and a few flags (#B, OV).
For the numeric fields, the bit order in parenthesis shows that the most-significant-bit (MSB) is encountered first when reading the word orderly from the first to the last bit. This byte order is called big-endian (often indicated by ‘>’).
This example can seem fairly convoluted, but is quite common in data from scientific instruments.
Let’s see how we can decode this data, easily and efficiently, using numpy.
Numpy magics: dtype system¶
This section is a brief digression to introduce numpy’s dtype system and arrays “views” of binary buffers.
One of the most powerful features of numpy is providing a flexible memory model to access numeric data in arbitrary layouts.
At the core of the numpy library there is the ndarray object, i.e. a multi-dimentional array of homogeneous data type.
The crucial point is: what does numpy accept as homogeneous data type?
We can certaintly use all the standard numeric type like integers (signed or unsigned with 8, 16, 32 or 64 bits) or floats (16, 32, 64) or even complex numbers (internally a pair of floats).
But what if we want to define a more complex structure?
Numpy allows defining structures to be used as array elements: we just
need to define a dtype describing such a structure. For example,
to define a structure of a unit16 followed by a float16 we write:
import numpy as np
custom_dtype = np.dtype([('time', np.int16), ('humidity', np.float16)])
custom_dtype
Note that we assigned a name to each field in the structure, in this case
“time” (a 16-bit integer) and “humidity” (a 16-bit float). The ‘<’ sign in
the representation indicated little-endian byte order.
For alternative syntax to define a new dtype see the comprehensive
dtype documentation.
We can build an array of such a structure specifying the dtype argument
(accepted by the majority of numpy functions returning arrays) like this:
data = np.ones(6, dtype=custom_dtype)
data
The previous command defined a 6-element array. Each element of the array is the previously defined structure. Arrays with custom dtypeas are called in numpy structured arrays.
Let’s see a few example of what we can do with such an array.
1. Scalar to array broadcast¶
Add 10 to all the time fields:
data['time'] += 10
data
2. Assign data¶
Assign data to time and humidity fields:
data['time'] = np.arange(6)*10
data['humidity'] = (np.random.randn(6) + 5)*10
data
3. Indexing and slicing¶
Take element number 3 (the 4th element) of the array:
data[3]
A single element contains two values of the two fields.
Next, let’s take a slice starting at 1 and taking every 2 elements:
data[1::2]
This is a new structure array with only the selected elements.
4. View with different dtype¶
custom_dtype_b = np.dtype([('byte1', 'u1'), ('byte2', 'u1'), ('humidity', '<f2')])
data_b = data.view(custom_dtype_b)
data_b
In this latter data_b points to the same memory as data, but it reinterprets the array element with a different structure (the integer is split in 2 bytes).
Let’s decode!¶
Before starting let’s define a dummy binay buffer:
buff = b'\xFF' * (6*4)
len(buff)
In a real world situation this buffer would be read from disc using something like:
with open('datafile.dat', 'br') as f:
buff = f.read()Coming back to the previous example, we recognize that data has MT and mT aligned at 32 bit and 16 bit boundaries with big-endian representation.
To access those fields we can simply interpret the binary data with the following dtype:
dtype_fields = np.dtype([('timestamps', '>u4'), ('nanotimes', '>u2')])
dtype_fields
We only need to copy the bits CH, OV, #B first, and then set them to 0 in the original buffer. This operation is easier if we had direct access to the byte contining the 3 bits to save. Easy enough, this byte can be accessed defining a second dtype:
dtype_bytes = np.dtype([('byte%d' % b, 'u1') for b in range(6)])
dtype_bytes
Now we can “view” this binary buffer according to the two different dtypes we previously defined:
data = np.frombuffer(buff, dtype=dtype_fields)
data
datab = np.frombuffer(buff, dtype=dtype_bytes)
datab
By creating an array from an already existing buffer the data is not
copied, is just interpreted according to the passed dtype. In this
example data and datab point to the same binary content
in memory, they simply “view” the data differently, or
they provide different interfaces to the same data.
We can use datab['byte1'] as a normal array containing uint8 elements:
datab['byte0']
Let start copying the 3 bits in another array:
bitfields = np.zeros_like(data, dtype='u1')
bitfields
bitfields = datab['byte0'].copy()
np.right_shift(bitfields, 5, out=bitfields)
Then we can set the 3 bit to 0 in the orginal buffer:
datab.flags.writeable = True
datab['byte1'] &= 0x00111111
Note that we need to set the array flag writable to True
in order to be alble to modify the buffer. Having modified
the buffer also data sees the modified values:
data
data['timestamps']