Skip to content

Utilities for serialising and deserialising streaming data type messages

License

Notifications You must be signed in to change notification settings

ess-dmsc/python-streaming-data-types

Repository files navigation

Python Streaming Data Types

Utilities for working with the FlatBuffers schemas used at the European Spallation Source ERIC for data transport.

https://github.com/ess-dmsc/streaming-data-types

FlatBuffer Schemas

name description
hs00 Histogram schema (deprecated in favour of hs01)
hs01 Histogram schema
ns10 NICOS cache entry schema
pl72 Run start
6s4t Run stop
f142 Log data (deprecated in favour of f144)
f144 Log data
ev42 Event data (deprecated in favour of ev44)
ev43 Event data from multiple pulses
ev44 Event data with signed data types
x5f2 Status messages
tdct Timestamps
ep00 EPICS connection info (deprecated in favour of ep01)
ep01 EPICS connection info
rf5k Forwarder configuration update (deprecated in favour of fc00)
fc00 Forwarder configuration update
answ File-writer command response
wrdn File-writer finished writing
NDAr Deprecated
ADAr EPICS areaDetector data
al00 Alarm/status messages used by the Forwarder and NICOS
senv Deprecated
json Generic JSON data
se00 Arrays with optional timestamps, for example waveform data. Replaces senv.
da00 Scipp-like data arrays, for histograms, etc.

hs00 and hs01

Schema for histogram data. It is one of the more complicated to use schemas. It takes a Python dictionary as its input; this dictionary needs to have correctly named fields.

The input histogram data for serialisation and the output deserialisation data have the same dictionary "layout". Example for a 2-D histogram:

hist = {
    "source": "some_source",
    "timestamp": 123456,
    "current_shape": [2, 5],
    "dim_metadata": [
        {
            "length": 2,
            "unit": "a",
            "label": "x",
            "bin_boundaries": np.array([10, 11, 12]),
        },
        {
            "length": 5,
            "unit": "b",
            "label": "y",
            "bin_boundaries": np.array([0, 1, 2, 3, 4, 5]),
        },
    ],
    "last_metadata_timestamp": 123456,
    "data": np.array([[1, 2, 3, 4, 5], [6, 7, 8, 9, 10]]),
    "errors": np.array([[5, 4, 3, 2, 1], [10, 9, 8, 7, 6]]),
    "info": "info_string",
}

The arrays passed in for data, errors and bin_boundaries can be NumPy arrays or regular lists, but on deserialisation they will be NumPy arrays.

Developer documentation

See README_DEV.md