ZeroMQ XOP

The ZeroMQ XOP allows to interface with Igor Pro over the network using ZeroMQ as messaging layer and JSON as message format. Reading and writing JSON documents can be done with the JSON XOP.

The XOP provides the following functions:

:cpp:func:`zeromq_client_connect()`
:cpp:func:`zeromq_client_recv()`
:cpp:func:`zeromq_client_send()`
:cpp:func:`zeromq_handler_start()`
:cpp:func:`zeromq_handler_stop()`
:cpp:func:`zeromq_pub_bind`
:cpp:func:`zeromq_pub_send`
:cpp:func:`zeromq_server_bind()`
:cpp:func:`zeromq_server_recv()`
:cpp:func:`zeromq_server_send()`
:cpp:func:`zeromq_set()`
:cpp:func:`zeromq_set_logging_template`
:cpp:func:`zeromq_stop()`
:cpp:func:`zeromq_sub_add_filter`
:cpp:func:`zeromq_sub_connect`
:cpp:func:`zeromq_sub_recv`
:cpp:func:`zeromq_sub_remove_filter`

This XOP primarily supports (and is tested on) Igor Pro versions 8 or above. The code in principle supports Igor Pro 6 and 7, but the test suite does not. Therefore, builds released for Igor 6/7 are considered EXPERIMENTAL and should be treated as such. Special instructions for Igor 6/7 are described at the end of this readme.

Installation

Here XX denotes your major Igor Pro version, e.g. 8 or 9.

Download the ZeroMQ-XOP*.zip file from the latest release.
Extract it to a folder

Windows

Quit Igor Pro
Create the following shortcuts in "$HOME\Documents\WaveMetrics\Igor Pro XX User Files"
- In "Igor Procedures" a shortcut pointing to "procedures"
- In "Igor Help Files" a shortcut pointing to "help"
- In "Igor Extensions" a shortcut pointing to "output/win/x86"
- In "Igor Extensions (64-bit)" a shortcut pointing to "output/win/x64"
Start Igor Pro

MacOSX

Quit Igor Pro
Unzip the files in "output/mac"
Create the following symbolic links (symlinks) in "$HOME/Documents/WaveMetrics/Igor Pro XX User Files"
- In "Igor Procedures" a symlink pointing to "procedures"
- In "Igor Help Files" a symlink pointing to "help"
- In "Igor Extensions" a symlink pointing to "output/mac/ZeroMQ"
- In "Igor Extensions (64-bit)" a symlink pointing to "output/mac/ZeroMQ-64"
Start Igor Pro

In the following the JSON message format is discussed.

Direction: World -> Igor Pro

Call Igor Pro functions and return the result

The following table lists all currently supported function parameter and return types. PRs adding support for new parameter/return types are welcome.

Type	by-value Parameter	by-ref Parameter	optional Parameter	Return value	Multiple return values
Variable aka double
Variable/C aka complex
Int/int64/uint64/uint
String
Wave
DFREF
FUNCREF
STRUCT

The Igor Pro function FooBar(string panelTitle, variable index) can be called by sending the following string

{
  "version"   : 1,
  "messageID" : "my first message",
   "CallFunction" : {
     "name" : "FooBar",
     "params" : [
        "ITC18USB_DEV_0",
        1
     ]
   }
}

Calling a function without parameters:

{
  "version" : 1,
   "CallFunction" : {
     "name" : "FooBarWithoutArgs"
   }
}

Possible responses:

{
  "errorCode" : {
   "value" : 0
  },
  "messageID" : "my first message",
  "result" : {
    "type" : "variable",
    "value" : 4711
  }
}

or

{
  "errorCode" : {
    "value" : 100,
    "msg" : "Function does not exist"
  },
  "messageID" : "my first message",
}

If the function has pass-by-reference parameters their results are returned as

{
  "errorCode": {
      "value": 0
  },
  "passByReference": [
    {
        "type": "variable",
        "value": 4711
    },
    {
        "type": "string",
        "value": "hi there"
    }
  ],
  "result": {
      "type": "variable",
      "value": 42
  }
}

Functions can also return datafolder references

{
  "errorCode" : {
   "value" : 0
  },
  "result" : {
    "type"  : "dfref",
    "value" : "root:MIES"
  }
}

result.value can also be free or null.

Functions with multiple return values

Since Igor Pro 8 functions can return multiple values.

Function [variable erroCode, string message] FooBarMRS()

   return [42, "Hi there!"]
End

The function FooBarMRS() will return the following message:

{
    "errorCode": {
        "value": 0
    },
    "result": [
        {
            "type": "variable",
            "value": 42
        },
        {
            "type": "string",
            "value": "Hi there!"
        }
    ]
}

Functions returning waves

Example wave contents (rows are vertical, colums are horizontal)

5	8
6	-inf
7	10

Waves with standard settings only:

{
  "errorCode" : {
   "value" : 0
  },
  "result" : {
    "type"  : "wave",
    "value" : {
      "type"     : "NT_FP64",
      "dimSize"  : [3, 2],
      "date"     : {
        "modification" : 10221232
        },
      "data" : {
        "raw" : [5, 6, 7, 8, "-inf", 10]
        }
      }
  }
}

In case the function returned an invalid wave reference $"":

{
  "errorCode" : {
   "value" : 0
  },
  "result" : {
    "type"  : "wave",
    "value" : null
  }
}

The following is an example where all additional settings are present because they differ from their default values:

{
  "errorCode" : {
   "value" : 0
  },
  "result" : {
    "type"  : "wave",
    "value" : {
      "type"     : "NT_FP64",
      "date"     : {
        "modification" : 10221232
        },
      "data" : {
        "raw"       : [5, 6, 7, 8, "-inf", 10],
         "unit"      : "m",
         "fullScale" : [5, 10]
        },
      "dimension" : {
        "size"  : [3, 2],
         "delta" : [1, 2.5],
         "offset": [1e5, 3e7],
         "unit"  : ["kHz", "s"],
         "label" : {
           "full"  : [ "some name", "blah" ],
           "each" : [ "..." ]
          }
      },
       "note" : "Hi there I'm a nice wave note and are encoded in \"UTF8\". With fancy things like ï or ß.",
    }
  }
}

Specification

Messages consist of JSON RFC7158 encoded strings with one speciality. NaN, Inf and -Inf are not supported by JSON, so we encode these non-normal numbers as strings, e.g. "NaN", "Inf", "+Inf" and "-Inf" (case insensitive).

Sent JSON message

Name	JSON type	Value	Description	Required
version	string	`v1`	global for the complete interface	Yes
operation	object	`CallFunction`	operation which should be performed	Yes
CallFunction.name	string	non-empty	ProcGlobal function without module and or independent module specification, i.e. without `#`.	Yes
CallFunction.params	array of strings/numbers	holds strings/numbers	function parameters, conversion will be done eagerly.	No
messageID	string	user settable	will be returned in the reply message if present	No

Received JSON message for operation `CallFunction`

Name	JSON type	Description
errorCode.value	number	indicates the success/error of the operation, see :cpp:any:`REQ_SUCCESS`
errorCode.msg	string	human readable error message, only set if errorCode.value != 0
history	string	Igor Pro history ouputted during function execution, only set if errorCode.value != 0
return	object or array	function result, will be an array when multiple return value syntax functions are called.
-> type	string	type of the function result, one of `string`, `variable`, `wave` or `dfref`, only for errorCode.value == 0
-> value	number, string or object	function result, only for errorCode.value == 0
passByReference	array of objects	Changed parameter values for pass-by-reference parameters.
-> type	string	type of the function result, one of `string`, `variable` or `dfref`
-> value	number or string	possibly changed input parameters, only for errorCode.value == 0
messageID	string	message ID from the sent message. This entry is not present if the sent message did not contain a message id.

Callers are encouraged to always check errorCode.value before processing the rest of the JSON. Functions returning waves will hold the wave data and metadata as object below value. All strings are UTF8 encoded. The messageID allows to correlate responses with requests.

Wave serialization format

When the serialization is done as part of the function call reply as shown above, one has to prefix each name with value..

Name	JSON type	Description
type	string	wave type; one of NT_FP32, NT_FP64, NT_I8, NT_I16, NT_I32, NT_I64, TEXT_WAVE_TYPE, WAVE_TYPE or DATAFOLDER_TYPE; or'ed with NT_UNSIGNED or NT_CMPLX if needed
dimension.size	array of 1 to 4 numbers	either "32-bit unsigned int" or "64-bit unsigned int" depending on Igor bitness. An empty wave has `[0]`.
dimension.delta	array of 1 to 4 numbers	delta for each dimension
dimension.offset	array of 1 to 4 numbers	offset for each dimension
dimension.label.full	array of 1 to 4 stringss	dimension labels for the full dimensions
dimension.label.each	array of strings	dimension labels for each row/column/layer/chunk, colum-major format as `result.data.raw`
dimension.unit	array of 1 to 4 strings	arbitrary strings denoting the unit for each dimension. The contents are most likely SI with prefix, but this is not guaranteed.
date.modification	number	time of last modification in seconds since unix epoch in UTC. 0 for free waves.
data.raw	array of numbers/strings	column-major format, read it with `np.array([5, 6, 7, 8, "-inf", 10]).reshape(3, 2, order='F')` using Python. For complex waves `raw` has two keys `real` and `imag` both holding arrays. For wave reference waves `raw` holds an array with wave objects or null.
data.unit	string	arbitrary strings denoting the unit. The contents are most likely SI with prefix, but this is not guaranteed.
data.fullScale	array of 2 numbers	min and max of the data (non-authorative)
note	string	wave note

Examples

Numeric wave with properties set to non-default values:

{
  "type"     : "NT_FP64",
  "data" : {
    "raw"       : [5, 6, 7, 8, "-inf", 10],
     "unit"      : "m",
     "fullScale" : [5, 10]
  },
  "date"     : {
    "modification" : 10221232
  },
  "dimension" : {
    "size"  : [3, 2],
     "delta" : [1, 2.5],
     "offset": [1e5, 3e7],
     "unit"  : ["kHz", "s"],
     "label" : {
       "full"  : [ "some name", "blah" ],
       "each" : [ "..." ]
      }
  },
  "note" : "Hi there I'm a nice wave note and are encoded in \"UTF8\". With fancy things like ï or ß."
}

Text wave:

{
  "data": {
      "raw": [ "abcd", "efgh" ]
  },
  "date": {
      "modification": 1685115358
  },
  "dimension": {
      "size": [ 2 ]
  },
  "type": "TEXT_WAVE_TYPE"
}

Wave reference wave:

{
  "data": {
      "raw": [
          {
              "data": {
                  "raw": [ 1, 2 ]
              },
              "date": {
                  "modification": 1685115583
              },
              "dimension": {
                  "size": [ 2 ]
              },
              "type": "NT_FP32"
          },
          {
              "data": {
                  "raw": [ 3, 4 ]
              },
              "date": {
                  "modification": 1685115598
              },
              "dimension": {
                  "size": [ 2 ]
              },
              "type": "NT_FP32"
          },
          null
      ]
  },
  "date": {
      "modification": 1685115607
  },
  "dimension": {
      "size": [ 3 ]
  },
  "type": "WAVE_TYPE"
}

Direction: Igor Pro -> World

The XOP implements Publisher/Subscriber sockets. This allows applications outside of Igor Pro to be notified about events in Igor Pro. The implementation uses plain PUB/SUB sockets, but XPUB/XSUB sockets should be compatible as well.

The published messages will be a multipart message with two frames, see also the official documentation:

Frame 1: Filter
Frame 2: Data

where Filter is the message type and Data the payload. No serialization format of Data is enforced, but users are encouraged to use standard serialization formats like JSON.

Subscriber sockets will only receive messages from their subscribed filters. By default there are no subscriptions to any filters.

One publisher message is sent out every five seconds, this is the "heartbeat" message with no data.

Users are encouraged to offer a list of available message filters via server/client sockets and calling a pre-agreed function which returns a text wave.

Dependencies

zeromq-xop has the following 3rd party dependencies, which must be installed to compile:

(Windows only) Visual Studio 2019 - Windows development environment.
(MacOSX only) Xcode - Mac OSX development environment.
CMake (version 3.15 or later) - build system.
XOPToolkit 8 - toolkit for creating XOPs (such as this one), to communicate with Igor Pro.

zeromq-xop also depends on a couple of additional repositories, which are included in the repository and do not require separate installation:

FMT formatting library.
JSON for Modern C++ JSON encoding/decoding in C++.
Caseymcc's CreateLaunchers (from Rylie's CMake Modules Collection) helper modules used by the build system.

Lastly, unit tests requires setup of the following (with instructions on doing so further below):

Igor Unit Testing Framework

Building the ZeroMQ XOP

To get set up, we must install prerequisites, clone our repository, set up our submodules, and 'position' the XOP toolkit.

We will use the following variable names for clarity below:

$xop-toolkit-dir is the path to the XOP Toolkit top-level directory; and
$zmq-xop-dir is the path to our ZeroMQ-XOP code;

Installing prerequisites

Before continuing, ensure you have installed the prerequisites listed in the 'Dependencies' section above. For a Windows system, ensure Visual Studio is installed; for a Mac system, ensure XCode is installed. For both, ensure you have cmake installed, and the XOP Toolkit downloaded.

Repository setup

To clone the repository (and clone the required submodules), perform the following:

git clone --recurse-submodules https://github.com/AllenInstitute/ZeroMQ-XOP.git

Here, --recurse-submodules is responsible for recursively initializing and updating the submodules (described above). If you have already cloned, init and update the modules via git submodule update --init --recursive.
If you are using SSH or another mechanism to obtain the repository, replace the http link above with your repository ID.

XOP toolkit setup

Our build system (cmake) must know where the XOP toolkit's main code files are (located in $xop-toolkit-dir/XOP Toolkit 8/IgorXOPs8/XOPSupport). By default, cmake will search for them in: $zmq-xop-dir/XOPSupport.

If using the default location, one should make a shortcut/symbolic link between $xop-toolkit-dir/XOP Toolkit 8/IgorXOPs8/XOPSupport and $zmq-xop-dir/XOPSupport:

# Windows (Note: mklink requires administrator privileges)
# {
mklink \d $zmq-xop-dir/XOPSupport "$xop-toolkit-dir/XOP Toolkit 8/IgorXOPs8/XOPSupport"
# }
# MacOSX
# {
ln -s "$xop-toolkit-dir/XOP Toolkit 8/IgorXOPs8/XOPSupport" $zmq-xop-dir/XOPSupport
# }

This can be alternatively be changed by changing cmake's ${XOP_SUPPORT_PATH} variable, either via the UI (cmake-gui for Windows, ccmake for Linux/Mac OSX), or when invoking the generator:

cmake -DXOP_SUPPORT_PATH="$xop-toolkit-dir/XOP Toolkit 8/IgorXOPs8/XOPSupport"

Compilation instructions

The compilation procedure involves:

cmake generates the environment-specific 'projects', based on its CMakeLists.txt files. This is achieved by the initial cmake call.
The development environment builds the XOP library, via the '--build' portion of the cmake call.
The development environment 'installs' the XOP library (and dependencies) in an install location (as defined in the CMakeLists). Note that 'install' here simply refers to a copy of appropriate files to a predefined location (and thus differs from our "Installation" instructions).

The commands below perform this. (See also .gitlab.ci.yml for up-do-date build instructions.)

# Windows
# {
cd $zmq-xop-dir/src
md build build-64
cd build
cmake -G "Visual Studio 16 2019" -A Win32 -S .. -B .
cmake --build . --config Release --target install
cd ../build-64
cmake -G "Visual Studio 16 2019" -A x64 -S .. -B .
cmake --build . --config Release --target install
# }

# MacOSX
# {
cmake -G Xcode -S .. -B .
cmake --build . --config Release --target install
# }

After cmake 'install', the created libraries will be located in $zmq-xop-dir/output/$os, where $os is mac for Mac, and win for Windows. For Mac, they will be in an xop directory, whereas for Windows they will be in an xop directory within a 'bitness' directory (x64 for 64-bit, x86 for 32-bit).

Debugging the XOP

When compiled from source, debugging launchers are created to allow easier debugging of the XOP.

For Windows, a number of launch-ZeroMQ-${CMAKE_BUILD_TYPE}.cmd scripts are created, with ${CMAKE_BUILD_TYPE} referring to a compilation mode of interest (e.g., Debug, Release). Running it will launch Igor with the ZeroMQ.xop in debugger mode. The user can then open their VS debugger and debug as needed.
For Mac OSX, a launch-ZeroMQ.sh script is created. Running it will start Igor and a gdb debugger, allowing similar debugging to be done as needed.

In both cases, a knowledge of where the Igor executable is located is necessary. The existing CMake files contain hardcoded assumptions for where they are, assuming Igor Pro 9 is installed. However, the user may explicit this path by setting the CMake variable ${igorPath} (see "XOP toolkit setup" for instructions on settings cmake variables).

Running the test suite

Clone the Igor Unit Testing Framework.
Create in "Igor Procedures" a shortcut pointing to the "procedures" directory of that repository.
Open $zmq-xop-dir/tests/RunTests.pxp
Execute in Igor run()
The test suite always passes without errors

ZeroMQ XOP implementation details

The XOP uses the Dealer (called Client in the XOP interface), Router (called Server in the XOP interface) and Publisher/Subscriber socket types.

The default socket options are:

ZMQ_LINGER = 0
ZMQ_SNDTIMEO = 0
ZMQ_RCVTIMEO = 0
ZMQ_ROUTER_MANDATORY = 1 (Router only)
ZMQ_MAXMSGSIZE = 1024 (in bytes, Router only)
ZMQ_IDENTITY = zeromq xop: dealer (Dealer only)

The Router/Server expects three frames (identity, empty, payload) and the Dealer/Client expects two frames (empty, payload) when sending/receiving messages. This format is used to be compatible with REP/REQ sockets.

The Publisher/Subscriber send/expect two frames (filter, payload). This is done so that there is no ambiguity between filter and payload. The payload can be empty.

The passed function in the JSON message is currently always executed in the main thread during IDLE events. IDLE events are generated by Igor Pro only when no functions are running. In case you want to execute a function during the time when functions are running the operation DoXOPIdle allows to force an IDLE event.

Logging

The XOP allows to log all incoming and outgoing messages to disk. This can be enabled via zeromq_set. The log format is JSONL. Additional static entries can be added to every line via zeromq_set_logging_template which allows to set a new template JSON text.

The location of the log file on Windows is C:\Users\$user\AppData\Roaming\WaveMetrics\Igor Pro $version\Packages\ZeroMQ\Log.jsonl.

Igor Pro 6/7 Support

As mentioned previously, this XOP supports Igor versions 6/7 in principle. However, the test suite does not. Thus, Igor6/7 builds are experimental and should be treated as such.

In what follows, we explicit the differences in installation and compilation for Igor6/7.

Installation

The differences in installation relate to Igor Pro 6/7's support for 32-bit or 64-bit extensions.

Igor Pro 6 and earlier are 32-bit applications, and thus require 32-bit extensions. Note that there is a special 64-bit Igor Pro 6 for Windows, but it is suggested only for special cases. Also note that Igor6 on Mac requires MacOS 10.14 or earlier (as 32-bit support ends with MacOS 10.15).
Igor Pro 7 installs both 32-bit and 64-bit versions, with the 64-bit recommended to be used.

In both cases, only a single "Igor Extensions" directory is provided in the user files directory (i.e., there is no "Igor Extensions (64-bit)"). As such, you must symlink the appropriate extensions folder depending on your Igor bitness:

If using 32-bit Igor, make a symbolic link/shortcut to "output/win/x86" for Windows, and "output/mac/ZeroMQ" for Mac.
If using 64-bit Igor, make a symbolic link/shortcut to "output/win/x64" for Windows, and "output/mac/ZeroMQ-64" for Mac.

Compilation

To compile for Igor Pro 6/7:

You must use the proper XOP Toolkit: Toolkit 7. Thus, in the XOP Toolkit Setup section, replace $xop-toolkit-dir/XOP Toolkit 8/IgorXOPs8/XOPSupport with $xop-toolkit-dir/XOP Toolkit 7/IgorXOPs7/XOPSupport throughout.
You must explicit Igor 6 in the cmake generation stage. In other words, your cmake -G .. calls (the first cmake call) must include -DXOP_MINIMUM_IGORVERSION=637 (indicating the XOP version).
On Windows, you should compile with the officially supported Visual Studio version for XOP Toolki 7: Visual Studio 15 2017. As such, your cmake generation stage should use cmake -G "Visual Studio 15 2017" .. (instead of 2019).

Putting these together, the generation steps are (note the lack of x64 build for Windows, as it is not supported generally):

# Windows
# {
cd $zmq-xop-dir/src
md build
cd build
cmake -G "Visual Studio 15 2017" -A Win32 -DCMAKE_BUILD_TYPE=Release -DXOP_MINIMUM_IGORVERSION=637 -S .. -B .
cmake --build . --config Release --target install
# }

# MacOSX
# {
cmake -G Xcode -DCMAKE_BUILD_TYPE=Release -DXOP_MINIMUM_IGORVERSION=637 -S .. -B .
cmake --build . --config Release --target install
# }

After compilation, the created libraries will be located in $zmq-xop-dir/output/$os.

Name		Name	Last commit message	Last commit date
Latest commit History 284 Commits
doc		doc
examples		examples
help		help
output		output
procedures		procedures
src		src
tests		tests
utf @ 1b0c98b		utf @ 1b0c98b
xop-stub-generator		xop-stub-generator
.clang-tidy		.clang-tidy
.gitattributes		.gitattributes
.gitlab-ci.yml		.gitlab-ci.yml
.gitmodules		.gitmodules
CONTRIBUTING.md		CONTRIBUTING.md
LICENSE.txt		LICENSE.txt
Readme.rst		Readme.rst
ctags.cnf		ctags.cnf

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Repository files navigation

ZeroMQ XOP

Installation

Windows

MacOSX

Direction: World -> Igor Pro

Call Igor Pro functions and return the result

Functions with multiple return values

Functions returning waves

Specification

Sent JSON message

Received JSON message for operation `CallFunction`

Wave serialization format

Examples

Direction: Igor Pro -> World

Dependencies

Building the ZeroMQ XOP

Installing prerequisites

Repository setup

XOP toolkit setup

Compilation instructions

Debugging the XOP

Running the test suite

ZeroMQ XOP implementation details

Logging

Igor Pro 6/7 Support

Installation

Compilation

About

Releases 2

Contributors 5

Languages

License

AllenInstitute/ZeroMQ-XOP

Folders and files

Latest commit

History

Repository files navigation

ZeroMQ XOP

Installation

Windows

MacOSX

Direction: World -> Igor Pro

Call Igor Pro functions and return the result

Functions with multiple return values

Functions returning waves

Specification

Sent JSON message

Received JSON message for operation CallFunction

Wave serialization format

Examples

Direction: Igor Pro -> World

Dependencies

Building the ZeroMQ XOP

Installing prerequisites

Repository setup

XOP toolkit setup

Compilation instructions

Debugging the XOP

Running the test suite

ZeroMQ XOP implementation details

Logging

Igor Pro 6/7 Support

Installation

Compilation

About

Resources

License

Stars

Watchers

Forks

Releases 2

Contributors 5

Languages

Received JSON message for operation `CallFunction`