Skip to content

Commit

Permalink
Merge pull request #2414 from zuo/jk/ssl-improvements-related-to-stom…
Browse files Browse the repository at this point in the history
…p-and-n6

STOMP+SSL-related changes: security fixes + convenience enhancement (+ doc updates/improvements)
  • Loading branch information
sebix authored Nov 21, 2023
2 parents 95e4fec + 0784db4 commit d14611c
Show file tree
Hide file tree
Showing 10 changed files with 446 additions and 109 deletions.
35 changes: 26 additions & 9 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,10 @@
if `auth_by_ssl_client_certificate` is *false*);
- `password` (STOMP authentication passcode, default: "guest"; to be used only
if `auth_by_ssl_client_certificate` is *false*).
- Add the possibility to set the `ssl_ca_certificate` configuration parameter for
`intelmq.bots.collectors.stomp.collector` and/or `intelmq.bots.outputs.stomp.output`
to an empty string - which means that the SSL machinery used for STOMP communication
will attempt to load the system’s default CA certificates (PR#2414 by Jan Kaliszewski).

### Core
- `intelmq.lib.message`: For invalid message keys, add a hint on the failure to the exception: not allowed by configuration or not matching regular expression (PR#2398 by Sebastian Wagner).
Expand All @@ -27,7 +31,7 @@
- `intelmq.lib.mixins`: Add a new class, `StompMixin` (defined in a new submodule: `stomp`),
which provides certain common STOMP-bot-specific operations, factored out from
`intelmq.bots.collectors.stomp.collector` and `intelmq.bots.outputs.stomp.output`
(PR#2408 by Jan Kaliszewski).
(PR#2408 and PR#2414 by Jan Kaliszewski).

### Development
- Makefile: Add codespell and test commands (PR#2425 by Sebastian Wagner).
Expand All @@ -36,11 +40,16 @@

### Bots
#### Collectors
- `intelmq.bots.collectors.stomp.collector` (PR#2408 by Jan Kaliszewski):
- Add support for authentication based on STOMP login and passcode,
introducing 3 new configuration parameters (see above: *Configuration*).
- `intelmq.bots.collectors.stomp.collector` (PR#2408 and PR#2414 by Jan Kaliszewski):
- Drop support for versions of `stomp.py` older than `4.1.12`.
- Update the code to support new versions of `stomp.py`, including the latest (`8.1.0`);
fixes [#2342](https://github.com/certtools/intelmq/issues/2342).
- Add support for authentication based on STOMP login and passcode, introducing three
new configuration parameters (see above: *Configuration*).
- Add support for loading the system’s default CA certificates, as an alternative to
specifying the CA certificate(s) file path explicitly (see above: *Configuration*).
- Fix (by carefully targeted monkey patching) certain security problems caused by
SSL-related weaknesses that some versions of `stomp.py` suffer from.
- Fix the reconnection behavior: do not attempt to reconnect after `shutdown`. Also,
never attempt to reconnect if the version of `stomp.py` is older than `4.1.21` (it
did not work properly anyway).
Expand All @@ -59,27 +68,35 @@
file permissions on socket file, if it is in use.

#### Outputs
- `intelmq.bots.outputs.stomp.output` (PR#2408 by Jan Kaliszewski):
- Add support for authentication based on STOMP login and passcode,
introducing 3 new configuration parameters (see above: *Configuration*).
- `intelmq.bots.outputs.stomp.output` (PR#2408 and PR#2414 by Jan Kaliszewski):
- Drop support for versions of `stomp.py` older than `4.1.12`.
- Update the code to support new versions of `stomp.py`, including the latest (`8.1.0`).
- Add support for authentication based on STOMP login and passcode, introducing three
new configuration parameters (see above: *Configuration*).
- Add support for loading the system’s default CA certificates, as an alternative to
specifying the CA certificate(s) file path explicitly (see above: *Configuration*).
- Fix (by carefully targeted monkey patching) certain security problems caused by
SSL-related weaknesses that some versions of `stomp.py` suffer from.
- Fix `AttributeError` caused by attempts to get unset attributes of `StompOutputBot`
(`ssl_ca_cert` et consortes).
- Add coercion of the `port` config parameter to `int`.
- Add implementation of the `check` hook (verifying, in particular, accessibility
of necessary file(s)).
- Add `stomp.py` version check (raise `MissingDependencyError` if not `>=4.1.8`).
- Add `stomp.py` version check (raise `MissingDependencyError` if not `>=4.1.12`).
- Minor fixes/improvements and some refactoring (see also above: *Core*...).

### Documentation
- Add a readthedocs configuration file to fix the build fail (PR#2403 by Sebastian Wagner).
- Add a guide of developing extensions packages (PR#2413 by Kamil Mankowski)
- Update/fix/improve the stuff related to the STOMP bots and integration with the *n6*'s
Stream API (PR#2408 by Jan Kaliszewski).
Stream API (PR#2408 and PR#2414 by Jan Kaliszewski).
- Complete documentation overhaul. Change to markdown format. Uses the mkdocs-material (PR#2419 by Filip Pokorný).

### Packaging
- Add `pendulum` to suggested packages, as it is required for the sieve bot (PR#2424 by Sebastian Wagner).
- `debian/control`: in `Suggests` field, replace ``python3-stomp.py (>= 4.1.9)`` with
``python3-stomp (>= 4.1.12)``, i.e., fix the package name by removing the `.py`
suffix and bump the minimum version to `4.1.12` (PR#2414 by Jan Kaliszewski).

### Tests

Expand Down
2 changes: 1 addition & 1 deletion debian/control
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,7 @@ Suggests: python3-geoip2 (>= 2.2.0),
python3-pyasn (>= 1.5.0),
python3-pymongo (>= 2.7.1),
python3-sleekxmpp (>= 1.3.1),
python3-stomp.py (>= 4.1.9),
python3-stomp (>= 4.1.12),
python3-pendulum
Description: Solution for IT security teams for collecting and processing security feeds
IntelMQ is a solution for IT security teams (CERTs, CSIRTs, abuse
Expand Down
123 changes: 80 additions & 43 deletions docs/user/bots.md
Original file line number Diff line number Diff line change
Expand Up @@ -1184,35 +1184,49 @@ Install the `stomp.py` library from PyPI:
pip3 install -r intelmq/bots/collectors/stomp/REQUIREMENTS.txt
```

Alternatively, you may want to install it using your OS's native
packaging tools, e.g.:

```bash
apt install python3-stomp
```

Apart from that, depending on what STOMP server you connect to, you may
need to obtain, from the organization or company owning the server, one
or more of the following security/authentication-related resources:

* CA certificate file;
* either: *client certificate* and *client certificate's key* files,
or: *username* (STOMP *login*) and *password* (STOMP *passcode*).

Also, you will need to know an appropriate STOMP *destination* (aka
*exchange point*), e.g. `/exchange/my.example.org/*.*.*.*`.

**Parameters (also expects [feed parameters](#feed-parameters)):**

**`server`**

(required, string) Hostname of the STOMP server.
(required, string) STOMP server's hostname or IP, e.g. "n6stream.cert.pl" (which is default)

**`port`**

(optional, integer) Defaults to 61614.
(optional, integer) STOMP server's port number (default: 61614)

**`exchange`**

(required, string) STOMP *destination* to subscribe to, e.g. "/exchange/my.org/*.*.*.*"
(required, string) STOMP *destination* to subscribe to, e.g. `"/exchange/my.org/*.*.*.*"`

**`username`**

(optional, string) Username to use.
**`heartbeat`**

**`password`**

(optional, string) Password to use.
(optional, integer) default: 6000

**`ssl_ca_certificate`**

(optional, string) Path to trusted CA certificate.
(optional, string) Path to CA file, or empty string to load system's default CA certificates

**`auth_by_ssl_client_certificate`**

(optional, boolean) Whether to authenticate using TLS certificate. (Set to false for new *n6* auth.) Defaults to true.
(optional, boolean) Default: true (note: false is needed for new *n6* auth)

**`ssl_client_certificate`**

Expand All @@ -1222,6 +1236,14 @@ pip3 install -r intelmq/bots/collectors/stomp/REQUIREMENTS.txt

(optional, string) Path to client private key to use for TLS connections.

**`username`**

(optional, string) Username to use.

**`password`**

(optional, string) Password to use.

---

### Twitter (REMOVE?) <div id="intelmq.bots.collectors.twitter.collector_twitter" />
Expand Down Expand Up @@ -5135,72 +5157,87 @@ This bot pushes data to any STOMP stream. STOMP stands for Streaming Text Orient
**Requirements**
Install the stomp.py library, e.g. [apt install python3-stomp.py] or [pip install stomp.py].
Install the `stomp.py` library from PyPI:
You need a CA certificate, client certificate and key file from the organization / server you are connecting to. Also
you will need a so called "exchange point".
```bash
pip3 install -r intelmq/bots/outputs/stomp/REQUIREMENTS.txt
```

**Parameters:**
Alternatively, you may want to install it using your OS's native
packaging tools, e.g.:

**`exchange`**
```bash
apt install python3-stomp
```

(optional, string) The exchange to push to. Defaults to `/exchange/_push`.
Apart from that, depending on what STOMP server you connect to, you may
need to obtain, from the organization or company owning the server, one
or more of the following security/authentication-related resources:

**`username`**
* CA certificate file;
* either: *client certificate* and *client certificate's key* files,
or: *username* (STOMP *login*) and *password* (STOMP *passcode*).

(optional, string) Username to use.
Also, you will need to know an appropriate STOMP *destination* (aka
*exchange point*), e.g. `/exchange/_push`.

**`password`**
**Parameters:**

(optional, string) Password to use.
**`server`**

**`ssl_ca_certificate`**
(optional, string) STOMP server's hostname or IP, e.g. "n6stream.cert.pl" or "127.0.0.1" (which is default)

(optional, string) Path to trusted CA certificate.
**`port`**

**`auth_by_ssl_client_certificate`**
(optional, integer) STOMP server's port number (default: 61614)

(optional, boolean) Whether to authenticate using TLS certificate. (Set to false for new *n6* auth.) Defaults to true.
**`exchange`**

(optional, string) STOMP *destination* to push at, e.g. ``"/exchange/_push"`` (which is default)

**`heartbeat`**

(optional, integer) Defaults to 60000.

**`message_hierarchical_output`**
**`ssl_ca_certificate`**

(optional, boolean) Defaults to false.
(optional, string) path to CA file, or empty string to load system's default CA certificates

**`message_jsondict_as_string`**
**`auth_by_ssl_client_certificate`**

(optional, boolean) Defaults to false.
(optional, boolean) default: true (note: false is needed for new *n6* auth)

**`message_with_type`**
**`ssl_client_certificate`**

(optional, boolean) Defaults to false.
(optional, string) Path to client certificate to use for TLS connections.

**`port`**
**`ssl_client_certificate_key`**

(optional, integer) Defaults to 61614.
(optional, string) Path to client private key to use for TLS connections.

**`server`**
**`username`**

(optional, string) Hostname of the STOMP server.
(optional, string) STOMP *login* (e.g., *n6* user login), used only if `auth_by_ssl_client_certificate` is false

**`single_key`**
**`password`**

(optional, string) Output only a single specified key. In case of `raw` key the data is base64 decoded. Defaults to null (output the whole message).
(optional, string) STOMP *passcode* (e.g., *n6* user API key), used only if `auth_by_ssl_client_certificate` is false

**`ssl_ca_certificate`**
**`message_hierarchical_output`**

(optional, string) Path to trusted CA certificate.
(optional, boolean) Defaults to false.

**`ssl_client_certificate`**
**`message_jsondict_as_string`**

(optional, string) Path to client certificate to use for TLS connections.
(optional, boolean) Defaults to false.

**`ssl_client_certificate_key`**
**`message_with_type`**

(optional, string) Path to client private key to use for TLS connections.
(optional, boolean) Defaults to false.

**`single_key`**

(optional, string) Output only a single specified key. In case of `raw` key the data is base64 decoded. Defaults to null (output the whole message).

---

Expand Down
16 changes: 8 additions & 8 deletions docs/user/feeds.md
Original file line number Diff line number Diff line change
Expand Up @@ -719,15 +719,15 @@ parameters:
### N6 Stomp Stream
N6 Collector - CERT.pl's N6 Collector - N6 feed via STOMP interface. Note that rate_limit does not apply for this bot as it is waiting for messages on a stream.
N6 Collector - CERT.pl's *n6* Stream API feed (via STOMP interface). Note that 'rate_limit' does not apply to this bot, as it is waiting for messages on a stream.
**Public:** no
**Revision:** 2023-09-23
**Revision:** 2023-10-08
**Documentation:** <https://n6.readthedocs.io/usage/streamapi/>
**Additional Information:** Contact cert.pl to get access to the feed.
**Additional Information:** Contact CERT.pl to get access to the feed. Note that the configuration parameter values suggested here are suitable for the new *n6* Stream API variant (with authentication based on 'username' and 'password'); for this variant, typically you can leave the 'ssl_ca_certificate' parameter's value empty - then the system's default CA certificates will be used; however, if that does not work, you need to set 'ssl_ca_certificate' to the path to a file containing CA certificates eligible to verify "*.cert.pl" server certificates (to be found among the publicly available CA certs distributed with modern web browsers/OSes). Also, note that the 'server' parameter's value (for the *new API variant*) suggested here, "n6stream-new.cert.pl", is a temporary domain; ultimately, it will be changed back to "stream.cert.pl". When it comes to the *old API variant* (turned off in November 2023!), you need to have the 'server' parameter set to the name "n6stream.cert.pl", 'auth_by_ssl_client_certificate' set to true, 'ssl_ca_certificate' set to the path to a file containing the *n6*'s legacy self-signed CA certificate (which is stored in file "intelmq/bots/collectors/stomp/ca.pem"), and the parameters 'ssl_client_certificate' and 'ssl_client_certificate_key' set to the paths to your-*n6*-client-specific certificate and key files (note that the 'username' and 'password' parameters are then irrelevant and can be omitted).
**Collector configuration**
Expand All @@ -736,14 +736,14 @@ N6 Collector - CERT.pl's N6 Collector - N6 feed via STOMP interface. Note that r
module: intelmq.bots.collectors.stomp.collector
parameters:
auth_by_ssl_client_certificate: False
exchange: {insert your exchange point as given by CERT.pl}
exchange: {insert your STOMP *destination* to subscribe to, as given by CERT.pl, e.g. /exchange/my.example.org/*.*.*.*}
name: N6 Stomp Stream
password: {insert n6 user's API key}
password: {insert your *n6* API key}
port: 61614
provider: CERT.PL
server: n6stream.cert.pl
ssl_ca_certificate: {insert path to CA file for CERT.pl's n6}
username: {insert n6 user's login}
server: n6stream-new.cert.pl
ssl_ca_certificate:
username: {insert your *n6* login, e.g. [email protected]}
```
**Parser configuration**
Expand Down
2 changes: 1 addition & 1 deletion intelmq/bots/collectors/stomp/REQUIREMENTS.txt
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# SPDX-FileCopyrightText: 2017 Sebastian Wagner
# SPDX-License-Identifier: AGPL-3.0-or-later

stomp.py>=4.1.8
stomp.py>=4.1.12
44 changes: 32 additions & 12 deletions intelmq/bots/collectors/stomp/collector.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,15 +4,19 @@

# -*- coding: utf-8 -*-

from intelmq.lib.bot import CollectorBot
from intelmq.lib.mixins import StompMixin

try:
import stomp
import stomp.exception
except ImportError:
stomp = None
else:
import stomp.exception

from intelmq.lib.bot import CollectorBot
from intelmq.lib.mixins import StompMixin


if stomp is not None:

class StompListener(stomp.PrintingListener):
"""
the stomp listener gets called asynchronously for
Expand Down Expand Up @@ -74,17 +78,33 @@ def connect_and_subscribe(conn, logger, destination, start=False, connect_kwargs
class StompCollectorBot(CollectorBot, StompMixin):
"""Collect data from a STOMP Interface"""
""" main class for the STOMP protocol collector """
exchange: str = ''

server: str = 'n6stream.cert.pl'
port: int = 61614
server: str = "n6stream.cert.pl"
auth_by_ssl_client_certificate: bool = True
username: str = 'guest' # ignored if `auth_by_ssl_client_certificate` is true
password: str = 'guest' # ignored if `auth_by_ssl_client_certificate` is true
ssl_ca_certificate: str = 'ca.pem' # TODO pathlib.Path
ssl_client_certificate: str = 'client.pem' # TODO pathlib.Path
ssl_client_certificate_key: str = 'client.key' # TODO pathlib.Path
exchange: str = ''
heartbeat: int = 6000

# Note: the `ssl_ca_certificate` configuration parameter must be set:
# * *either* to the server's CA certificate(s) file path,
# * *or* to an empty string -- dictating that the SSL tools employed
# by the `stomp.py`'s machinery will attempt to load the system’s
# default CA certificates.
# The latter, if applicable, is more convenient -- by avoiding the
# need to manually update the CA certificate(s) file.
ssl_ca_certificate: str = 'ca.pem' # <- TODO: change to '' (+ remove "ca.pem*" legacy files)
# (^ TODO: could also be pathlib.Path)

auth_by_ssl_client_certificate: bool = True

# Used if `auth_by_ssl_client_certificate` is true (otherwise ignored):
ssl_client_certificate: str = 'client.pem' # (cert file path)
ssl_client_certificate_key: str = 'client.key' # (cert's key file path)
# (^ TODO: could also be pathlib.Path)

# Used if `auth_by_ssl_client_certificate` is false (otherwise ignored):
username: str = 'guest' # (STOMP auth *login*)
password: str = 'guest' # (STOMP auth *passcode*)

_collector_empty_process: bool = True
__conn = False # define here so shutdown method can check for it

Expand Down
Loading

0 comments on commit d14611c

Please sign in to comment.