The purpose of detnetctl is to coordinate different applications requiring real-time communication (in the sense of TSN or DetNet) running on the same Linux system. The main focus is to avoid interference between different networking applications, even if they can not be fully trusted to be cooperative. For example, this prevents the situation that two applications send to the same TSN stream due to misconfiguration, bugs or security issues and thus might sent their traffic in the same time slot leading to missed deadlines.
In its current status, this software should be classified as demonstrator or research prototype intended for collecting experience with the requirements. For feedback or if you have a related productive use case, please contact Linutronix.
The detnetctl software is split up into features that can be individually enabled to make it possible to try it out without the full set of dependencies.
The features are introduced one by one below, but you should be able to mix and match by adapting the respective cargo build commands.
- Oneshot Dry-Run (Minimal Feature Set) - using
--oneshot - D-Bus Interface - requires
dbusfeature, preferred over oneshot - Interface setup - requires
iproute2feature, skip at runtime via--no-interface-setup - eBPF Dispatcher - requires
bpffeature, skip at runtime via--no-dispatcher - PTP Configuration and Status - requires
ptpfeature, skip at runtime via--no-ptp-config - TAPRIO Queue setup - requires
iproute2feature, skip at runtime via--no-queue-setup - Configuration with sysrepo (YANG/NETCONF) - requires
sysrepofeature, if requested with--sysrepo
detnetctl itself (i.e. everything under the src directory) is published under the terms of the GNU General Public License v3.0 or later. We are happy about every contribution to the main repository to improve detnetctl for everyone!
However, just interfacing with detnetctl via the D-Bus interface does not count as a derived work, so there is no need to publish your proprietary application that just uses detnetctl. In order to support that, the examples are licensed under the terms of the "BSD Zero Clause License" and thus can be easily used as starting point for your integration. We still encourage you to share examples and use cases with us if they could be interesting for the community. However, pay attention that the examples focus on clarity so you need to extend the code to match the requirements for error handling and resilience of your application.
Please note that detnetctl has several dependencies with their own licenses. The cargo-license tool can give you an overview. Also note that config/yang/schemas is a Git submodule referring to the Github YANG collection and for using its YANG models you need to conform to their respective licenses.
A TSN/DetNet Node Controller with Interference Protection
Usage: detnetctl [OPTIONS] [FILE]
Arguments:
[FILE] YAML configuration file. Mandatory if --sysrepo is not provided. If both is provided, configuration of
file and sysrepo is merged
Options:
-o, --oneshot Oneshot setup, i.e. do not spawn D-Bus service
--no-queue-setup Skip queue setup
--no-dispatcher Skip installing eBPFs - no interference protection!
--bpf-debug-output Print eBPF debug output to kernel tracing
--no-interface-setup Skip setting up the link
--no-ptp-config Skip PTP configuration
-s, --sysrepo Load Sysrepo configuration
-h, --help Print help
-V, --version Print versionOnly performs a one-shot dry-run reading the configuration from a YAML file.
- Install Rust
- Start the build in the detnetctl directory
cargo build --no-default-featuresIn the detnetctl directory run the following command
./target/debug/detnetctl --no-queue-setup --no-dispatcher --no-interface-setup --no-ptp-config --oneshot config/yaml/example.ymlThis will only read the configurations from the configuration file, performs a dry run setup as well as pretending to installing protection for app0 and prints out for example the following output:
Setup of DetNet system
Fetched from configuration module: {
"enp86s0": Interface {
schedule: Some(
Schedule {
number_of_traffic_classes: Some(
4,
),
priority_map: Some(
{
0: 0,
1: 0,
2: 1,
3: 1,
4: 2,
5: 2,
6: 3,
7: 3,
},
),
basetime_ns: Some(
0,
),
control_list: Some(
[
GateControlEntry {
operation: Some(
SetGates,
),
time_interval_ns: Some(
99040,
),
traffic_classes: Some(
[
3,
],
),
},
GateControlEntry {
operation: Some(
SetGates,
),
time_interval_ns: Some(
960,
),
traffic_classes: Some(
[
2,
],
),
},
],
),
},
),
taprio: Some(
TaprioConfig {
mode: Some(
FullOffload,
),
clock: None,
txtime_delay: None,
queues: Some(
[
QueueMapping {
count: 1,
offset: 0,
},
QueueMapping {
count: 1,
offset: 1,
},
QueueMapping {
count: 1,
offset: 2,
},
QueueMapping {
count: 1,
offset: 3,
},
],
),
},
),
pcp_encoding: Some(
PcpEncodingTable {
map: Some(
{
0: 0,
1: 1,
2: 2,
3: 3,
4: 4,
5: 5,
6: 6,
7: 7,
},
),
},
),
ip_addresses: None,
mac_address: None,
},
"enp86s0.5": Interface {
schedule: None,
taprio: None,
pcp_encoding: Some(
PcpEncodingTable {
map: Some(
{
0: 0,
1: 1,
2: 2,
3: 3,
4: 4,
5: 5,
6: 6,
7: 7,
},
),
},
),
ip_addresses: Some(
[
(
10.5.1.1,
24,
),
],
),
mac_address: None,
},
} {
"app0": AppConfig {
bind_interface: Some(
"enp86s0.5",
),
physical_interface: Some(
"enp86s0",
),
stream: Some(
StreamIdentification {
destination_address: Some(
MacAddress("48:21:0b:56:db:da"),
),
vid: Some(
5,
),
},
),
cgroup: Some(
"/user.slice/",
),
priority: Some(
7,
),
},
"app1": AppConfig {
bind_interface: Some(
"enp86s0.3",
),
physical_interface: Some(
"enp86s0",
),
stream: Some(
StreamIdentification {
destination_address: Some(
MacAddress("48:21:0b:56:db:da"),
),
vid: Some(
3,
),
},
),
cgroup: None,
priority: Some(
5,
),
},
}
Added 10.5.1.1/24 to enp86s0.5
Interface enp86s0 down
Queues set up
Dispatcher installed for stream StreamIdentification {
destination_address: Some(
MacAddress("48:21:0b:56:db:da"),
),
vid: Some(
5,
),
} with priority 7 on enp86s0
with protection for cgroup "/user.slice/"
VLAN interface enp86s0.5 properly configured
Dispatcher installed for stream StreamIdentification {
destination_address: Some(
MacAddress("48:21:0b:56:db:da"),
),
vid: Some(
3,
),
} with priority 5 on enp86s0
VLAN interface enp86s0.3 properly configured
Interface enp86s0 up
Interface enp86s0.5 up
Interface enp86s0.3 up
Finished after 883.6µsInstead of using the oneshot mode, you usually want to run detnetctl as daemon to dynamically react to new requests. In the following, it is shown how to use the D-Bus interface to send protect requests (even if they will have no effect until the eBPF Dispatcher is also used). This can be done by the application itself (for the interface description see [facade]) or via the detnetctl-run tool. For the former, the application needs to take care to put itself in an adequate cgroup (see eBPF Dispatcher), while the detnetctl-run tool requires a running systemd user session.
- Make sure you have D-Bus and systemd running on your system
- Make yourself familiar with the D-Bus policy in
config/dbus/detnetctl.confand change the policy user for the application to your username. Finally install it with
sudo cp config/dbus/detnetctl.conf /etc/dbus-1/system.d/- Install the build dependencies for D-Bus applications, e.g.
sudo apt install libdbus-1-dev libdbus-1-3 build-essential pkg-config- Build detnetctl
cargo build --no-default-features --features dbus- Build the example application
sudo apt install libxdp-dev
SETCAPS=1 make -C examplesThe SETCAPS sets the required capabilities and for that calls sudo setcap, so you might get a password prompt.
Copy and adapt the configuration file according to your preference, especially the logical interface needs to be bindable from the application and should be able to reach the hostname you specify below. A minimal configuration file without VLAN and TSN settings would look like this:
version: 0.8.0
unbridged_apps:
app0:
bind_interface: enp86s0
physical_interface: enp86s0
stream:
vid: null
interfaces:
enp86s0:
schedule: nullStart the service with
sudo ./target/debug/detnetctl --no-queue-setup --no-dispatcher --no-interface-setup myconfig.ymlsudo is required here, since the D-Bus policy above only allows root to own org.detnet.detnetctl. You can adapt the policy accordingly if you like.
Then in a second terminal start the sample application with
./target/debug/detnetctl-run app0 ./examples/simple/simple example.orgHere app0 references the application in the configuration and everything else is the command and its arguments that will be started by detnetctl-run.
Up to now the transmission took place directly via the physical interface. Now, change the logical interface in the configuration to a VLAN interface (e.g. enp86s0.5) and set the VLAN ID (5 in this case). The VLAN interface will be automatically added by detnetctl.
Adapt the configuration to include the interface configuration, e.g.
version: 0.8.0
unbridged_apps:
app0:
bind_interface: enp86s0.5
physical_interface: enp86s0
stream:
vid: 5
interfaces:
enp86s0:
schedule: null
enp86s0.5:
ip_addresses: [[10.5.1.1, 24]]In order to run the simple example, you also need to make sure that a webserver can be reached via this VLAN. For this, you can configure a VLAN interface on a second computer (referred to with hostname webserver and IP address 10.5.1.2 in the following) with e.g.
webserver:~$ sudo ip link add link enp86s0 name enp86s0.5 type vlan id 5
webserver:~$ sudo ip address add 10.5.1.2/24 dev enp86s0.5
webserver:~$ sudo ip link set dev enp86s0.5 upand install a webserver (like lighttpd) or just run one temporarily with
webserver:~$ sudo python3 -m http.server 80Again at the first computer start the build:
cargo build --no-default-features --features dbus,iproute2Start the service with
sudo ./target/debug/detnetctl --no-queue-setup --no-dispatcher myconfig.yml And in a second terminal
./target/debug/detnetctl-run app0 ./examples/simple/simple 10.5.1.2The sample application will now send over the VLAN. However, if you now start a second application with
./examples/simple/simple 10.5.1.2it will happily connect, too. That is normal for general networking, but in DetNet/TSN context this implies that two applications generate traffic for the same DetNet flow or TSN stream respectively. Since the realtime guarantees for a DetNet flow or TSN stream are always only provided for a given amount of traffic, this can be very problematic and thus the access will be restricted in the next section.
Installs an eBPF at tc egress so that after an application has registered, only the application(s) in a certain cgroup can transmit for the given TSN stream. The respective cgroup will be provided during protection by the caller, so it is its responsibility to make sure that all applications within the given cgroup are permitted to send to the TSN stream. This usually means that the relevant application is isolated in its own cgroup (similar to isolating applications in cgroups for controlling CPU and memory utilization).
How the cgroups are managed is system-dependent. Today, this is usually the responsibility of a dedicated service like systemd (reasons for this approach and how systemd handles it), but there are other options like the (unfavored) option for direct interfacing with the kernel cgroup interface. For this documentation, we assume systemd is used, but detnetctl itself has no dependency on systemd (except for the detnetctl-run tool) and can be used with other means of managing cgroups.
To properly identify the TSN stream in the dispatcher and to set the PCP, we now also need to add the destination MAC address and the PCP to the configuration:
version: 0.8.0
unbridged_apps:
app0:
bind_interface: enp86s0.5
physical_interface: enp86s0
stream:
vid: 5
destination_address: 48:21:0b:56:db:da
interfaces:
enp86s0:
schedule: null
enp86s0.5:
ip_addresses: [[10.5.1.1, 24]]- Install the build dependencies for eBPF applications, i.e.
sudo apt install libelf-dev clang- Build detnetctl
cargo build --no-default-features --features dbus,iproute2,bpfStart the service with
sudo ./target/debug/detnetctl --no-queue-setup --bpf-debug-output myconfig.ymlThen in a second terminal start the sample application with
./target/debug/detnetctl-run app0 ./examples/simple/simple 10.5.1.2and start a second sample application with
./examples/simple/simple 10.5.1.2While the first application should happily connect, the second application should be blocked, that is it will not be able to establish a connection since all its traffic gets dropped. You can monitor the filter with
sudo cat /sys/kernel/debug/tracing/trace_pipe- Install
linuxptp, configure and runptp4landphc2sys, either from your packet repository or from source as described at https://tsn.readthedocs.io/timesync.html. - Build detnetctl
cargo build --no-default-features --features dbus,iproute2,bpf,ptpAdapt the configuration according to your needs. For the YAML file, the relevant section is ptp as shown below that should be added to your myconfig.yml. For YANG (see below) the relevant section is ieee1588-ptp-tt:ptp. There can be multiple PTP instances in the configuration file that will be selected by the active_instance parameter. While the instances themselves can also be configured via YANG (see below), the active_instance parameter is only available in the YAML file that can be supplied at the same time as loading the YANG file. If it is not provided, no configuration will be applied, but the PTP status can still be requested.
...
ptp:
active_instance: 1
instances:
1:
clock_class: 248
clock_accuracy: 0x31
offset_scaled_log_variance: 65535
current_utc_offset: 37
current_utc_offset_valid: true
leap59: false
leap61: false
time_traceable: true
frequency_traceable: false
ptp_timescale: true
time_source: 0xA0
gptp_profile: trueThe most important setting that ensures the configuration is applied correctly is if the gPTP profile (IEEE 802.1AS) is used. If you are unsure, have a look at the transportSpecific field of the ptp4l configuration and the --transportSpecific argument of phc2sys. If it is 1, you should set gptp_profile in the YAML file to true and in the YANG file the sdo-id to 256 (i.e. 0x100). Otherwise, set it to false and 0, respectively.
Then start detnetctl for YAML configuration as
sudo ./target/debug/detnetctl --no-queue-setup myconfig.ymlAt the start, the settings will be sent to ptp4l/phc2sys. It might take up to 1 minute until they are fully applied.
Since the PTP status depends on the interface to use, provide the VLAN (!) interface to the application like
./target/debug/detnetctl-run app0 ./examples/simple/simple 10.5.1.2 enp86s0.5You should see the PTP status printed every few seconds.
In order to actually distribute the TSN streams into different timeslots according to Enhancements for Scheduled Traffic (EST) aka IEEE 802.1Qbv, detnetctl can set up the queues / qdiscs according to the configuration to enable TSN communication using TAPRIO Qdiscs.
You usually want to use a network card with hardware support for TAPRIO (e.g. Intel® Ethernet Controller I225-LM), but the is also a software offload option available.
In order to avoid interference, the configuration needs to be set up in a way that traffic that could disturb each other should not end up in the same queue. If nothing else is specified for an app, its traffic gets assigned to the best-effort priority 0 shared with many other traffic sources.
The following mapping is the default one applicable to the I225 NIC that has four hardware queues, i.e. two priorities share one queue, respectively:
| Priority | Traffic Class | Queue |
|---|---|---|
| 0 | 0 | 0 |
| 1 | ||
| 2 | 1 | 1 |
| 3 | ||
| 4 | 2 | 2 |
| 5 | ||
| 6 | 3 | 3 |
| 7 |
In the default, there is a 1:1 mapping between queue and traffic class, so the number of traffic classes corresponds to the number of queues, but it can be changed e.g. like this
interfaces:
enp86s0:
schedule:
number_of_traffic_classes: 2
taprio:
queues:
- count: 2
offset: 0
- count: 2
offset: 2The mapping between priorities and traffic classes corresponds to the default mapping according to IEEE 802.1Q-2022 corresponding to the configured number_of_traffic_classes. But also that can be changed like
interfaces:
enp86s0:
schedule:
number_of_traffic_classes: 2
priority_map:
0: 0
1: 0
2: 0
3: 1
4: 0
5: 0
6: 1
7: 0Apart from the priority, the schedule needs to be configured, like
version: 0.8.0
apps:
app0:
bind_interface: enp86s0.5
physical_interface: enp86s0
stream:
vid: 5
destination_address: 48:21:0b:56:db:da
priority: 2
interfaces:
enp86s0:
schedule:
number_of_traffic_classes: 4
control_list:
- time_interval_ns: 5000
traffic_classes: [0]
- time_interval_ns: 5000
traffic_classes: [1]
enp86s0.5:
ip_addresses: [[10.5.1.1, 24]]There are several other configuration options:
The basetime_ns option (default 0) of the schedule can be used to align the schedule in time (in nanoseconds relative to Unix epoch). This is relevant when aligning the schedules from several nodes (common in real-world deployments, but often unnecessary for smaller demos).
The mode option of taprio has FullOffload as default, but Software can be used without hardware support and TxTimeAssist to emulate TAPRIO in software, but send the packets with TX timestamps to the NIC, which requires NIC support and the txtime_delay option. Also clock is to be set (with the options Tai, Realtime, Monotonic and Boottime) unless full offload is used.
cargo build --no-default-features --features dbus,iproute2,bpf,ptpsudo ./target/debug/detnetctl myconfig.ymlFor the simple example, there should be no noticable difference when now transmitting via the TAPRIO Qdisc. For a more complex example that tracks the timestamps have a look at the timestamp example.
When changing the TAPRIO qdisc after one was already installed, you might often get the following error
Error: Setting up the queue failed
Caused by:
UnknownErrno: Unknown errno (524) Changing the traffic mapping of a running schedule is not supportedThis is expected behavior, because the TAPRIO qdisc is protected against certain on-the-fly changes that might lead to packet drops. To reset it manually (and accept potential drops), you can for example execute
sudo tc qdisc replace dev enp86s0 root pfifo_fastInstead of using a YAML file, the configuration can also be made via sysrepo that uses YANG data models and can be configured via NETCONF.
- Install sysrepo and its dependencies. You might need to build from source, because most available packages are too old. It was successfully tested with the following versions:
libsysrepo-dev: 2.2.36
libsysrepo7: 2.2.36
sysrepo-tools: 2.2.36
libyang2: 2.1.30.1
libyang2-dev: 2.1.30.1
libyang2-tools: 2.1.30.1
netopeer2: 2.1.49
libnetconf2-3: 2.1.28- Clone submodule to get YANG schema definitions (
--recursiveis not necessary, no modules from recursive submodules are used at the moment and they are huge).
git submodule update --init- Build detnetctl
cargo build --no-default-features --features dbus,iproute2,bpf,ptp,sysrepoor equivalent
cargo buildLoad the YANG configuration from config/yang/example.json after adapting it to your needs:
sudo sysrepocfg --import=config/yang/example.jsonIn case of errors, load the missing schemas from config/yang/schemas, e.g.
sudo sysrepoctl -i config/yang/schemas/standard/ietf/RFC/[email protected]
sudo sysrepoctl -i config/yang/schemas/standard/ieee/draft/1588/ieee1588-ptp-tt.yang
sudo sysrepoctl -i config/yang/schemas/standard/ietf/RFC/[email protected]
sudo sysrepoctl -i config/yang/schemas/standard/ietf/RFC/[email protected]
sudo sysrepoctl -i config/yang/schemas/standard/ietf/RFC/[email protected]
sudo sysrepoctl -i config/yang/schemas/standard/ieee/published/802.1/ieee802-dot1q-types.yang
sudo sysrepoctl -i config/yang/ietf-detnet.yang
sudo sysrepoctl -i config/yang/schemas/standard/iana/[email protected]
sudo sysrepoctl -i config/yang/schemas/experimental/ietf-extracted-YANG-modules/[email protected] -e sub-interfaces
sudo sysrepoctl -i config/yang/schemas/standard/ietf/RFC/ietf-ip.yang
sudo sysrepoctl -i config/yang/schemas/standard/ieee/published/802.1/ieee802-dot1q-tsn-types.yang
sudo sysrepoctl -i config/yang/schemas/standard/ieee/published/802.1/ieee802-dot1cb-stream-identification-types.yang
sudo sysrepoctl -i config/yang/schemas/standard/ieee/published/802.1/ieee802-dot1cb-stream-identification.yang
sudo sysrepoctl -i config/yang/schemas/standard/ieee/published/802.1/ieee802-dot1q-bridge.yang
sudo sysrepoctl -i config/yang/ieee802-dot1q-sched.yang
sudo sysrepoctl -i config/yang/schemas/standard/ieee/published/802.1/ieee802-dot1q-sched-bridge.yangNote that an adapted version of ieee802-dot1q-sched.yang is loaded due to a potential bug in the standard: https://mailarchive.ietf.org/arch/msg/netmod/IxJ3uPRQYJgVb91fuhfhz4TDLB0/
Start detnetctl as
sudo ./target/debug/detnetctl --sysrepoas well as the application like before.