Combination of Time Aware Shaper with Frame Preemption

[PhilippComet]

Hello everybody,

my goal is to combine Time Aware Shaping with Frame Preemption. Scheduled Traffic here has higher priority than AVB Traffic. The expected behaviour should look like it is shown in figure b of this graph:

As you can see in the following picture, the queues are growing, but transmission selection does not dequeue frames.

Do you have an idea what could be the reason and what has to be modified, to get this combination running?
I'm looking forward to your responses

Philipp

[levy]

The queue module and the classifier modules are both passive modules, they do not initiate activity. There are queue submodules in the expressMacLayer and preemptableMacLayer submodules by default, I don't know if they are still there. If they are still there the then server submodules in the inner MAC layers cannot pull packets from the outer queue because of the innert queue.

BTW, this scenario has never been tried and most likely doesn't work out of the box.

[PhilippComet]

Hi Levente,
thank for your fast response. The queue submodules in the expressMacLayer and preemptableMacLayer are still there.

Ok, so do I understand it right, that in this case the preempingServer pulls packets from the queue in the expressMacLayer and preemptableMacLayer?
Am I right, that for pulling packets from the outer queue the server needs to be modified/extended or another server module has to be added?

Parallely I'm working on getting more into depth into the code to understand the mechanisms

[levy]

The servers pull packets from the directly connected modules on their input.

Perhaps you could try putting a server module right after the outer queue. This would allow the TAS to send packets to the outbound classifier according to the gate schedule.

[PhilippComet]

Hello Levente,

Thanks for the hint using the server module. Using an InstantServer module directly behind the queue module worked. But in the meantime I figuered out, that one step further to the solution should be to use the queue submodules in the sub-MAC layers in connection with a periodic gate.

Very similar to the TimeAwareShaper module the outboundClassifier can be exchanged by a "PcpTrafficClassClassifier" to realize a PCP priority dependant classification instead of using different VLAN IDs (as in the preemption showcase). So it remains only to add PeriodicGates at the queue submodules in the sub-MAC layers.

I have already tried this modification - but now I have the problem that no packets from the queue submodule of the preemptableMacLayer are dequeued. If I leave the PeriodicGate constantly open, frame preemption works.

My guess is, that the preempting server schedules a timer event at point in time when the transmission of an express packet is expected to be ended to trigger the resumption of the preempted packet. But the PeriodicGate does allow a transmission only at a later point in time - and so the incoming timer event has no effect.

Do you have any idea what could be the cause of the problem?
Please let me know when I can provide you more information.

[levy]

Without having the example I cannot tell what is the cause of this behavior. Let's add a frame preemption combined with time-aware shaper showcase to the repository. Could you please send what you have right now? I'd like to create a new branch for this example and see if we can make it work.

[PhilippComet]

Thank you for your offered help!
here you can find the files that I modified to get to the point I showed above:

tasandfp_modifications.zip (updated 6:28pm GMT+2)

simpleTestNetwork/omnet.ini : To cause frame preemption I configured two apps sending packets with a temporal distribution as shown in app_sending_intervals.jpg

TSNDevice.ned and GatedQueue.ned : I introduced a new module called "GatedQueue" that simply combines a PacketQueue with a PeriodicGate. Changes in TsnDevice still have to be made to consider all existing configurations.

EthernetPreemptingMacLayer.ned : removal of the queue sub module to use queue submodules in sub-MAC layers and exchanging outboundClassifier to classify according to PCP

edit: I'm still working on comprehension of all the mechanisms, but my first guess is that the preemptableServer does not start streaming from the preemptableMacLayer, because it has no knowledge about when gates open/close. I could also imagine that using packet based PeriodicGates in connection with the streaming based preemptingServer could be disfunctional.

[levy]

Thanks for the source files. I noticed that you have changed TsnDevice and EthernetPreemptingMacLayer. I think we can make this work with minor changes and move the rest to the INI file. I will create a new branch in the INET repository called topic/fpandtas that will contain an example in showcases/tsn/combiningfeatures/fpandtas. I replaced the network in the INI file to use the default TsnLinearNetwork that contains a client/server and switch. I assume you only want to have FP and TAS work together in the switch, right?

[PhilippComet]

Thank you for the update. In the long term I would like to also insert the CBS in the queue submodule of the preemtableMacLayer. (Maybe you saw the uncommented part in the .ini file).

In the meantime I had a successful test run with the above shown mods, but only with no guard bands active. I think we have to additionally modify the guardband mechanism. Because now with frame preemption the guardband duration reduces to a new minimum (what can also be seen in the image in the initial post).

Btw, the reason why the AVB packets weren't dequeued in the post above was, that I had a typo in the .ini file. I missspelled the path for the parameter for the implicit guardband and so the default value "true" was set and the guardband has blocked the forwarding. I added some code in PeriodicGate.cc to constantly assume 1500Bytes as guardband before gate closing. So my 100us AVB open cycles were to short to allow transmission.

[levy]

This sequence chart shows almost what you wanted. The implicit guard band is disabled, so the gate state of the express mac layer periodic gate is somewhat misleading, especially in the switch. Nevertheless, the frame preemption happens in the client right at the moment the gate opens.

[PhilippComet]

Yes, unfortunately the drawback with executing the frame preemption when the express mac layer gate opens is, that the express packets will be delayed by a small amount of time. This introduces jitter to the TT-Traffic. I would like to isolate this traffic from lower priority traffic by using the guardband. In the picture the TT-0 frame should start with this modification at 100us with a period of 400us.

My solution approach to get to the point shown in the figure of the initial post would be to end streaming the preemptable traffic a little bit earlier so that the TT (Time Triggered) - Traffic can start reliable at predefined times configured by the GCL (or .transmissionGate.durations parameter)

edit: I'm not sure yet where a trigger signal for earlier stream ending could be implemented, but would perhaps the periodicGate be the right place for it? Because there we have knowledge about the gate switching?

[levy]

The preemption happens exactly at 100us when the TT frame becomes available. The 960ns interval from # 19 to # 21 is there because the preemption happens at the MAC layer and the AVB frame has to be completed with an FCS. Compared to the MAC layer, the packet on the wire contains an extra 8B PHY header, with the 4B FCS it is 96b altogether. This could be reduced to 32b if the PHY overhead would be taken into account in the PreemtableStreamer module. The other 960ns interval from # 21 to # 23 is an IFG, and this cannot be removed. So the total 1920ns could be reduced by 64ns if the implementation would be more sophisticated.

Another option would be to preempt the AVB frame such that it ends at the moment the TT gate opens. Implementing this behavior is more complicated, because there's no direct communication between the gates and the preemption mechanism.

I updated the topic/fpandtas branch and only very little changes were required for the basic approach to work.

[PhilippComet]

Thank you for the detailed explanation about the internal mechanism with the added Bytes to the preempted frame and updating the branch.

As a conclusion, I would suggest that we need to execute the preemption of the AVB frame 1920ns earlier. What do you think about the idea to activate the guardband in periodicGate.cc constantly by this amount of time before the AVB gate closing? We could subscribe the "guardBandStateChangedSignal" in the preemptingServer and call endStreaming() and cancelClockEvent() depending on it.

I already extended the periodicGate.cc code to activate the guardband by constant amount of duration before gate closing.
periodicGate_files.zip (only a draft - I have to still clean up everything)

(for your information: I updated initiallyOpen = true for TT gate and initiallyOpen = false for AVB gate and changed the productionOffset of both sending apps accordingly in the graph. That is different in the .ini file that I sent to you)

[levy]

The guard band of PeriodicGate is active if and only if the gate is open and the next packet cannot flow through the gate before it closes. This is called the implicit guard band, because there are no parameters to control the size of the guard band, other than it's possible to disable it completely. The credit-based shaper uses the guard band signal to update the credit gain rate based on whether the implicit guard band is active or not.

I'm not sure if it's a good idea to change this mechanism to achieve the preemption of a packet flowing through a periodic gate somewhat earlier than the gate is actually closing. I'm still thinking about this a bit more.

[PhilippComet]

Ok, I understand. So now if we want to use the implicit guardband reasonably with FP+TAS, canPacketFlowThrough() needs further consideration of preemption, right?

I would like very much to experiment with the approach to preempt AVB frames earlier, could you please give me a hint how it could work to get information about guardband active/inactive from the periodicGate (in the preemptableMacLayer) to the preemptingServer module?

[levy]

If you want to preempt the AVB frame earlier, then yes, you need to change the PreemptingServer module. I don't know what is the exact condition upon which you want to preempt the AVB frame. Do you want to preempt it unconditionally when the time window ends even if no higher priority frame is available? So what is the exact condition that you are trying to implement?

Currently, preemption is supposed to happen when a new higher priority frame becomes available in the scheduler. The server immediately preempts the current streaming operation and starts a new one.

Edit: Note that you can't just bring the preemption with a certain amount of time earlier, because the higher priority frame is not available yet, so the condition for preemption cannot be evaluated.

[PhilippComet]

The condition, upon which I want to preempt the AVB frame, is the AVB gate closing event, because I want the higher express frames to be completely isolated temporally. (As we know, the main advantage of TAS)

My aim is to keep reserved TT-time windows free from other traffic. Strictly speaking, these could even remain unused if the network configurator would turn off TT sending apps.

My assumption is, that there is always a higher priority express TT frame availible when the TT gate opens, because my Gate Control List and app sending intervals are externally calculated and synchronized. The goal is to be able to predict where and when TT Traffic is at which node in the network.

Thus, the condition for preemption when using FP+TAS together for my desired purposes should be only the preemptableMacLayer's gate closing event.
In several papers this is called "Fame Preemption with HOLD/RELEASE". And when considering the current implementation state until know this is called "without HOLD/RELEASE"

[levy]

Thanks for the clarification. I think you should then modify the PreemtingServer to subscribe for the gateStateChanged signal and preempt the AVB frame if the gate closed signal is from the right gate. Also, if you bring the gate close time a bit earlier, then the AVB frame will end the transmission and even the IFG period will elapse by the time the TT frame arrives and the TT gate opens.

[PhilippComet]

Thank you Levente for your suggestion and fast response. The approach with subscribing to a signal sent by the respective periodicGate worked for me already well partially as you can see in the diagram below

There pops up another aspect that I also think is important to consider. Right now the fragment of the preempted frame is sent immediately after the express frame, even when its time window only comes later:

I found this topic exactly described in the current IEEE 802.1Q Standard in Annex S.3 (FP without HOLD/RELEASE) & S.4 (FP with HOLD/RELEASE)
8021Q-2018.pdf

If there would occur another following TT frame in the TT time window this would in adding avoidable delay.

How could the preempted fragment further be delayed up to its associated time window?

[PhilippComet]

I could already solve this issue with a first approach by subscribing the gateStateChangedSignal in the preemptableStreamer.Transmission of the fragment (remainingPacket) is disabled as long as the associated gate is closed.

[levy]

In the current solution the order of modules is: queue -> gate -> streamer -> scheduler -> server. When the server pulls a packet the following happens:

• the server pulls the start of a packet from the scheduler
• the scheduler pulls the start of a packet from the highest priority streamer which can be pulled
• the streamer pulls a whole packet from the gate which must be open
• the gate pulls a whole packet from the queue

The packet ends up in the streamer which stores it and starts the streaming operation. This operation can be later interrupted by the server by pulling the end of the packet. The remaining part of the packet stays in the streamer. When the server pulls the start of a packet the next time, the streamer will start streaming the remaining stored part without asking the gate.

This operation results in the diagram you have sent earlier where the second part of the AVB frame is immediately transmitted after the TT frame, ignoring the gate closed state of the AVB gate.

[levy]

Perhaps the module hierarchy could be changed to: queue -> streamer -> gate -> scheduler -> server. In this case the remaining part of the packet in the streamer could not be pulled when the gate is closed. There are two issues with this that I see right now:

• The first issue is that what should happen when a gate closes while a streaming operation is in progress? The natural answer could be that the gate should stop the streaming operation effectively preempting the packet transmission. Unfortunately, the gate is a passive component from the server's point of view in this pull operation, so I'm not sure how to notify the server about this. Besides, this would mean that whenever the AVB gate closes, independently of the availability of other higher priority frames, the transmission of the AVB frame would be preempted.
• The second issue is that the gate is currently part of the queue module, but the streamer is not. The queue module always passes whole frames both in and out. So I guess another gate module should be optionally added after the streamer module in the EthernetFragmentingMacLayer module?

[PhilippComet]

Thank you for the clear summary of the module behaviour.

about the first issue: How about using the gateStateChangedSignal (of guardBandSTateChangedSIgnal) to notify the server to end the streaming process? I can already guess that subscribing signals is not the proper way for this, but what is the reason behind?

• I think when using TAS in combination with FP, preemption should happen dependant on gate closing to overall protect the highest order traffic even when it's possibly absent. This is also useful to protect sporadic traffic at all cost (e.g control data in an in-vehicle network, etc.)
• When using only FP without TAS the preemption should of course happen, when a higher order frame is availible (As the framepreemption showcase already shows very good)

about the second issue: Another gate module could be a solution, but is it worth to only use it for hindering the fragment from beeing transmitted?

[levy]

The problem with the gateStateChangedSignal is that the server can't be sure that the gate which is sending the signal is in the path of the current packet that is being streamed. For example, in a more complex module structure there could be multiple gates one being on the path, the other one is not. Both can send gateStateChangedSignals and the server is unable to decide when to preempt the current packet streaming operation. There should be a way to communicate this along the connections using the various queueing interfaces. This allows intermediate modules to interact with this process, etc.

Besides, there's this idea that modules should be as simple as possible, and modules should assume the least amount of knowledge about their environment. I feel like having a PreemptingServer that assumes gates connected in a certain way to queues and the scheduler, etc. greatly reduces composability, so it's kinda wrong. It's not clear where should the server subscribe for the gateStateChangedSignal and when should that signal mean that the current streaming operation should be preempted.

I don't yet know what is the right answer. I'm experimenting currently.

[levy]

Here is a simplified version of what I think right now. The parts marked with ??? is not well understood.

I'm using the following simplified network:

queueTT  -> streamerTT                -> gateTT  ->
                                                    -> scheduler -> server (preempting) -> transmitter
queueAVB -> streamerAVB (preemptable) -> gateAVB ->

AVB gate is open, AVB frame becomes available:

  queueAVB.pushPacket()
    streamerAVB.handlePullPacketChanged()
      gateAVB.handlePullPacketChanged()
        scheduler.handlePullPacketChanged()
          server.handlePullPacketChanged()
            scheduler.pullPacketStart()
              gateTT.canPullPacket()
              gateAVB.canPullPacket()
              gateAVB.pullPacketStart()
                streamerAVB.pullPacketStart()
                  queueAVB.pullPacket()
            transmitter.pushPacketStart()

AVB streaming is in progress, AVB gate closes:

  gateAVB.close()
    scheduler.handlePullPacketChanged()
      server.handlePullPacketChanged()
        scheduler.???
        scheduler.pullPacketEnd()
          gateAVB.pullPacketEnd()
            streamerAVB.pullPacketEnd()
              streamerAVB.splitStreamedPacket()
        transmitter.pushPacketEnd()

AVB streaming is in progress, TT gate opens but no TT frame is available:

  gateTT.open()
    scheduler.handlePullPacketChanged()
      server.handlePullPacketChanged()
        scheduler.???

AVB streaming is in progress, TT gate is open, TT frame becomes available:

  queueTT.pushPacket()
    streamerTT.handlePullPacketChanged()
      gateTT.handlePullPacketChanged()
        scheduler.handlePullPacketChanged()
          server.handlePullPacketChanged()
            scheduler.???
            scheduler.pullPacketEnd()
              gateAVB.pullPacketEnd()
                streamerAVB.pullPacketEnd()
                  streamerAVB.splitStreamedPacket()
            transmitter.pushPacketEnd()
            scheduler.pullPacketStart()
              gateTT.canPullPacket()
              gateTT.pullPacketStart()
                streamerTT.pullPacketStart()
                  queueTT.pullPacket()
            transmitter.pushPacketStart()

Do you have any other interesting event that should be thought of?

[PhilippComet]

Ok, I get the point with the signals. This gives a very good overview.

In my case I only considered exclusive gating, but as you showed above it is better to also consider two gates open at the same time.

Additionally in the longterm I would suggest to prepare the architecture for multiple queues within the sub mac layers (with priority scheduling). So that for example combinations like TT, AVB_A in the express MAC and AVB_B and Best-Effort Traffic in the preemptable MAC can be configured.

I'm using the following simplified network:

queueTT  -> streamerTT (preemptable)  -> gateTT  ->
                                                    -> scheduler -> server (preempting) -> transmitter
queueAVB -> streamerAVB (preemptable) -> gateAVB ->

why did you consider the TT streamer also to be preemptable?

[levy]

You are right, the TT streamer should not be preemptable.

Having multiple queues in the sub MAC layers is not a problem. A classifier with multiple subqueues and a scheduler acts like a queue (compound queue) and can be used instead of queueTT or queueAVB in the above structure. I think even traffic shapers could be used there.

[levy]

This chart looks better in terms of delaying the second AVB frame fragment:

The AVB frame is preempted by the TT frame. The second AVB fragment is sent only when the AVB gate is open. The open/closed gate states depicted on the chart belong to the TT gate.

[PhilippComet]

Having multiple queues in the sub MAC layers is not a problem. A classifier with multiple subqueues and a scheduler acts like a queue (compound queue) and can be used instead of queueTT or queueAVB in the above structure. I think even traffic shapers could be used there.

Sounds good, so I assume we can go back to the original TAS module in the queue instead of the GatingQueue.ned that was suggested. I already ran this morning a simulation under this constellation sucessfully with using an additional CBS.

[levy]

In the above latest example, the periodic gate (TAS) is not part of the queue module anymore, it is right after the streamer. That's because the preemptable streamer is the one which stores the remaining packet when the preemption occurs. This structure prevents the transmission of the remaining packet when the TAS is closed.

I have to switch to a different topic temporarily, I'll come back to this issue when I'm finished.

[PhilippComet]

Ok, thank you already for your effort.

[PhilippComet]

Hi Levente, can you please give me some advice on how to get to this graph. I guess it has to do with the pcap recording, right?

[levy]

You should add the line record-eventlog = true to your INI file. Run the simulation and open the elog file in the IDE from the results folder. Select the Preset Configuration / Network Node menu item, then Timeline Mode / Linear, then zoom out a little, then Axis Ordering Mode / Manual, then move the switch up, then Configure Style, enable Message Sends, add two lines with AVB* blue and TT* darkgreen, finally you should attach the vector data by right clicking on the axis and selecting the appropriate gateState vector, you may also need to enable Show/Hide / Show Axis Vector Data.

I hope this helps!

[levy]

The topic/fpandtas branch contains a proposal for the necessary changes for FP and TAS to work together.
Here is a summary:

• Added an optional gate submodule after the streamer submodule in Ethernet MAC layers which contain a streamer. This change allows inserting the PeriodicGate submodule after the streamer submodule when using both FP and TAS. This structure effectively prevents a remaining fragment to be sent in the closed period of the gate.
• Changed the PacketGateBase class to stop the packet streaming operation when a gate closes and a packet streaming is in progress. The gate has an invariant which says that no packet streaming is allowed if the gate is closed. This is similar to the rule which says that no packets should pass the gate if the gate is closed. So, when the gate actually closes, it also stops the current packet streaming operation by notifying its connected source (by pulling the end) and its connected sink (by pushing the end).
• Changed the PriorityScheduler class to stop the lower priority packet streaming operation if a higher priority packet becomes available. The priority scheduler has an invariant which says that no lower priority packet streaming is allowed if a higher priority packet is available. This is similar to the rule which says that the highest priority packet should come first. So, when a higher priority packet becomes available, the scheduler stops the current packet streaming operation by notifying its connected source (by pulling the end) and its connected sink (by pushing the end).
• Changed the PreemptingServer class to not stop the packet streaming operation if a handleCanPullPacketChanged notification comes. This module cannot be sure whether the current streaming operation needs to be stopped or not. It still gets notified about the new packet becoming available and starts a new streaming operation if there's no operation is in progress.

You can see an FP and TAS configuration example in showcases/combiningfeatures/fpandtas. Could you please check if you see some incorrect behavior in your example?

[PhilippComet]

Hi Levente,
thank you very much for the detailed description how to get to the visualization of the graph with the .elog recording and the summary!

I will have a closer look as fast as possible to the fpandtas showcase.

[gimmyy]

Hi, Thank you for your inspiring works and contributions. I tried the latest fpandtas example in showcases and it works really well. However, for the more realistic case specified in IEEE802.1 standards, the number of queues in expressMacLayer and preemptableMacLayer is not limited to 1. The case I want to implement is like the following figure, where TT flows can preempt the AVB_A, AVB_B, and BestEffort(BE) flows.

To implement the case, as I understand, the preemptableMacLayer should be modified into multiple queues as follows.

Here I only display two queues in the preemptableMacLayer for simplicity. If the two queues are for AVB flows in the first figure, the transmission algorithm can be configured as "CredictBasedShaper".

However, when I modified the number of queues in preemptableMacLayer using queue.numTrafficClasses=2 in the ini file, and assign pcp values to different flows based on pcpClassClassifier, one of the AVB flows goes to the queue in expressMacLayer. It seems like the multi-queue mechanism in preemptableMacLayer can not be implemented this way.

Could you please tell me how to realize multi-queue in preemptableMacLayer and also expressMacLayer? Thank you so much! Looking forward to your response.

[levy]

You are on the right track, I think this should be doable. The EthernetPreemptingMacLayer has its own classifier that decides if a packet goes to the express or the preemptable MAC layer. You need to make sure that this classifier is correctly configured. There is another classifier inside the express MAC layer that decides which queue the packet goes into. Similarly, there's another classifier inside the preemptable MAC layer. So there are 3 classifiers in total and each should be correctly configured.

I hope this helps!

[gimmyy]

Thanks for your timely response! It's of great help. However, the problem may lie in the multi-queue configuration in preemptableMacLayer. Setting the numTrafficClasses of preemptableMacLayer.queue seems to be insufficient because there is no client. eth[0].macLayer.preemptableMacLayer.queue.queue[0] or client. eth[0].macLayer.preemptableMacLayer.queue.queue[1] from the modules in Browse Data in the results folder. It seems that there is still one queue in the preemptableMacLayer.

Can I attach my source code here for better troubleshooting and community contribution? Could you please help fix the following two issues when you are available?

• How to configure Multi-queue in both preemptableMacLayer and expressMacLayer.
• How to configure the outbound Classifier to make different flow types go through the corresponding queue in the multi-queue preemptableMacLayer and expressMacLayer.

The source code is here
fp_tas_cbs.zip.

The network setting is in TsnLinearNetwork.ned. The scenario illustration is in INI_CBS_illustration.jpg. There is also readme to introduce in detail.

I would appreciate it if you can fix the above problems. Thank you so much!