Re: [RFC, net-next] net: qos: introduce a frer action to implement 802.1CB

[Ferenc Fejes]

Hi Vladimir!

I adjusted the CC list too with my colleagues.

On 2022. 05. 06. 14:23, Vladimir Oltean wrote:
> Hi Ferenc,
>
> (I adjusted the CC list)
>
> On Fri, May 06, 2022 at 11:55:56AM +0000, Ferenc Fejes wrote:
>> On 2021. 09. 28. 13:44, Xiaoliang Yang wrote:
>>> This patch introduce a frer action to implement frame replication and
>>> elimination for reliability, which is defined in IEEE P802.1CB.
>> HiXiaoliang!
>>
>> thanks for your efforts to introduce afreraction to implement frame
>> replication and elimination for reliability, which is defined in IEEE
>> P802.1CB-2017. I would like to relay a small comment from our team,
>> regarding to the FRER, not particularly to the code.
>>
>> Support of RTAG format is very straightforward.
>>
>> Since 2017, several maintenance items were opened regarding IEEE
>> P802.1CB-2017 to fix some errors in the standard. Discussions results
>> will be published soon e.g., in IEEE P802.1CBdb
>> (https://protect2.fireeye.com/v1/url?k=31323334-501d5122-313273af-454445555731-a50148cf7cc14d37&q=1&e=bac1cc80-3c80-4916-be67-352e21564815&u=https%3A%2F%2F1.ieee802.org%2Ftsn%2F802-1cbdb%2F).
>>
>> One of the maintenance items impacts the vector recovery algorithm itself.
>>
>> Details on the problem and the solution are here:
>>
>> -https://protect2.fireeye.com/v1/url?k=31323334-501d5122-313273af-454445555731-7de6c1f45efd0a2c&q=1&e=bac1cc80-3c80-4916-be67-352e21564815&u=https%3A%2F%2Fwww.802-1.org%2Fitems%2F370
>>
>> -https://protect2.fireeye.com/v1/url?k=31323334-501d5122-313273af-454445555731-cc75f6c9f6a68939&q=1&e=bac1cc80-3c80-4916-be67-352e21564815&u=https%3A%2F%2Fwww.ieee802.org%2F1%2Ffiles%2Fpublic%2Fdocs2020%2Fmaint-varga-257-FRER-recovery-window-0320-v01.pdf
>> <https://protect2.fireeye.com/v1/url?k=31323334-501d5122-313273af-454445555731-cc75f6c9f6a68939&q=1&e=bac1cc80-3c80-4916-be67-352e21564815&u=https%3A%2F%2Fwww.ieee802.org%2F1%2Ffiles%2Fpublic%2Fdocs2020%2Fmaint-varga-257-FRER-recovery-window-0320-v01.pdf>
>>
>> It is a small but important fix. There is an incorrect reference to the
>> size of the recovery window, when a received packet is checked to be
>> out-of-range or not. Without this fix the vector recovery algorithm do
>> not work properly in some scenarios.
>>
>> Please consider to update your patch to reflect the maintenance efforts
>> of IEEE to correct .1CB-2017 related issues.
>>
>>> There are two modes for frer action: generate and push the tag, recover
>>> and pop the tag. frer tag has three types: RTAG, HSR, and PRP. This
>>> patch only supports RTAG now.
>>>
>>> User can push the tag on egress port of the talker device, recover and
>>> pop the tag on ingress port of the listener device. When it's a relay
>>> system, push the tag on ingress port, or set individual recover on
>>> ingress port. Set the sequence recover on egress port.
>>>
>>> Use action "mirred" to do split function, and use "vlan-modify" to do
>>> active stream identification function on relay system.
>>>
>> All of our research in the topic based on a in-house userspace FRER
>> implementation but we are looking forward to test your work in the future.
>>
>> Thanks,
>>
>> Ferenc
> Glad to see someone familiar with 802.1CB. I have a few questions and
> concerns if you don't mind.
I CCd Balazs Varga  and Janos Farkas, experts of the TSN topics 
including 802.1CB as well. Istvan Moldovan's can also give valuable 
feedback as the author of our in-house userspace FRER. I'll also try my 
best to answer but I'm the least competent in the topic.
>
> I think we are seeing a bit of a stall on the topic of FRER modeling in
> the Linux networking stack, in no small part due to the fact that we are
> working with pre-standard hardware.
>
> The limitation with Xiaoliang's proposal here (to model FRER stream
> replication and recovery as a tc action) is that I don't think it works
> well for traffic termination - it only covers properly the use case of a
> switch. More precisely, there isn't a single convergent termination
> point for either locally originating traffic, or locally received
> traffic (i.e. you, as user, don't know on which interface of several
> available to open a socket).
>
> In our hardware, this limitation isn't really visible because of the way
> in which the Ethernet switch is connected inside the NXP LS1028A.
We have some NXP LS1028As as well so at least I familiar with the box :-)
> It is something like this:
>
>    +---------------------------------------+
>    |                                       |
>    |           +------+ +------+           |
>    |           | eno2 | | eno3 |           |
>    |           +------+ +------+           |
>    |              |         |              |
>    |           +------+ +------+           |
>    |           | swp4 | | swp5 |           |
>    |           +------+ +------+           |
>    |  +------+ +------+ +------+ +------+  |
>    |  | swp0 | | swp1 | | swp2 | | swp3 |  |
>    +--+------+-+------+-+------+-+------+--+
>
> In the above picture, the switch ports swp0-swp3 have eno3 as a DSA
> master (connected to the internal swp5, a CPU port). The other internal
> port, swp5, is configured as a DSA user port, so it has a net device.
> Analogously, while eno3 is a DSA master and receives DSA-tagged traffic
> (so it is useless for direct IP termination), eno2 receives DSA untagged
> traffic and is therefore an IP termination endpoint into a switched
> network.

Unfortunately I'm not familiar with the distributed switch architecture 
(I only read a netdev paper from that and thats all) but I try to grasp 
on the problem.
In my understanding, the main issue is the distinction between the 
locally terminated and forwarded TSN streams, because currently the DSA 
metadata tags are required to do that? Can you explain the problem for 
one who not familiar with DSA?

>
> What we do in this case is put tc-frer rules for stream replication and
> recovery on swp4 itself, and we use eno2 as the convergence point for
> locally terminated streams.
>
> However, naturally, a hardware design that does not look like this can't
> terminate traffic like this.
Yes, this is my concern too. What would be a nice to have thing if the 
user can configure the SW implementation and the HW offload with the 
same commands and the original tc-frer approach fits well to this 
concept. Anything towards that direction is the way forward IMO, even if 
the underlying implementation will change.
>
> My idea was that it might be better if FRER was its own virtual network
> interface (like a bridge), with multiple slave interfaces. The FRER net
> device could keep its own database of streams and actions (completely
> outside of tc) which would be managed similar to "bridge fdb add ...".
> This way, the frer0 netdevice would be the local termination endpoint,
> logically speaking.
Interesting approach. To be honest I dont see the long term implications 
of this solution, others might have ideas about the pros and cons, but 
that looks like a solution where local stream termination is trivial.
> What I don't know for sure is if a FRER netdevice is supposed to forward
> frames which aren't in its list of streams (and if so, by which rules).
Yes this sounds correct, somehow non-local packets should be forwarded 
too with a bridge. Is it possible to the linux bridge recognize if one 
port is a frer0 port (or on the frer0 if that is enslaved) and do the 
forwarding of the streams? Re-implementing bridge functions just for the 
frer device would be redundant. Unfortunately I never dug myself deep 
enough into the linux bridge code, just when debugged VXLAN ARP 
suppression for EVPN, but I think it would be possible to exchange some 
metadatas between the bridge and the frer device to do the 
forwarding/terminating decision, something like here [0]
> Because if a FRER netdevice is supposed to behave like a regular bridge
> for non-streams, the implication is that the FRER logic should then be
> integrated into the Linux bridge.

This is (for me) more appealing. Also we can keep that in mind when 
Linux will support deterministic layer3 networking (IETF DetNet WG RFCs) 
it would be nice to have mapping between TSN and DetNet streams, then 
forward the packets on DetNet tunnels as well (with different 
endpoints). This is something our team researching so Balazs and Istvan 
might give you some info about that. But I admit that thinking about 
playing nicely with DetNet in regard of the current linux FRER 
implementation is more than overwhelming, but the Linux bridge would be 
a nice place to map TSN flows to DetNet flow like currently EVPN maps 
VLANs to VXLANs.

> Also, this new FRER software model complicates the offloading on NXP
> LS1028A, but let's leave that aside, since it shouldn't really be the
> decisive factor on what should the software model look like.
>
> Do you have any comments on this topic?
I would like to see if others can join to the discussion as well, I will 
try to think about this problem more too.

[0] https://lore.kernel.org/netdev/20220301050439.31785-10-roopa@xxxxxxxxxx/

Best,
Ferenc