Re: [PATCH net-next v2 0/9] Add support for OPEN Alliance 10BASE-T1x MACPHY Serial Interface

[Parthiban.Veerasooran]

Hi Benjamin Bigler,

Thank you for your testing and feedback. It would be really helpful to 
bring the driver to a good shape. We really appreciate your efforts on this.

On 24/03/24 5:25 pm, Benjamin Bigler wrote:
> [Some people who received this message don't often get email from benjamin@xxxxxxxxxx. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]
> 
> EXTERNAL EMAIL: Do not click links or open attachments unless you know the content is safe
> 
> Hi Parthiban
> 
> I hope I send this in the right context as it is not related to just one patch or
> some specific code.
> 
> I conducted UDP load testing using three i.MX8MM boards in conjunction with the
> LAN8651. The setup involved one board functioning as a server, which is just
> echoing back received data, while the remaining two boards acted as clients,
> sending UDP packets of different sizes in various bursts to the server.
> Due to hardware constraints, the SPI bus speed was limited to 15 MHz, which might
> have influenced the results.
> 
> During the tests I experienced some issues:
> 
> - The boards just start receiving after first sending something (ping another board).
>    Some measurements showed that the irq stays asserted after init. This makes sense
>    as far as I understand the chapter 7.7 of the specification, the irq is deasserted
>    on reception of the first data header following CSn being asserted. As a workaround
>    I trigger the thread at the end of oa_tc6_init.
It looks like the IRQ is asserted on RESET completion and expects a data
chunk from host to deassert the IRQ. I used to test the driver in RPI 4
using iperf3. For some reason I never faced this issue, may be when the
network device is being registered there might be some packet 
transmission which leads to deliver a data chunk so that the IRQ is
deasserted. Thanks for the workaround. I think that would be the 
solution to solve this issue. Adding the below lines in the end of the 
function oa_tc6_init() will trigger the oa_tc6_spi_thread_handler() to 
perform an empty data chunk transfer which will deassert the IRQ before 
starting the actual data transfer.

/* oa_tc6_sw_reset_macphy() function resets and clears the MAC-PHY reset
  * complete status. IRQ is also asserted on reset completion and it is
  * remain asserted until MAC-PHY receives a data chunk. So performing an
  * empty data chunk transmission will deassert the IRQ. Refer section
  * 7.7 and 9.2.8.8 in the OPEN Alliance specification for more details.
  */
tc6->int_flag = true;
wake_up_interruptible(&tc6->spi_wq);
> 
> - If there is a lot of traffic, the receive buffer overflow error spams the log.
> 
> - If there is a lot of traffic, I got various kernel panics in oa_tc6_update_rx_skb.
>    Mostly because more data to rx_skb is added than allocated and sometimes because
>    rx_skb is null in oa_tc6_update_rx_skb or oa_tc6_prcs_rx_frame_end. Some debugging
>    with a logic analyzer showed that the chip is not behave correctly. There is more
>    bytes between start_valid and end_valid than there should be. Also there
>    seems to be 2 end_valid without a start_valid between. What is common is that the incorrect
>    frame starts in a chunk where end_valid and start_valid is set.
>    In my opinion its a problem in the chip (maybe related to the errata in the next point)
>    but the driver should be resilent and just drop the packet and not cause a kernel panic.
Usually I run into this issue "receive buffer overflow" when I run RPI 4
with default cpu governor setting which is "ondemand". In this case, 
even though if I set SPI clock speed as 15 MHz the RPI 4 core clock is
clocking down when it is idle which leads delivering half of the
configured SPI clock speed around 5.9 MHz. So the systems like RPI 4 
need performance mode enabled to get the proper clock speed for SPI. 
Refer below link for more details.

https://github.com/raspberrypi/linux/issues/3381#issuecomment-1144723750

I used to enable performance mode using the below command.

echo performance | sudo tee 
/sys/devices/system/cpu/cpu0/cpufreq/scaling_governor > /dev/null

So please ensure the SPI clock speed using a logic analyzer to get the
maximum throughput without receive buffer overflow.

Of course, I agree that the driver should not crash in case of receive
buffer overflow. By referring your investigations, I understand that the
buffers in the MAC-PHY is being continuously overwritten again and again
as the host is very slow to read the data from the MAC-PHY buffers
through SPI which alters the descriptors. There might be two reasons why
we run into this situation.
1. The host is busy doing something else and delays to initiate SPI even
    though SPI clock speed is 15 MHz.
2. The SPI clock speed is less than 15 MHz.

I use the below iperf3 setup for my testing and never faced the driver
crash issue even though faced "receive buffer overflow" error when I run
RPI 4 with "ondemand" default mode.

Node 0 - Raspberry Pi 4 with LAN8650 MAC-PHY
  $ iperf3 -s
Node 1 - Raspberry Pi 4 with EVB-LAN8670-USB USB Stick
  $ iperf3 -c 192.168.5.100 -u -b 10M -i 1 -t 0

and vice versa.

I never faced "receive buffer overflow" error when I run RPI 4 with
"performance" mode enabled and even though all the cores are stressed
using the below command,

$ yes >/dev/null & yes >/dev/null & yes >/dev/null & yes >/dev/null &

Can you share more details about your testing setup and applications you
use, so that I will try to reproduce the issue in my setup to debug the
driver?
> 
> - Sometimes the chip stops working. It always asserts the irq but there is no data (rca=0)
>    and also exst is not active. I found out that there is an errata (DS80001075) point s3
>    that explains this. I set the ZARFE bit in CONFIG0. This also fixes the point above.
>    The driver now works since about 2.5 weeks with various load with just one loss of frame
>    error where I had to reboot the system after about 4 days.
It is good to hear that the driver works fine with the above changes. As 
mentioned in the errata, this continuous interrupt issue is a known
issue with LAN8651 Rev.B0. Switching to LAN8651 Rev.B1 will solve this
issue and no need of any workaround. Setting ZARFE bit in the CONFIG0
will solve the continuous interrupt issue but don't know how the above
"receive buffer overflow" issue also solved. I think it is a good idea 
to test with LAN8651 Rev.B1 without setting ZARFE bit once. It would be 
interesting to see the result. I am always using LAN8651 Rev.B1 for my 
testing.

I should be able to reproduce the "receive buffer overflow" issue and 
consequently kernel crash in my setup with LAN8651 Rev.B1 so that I can 
investigate the issue further. As I am not able to reproduce in my RPI 
4, I need your support for the tests and applications you used in your 
setup.
> 
> Is there a reason why you removed the netdev watchdog which was active in v2?
When the timeout occurs, there is no further action except increasing
tx_errors. Not seeing this except USB-to-Ethernet which can be removed
unexpectedly. But this is SPI interface which will not be removed
unexpectedly as it is a platform device. That's why we removed this.

Best regards,
Parthiban V
> 
> Thanks,
> Benjamin Bigler
>