Re: DM3730 Bluetooth Performance differences between SERIAL_8250_OMAP vs SERIAL_OMAP

[Adam Ford]

On Sat, Oct 5, 2019 at 8:23 PM Adam Ford <aford173@xxxxxxxxx> wrote:
>
> On Fri, Oct 4, 2019 at 10:45 PM Adam Ford <aford173@xxxxxxxxx> wrote:
> >
> > On Fri, Oct 4, 2019 at 11:51 AM Adam Ford <aford173@xxxxxxxxx> wrote:
> > >
> > > On Fri, Oct 4, 2019 at 9:08 AM Adam Ford <aford173@xxxxxxxxx> wrote:
> > > >
> > > > On Fri, Oct 4, 2019 at 7:27 AM Yegor Yefremov
> > > > <yegorslists@xxxxxxxxxxxxxx> wrote:
> > > > >
> > > > > Hi Adam,
> > > > >
> > > > > On Fri, Oct 4, 2019 at 12:39 PM Adam Ford <aford173@xxxxxxxxx> wrote:
> > > > > >
> > > > > > On Fri, Oct 4, 2019 at 5:02 AM Adam Ford <aford173@xxxxxxxxx> wrote:
> > > > > > >
> > > > > > > I am running Kernel 5.3.2 trying to troubleshoot some intermittent
> > > > > > > Bluetooth issues, and I think I have narrowed it down to the serial
> > > > > > > driver in use.
> > > > > >
> > > > > > I should have also noted that it's using UART2 with CTS and RTS on the
> > > > > > DM3730 (omap3630) and its configured with a baud rate of 3M.
> > > > > > I tried slowing it to 115200, but that didn't help.  I tried disabling
> > > > > > the DMA hooks from the device tree, and that didn't help.
> > > > > >
> > > > > > > By default, omap2plus_defconfig enables both SERIAL_8250_OMAP and
> > > > > > > SERIAL_OMAP.  I have my console device configured as  ttyS0, and all
> > > > > > > appears fine.  When I enable Bluetooth, however, I get intermittent
> > > > > > > errors on an DM3730 / OMAP3630.
> > > > > > >
> > > > > > > Using the 8250 driver for Blueotooth I get intermittent frame errors
> > > > > > > and data loss.
> > > > > > >
> > > > > > > Scanning ...
> > > > > > > [   28.482452] Bluetooth: hci0: Frame reassembly failed (-84)
> > > > > > > [   36.162170] Bluetooth: hci0: Frame reassembly failed (-84)
> > > > > > >         F4:4E:FC:C9:2F:57       BluJax
> > > > > > > # l2ping F4:4E:FC:C9:2F:57
> > > > > > > Ping: F4:4E:FC:C9:2F:57 from 00:18:30:49:7D:63 (data size 44) ...
> > > > > > > 44 bytes from F4:4E:FC:C9:2F:57 id 0 time 8.27ms
> > > > > > > no response from F4:4E:FC:C9:2F:57: id 1
> > > > > > > ^C2 sent, 1 received, 50% loss
> > > > > > >
> > > > > > > (after a fairly long hang, I hit control-c)
> > > > > > >
> > > > > > > However, disabling the 8250 driver and using the only SERIAL_OMAP and
> > > > > > > the console routed to ttyO0, the Bluetooth works well, so I believe it
> > > > > > > to be a serial driver issue and not a Bluetooth error.
> > > > > > >
> > > > > > > # hcitool scan
> > > > > > > Scanning ...
> > > > > > >         F4:4E:FC:C9:2F:57       BluJax
> > > > > > > ^C
> > > > > > > # l2ping F4:4E:FC:C9:2F:57
> > > > > > > Ping: F4:4E:FC:C9:2F:57 from 00:18:30:49:7D:63 (data size 44) ...
> > > > > > > 44 bytes from F4:4E:FC:C9:2F:57 id 0 time 6.90ms
> > > > > > > ...
> > > > > > > 44 bytes from F4:4E:FC:C9:2F:57 id 14 time 28.29ms
> > > > > > > ^C15 sent, 15 received, 0% loss
> > > > > > > #
> > > > > > >
> > > > > > > 0% loss and regular, repeatable communication without any Frame
> > > > > > > reassembly errors.
> > > > > > >
> > > > > >
> > > > > > I tried disabling SERIAL_OMAP and using only SERIAL_8250_OMAP, but
> > > > > > that didn't help.  Because the issue goes away when I disable
> > > > > > SERIAL_8250_OMAP, I am wondering if something is either being
> > > > > > misconfigured or some IRQ or DMA integration is missing that may be
> > > > > > present with the older SERIAL_OMAP driver.
> > > > > >
> > > > > > > Any suggestions on how to troubleshoot or what might cause the
> > > > > > > difference between the two drivers?
> > > > >
> > > > > Can it be related to this issue [1]? Can you confirm that 5.2 is
> > > > > working as expected with the 8250 driver?
> > > > >
> > > > > [1] https://marc.info/?l=linux-serial&m=156965039008649&w=2
> > > >
> > > > I reverted the whole 8250 directory to d99482673f95 ("serial:
> > > > mctrl_gpio: Check if GPIO property exisits before requesting it") and
> > > > it is somewhat better, but it's not as good as the stock OMAP serial
> > > > driver.  I get some frame errors and eventually, I get some timeouts,
> > > > but it's not as bad.  I'll try to implement the RTS and CTS as gpio
> > > > pins and change the device tree accordingly.  It might shed some light
> > > > on the situation.
> > >

Yegor,

Can you take a look at [1]?  I am still investigating the DMA issue
that I have where the serial port does not correctly work with DMA
enabled, but I submitted a patch that appears to fix my serial port
issue (as long as 8250 DMA is off).
I tested this against 5.4-rc1, and Tony submitted a different patch
that seems to help as well [2] that seems to help with the 5.2 and 5.3
kernels.

thanks

adam

[1] - https://patchwork.kernel.org/patch/11175969/

And for trees prior to 5.4-rc1):
[2] - https://patchwork.kernel.org/patch/11054387/



> > > I tried to manually setup RTS and CTS pins as GPIO, but that didn't
> > > work, so I changed it back.
> > >
> > > It looks like the clocking is correct.  I don't know enough about the
> > > DMA or the IRQ to know if it's working correctly.
> > >
> > > I was wondering if the problem is in the handshaking or not.
> > > I added " uart-has-rtscts;" to by uart node thinking it might help,
> > > but it did not.
> > >
> > > >
> > > 8250_omap.c has some checks to see if we can enable autoRTS:
> > >
> > > if (termios->c_cflag & CRTSCTS && up->port.flags & UPF_HARD_FLOW &&
> > >     !up->gpios) {
> > >      /* Enable AUTOCTS (autoRTS is enabled when RTS is raised) */
> > >      ...
> > > }
> > >
> > > Based on this, I would expect up->gpios to always be zero if we want
> > > auto RTS CTS.
> > >
> > > I threw some debug code into the serial driver to look at the status
> > > of the various flags that go into setting up auto RTS/CTS.
> > >
> > > [   13.837005] termios->c_cflag & CRTSCTS = 80000000
> > > [   13.841888] up->port.flags & UPF_HARD_FLOW = 300000
> > > [   13.846801] up->gpios = ce3f3cc0
> > > [   17.166595] termios->c_cflag & CRTSCTS = 0
> > > [   17.170745] up->port.flags & UPF_HARD_FLOW = 300000
> > > [   17.175781] up->gpios = ce3f3d40
> > >
> > > I don't know which port is which, but I only have two active uarts
> > > (console and Bluetooth)
> > > The Bluetooth port should use handshaking and the console should not.
> > >
> > > From the look of the dump, up->gpios is never 0, so the last
> > > evaluation would always be false unless I am misinterpreting
> > > something.
> > >
> > > I tried to modify the check to remove !up->gpios check, but that
> > > didn't help either, but it made it a little better.
> >
> > I have it working now on the 5.3.y branch, but I had to do 4 things.
> >
> > 1)   Disable the older OMAP serial driver
> > 2)   Apply [1] patch
> > 3)   Disable 8250 DMA support
> > 4)  Remove checks for !up->gpios in the 8250-omap.c code.
> >
> > With all those changes, I am able to successfully and repeatedly scan
> > and l2ping Bluetooth devices.  Without any one of those, I get frame
> > errors or lost packets.
> >
> > I wonder if we can dump the legacy omap serial driver once  the 8250
> > driver is working better.  That would eliminate at least one of the 4.
> > I am not sure what's up with the DMA, and I've already stated some
> > thoughts on item 4 in a previous e-mail.
> >
> > I think the [1] patch is probably going to help fix some other issues
> > and should probably be backported, but by itself it isn't enough to
> > fix the Bluetooth problem on the OMAP3630.
> >
> > [1] - https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/drivers/tty/serial/8250/8250_omap.c?h=v5.4-rc1&id=627a545c6bb0c7de09208e6546f5111290477261
> >
> In the latest 'master' branch of 5.4-RC1 (43b815c6a8e7), I have
> confirmed that by removing the DMA and forcing the checks for
> !up->gpios to be true to keep the hardware flow control working, I can
> repeatably make Bluetooth work.
>
> I am not sure what's broken in the DMA, but the check for  !up->gpios
> is never false, so it never actually does the code inside to address
> handshaking
>
> index c68e2b3a1634..884fcba61303 100644
> --- a/drivers/tty/serial/8250/8250_omap.c
> +++ b/drivers/tty/serial/8250/8250_omap.c
> @@ -141,7 +141,7 @@ static void omap8250_set_mctrl(struct uart_port
> *port, unsigned int mctrl)
>
>         serial8250_do_set_mctrl(port, mctrl);
>
> -       if (!up->gpios) {
> +       if (1 /* !up->gpios */) {
>                 /*
>                  * Turn off autoRTS if RTS is lowered and restore autoRTS
>                  * setting if RTS is raised
> @@ -455,8 +455,7 @@ static void omap_8250_set_termios(struct uart_port *port,
>         priv->efr = 0;
>         up->port.status &= ~(UPSTAT_AUTOCTS | UPSTAT_AUTORTS | UPSTAT_AUTOXOFF);
>
> -       if (termios->c_cflag & CRTSCTS && up->port.flags & UPF_HARD_FLOW &&
> -           !up->gpios) {
> +       if (termios->c_cflag & CRTSCTS && up->port.flags & UPF_HARD_FLOW) {
>                 /* Enable AUTOCTS (autoRTS is enabled when RTS is raised) */
>                 up->port.status |= UPSTAT_AUTOCTS | UPSTAT_AUTORTS;
>                 priv->efr |= UART_EFR_CTS;
>
> I know this is wrong to do what I did, but whatever is setting gpios
> is clearly not setting it to NULL, so I am guessing these checks here
> to be something else or something needs to be able to set these to
> NULL when there are no manually configured gpio pints for cts/rts.
>
> adam
>
> > adam
> > >
> > >
> > > > adam
> > > > >
> > > > > Cheers,
> > > > > Yegor