USB-C DisplayPort display failing to stay active with Intel Barlow Ridge USB4 controller, power-management related issue?

[Aaron Rainbolt]

We're experiencing a Linux kernel bug affecting multiple Clevo X370SNx1
laptops (specifically the X370SNW1 variant). The bug appears to be
present in kernels greater than or equal to 6.5, worsening
significantly with kernel 6.11.2 (latest stable at time of this
writing). It is unclear if all of the issues encountered are the same
bug, however the primary problem we've run into appears to be a
consequence of the power management code involving Intel Barlow Ridge
controllers and DisplayPort. The issue occurs with in-kernel Nouveau
drivers and also with proprietary NVIDIA drivers.

When a DisplayPort monitor is attached to these laptops via a USB-C
connection, the monitor is recognized by the system and comes on for
approximately 15 seconds. It then blanks out and is automatically
disconnected from the system as if it had been unplugged. It will
remain that way indefinitely until unplugged and replugged, or until
something "jiggles" (for lack of a better term) the thunderbolt driver.
When either of these things occur, the display will re-attach and come
back on for 15 seconds, then blank out and detach again. There are
various different things that can "jiggle" the thunderbolt driver,
including but not limited to:

* Running `lspci -k` (this one came as a particular surprise)
* Removing and re-inserting the thunderbolt driver (`sudo modprobe -r
  thunderbolt; sleep 1; sudo modprobe thunderbolt`)
* Running `nvidia-detector` while proprietary NVIDIA drivers are loaded

It is possible to mitigate this issue by simply running
`sudo modprobe -r thunderbolt` or `sudo rmmod thunderbolt` and then
leaving the driver unloaded. USB-C displays become stable after this -
they are recognized when attached and remain recognized and functional
indefinitely as one would expect.

We believe this is related to the Intel Barlow Ridge USB4 controller
because:

* Removing the thunderbolt driver restores normal display operation.
* This issue was *not* a problem on Clevo X370SNx machines, which are
  identical to the X370SNx1 except for the Maple Ridge TBT controller
  on the board has been replaced with a Barlow Ridge USB4 controller.
* This problem does not occur on the affected models with the 6.1
  kernel. It occurs with the 6.5 kernel and on all newer kernels we
  have tried.

Furthermore, from inspecting the Thunderbolt driver code, we believe
this is related to the power management features of the driver, because:

* There is only one 15-second timeout defined in the driver source
  code, that being TB_AUTOSUSPEND_DELAY in drivers/thunderbolt/tb.h
* On earlier kernels (Ubuntu’s variant of 6.8 at least), displays are
  stable even when the thunderbolt driver is loaded if we:
  * Remove the thunderbolt driver
  * Attach a USB-C dock
  * Attach displays to the dock (we used 2 4K HDMI monitors)
  * Reload the thunderbolt driver

During our investigation, we discovered commit
a75e0684efe567ae5f6a8e91a8360c4c1773cf3a (patch on mailing list at
https://lore.kernel.org/linux-usb/20240213114318.3023150-1-mika.westerberg@xxxxxxxxxxxxxxx/)
which appears to be a fix for this exact problem. It adds a quirk for
Intel Barlow Ridge controllers, which detects when a DisplayPort device
has been plugged directly into the USB4 port (thus using "redrive"
mode), and instructs the power management subsystem to not power the
chip down during this time if so. Unfortunately, this quirk seems to be
silently ignored, as we built a custom kernel with some `printk` lines
added to the `tb_enter_redrive` and `tb_exit_redrive` functions to
announce when they were called, and nothing in the dmesg log indicated
that they had been called when we did this.

This bug is easily reproducible using the stock kernels in Kubuntu
22.04, Kubuntu 24.04, Kali Linux 2024.2, and Fedora Workstation
Rawhide. Similar behavior is observed across all of these distributions.

We built the 6.11.2 kernel from source and tested it on Kubuntu 24.04,
but while the kernel built, installed, and functioned properly in most
respects, it actually made the problem with USB-C displays worse. As
long as the thunderbolt driver was loaded, no displays were detected
when plugged in (not for even a short length of time), and when the
thunderbolt driver was unloaded, displays would only be recognized and
function if there was only one display attached. Attaching a second
display resulted in the first external display becoming detached and
the second display not coming on. Unplugging the second display
resulted in the first display reattaching. This machine supports up to
three external displays and this has proven to be achievable and stable
with earlier kernels. No valuable error messages were logged in dmesg
when these problems occurred.

Our testing has been limited to the Clevo X370SNW1 model, however we
expect that the X370SNV1 model will exhibit the same issues as it uses
very similar internal components on the system board.

This is basically the extent of our knowledge at this point. We
attempted various patches on Ubuntu's 6.8 kernel to resolve the issue,
all without success:

* We attempted reverting fd4d58d1fef9ae9b0ee235eaad73d2e0a6a73025
  (thunderbolt: Enable CL2 low power state), which had no effect.
* We noticed that one of the Barlow Ridge bridge controllers
  listed by `lspci -k` appeared to not have its device ID in
  drivers/thunderbolt/nhi.h and there was a corresponding quirk in
  drivers/thunderbolt/quirks.c that looked like it might be vaguely
  related to the issue (specifically quirk_usb3_maximum_bandwidth), so
  we tried adding that device to the appropriate files in order to make
  that quirk apply to that device as well, this had no visible effect
  on the kernel's operation and did not resolve the issue.
* After narrowing it down to `quirk_block_rpm_in_redrive`, we attempted
  adding a new `thunderbolt.kf_force_redrive` kernel parameter in
  drivers/thunderbolt/tb.c that forced the code in
  `tb_enter_redrive` and `tb_exit_redrive` to be executed even *if* the
  device didn't have the appropriate quirk bit set, in the hopes that
  this would make the quirk execute and resolve the issue. What ended
  up happening was somehow `tb_enter_redrive` was never called at all
  and `tb_exit_redrive` was called. This in turn made it so that no
  USB-C displays would even be recognized for a short period of time if
  the thunderbolt driver was loaded.
* Looking at PCI vendor IDs, we noticed that the PCI vendor ID used to
  recognize all Intel controllers in drivers/thunderbolt/quirks.c was
  0x8087, whereas the Barlow Ridge controller in our device reported a
  vendor ID of 0x8086. On the off chance that this was a typo of epic
  proportions, we tried adjusting all of the occurrences of 0x8087 in
  the tb_quirks[] array to PCI_VENDOR_ID_INTEL (which is defined as
  0x8086 in include/linux/pci_ids.h). This has no visible effect on the
  kernel's behavior, and did not resolve the issue. (Presumably there's
  something going on with the IDs there that we're not aware of.)

As to my speculation as to what's wrong, I believe this is likely a
combination of two things:

* Some data in the `tb_quirks` array in drivers/thunderbolt/quirks.c is
  incorrect and leading to the Barlow Ridge controllers not being
  recognized as needing the DisplayPort redrive mode quirk.
* The code in drivers/thunderbolt/tb.c `tb_dp_resource_unavailable`
  that controls whether or not to run `tb_enter_redrive` is faulty in
  some way and is not calling `tb_enter_redrive` in all scenarios where
  it is necessary. To be clear, the exact code I'm talking about is
  this chunk from the aforementioned function:

        tunnel = tb_find_tunnel(tb, TB_TUNNEL_DP, in, out);
        if (tunnel)
                tb_deactivate_and_free_tunnel(tunnel);
        else 
                tb_enter_redrive(port);

Finally, this is probably a result of me misreading the driver code
somehow, but I was surprised by the following conditional at the top
of `tb_enter_redrive`:

        if (!(sw->quirks & QUIRK_KEEP_POWER_IN_DP_REDRIVE))
                return;

To me this reads as "if the DP redrive quirk bit is set, return and do
nothing. Otherwise, if the bit is not set, run the quirk function."
This is the opposite of what I would expect - shouldn't the code run if
the bit is set, not if it is clear? Or does the bit being unset mean
that the quirk is active? (I do not believe that this is the root cause
of the issue because even when I forced this function to run any time
it was invoked, it wasn't being invoked at all.)

This issue has only been definitively reproduced on already-EOL kernels
due to the (potentially related) problem encountered with 6.11.2.
However based on a code comparison it appears all of the apparently
relevant code (that which deals with the DP quirk) is identical between
Ubuntu's variation of the 6.8 kernel and the tip of the mainline master
branch. Therefore I believe this issue very likely impacts the latest
mainline kernel.