[REGRESSION] Bluetooth: ASUS USB-BT540 (0b05:1bef) fails after BTUSB_REALTEK quirk

[Niko]

Hello,

I would like to report a regression affecting an ASUS USB-BT540 Bluetooth
adapter (USB ID 0b05:1bef).

The regression appears to be caused by commit:

980084de4d9b25193398d89a1c0430ba3501b683
Bluetooth: btusb: Add ASUS USB-BT540 for Realtek 8761CU

which adds BTUSB_REALTEK | BTUSB_WIDEBAND_SPEECH for this USB ID.

Hardware
========

Host:
  Raspberry Pi
  Debian 13 (Trixie)
  Raspberry Pi kernel: 6.18.50+rpt-rpi-v8

Adapter:
  ASUS USB-BT540
  USB ID: 0b05:1bef

USB descriptors:

  bcdUSB               1.10
  bcdDevice            2.00
  iManufacturer           1 Realtek
  iProduct                2 Bluetooth Controller
  iSerial                 0

The descriptors appear to match the device reported in the original patch,
including bcdUSB 1.10 and bcdDevice 2.00.

Problem
=======

With the stock btusb driver containing:

  { USB_DEVICE(0x0b05, 0x1bef), .driver_info = BTUSB_REALTEK |
                                       BTUSB_WIDEBAND_SPEECH },

the adapter fails during Realtek initialization.

The kernel reports:

  Bluetooth: hci1: RTL: examining ...
  Bluetooth: hci1: RTL: Read reg16 failed (-32)

I have also observed:

  Bluetooth: hci1: RTL: Read reg16 failed (-110)

The device then disconnects/re-enumerates and does not become a usable
Bluetooth controller.

I reproduced the failure after cold boots and using different Raspberry Pi
USB ports.

The same behavior was also observed with Raspberry Pi kernels 6.12.109 and
6.18.50.

Power/undervoltage problems were also checked on the Raspberry Pi:

  vcgencmd get_throttled
  throttled=0x0

Test / workaround
=================

To isolate the problem, I built btusb.ko from the exact Raspberry Pi 6.18.50
kernel source.

I made only one functional change to drivers/bluetooth/btusb.c: I 
removed the
0b05:1bef entry which forces this device through BTUSB_REALTEK.

The adjacent ASUS USB-BT600 0b05:1d70 entry and all other btusb code were
left unchanged.

With 0b05:1bef falling back to the generic btusb path, the same physical
USB-BT540 works correctly.

BlueZ detects the USB-BT540 as a second controller.

The working controller reports:

  Type: Primary
  Bus: USB
  HCI Version: 5.4 (0xd)
  HCI Revision: 0x000e
  LMP Version: 5.4 (0xd)
  LMP Subversion: 0x8761
  Manufacturer: Realtek Semiconductor Corporation (93)

btmgmt also reports:

  version 13
  manufacturer 93

and the controller remains powered and operational.

hciconfig reports:

  UP RUNNING PSCAN
  RX errors: 0
  TX errors: 0

BLE operation
=============

This is not only an HCI enumeration test.

With the generic btusb path the adapter successfully performs BLE scanning
and discovers BLE devices.

It is also currently being used continuously by a Python application using
Bleak to communicate with Ensto ELTE6-BT BLE heating controllers.

The complete path:

  USB-BT540
    -> generic btusb
    -> BlueZ
    -> Bleak
    -> BLE heating controller
    -> MQTT

is working reliably.

Therefore the controller appears to be fully usable through the generic
btusb path rather than merely enumerating successfully.

Test on another computer
========================

I also tested the same physical ASUS USB-BT540 on a separate Pop!_OS system
running kernel 7.1.1.

On that system the adapter worked as a second BlueZ controller using the
generic btusb path.

The only notable message there was:

  Bluetooth: hci1: Failed to read codec capabilities (-22)

but the controller itself was operational.

Observation
===========

This is interesting because the USB descriptors of this adapter appear to
match those from the device used for the original BTUSB_REALTEK patch:

  VID:PID       0b05:1bef
  bcdUSB        1.10
  bcdDevice     2.00
  Manufacturer  Realtek
  Product       Bluetooth Controller

However, the behavior of this unit appears to be the opposite of the unit
described in the original patch.

On this adapter:

  BTUSB_REALTEK
    -> Realtek initialization
    -> Read reg16 fails (-32 or -110)
    -> controller unusable

while:

  generic btusb
    -> controller initializes
    -> HCI operational
    -> BLE scanning works
    -> BLE GATT communication works reliably

I am therefore not suggesting simply reverting the 0b05:1bef entry, since
the USB-BT540 used for the original patch apparently required the Realtek
firmware path to operate.

It seems possible that ASUS/Realtek USB-BT540 adapters with the same VID:PID
and bcdDevice exist with different internal hardware, ROM or firmware
revisions, or that there is another distinction which is not represented by
the USB descriptors.

For comparison, this working unit identifies itself as:

  HCI Version:    5.4
  HCI Revision:   0x000e
  LMP Version:    5.4
  LMP Subversion: 0x8761
  Manufacturer:   Realtek (93)

I can provide additional logs, USB/HCI information, and test proposed driver
changes if needed.

Regards,
Niko