Re: [BUG] misc/mei: Race between mei_release() disconnect and mei_ioctl_connect_vtag() causes CSME reset storm and i915 freeze

[nirbhayykumarr]

On Saturday, August 29th, 2026 at 1:00 PM, gregkh@xxxxxxxxxxxxxxxxxxx <gregkh@xxxxxxxxxxxxxxxxxxx> wrote:

> On Sat, Aug 29, 2026 at 07:14:10AM +0000, nirbhayykumarr@xxxxxxxxx wrote:
> > Hi all,
> >
> > This issue was discovered using a custom multi-threaded C fuzzer
> > designed to stress-test MEI Virtual Tag (vtag) client lifecycles and
> > multiplexing over /dev/mei0. By concurrently racing rapid vtag
> > connections against file descriptor closures and streaming I/O, a
> > race condition is triggered during client teardown.
> 
> But that's not a normal use case, right?

Yes, you are right. It is definitely not a normal, everyday
workload. it is an extreme edge case that I caught unintentionally while
stress-testing vtag multiplexing during a broader CSME research project.
But because it results in an unrecoverable full system kernel and
DRM/i915 display freeze when hit, I thought it was important to document
and report it.

> > In mei_release(), closing the last file descriptor holding a virtual tag
> > invokes mei_cl_disconnect(). Inside __mei_cl_disconnect(), dev->device_lock
> > is dropped while awaiting the firmware disconnect ACK on cl->wait.
> >
> > During this lock-drop window, a concurrent IOCTL_MEI_CONNECT_CLIENT_VTAG
> > call on the same UUID (e.g. MKHI) scans dev->file_list, matches the tearing-
> > down client 'pos' (in MEI_FILE_DISCONNECTING), repoints file->private_data
> > to pos, and adds its new vtag to pos->vtag_map.
> >
> > When the disconnect ACK arrives, __mei_cl_disconnect() calls
> > mei_cl_set_disconnected(cl), setting cl->me_cl = NULL and
> > cl->state = MEI_FILE_DISCONNECTED. Because pos->vtag_map now contains the
> > second thread's tag, mei_release() skips unlinking/freeing cl. The second
> > thread then wakes up and attempts to reconnect via mei_ioctl_connect_client().
> >
> > Additionally, shared clients lack tag based demuxing on cl->rd_pending in
> > drivers/misc/mei/interrupt.c:
> >   cb = list_first_entry_or_null(&cl->rd_pending, struct mei_cl_cb, list);
> >
> > Incoming packets are matched to the head of the FIFO queue regardless of
> > vtag, causing -EPROTO errors, dropped packets, and out-of-order HBM command
> > sequences. This triggers continuous CSME hardware link resets:
> >   mei mei0: FW not ready: resetting: dev_state = 3
> >   mei mei0: unexpected reset: dev_state = ENABLED fw status = ...
> >
> > During each reset, child client drivers (mei_hdcp, mei_pxp) unbind and rebind
> > with i915 DRM. With resets looping at hundreds of cycles per second (>9,800
> > events in 27s), mei_cldev_enable() repeatedly fails with -EFAULT / -ENODEV,
> > deadlocking i915 display worker mutexes in TASK_UNINTERRUPTIBLE and causing
> > an unrecoverable full system freeze.
> >
> > Proposed Fix:
> > - Prevent vtag reuse during teardown: in mei_ioctl_connect_vtag(),
> >    ignore existing clients on dev->file_list if they are in
> >    MEI_FILE_DISCONNECTING or MEI_FILE_DISCONNECTED states or undergoing
> >    teardown.
> > - Implement proper reference counting / lifecycle synchronization on
> >    shared struct mei_cl instances.
> > - In drivers/misc/mei/interrupt.c, demux cl->rd_pending by matching the
> >    incoming packet's vtag header to the corresponding callback rather
> >    than assuming FIFO order.
> > - Add rate-limiting / backoff to MEI client reprobing during hardware
> >    link resets to prevent cascading bus storms into DRM / i915.
> 
> Please send patches for this if you wish to see these issues addressed.
> As you have a reproducer, it should be simple for you to do so.

I appreciate the nudge. While the reproducer is straightforward, safely 
resolving the cross-subsystem interaction between MEI and i915 without 
introducing regressions seemed complex enough that I initially deferred 
to the Intel maintainers. 

However, since I've already mapped out the race in mei_ioctl_connect_vtag(), 
I am happy to take a shot at it. I will work on implementing the localized 
state checks to prevent vtag reuse during teardown, and I'll submit a 
patch for review in the next few days.

> And how does this differ from the patch you have already sent but has
> not yet been merged?

The previous patch addressed a single-client teardown ordering issue where 
incoming IRQ completions populated cl->rd_completed after mei_cl_flush_queues() 
was called, triggering a WARN_ON() in mei_cl_unlink().

This issue is a separate, multi-client race specific to Virtual Tag (vtag)
multiplexing in mei_ioctl_connect_vtag(). While mei_release() drops
dev->device_lock to wait for firmware disconnect, a concurrent thread
requesting a new vtag matches the tearing-down client on dev->file_list.
When the disconnect finishes, cl->me_cl is cleared/NULL'd while the client
is actively reused, resulting in CSME hardware reset loops and cascading
DRM/i915 display freezes.


Thanks,
Nirbhay Kumar