Re: [PATCH] drivers: base: Introduce a new kernel parameter driver_sync_probe=

[Yafang Shao]

On Thu, Dec 7, 2023 at 6:19 PM Greg KH <gregkh@xxxxxxxxxxxxxxxxxxx> wrote:
>
> On Wed, Dec 06, 2023 at 10:08:40PM +0800, Yafang Shao wrote:
> > On Wed, Dec 6, 2023 at 9:31 PM Greg KH <gregkh@xxxxxxxxxxxxxxxxxxx> wrote:
> > >
> > > On Wed, Dec 06, 2023 at 11:53:55AM +0000, Yafang Shao wrote:
> > > > After upgrading our kernel from version 4.19 to 6.1, certain regressions
> > > > occurred due to the driver's asynchronous probe behavior. Specifically,
> > > > the SCSI driver transitioned to an asynchronous probe by default, resulting
> > > > in a non-fixed root disk behavior. In the prior 4.19 kernel, the root disk
> > > > was consistently identified as /dev/sda. However, with kernel 6.1, the root
> > > > disk can be any of /dev/sdX, leading to issues for applications reliant on
> > > > /dev/sda, notably impacting monitoring systems monitoring the root disk.
> > >
> > > Device names are never guaranteed to be stable, ALWAYS use a persistant
> > > names like a filesystem label or other ways.  Look at /dev/disk/ for the
> > > needed ways to do this properly.
> >
> > The root disk is typically identified as /dev/sda or /dev/vda, right?
>
> Depends on your system.  It can also be identified, in the proper way,
> as /dev/disk/by-uuid/eef0abc1-4039-4c3f-a123-81fc99999993 if you want
> (note, fake uuid, use your own disk uuid please.)
>
> Why not do that?  That's the most stable and recommended way of doing
> things.

Adapting to this change isn't straightforward, especially for a large
fleet of servers. Our monitoring system needs to accommodate and
adjust accordingly.

>
> > This is because the root disk, which houses the operating system,
> > cannot be removed or hotplugged.
>
> Not true at all, happens for many systems (think about how systems that
> run their whole OS out of ram work...)
>
> > Therefore, it usually remains as the
> > first disk in the system. With the synchronous probe, the root disk
> > maintains a stable and consistent identification.
> >
> > >
> > > > To address this, a new kernel parameter 'driver_sync_probe=' is introduced
> > > > to enforce synchronous probe behavior for specific drivers.
> > >
> > > This should be a per-bus thing, not a driver-specific thing as drivers
> > > for the same bus could have differing settings here which would cause a
> > > mess.
> > >
> > > Please just revert the scsi bus functionality if you have had
> > > regressions here, it's not a driver-core thing to do.
> >
> > Are you suggesting a reversal of the asynchronous probe code in the
> > SCSI driver?
>
> For your broken scsi driver, yes.
>
> > While reverting to synchronous probing could ensure
> > stability, it's worth noting that asynchronous probing can potentially
> > shorten the reboot duration under specific conditions. Thus, there
> > might be some resistance to reverting this change as it offers
> > performance benefits in certain scenarios. That's why I prefer to
> > introduce a kernel parameter for it.
>
> I don't want to add a new parameter that we need to support for forever
> and add to the complexity of the system unless it is REALLY needed.

BTW, since there's already a 'driver_async_probe=', introducing
another 'driver_sync_probe=' wouldn't significantly increase the
maintenance overhead.

> Please work with the scsi developers to resolve the issue for your
> hardware, as it's been working for everyone else for well over a year
> now, right?

The SCSI guys are added to this mail thread.

I'm uncertain whether it's possible to add SCSI kernel parameters
selectively. If that's not feasible, we'll need to maintain the
following modification in our local kernel:

diff --git a/drivers/scsi/sd.c b/drivers/scsi/sd.c
index e934779..8148d12 100644
--- a/drivers/scsi/sd.c
+++ b/drivers/scsi/sd.c
@@ -607,7 +607,7 @@ static void sd_set_flush_flag(struct scsi_disk *sdkp)
                .name           = "sd",
                .owner          = THIS_MODULE,
                .probe          = sd_probe,
-               .probe_type     = PROBE_PREFER_ASYNCHRONOUS,
+               .probe_type     = PROBE_PREFER_SYNCHRONOUS,
                .remove         = sd_remove,
                .shutdown       = sd_shutdown,
                .pm             = &sd_pm_ops,




--
Regards
Yafang