Re: [PATCH v2 1/2] coresight: trbe: Fix TRBE potential sleep in atomic context

[hejunhao]

Hi Anshuman Khandual,


On 2023/8/17 15:13, Anshuman Khandual wrote:
> Hello Junhao,
>
> On 8/16/23 19:40, Suzuki K Poulose wrote:
> > From: Junhao He <hejunhao3@xxxxxxxxxx>
> >
> > smp_call_function_single() will allocate an IPI interrupt vector to
> > the target processor and send a function call request to the interrupt
> > vector. After the target processor receives the IPI interrupt, it will
> > execute arm_trbe_remove_coresight_cpu() call request in the interrupt
> > handler.
> >
> > According to the device_unregister() stack information, if other process
> > is useing the device, the down_write() may sleep, and trigger deadlocks
> > or unexpected errors.
> >
> >    arm_trbe_remove_coresight_cpu
> >      coresight_unregister
> >        device_unregister
> >          device_del
> >            kobject_del
> >              __kobject_del
> >                sysfs_remove_dir
> >                  kernfs_remove
> >                    down_write ---------> it may sleep
> But how did you really detect this problem ? Does this show up as an warning when
> you enable lockdep debug ? OR it really happened during a real workload execution
> followed by TRBE module unload. Although the problem seems plausible (which needs
> fixing), just wondering how did we trigger this.

Yes, it really happened during a real workload.

If the TRBE driver is loaded and unloaded cyclically. the test script 
following:

  for ((i=0;i<99999;i++))
  do
  insmod coresight-trbe.ko;
  rmmod coresight-trbe.ko;
  echo "loop $i";
  done

The kernel will report a panic.

> > Add a helper arm_trbe_disable_cpu() to disable TRBE precpu irq and reset
> > per TRBE.
> > Simply call arm_trbe_remove_coresight_cpu() directly without useing the
> > smp_call_function_single(), which is the same as registering the TRBE
> > coresight device.
> >
> > Fixes: 3fbf7f011f24 ("coresight: sink: Add TRBE driver")
> > Signed-off-by: Junhao He <hejunhao3@xxxxxxxxxx>
> > Link: https://lore.kernel.org/r/20230814093813.19152-2-hejunhao3@xxxxxxxxxx
> > [ Remove duplicate cpumask checks during removal ]
> > Signed-off-by: Suzuki K Poulose <suzuki.poulose@xxxxxxx>
> > ---
> >   drivers/hwtracing/coresight/coresight-trbe.c | 33 +++++++++++---------
> >   1 file changed, 18 insertions(+), 15 deletions(-)
> >
> > diff --git a/drivers/hwtracing/coresight/coresight-trbe.c b/drivers/hwtracing/coresight/coresight-trbe.c
> > index 7720619909d6..025f70adee47 100644
> > --- a/drivers/hwtracing/coresight/coresight-trbe.c
> > +++ b/drivers/hwtracing/coresight/coresight-trbe.c
> > @@ -1225,6 +1225,17 @@ static void arm_trbe_enable_cpu(void *info)
> >   	enable_percpu_irq(drvdata->irq, IRQ_TYPE_NONE);
> >   }
> >   
> > +static void arm_trbe_disable_cpu(void *info)
> > +{
> > +	struct trbe_drvdata *drvdata = info;
> > +	struct trbe_cpudata *cpudata = this_cpu_ptr(drvdata->cpudata);
> > +
> > +	disable_percpu_irq(drvdata->irq);
> > +	trbe_reset_local(cpudata);
> > +	cpudata->drvdata = NULL;
> > +}
> > +
> > +
> >   static void arm_trbe_register_coresight_cpu(struct trbe_drvdata *drvdata, int cpu)
> >   {
> >   	struct trbe_cpudata *cpudata = per_cpu_ptr(drvdata->cpudata, cpu);
> > @@ -1326,18 +1337,12 @@ static void arm_trbe_probe_cpu(void *info)
> >   	cpumask_clear_cpu(cpu, &drvdata->supported_cpus);
> >   }
> >   
> > -static void arm_trbe_remove_coresight_cpu(void *info)
> > +static void arm_trbe_remove_coresight_cpu(struct trbe_drvdata *drvdata, int cpu)
> >   {
> > -	int cpu = smp_processor_id();
> > -	struct trbe_drvdata *drvdata = info;
> > -	struct trbe_cpudata *cpudata = per_cpu_ptr(drvdata->cpudata, cpu);
> >   	struct coresight_device *trbe_csdev = coresight_get_percpu_sink(cpu);
> >   
> > -	disable_percpu_irq(drvdata->irq);
> > -	trbe_reset_local(cpudata);
> >   	if (trbe_csdev) {
> >   		coresight_unregister(trbe_csdev);
> > -		cpudata->drvdata = NULL;
> >   		coresight_set_percpu_sink(cpu, NULL);
> >   	}
> >   }
> > @@ -1366,8 +1371,10 @@ static int arm_trbe_remove_coresight(struct trbe_drvdata *drvdata)
> >   {
> >   	int cpu;
> >   
> > -	for_each_cpu(cpu, &drvdata->supported_cpus)
> > -		smp_call_function_single(cpu, arm_trbe_remove_coresight_cpu, drvdata, 1);
> > +	for_each_cpu(cpu, &drvdata->supported_cpus) {
> > +		smp_call_function_single(cpu, arm_trbe_disable_cpu, drvdata, 1);
> > +		arm_trbe_remove_coresight_cpu(drvdata, cpu);
> > +	}
> >   	free_percpu(drvdata->cpudata);
> >   	return 0;
> >   }
> > @@ -1406,12 +1413,8 @@ static int arm_trbe_cpu_teardown(unsigned int cpu, struct hlist_node *node)
> >   {
> >   	struct trbe_drvdata *drvdata = hlist_entry_safe(node, struct trbe_drvdata, hotplug_node);
> >   
> > -	if (cpumask_test_cpu(cpu, &drvdata->supported_cpus)) {
> > -		struct trbe_cpudata *cpudata = per_cpu_ptr(drvdata->cpudata, cpu);
> > -
> > -		disable_percpu_irq(drvdata->irq);
> > -		trbe_reset_local(cpudata);
> > -	}
> > +	if (cpumask_test_cpu(cpu, &drvdata->supported_cpus))
> > +		arm_trbe_disable_cpu(drvdata);
> This code hunk seems unrelated to the context here other than just finding another use case
> for arm_trbe_disable_cpu(). The problem is - arm_trbe_disable_cpu() resets cpudata->drvdata
> which might not get re-initialized back in arm_trbe_cpu_startup(), as there will still be a
> per cpu sink associated as confirmed with coresight_get_percpu_sink(). I guess it might be
> better to drop this change and just keep everything limited to SMP IPI callback reworking in
> arm_trbe_remove_coresight().

OK, will fix it. The change is just to simplify the code of cpu_teardown.
Maybe we can consider whether we need to set "cpudata->drvdata = NULL"
in arm_trbe_disable_cpu()?  If it's not necessary, This can be kept.
Then drop the release cpudata->drvdata from arm_trbe_disable_cpu().

Best regards,
Junhao.

> >   	return 0;
> >   }
> >   
> .