Re: [PATCH 1/2] coresight: trbe: Fix TRBE potential sleep in atomic context

[hejunhao]

Hi Suzuki


On 2023/8/15 6:57, Suzuki K Poulose wrote:
> On 14/08/2023 14:32, hejunhao wrote:
> > Hi Suzuki
> >
> >
> > On 2023/8/14 18:34, Suzuki K Poulose wrote:
> > > Hi Junhao
> > >
> > > On 14/08/2023 10:38, Junhao He wrote:
> > > > smp_call_function_single() will allocate an IPI interrupt vector to
> > > > the target processor and send a function call request to the interrupt
> > > > vector. After the target processor receives the IPI interrupt, it will
> > > > execute arm_trbe_remove_coresight_cpu() call request in the interrupt
> > > > handler.
> > > >
> > > > According to the device_unregister() stack information, if other 
> > > > process
> > > > is useing the device, the down_write() may sleep, and trigger 
> > > > deadlocks
> > > > or unexpected errors.
> > > >
> > > >    arm_trbe_remove_coresight_cpu
> > > >      coresight_unregister
> > > >        device_unregister
> > > >          device_del
> > > >            kobject_del
> > > >              __kobject_del
> > > >                sysfs_remove_dir
> > > >                  kernfs_remove
> > > >                    down_write ---------> it may sleep
> > > >
> > > > Add a helper arm_trbe_disable_cpu() to disable TRBE precpu irq and 
> > > > reset
> > > > per TRBE.
> > > > Simply call arm_trbe_remove_coresight_cpu() directly without useing 
> > > > the
> > > > smp_call_function_single(), which is the same as registering the TRBE
> > > > coresight device.
> > > >
> > > > Fixes: 3fbf7f011f24 ("coresight: sink: Add TRBE driver")
> > > > Signed-off-by: Junhao He <hejunhao3@xxxxxxxxxx>
> > > > ---
> > > >   drivers/hwtracing/coresight/coresight-trbe.c | 35 
> > > > +++++++++++---------
> > > >   1 file changed, 20 insertions(+), 15 deletions(-)
> > > >
> > > > diff --git a/drivers/hwtracing/coresight/coresight-trbe.c 
> > > > b/drivers/hwtracing/coresight/coresight-trbe.c
> > > > index 7720619909d6..ce1e6f537b8d 100644
> > > > --- a/drivers/hwtracing/coresight/coresight-trbe.c
> > > > +++ b/drivers/hwtracing/coresight/coresight-trbe.c
> > > > @@ -1225,6 +1225,17 @@ static void arm_trbe_enable_cpu(void *info)
> > > >       enable_percpu_irq(drvdata->irq, IRQ_TYPE_NONE);
> > > >   }
> > > >   +static void arm_trbe_disable_cpu(void *info)
> > > > +{
> > > > +    struct trbe_drvdata *drvdata = info;
> > > > +    struct trbe_cpudata *cpudata = this_cpu_ptr(drvdata->cpudata);
> > > > +
> > > > +    disable_percpu_irq(drvdata->irq);
> > > > +    trbe_reset_local(cpudata);
> > > > +    cpudata->drvdata = NULL;
> > > > +}
> > > > +
> > > > +
> > > >   static void arm_trbe_register_coresight_cpu(struct trbe_drvdata 
> > > > *drvdata, int cpu)
> > > >   {
> > > >       struct trbe_cpudata *cpudata = per_cpu_ptr(drvdata->cpudata, 
> > > > cpu);
> > > > @@ -1326,18 +1337,12 @@ static void arm_trbe_probe_cpu(void *info)
> > > >       cpumask_clear_cpu(cpu, &drvdata->supported_cpus);
> > > >   }
> > > >   -static void arm_trbe_remove_coresight_cpu(void *info)
> > > > +static void arm_trbe_remove_coresight_cpu(struct trbe_drvdata 
> > > > *drvdata, int cpu)
> > > >   {
> > > > -    int cpu = smp_processor_id();
> > > > -    struct trbe_drvdata *drvdata = info;
> > > > -    struct trbe_cpudata *cpudata = per_cpu_ptr(drvdata->cpudata, 
> > > > cpu);
> > > >       struct coresight_device *trbe_csdev = 
> > > > coresight_get_percpu_sink(cpu);
> > > >   -    disable_percpu_irq(drvdata->irq);
> > > > -    trbe_reset_local(cpudata);
> > > >       if (trbe_csdev) {
> > > >           coresight_unregister(trbe_csdev);
> > > > -        cpudata->drvdata = NULL;
> > > >           coresight_set_percpu_sink(cpu, NULL);
> > >
> > > I am a bit concerned about "resetting" the sink from a different CPU.
> > > Could we instead, schedule a delayed work to unregister the trbe_csdev?
> >
> > Yes, I will try to do that.
> > Sorry for my following questions.
> > As you mean, do we need to take the same care when setting the percpu 
> > sink
> > in the register trbe_csdev ?
>
> Apologies, having taken another look, we set the percpu_sink for
> a cpu outside smp_call_function(). So, I think your patch is fine.
>
>
> > Best regards,
> > Junhao.
> >
> > > >       }
> > > >   }
> > > > @@ -1366,8 +1371,12 @@ static int arm_trbe_remove_coresight(struct 
> > > > trbe_drvdata *drvdata)
> > > >   {
> > > >       int cpu;
> > > >   -    for_each_cpu(cpu, &drvdata->supported_cpus)
> > > > -        smp_call_function_single(cpu, 
> > > > arm_trbe_remove_coresight_cpu, drvdata, 1);
> > > > +    for_each_cpu(cpu, &drvdata->supported_cpus) {
> > > > +        if (cpumask_test_cpu(cpu, &drvdata->supported_cpus))
> > > > +            smp_call_function_single(cpu, arm_trbe_disable_cpu, 
> > > > drvdata, 1);
> > > > +        if (cpumask_test_cpu(cpu, &drvdata->supported_cpus))
> > > > +            arm_trbe_remove_coresight_cpu(drvdata, cpu);
>
> Do we need to test the cpu here in both places ? We already check that
> in the loop entry. The reason why we repeat the check during the probe,
> is to skip any CPUs that may have a TRBE not accessible.
>
> Suzuki

Ok, Will fix in next version.

Best regards,
Junhao.

> > > > +    }
> > > >       free_percpu(drvdata->cpudata);
> > > >       return 0;
> > > >   }
> > > > @@ -1406,12 +1415,8 @@ static int arm_trbe_cpu_teardown(unsigned 
> > > > int cpu, struct hlist_node *node)
> > > >   {
> > > >       struct trbe_drvdata *drvdata = hlist_entry_safe(node, struct 
> > > > trbe_drvdata, hotplug_node);
> > > >   -    if (cpumask_test_cpu(cpu, &drvdata->supported_cpus)) {
> > > > -        struct trbe_cpudata *cpudata = 
> > > > per_cpu_ptr(drvdata->cpudata, cpu);
> > > > -
> > > > -        disable_percpu_irq(drvdata->irq);
> > > > -        trbe_reset_local(cpudata);
> > > > -    }
> > > > +    if (cpumask_test_cpu(cpu, &drvdata->supported_cpus))
> > > > +        arm_trbe_disable_cpu(drvdata);
> > > >       return 0;
> > > >   }
> > >
> > >
> > > .
>
>
> .