Re: debugging rmmod failures

[Stephen Brennan]

On 9/16/22 13:13, Steve French wrote:
> Any suggestions on how to debug why rmmod fails with EBUSY?
>
> I was trying to debug a problem where rmmod fails during an xfstest
> run but couldn't find useful information on how to get ideas 'why'
> rmmod is failing (in this case with EBUSY) and I could only find two
> places where EBUSY would plausibly be returned but don't see the
> pr_debug message in the log (e.g. "already dying") that I would expect
> to get in those cases.
>
> It also fails in this test scenario (which takes a while to reproduce
> so isn't trivial to repro) with EBUSY when doing "rmmod --verbose
> --force"  and the --verbose didn't display any additional info.
>
> I also tried "trace-cmd record -e *module*" which showed it (one call)
> returning 0xFFFFFFF0 but nothing useful that I could see.

In situations like this I sometimes find it useful to use the ftrace
function_graph tracer. It can show you every single function call,
starting from some root function. Of course it's a bit of a footgun, so
be careful with it :)

I used it to trace the delete_module system call like so:

# trace-cmd record -p function_graph -g __x64_sys_delete_module \
                   -F rmmod $MODULE_TO_REMOVE
...
# trace-cmd report
           rmmod-41656 [007]  5506.963846: funcgraph_entry:                   |  __x64_sys_delete_module() {
           rmmod-41656 [007]  5506.963846: funcgraph_entry:                   |    capable() {
           rmmod-41656 [007]  5506.963846: funcgraph_entry:                   |      security_capable() {
...                                                                                ...
           rmmod-41656 [007]  5506.965521: funcgraph_entry:                   |    free_module() {
           rmmod-41656 [007]  5506.965521: funcgraph_entry:                   |      mod_sysfs_teardown() {
           rmmod-41656 [007]  5506.965522: funcgraph_entry:                   |        mutex_lock() {


Of course, my module teardown succeeded, but you should be able to
cross reference your function calls with the source code to see where
an error may have occurred.

Regards,
Stephen

> Any ideas on how to debug *why* an rmmod fails?