Re: [PATCH v12 00/12]

[Petr Mladek]

On Thu 2018-08-30 13:58:15, Miroslav Benes wrote:
> On Tue, 28 Aug 2018, Petr Mladek wrote:
> 
> > livepatch: Atomic replace feature
> > 
> > The atomic replace allows to create cumulative patches. They
> > are useful when you maintain many livepatches and want to remove
> > one that is lower on the stack. In addition it is very useful when
> > more patches touch the same function and there are dependencies
> > between them.
> > 
> > This version does another big refactoring based on feedback against
> > v11[*]. In particular, it removes the registration step, changes
> > the API and handling of livepatch dependencies. The aim is
> > to keep the number of possible variants on a sane level.
> > It helps the keep the feature "easy" to use and maintain.
> > 
> > [*] https://lkml.kernel.org/r/20180323120028.31451-1-pmladek@xxxxxxxx
> 
> Hi,
> 
> I've started to review the patch set. Running selftests with lockdep 
> enabled gives me...
> 
> ======================================================
> WARNING: possible circular locking dependency detected
> 4.17.0-rc1-klp_replace_v12-117114-gfedb3eba611d #218 Tainted: G              
> K  
> ------------------------------------------------------
> kworker/1:1/49 is trying to acquire lock:
> 00000000bb88dc17 (kn->count#186){++++}, at: kernfs_remove+0x23/0x40
> 
> but task is already holding lock:
> 0000000073632424 (klp_mutex){+.+.}, at: klp_transition_work_fn+0x17/0x40
> 
> which lock already depends on the new lock.
> 
> 
> the existing dependency chain (in reverse order) is:
> 
> -> #1 (klp_mutex){+.+.}:
>        lock_acquire+0xd4/0x220
>        __mutex_lock+0x75/0x920
>        mutex_lock_nested+0x1b/0x20
>        enabled_store+0x47/0x150
>        kobj_attr_store+0x12/0x20
>        sysfs_kf_write+0x4a/0x60
>        kernfs_fop_write+0x123/0x1b0
>        __vfs_write+0x2b/0x150
>        vfs_write+0xc7/0x1c0
>        ksys_write+0x49/0xa0
>        __x64_sys_write+0x1a/0x20
>        do_syscall_64+0x62/0x1b0
>        entry_SYSCALL_64_after_hwframe+0x49/0xbe
> 
> -> #0 (kn->count#186){++++}:
>        __lock_acquire+0xe9d/0x1240
>        lock_acquire+0xd4/0x220
>        __kernfs_remove+0x23c/0x2c0
>        kernfs_remove+0x23/0x40
>        sysfs_remove_dir+0x51/0x60
>        kobject_del+0x18/0x50
>        kobject_cleanup+0x4b/0x180
>        kobject_put+0x2a/0x50
>        __klp_free_patch+0x5b/0x60
>        klp_free_patch_nowait+0x12/0x30
>        klp_try_complete_transition+0x13e/0x180
>        klp_transition_work_fn+0x26/0x40
>        process_one_work+0x1d8/0x5d0
>        worker_thread+0x4d/0x3d0
>        kthread+0x113/0x150
>        ret_from_fork+0x3a/0x50
> 
> other info that might help us debug this:
> 
>  Possible unsafe locking scenario:
> 
>        CPU0                    CPU1
>        ----                    ----
>   lock(klp_mutex);
>                                lock(kn->count#186);
>                                lock(klp_mutex);
>   lock(kn->count#186);

Sigh, I overestimated the power of kobjects. I thought that this
must have been a false positive but it was not.

1. kernfs_fop_write() ignores kobj->kref. It takes care only
   of its own reference count, see kernfs_get_active().

2. kobj_put() takes care only of kobj->kref. The following
   code is called when the reference count reaches zero:

   + kobj_put()
     + kref_put()
       + kobject_release()
         + kobject_cleanup()
           + kobject_del()
	     + sysfs_remove_dir()
	       + kernfs_remove()
	         + __kernfs_remove().
		   + kernfs_drain()

    , where kernfs_drain() waits until all opened files
    are closed.

Now, we call kobj_put() under klp_mutex() when the sysfs
interface still exists. Files can be opened for
writing. As a result:

   + enabled_store() might wait for klp_mutex

   + kernfs_drain() would wait for enabled_store()
     with klp_mutex() taken.


I have reproduced this with some extra sleeps.

I am going to work on another solution.

Best Regards,
Petr