Re: [PATCH v1] mm/memory_hotplug: Don't take the cpu_hotplug_lock

[David Hildenbrand]

On 25.09.19 20:20, Qian Cai wrote:
> On Wed, 2019-09-25 at 19:48 +0200, Michal Hocko wrote:
>> On Wed 25-09-19 12:01:02, Qian Cai wrote:
>>> On Wed, 2019-09-25 at 09:02 +0200, David Hildenbrand wrote:
>>>> On 24.09.19 20:54, Qian Cai wrote:
>>>>> On Tue, 2019-09-24 at 17:11 +0200, Michal Hocko wrote:
>>>>>> On Tue 24-09-19 11:03:21, Qian Cai wrote:
>>>>>> [...]
>>>>>>> While at it, it might be a good time to rethink the whole locking over there, as
>>>>>>> it right now read files under /sys/kernel/slab/ could trigger a possible
>>>>>>> deadlock anyway.
>>>>>>>
>>>>>>
>>>>>> [...]
>>>>>>> [ÂÂ442.452090][ T5224] -> #0 (mem_hotplug_lock.rw_sem){++++}:
>>>>>>> [ÂÂ442.459748][ T5224]ÂÂÂÂÂÂÂÂvalidate_chain+0xd10/0x2bcc
>>>>>>> [ÂÂ442.464883][ T5224]ÂÂÂÂÂÂÂÂ__lock_acquire+0x7f4/0xb8c
>>>>>>> [ÂÂ442.469930][ T5224]ÂÂÂÂÂÂÂÂlock_acquire+0x31c/0x360
>>>>>>> [ÂÂ442.474803][ T5224]ÂÂÂÂÂÂÂÂget_online_mems+0x54/0x150
>>>>>>> [ÂÂ442.479850][ T5224]ÂÂÂÂÂÂÂÂshow_slab_objects+0x94/0x3a8
>>>>>>> [ÂÂ442.485072][ T5224]ÂÂÂÂÂÂÂÂtotal_objects_show+0x28/0x34
>>>>>>> [ÂÂ442.490292][ T5224]ÂÂÂÂÂÂÂÂslab_attr_show+0x38/0x54
>>>>>>> [ÂÂ442.495166][ T5224]ÂÂÂÂÂÂÂÂsysfs_kf_seq_show+0x198/0x2d4
>>>>>>> [ÂÂ442.500473][ T5224]ÂÂÂÂÂÂÂÂkernfs_seq_show+0xa4/0xcc
>>>>>>> [ÂÂ442.505433][ T5224]ÂÂÂÂÂÂÂÂseq_read+0x30c/0x8a8
>>>>>>> [ÂÂ442.509958][ T5224]ÂÂÂÂÂÂÂÂkernfs_fop_read+0xa8/0x314
>>>>>>> [ÂÂ442.515007][ T5224]ÂÂÂÂÂÂÂÂ__vfs_read+0x88/0x20c
>>>>>>> [ÂÂ442.519620][ T5224]ÂÂÂÂÂÂÂÂvfs_read+0xd8/0x10c
>>>>>>> [ÂÂ442.524060][ T5224]ÂÂÂÂÂÂÂÂksys_read+0xb0/0x120
>>>>>>> [ÂÂ442.528586][ T5224]ÂÂÂÂÂÂÂÂ__arm64_sys_read+0x54/0x88
>>>>>>> [ÂÂ442.533634][ T5224]ÂÂÂÂÂÂÂÂel0_svc_handler+0x170/0x240
>>>>>>> [ÂÂ442.538768][ T5224]ÂÂÂÂÂÂÂÂel0_svc+0x8/0xc
>>>>>>
>>>>>> I believe the lock is not really needed here. We do not deallocated
>>>>>> pgdat of a hotremoved node nor destroy the slab state because an
>>>>>> existing slabs would prevent hotremove to continue in the first place.
>>>>>>
>>>>>> There are likely details to be checked of course but the lock just seems
>>>>>> bogus.
>>>>>
>>>>> Check 03afc0e25f7f ("slab: get_online_mems for
>>>>> kmem_cache_{create,destroy,shrink}"). It actually talk about the races during
>>>>> memory as well cpu hotplug, so it might even that cpu_hotplug_lock removal is
>>>>> problematic?
>>>>>
>>>>
>>>> Which removal are you referring to? get_online_mems() does not mess with
>>>> the cpu hotplug lock (and therefore this patch).
>>>
>>> The one in your patch. I suspect there might be races among the whole NUMA node
>>> hotplug, kmem_cache_create, and show_slab_objects(). See bfc8c90139eb ("mem-
>>> hotplug: implement get/put_online_mems")
>>>
>>> "kmem_cache_{create,destroy,shrink} need to get a stable value of cpu/node
>>> online mask, because they init/destroy/access per-cpu/node kmem_cache parts,
>>> which can be allocated or destroyed on cpu/mem hotplug."
>>
>> I still have to grasp that code but if the slub allocator really needs
>> a stable cpu mask then it should be using the explicit cpu hotplug
>> locking rather than rely on side effect of memory hotplug locking.
>>
>>> Both online_pages() and show_slab_objects() need to get a stable value of
>>> cpu/node online mask.
>>
>> Could tou be more specific why online_pages need a stable cpu online
>> mask? I do not think that show_slab_objects is a real problem because a
>> potential race shouldn't be critical.
> 
> build_all_zonelists()
>   __build_all_zonelists()
>     for_each_online_cpu(cpu)
> 

Two things:

a) We currently always hold the device hotplug lock when onlining memory
and when onlining cpus (for CPUs at least via user space - we would have
to double check other call paths). So theoretically, that should guard
us from something like that already.

b)

commit 11cd8638c37f6c400cc472cc52b6eccb505aba6e
Author: Michal Hocko <mhocko@xxxxxxxx>
Date:   Wed Sep 6 16:20:34 2017 -0700

    mm, page_alloc: remove stop_machine from build_all_zonelists

Tells me:

"Updates of the zonelists happen very seldom, basically only when a zone
 becomes populated during memory online or when it loses all the memory
 during offline.  A racing iteration over zonelists could either miss a
 zone or try to work on one zone twice.  Both of these are something we
 can live with occasionally because there will always be at least one
 zone visible so we are not likely to fail allocation too easily for
 example."

Sounds like if there would be a race, we could live with it if I am not
getting that totally wrong.

-- 

Thanks,

David / dhildenb