Re: [PATCH v1 05/29] virtio-mem: generalize check for added memory

[David Hildenbrand]

On 15.10.20 10:28, Wei Yang wrote:
> On Mon, Oct 12, 2020 at 02:52:59PM +0200, David Hildenbrand wrote:
>> Let's check by traversing busy system RAM resources instead, to avoid
>> relying on memory block states.
>>
>> Don't use walk_system_ram_range(), as that works on pages and we want to
>> use the bare addresses we have easily at hand.
>>
>> Cc: "Michael S. Tsirkin" <mst@xxxxxxxxxx>
>> Cc: Jason Wang <jasowang@xxxxxxxxxx>
>> Cc: Pankaj Gupta <pankaj.gupta.linux@xxxxxxxxx>
>> Signed-off-by: David Hildenbrand <david@xxxxxxxxxx>
>> ---
>> drivers/virtio/virtio_mem.c | 19 +++++++++++++++----
>> 1 file changed, 15 insertions(+), 4 deletions(-)
>>
>> diff --git a/drivers/virtio/virtio_mem.c b/drivers/virtio/virtio_mem.c
>> index b3eebac7191f..6bbd1cfd10d3 100644
>> --- a/drivers/virtio/virtio_mem.c
>> +++ b/drivers/virtio/virtio_mem.c
>> @@ -1749,6 +1749,20 @@ static void virtio_mem_delete_resource(struct virtio_mem *vm)
>> 	vm->parent_resource = NULL;
>> }
>>
>> +static int virtio_mem_range_has_system_ram(struct resource *res, void *arg)
>> +{
>> +	return 1;
>> +}
>> +
>> +static bool virtio_mem_has_memory_added(struct virtio_mem *vm)
>> +{
>> +	const unsigned long flags = IORESOURCE_SYSTEM_RAM | IORESOURCE_BUSY;
>> +
>> +	return walk_iomem_res_desc(IORES_DESC_NONE, flags, vm->addr,
>> +				   vm->addr + vm->region_size, NULL,
>> +				   virtio_mem_range_has_system_ram) == 1;
>> +}
>> +
>> static int virtio_mem_probe(struct virtio_device *vdev)
>> {
>> 	struct virtio_mem *vm;
>> @@ -1870,10 +1884,7 @@ static void virtio_mem_remove(struct virtio_device *vdev)
>> 	 * the system. And there is no way to stop the driver/device from going
>> 	 * away. Warn at least.
>> 	 */
>> -	if (vm->nb_mb_state[VIRTIO_MEM_MB_STATE_OFFLINE] ||
>> -	    vm->nb_mb_state[VIRTIO_MEM_MB_STATE_OFFLINE_PARTIAL] ||
>> -	    vm->nb_mb_state[VIRTIO_MEM_MB_STATE_ONLINE] ||
>> -	    vm->nb_mb_state[VIRTIO_MEM_MB_STATE_ONLINE_PARTIAL]) {
>> +	if (virtio_mem_has_memory_added(vm)) {
> 
> I am not sure this would be more efficient.

In general, no. However, this is a preparation for Big Block Mode, which
won't have memory block states.

(this path only triggers when unloading the driver - which most probably
only ever happens during my testing ... :) and we don't really care
about performance there)

> 
>> 		dev_warn(&vdev->dev, "device still has system memory added\n");
>> 	} else {
>> 		virtio_mem_delete_resource(vm);
> 
> BTW, I got one question during review.
> 
> Per my understanding, there are 4 states of a virtio memory block
> 
>   * OFFLINE[_PARTIAL]
>   * ONLINE[_PARTIAL]
> 
> While, if my understanding is correct, those two offline states are transient.
> If the required range is onlined, the state would be change to
> ONLINE[_PARTIAL] respectively. If it is not, the state is reverted to UNUSED
> or PLUGGED.

Very right.

> 
> What I am lost is why you do virtio_mem_mb_remove() on OFFLINE_PARTIAL memory
> block? Since we wait for the workqueue finish its job.

That's an interesting corner case. Assume you have a 128MB memory block
but only 64MB are plugged.

As long as we have our online_pages callback in place, we can hinder the
unplugged 64MB from getting exposed to the buddy
(virtio_mem_online_page_cb()). However, once we unloaded the driver,
this is no longer the case. If someone would online that memory block,
we would expose unplugged memory to the buddy - very bad.

So we have to remove these partially plugged, offline memory blocks when
losing control over them.

I tried to document that via:

"After we unregistered our callbacks, user space can online partially
plugged offline blocks. Make sure to remove them."

> 
> Also, during virtio_mem_remove(), we just handle OFFLINE_PARTIAL memory block.
> How about memory block in other states? It is not necessary to remove
> ONLINE[_PARTIAL] memroy blocks?

Blocks that are fully plugged (ONLINE or OFFLINE) can get
onlined/offlined without us having to care. Works fine - we only have to
care about partially plugged blocks.

While we *could* unplug OFFLINE blocks, there is no way we can
deterministically offline+remove ONLINE blocks. So that memory has to
stay, even after we unloaded the driver (similar to the dax/kmem driver).

ONLINE_PARTIAL is already taken care of: it cannot get offlined anymore,
as we still hold references to these struct pages
(virtio_mem_set_fake_offline()), and as we no longer have the memory
notifier in place, we can no longer agree to offline this memory (when
going_offline).

I tried to document that via

"After we unregistered our callbacks, user space can no longer offline
partially plugged online memory blocks. No need to worry about them."


> 
> Thanks in advance, since I may missed some concepts.

(force) driver unloading is a complicated corner case.

Thanks!

-- 
Thanks,

David / dhildenb