Re: [RFC PATCH v3 0/13] memory-hotplug : hot-remove physical memory

[Yasuaki Ishimatsu]

Hi Jiang,

2012/07/11 9:21, Jiang Liu wrote:
> On 07/11/2012 08:09 AM, Yasuaki Ishimatsu wrote:
> > Hi Jiang,
> >
> > 2012/07/11 1:50, Jiang Liu wrote:
> > > On 07/10/2012 05:58 PM, Yasuaki Ishimatsu wrote:
> > > > Hi Christoph,
> > > >
> > > > 2012/07/10 0:18, Christoph Lameter wrote:
> > > > > On Mon, 9 Jul 2012, Yasuaki Ishimatsu wrote:
> > > > >
> > > > > > Even if you apply these patches, you cannot remove the physical memory
> > > > > > completely since these patches are still under development. I want you to
> > > > > > cooperate to improve the physical memory hot-remove. So please review these
> > > > > > patches and give your comment/idea.
> > > > >
> > > > > Could you at least give a method on how you want to do physical memory
> > > > > removal?
> > > >
> > > > We plan to release a dynamic hardware partitionable system. It will be
> > > > able to hot remove/add a system board which included memory and cpu.
> > > > But as you know, Linux does not support memory hot-remove on x86 box.
> > > > So I try to develop it.
> > > >
> > > > Current plan to hot remove system board is to use container driver.
> > > > Thus I define the system board in ACPI DSDT table as a container device.
> > > > It have supported hot-add a container device. And if container device
> > > > has _EJ0 ACPI method, "eject" file to remove the container device is
> > > > prepared as follow:
> > > >
> > > > # ls -l /sys/bus/acpi/devices/ACPI0004\:01/eject
> > > > --w-------. 1 root root 4096 Jul 10 18:19 /sys/bus/acpi/devices/ACPI0004:01/eject
> > > >
> > > > When I hot-remove the container device, I echo 1 to the file as follow:
> > > >
> > > > #echo 1 > /sys/bus/acpi/devices/ACPI0004\:02/eject
> > > >
> > > > Then acpi_bus_trim() is called. And it calls acpi_memory_device_remove()
> > > > for removing memory device. But the code does not do nothing.
> > > > So I developed the continuation of the function.
> > > >
> > > > > You would have to remove all objects from the range you want to
> > > > > physically remove. That is only possible under special circumstances and
> > > > > with a limited set of objects. Even if you exclusively use ZONE_MOVEABLE
> > > > > you still may get cases where pages are pinned for a long time.
> > > >
> > > > I know it. So my memory hot-remove plan is as follows:
> > > >
> > > > 1. hot-added a system board
> > > >      All memory which included the system board is offline.
> > > >
> > > > 2. online the memory as removable page
> > > >      The function has not supported yet. It is being developed by Lai as follow:
> > > >      http://lkml.indiana.edu/hypermail/linux/kernel/1207.0/01478.html
> > > >      If it is supported, I will be able to create movable memory.
> > > >
> > > > 3. hot-remove the memory by container device's eject file
> > > We have implemented a prototype to do physical node (mem + CPU + IOH) hotplug
> > > for Itanium and is now porting it to x86. But with currently solution, memory
> > > hotplug functionality may cause 10-20% performance decrease because we concentrate
> > > all DMA/Normal memory to the first NUMA node, and all other NUMA nodes only
> > > hosts ZONE_MOVABLE. We are working on solution to minimize the performance
> > > drop now.
> >
> > Thank you for your interesting response.
> >
> > I have a question. How do you move all other NUMA nodes to ZONE_MOVABLE?
> > To use ZONE_MOVABLE, we need to use boot options like kernelcore or movablecore.
> > But it is not enough, since the requested amount is spread evenly throughout
> > all nodes in the system. So I think we do not have way to move all other NUMA
> > node to ZONE_MOVABLE.
> We have modified the ZONE_MOVABLE spreading and bootmem allocation. If the kernelcore
> or movablecore kernel parameters are present, we follow current behavior. If those
> parameter are absent and the platform supports physical hotplug, we will concentrate
> DMA/NORMAL memory to specific nodes.

That's interesting. I want to know more details, if you do not mind.
Current kernel doesn't do the behavior, does it? So I think you have some
patches for changing the behavior. Will you merge these patches into
community kernel?

Thanks,
Yasuaki Ishimatsu

> > Thanks,
> > Yasuaki Ishimatsu
> >
> > > > Thanks,
> > > > Yasuaki Ishimatsu
> > > >
> > > > > I am not sure that these patches are useful unless we know where you are
> > > > > going with this. If we end up with a situation where we still cannot
> > > > > remove physical memory then this patchset is not helpful.



--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/