Re: [PATCH v6 00/15] memory-hotplug: hot-remove physical memory
From: Tang Chen
Date: Wed Jan 30 2013 - 05:16:16 EST
Hi Simon,
Please see below. :)
On 01/29/2013 08:52 PM, Simon Jeons wrote:
Hi Tang,
On Wed, 2013-01-09 at 17:32 +0800, Tang Chen wrote:
Here is the physical memory hot-remove patch-set based on 3.8rc-2.
Some questions ask you, not has relationship with this patchset, but is
memory hotplug stuff.
1. In function node_states_check_changes_online:
comments:
* If we don't have HIGHMEM nor movable node,
* node_states[N_NORMAL_MEMORY] contains nodes which have zones of
* 0...ZONE_MOVABLE, set zone_last to ZONE_MOVABLE.
How to understand it? Why we don't have HIGHMEM nor movable node and
node_staes[N_NORMAL_MEMORY] contains 0...ZONE_MOVABLE, IIUC,
N_NORMAL_MEMORY only means the node has regular memory.
First of all, I think we need to understand why we need N_MEMORY.
In order to support movable node, which has only ZONE_MOVABLE (the last
zone),
we introduce N_MEMORY to represent the node has normal, highmem and
movable memory.
Here, "we have movable node" means you configured CONFIG_MOVABLE_NODE.
This config option doesn't mean we don't have movable pages, (NO)
it means we don't have a node which has only movable pages (only have
ZONE_MOVABLE). (YES)
Here, if we don't have CONFIG_MOVABLE_NODE (we don't have movable node),
we don't need a
separate node_states[] element to represent a particular node because we
won't have a node
which has only ZONE_MOVABLE.
So,
1) if we don't have highmem nor movable node, N_MEMORY == N_HIGH_MEMORY
== N_NORMAL_MEMORY,
which means N_NORMAL_MEMORY effects as N_MEMORY. If we online pages
as movable, we need
to update node_states[N_NORMAL_MEMORY].
Please refer to the definition of enum zone_type, if we don't have
CONFIG_HIGHMEM, we won't
have ZONE_HIGHMEM, but ZONE_NORMAL and ZONE_MOVABLE will always there.
So we can have movable
pages, and the zone_last should be ZONE_MOVABLE.
Again, because we won't have a node only having ZONE_MOVABLE, so we just
need to update
node_states[N_NORMAL_MEMORY].
* If we don't have movable node, node_states[N_NORMAL_MEMORY]
* contains nodes which have zones of 0...ZONE_MOVABLE,
* set zone_last to ZONE_MOVABLE.
How to understand?
2) this code is in #ifdef CONFIG_HIGHMEM, which means we have highmem,
so if we don't have
movable node, N_MEMORY == N_HIGH_MEMORY, and N_HIGH_MEMORY effects
as N_MEMORY. If we
online pages as movable, we need to update node_states[N_NORMAL_MEMORY].
2. In function move_pfn_range_left, why end<= z2->zone_start_pfn is not
correct? The comments said that must include/overlap, why?
This one is easy, if I understand you correctly.
move_pfn_range_left() is used to move the left most part [start_pfn,
end_pfn) of z2 to z1.
So if end_pfn<= z2->zone_start_pfn, it means [start_pfn, end_pfn) is not
part of z2.
Then it fails.
3. In function online_pages, the normal case(w/o online_kenrel,
online_movable), why not check if the new zone is overlap with adjacent
zones?
Can a zone overlap with the others ? I don't think so.
One pfn could only be in one zone,
zone = page_zone(pfn_to_page(pfn));
it could overlap with others, I think. :)
But maybe I misunderstand you. :)
4. Could you summarize the difference implementation between hot-add and
logic-add, hot-remove and logic-remove?
Sorry, I don't quite understand what do you mean by logic-add/remove.
Would you please explain more ?
If you meant the sys fs interfaces, I think they are just another set of
entrances
of memory hotplug.
Thanks. :)
This patch-set aims to implement physical memory hot-removing.
The patches can free/remove the following things:
- /sys/firmware/memmap/X/{end, start, type} : [PATCH 4/15]
- memmap of sparse-vmemmap : [PATCH 6,7,8,10/15]
- page table of removed memory : [RFC PATCH 7,8,10/15]
- node and related sysfs files : [RFC PATCH 13-15/15]
Existing problem:
If CONFIG_MEMCG is selected, we will allocate memory to store page cgroup
when we online pages.
For example: there is a memory device on node 1. The address range
is [1G, 1.5G). You will find 4 new directories memory8, memory9, memory10,
and memory11 under the directory /sys/devices/system/memory/.
If CONFIG_MEMCG is selected, when we online memory8, the memory stored page
cgroup is not provided by this memory device. But when we online memory9, the
memory stored page cgroup may be provided by memory8. So we can't offline
memory8 now. We should offline the memory in the reversed order.
When the memory device is hotremoved, we will auto offline memory provided
by this memory device. But we don't know which memory is onlined first, so
offlining memory may fail.
In patch1, we provide a solution which is not good enough:
Iterate twice to offline the memory.
1st iterate: offline every non primary memory block.
2nd iterate: offline primary (i.e. first added) memory block.
And a new idea from Wen Congyang<wency@xxxxxxxxxxxxxx> is:
allocate the memory from the memory block they are describing.
But we are not sure if it is OK to do so because there is not existing API
to do so, and we need to move page_cgroup memory allocation from MEM_GOING_ONLINE
to MEM_ONLINE. And also, it may interfere the hugepage.
How to test this patchset?
1. apply this patchset and build the kernel. MEMORY_HOTPLUG, MEMORY_HOTREMOVE,
ACPI_HOTPLUG_MEMORY must be selected.
2. load the module acpi_memhotplug
3. hotplug the memory device(it depends on your hardware)
You will see the memory device under the directory /sys/bus/acpi/devices/.
Its name is PNP0C80:XX.
4. online/offline pages provided by this memory device
You can write online/offline to /sys/devices/system/memory/memoryX/state to
online/offline pages provided by this memory device
5. hotremove the memory device
You can hotremove the memory device by the hardware, or writing 1 to
/sys/bus/acpi/devices/PNP0C80:XX/eject.
Is there a similar knode to hot-add the memory device?
Note: if the memory provided by the memory device is used by the kernel, it
can't be offlined. It is not a bug.
Changelogs from v5 to v6:
Patch3: Add some more comments to explain memory hot-remove.
Patch4: Remove bootmem member in struct firmware_map_entry.
Patch6: Repeatedly register bootmem pages when using hugepage.
Patch8: Repeatedly free bootmem pages when using hugepage.
Patch14: Don't free pgdat when offlining a node, just reset it to 0.
Patch15: New patch, pgdat is not freed in patch14, so don't allocate a new
one when online a node.
Changelogs from v4 to v5:
Patch7: new patch, move pgdat_resize_lock into sparse_remove_one_section() to
avoid disabling irq because we need flush tlb when free pagetables.
Patch8: new patch, pick up some common APIs that are used to free direct mapping
and vmemmap pagetables.
Patch9: free direct mapping pagetables on x86_64 arch.
Patch10: free vmemmap pagetables.
Patch11: since freeing memmap with vmemmap has been implemented, the config
macro CONFIG_SPARSEMEM_VMEMMAP when defining __remove_section() is
no longer needed.
Patch13: no need to modify acpi_memory_disable_device() since it was removed,
and add nid parameter when calling remove_memory().
Changelogs from v3 to v4:
Patch7: remove unused codes.
Patch8: fix nr_pages that is passed to free_map_bootmem()
Changelogs from v2 to v3:
Patch9: call sync_global_pgds() if pgd is changed
Patch10: fix a problem int the patch
Changelogs from v1 to v2:
Patch1: new patch, offline memory twice. 1st iterate: offline every non primary
memory block. 2nd iterate: offline primary (i.e. first added) memory
block.
Patch3: new patch, no logical change, just remove reduntant codes.
Patch9: merge the patch from wujianguo into this patch. flush tlb on all cpu
after the pagetable is changed.
Patch12: new patch, free node_data when a node is offlined.
Tang Chen (6):
memory-hotplug: move pgdat_resize_lock into
sparse_remove_one_section()
memory-hotplug: remove page table of x86_64 architecture
memory-hotplug: remove memmap of sparse-vmemmap
memory-hotplug: Integrated __remove_section() of
CONFIG_SPARSEMEM_VMEMMAP.
memory-hotplug: remove sysfs file of node
memory-hotplug: Do not allocate pdgat if it was not freed when
offline.
Wen Congyang (5):
memory-hotplug: try to offline the memory twice to avoid dependence
memory-hotplug: remove redundant codes
memory-hotplug: introduce new function arch_remove_memory() for
removing page table depends on architecture
memory-hotplug: Common APIs to support page tables hot-remove
memory-hotplug: free node_data when a node is offlined
Yasuaki Ishimatsu (4):
memory-hotplug: check whether all memory blocks are offlined or not
when removing memory
memory-hotplug: remove /sys/firmware/memmap/X sysfs
memory-hotplug: implement register_page_bootmem_info_section of
sparse-vmemmap
memory-hotplug: memory_hotplug: clear zone when removing the memory
arch/arm64/mm/mmu.c | 3 +
arch/ia64/mm/discontig.c | 10 +
arch/ia64/mm/init.c | 18 ++
arch/powerpc/mm/init_64.c | 10 +
arch/powerpc/mm/mem.c | 12 +
arch/s390/mm/init.c | 12 +
arch/s390/mm/vmem.c | 10 +
arch/sh/mm/init.c | 17 ++
arch/sparc/mm/init_64.c | 10 +
arch/tile/mm/init.c | 8 +
arch/x86/include/asm/pgtable_types.h | 1 +
arch/x86/mm/init_32.c | 12 +
arch/x86/mm/init_64.c | 390 +++++++++++++++++++++++++++++
arch/x86/mm/pageattr.c | 47 ++--
drivers/acpi/acpi_memhotplug.c | 8 +-
drivers/base/memory.c | 6 +
drivers/firmware/memmap.c | 96 +++++++-
include/linux/bootmem.h | 1 +
include/linux/firmware-map.h | 6 +
include/linux/memory_hotplug.h | 15 +-
include/linux/mm.h | 4 +-
mm/memory_hotplug.c | 459 +++++++++++++++++++++++++++++++---
mm/sparse.c | 8 +-
23 files changed, 1094 insertions(+), 69 deletions(-)
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@xxxxxxxxxx For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email:<a href=mailto:"dont@xxxxxxxxx";> email@xxxxxxxxx</a>
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/