Re: [PATCH RFC 00/37] Add support for arm64 MTE dynamic tag storage reuse
From: Kuan-Ying Lee (李冠穎)
Date: Wed Sep 13 2023 - 04:12:25 EST
On Wed, 2023-08-23 at 14:13 +0100, Alexandru Elisei wrote:
> Introduction
> ============
>
> Arm has implemented memory coloring in hardware, and the feature is
> called
> Memory Tagging Extensions (MTE). It works by embedding a 4 bit tag in
> bits
> 59..56 of a pointer, and storing this tag to a reserved memory
> location.
> When the pointer is dereferenced, the hardware compares the tag
> embedded in
> the pointer (logical tag) with the tag stored in memory (allocation
> tag).
>
> The relation between memory and where the tag for that memory is
> stored is
> static.
>
> The memory where the tags are stored have been so far unaccessible to
> Linux.
> This series aims to change that, by adding support for using the tag
> storage
> memory only as data memory; tag storage memory cannot be itself
> tagged.
>
>
> Implementation
> ==============
>
> The series is based on v6.5-rc3 with these two patches cherry picked:
>
> - mm: Call arch_swap_restore() from unuse_pte():
>
>
> https://lore.kernel.org/all/20230523004312.1807357-3-pcc@xxxxxxxxxx/
>
> - arm64: mte: Simplify swap tag restoration logic:
>
>
> https://lore.kernel.org/all/20230523004312.1807357-4-pcc@xxxxxxxxxx/
>
> The above two patches are queued for the v6.6 merge window:
>
>
> https://lore.kernel.org/all/20230702123821.04e64ea2c04dd0fdc947bda3@xxxxxxxxxxxxxxxxxxxx/
>
> The entire series, including the above patches, can be cloned with:
>
> $ git clone https://gitlab.arm.com/linux-arm/linux-ae.git \
> -b arm-mte-dynamic-carveout-rfc-v1
>
> On the arm64 architecture side, an extension is being worked on that
> will
> clarify how MTE tag storage reuse should behave. The extension will
> be
> made public soon.
>
> On the Linux side, MTE tag storage reuse is accomplished with the
> following changes:
>
> 1. The tag storage memory is exposed to the memory allocator as a new
> migratetype, MIGRATE_METADATA. It behaves similarly to MIGRATE_CMA,
> with
> the restriction that it cannot be used to allocate tagged memory (tag
> storage memory cannot be tagged). On tagged page allocation, the
> corresponding tag storage is reserved via alloc_contig_range().
>
> 2. mprotect(PROT_MTE) is implemented by changing the pte prot to
> PAGE_METADATA_NONE. When the page is next accessed, a fault is taken
> and
> the corresponding tag storage is reserved.
>
> 3. When the code tries to copy tags to a page which doesn't have the
> tag
> storage reserved, the tags are copied to an xarray and restored in
> set_pte_at(), when the page is eventually mapped with the tag storage
> reserved.
>
> KVM support has not been implemented yet, that because a non-MTE
> enabled VMA
> can back the memory of an MTE-enabled VM. After there is a consensus
> on the
> right approach on the memory management support, I will add it.
>
> Explanations for the last two changes follow. The gist of it is that
> they
> were added mostly because of races, and it my intention to make the
> code
> more robust.
>
> PAGE_METADATA_NONE was introduced to avoid races with
> mprotect(PROT_MTE).
> For example, migration can race with mprotect(PROT_MTE):
> - thread 0 initiates migration for a page in a non-MTE enabled VMA
> and a
> destination page is allocated without tag storage.
> - thread 1 handles an mprotect(PROT_MTE), the VMA becomes tagged, and
> an
> access turns the source page that is in the process of being
> migrated
> into a tagged page.
> - thread 0 finishes migration and the destination page is mapped as
> tagged,
> but without tag storage reserved.
> More details and examples can be found in the patches.
>
> This race is also related to how tag restoring is handled when tag
> storage
> is missing: when a tagged page is swapped out, the tags are saved in
> an
> xarray indexed by swp_entry.val. When a page is swapped back in, if
> there
> are tags corresponding to the swp_entry that the page will replace,
> the
> tags are unconditionally restored, even if the page will be mapped as
> untagged. Because the page will be mapped as untagged, tag storage
> was
> not reserved when the page was allocated to replace the swp_entry
> which has
> tags associated with it.
>
> To get around this, save the tags in a new xarray, this time indexed
> by
> pfn, and restore them when the same page is mapped as tagged.
>
> This also solves another race, this time with copy_highpage. In the
> scenario where migration races with mprotect(PROT_MTE), before the
> page is
> mapped, the contents of the source page is copied to the destination.
> And
> this includes tags, which will be copied to a page with missing tag
> storage, which can to data corruption if the missing tag storage is
> in use
> for data. So copy_highpage() has received a similar treatment to the
> swap
> code, and the source tags are copied in the xarray indexed by the
> destination page pfn.
>
>
> Overview of the patches
> =======================
>
> Patches 1-3 do some preparatory work by renaming a few functions and
> a gfp
> flag.
>
> Patches 4-12 are arch independent and introduce MIGRATE_METADATA to
> the
> page allocator.
>
> Patches 13-18 are arm64 specific and add support for detecting the
> tag
> storage region and onlining it with the MIGRATE_METADATA migratetype.
>
> Patches 19-24 are arch independent and modify the page allocator to
> callback into arch dependant functions to reserve metadata storage
> for an
> allocation which requires metadata.
>
> Patches 25-28 are mostly arm64 specific and implement the reservation
> and
> freeing of tag storage on tagged page allocation. Patch #28 ("mm:
> sched:
> Introduce PF_MEMALLOC_ISOLATE") adds a current flag,
> PF_MEMALLOC_ISOLATE,
> which ignores page isolation limits; this is used by arm64 when
> reserving
> tag storage in the same patch.
>
> Patches 29-30 add arch independent support for doing
> mprotect(PROT_MTE)
> when metadata storage is enabled.
>
> Patches 31-37 are mostly arm64 specific and handle the restoring of
> tags
> when tag storage is missing. The exceptions are patches 32 (adds the
> arch_swap_prepare_to_restore() function) and 35 (add
> PAGE_METADATA_NONE
> support for THPs).
>
> Testing
> =======
>
> To enable MTE dynamic tag storage:
>
> - CONFIG_ARM64_MTE_TAG_STORAGE=y
> - system_supports_mte() returns true
> - kasan_hw_tags_enabled() returns false
> - correct DTB node (for the specification, see commit "arm64: mte:
> Reserve tag
> storage memory")
>
> Check dmesg for the message "MTE tag storage enabled" or grep for
> metadata
> in /proc/vmstat.
>
> I've tested the series using FVP with MTE enabled, but without
> support for
> dynamic tag storage reuse. To simulate it, I've added two fake tag
> storage
> regions in the DTB by splitting a 2GB region roughly into 33 slices
> of size
> 0x3e0_0000, and using 32 of them for tagged memory and one slice for
> tag
> storage:
>
> diff --git a/arch/arm64/boot/dts/arm/fvp-base-revc.dts
> b/arch/arm64/boot/dts/arm/fvp-base-revc.dts
> index 60472d65a355..bd050373d6cf 100644
> --- a/arch/arm64/boot/dts/arm/fvp-base-revc.dts
> +++ b/arch/arm64/boot/dts/arm/fvp-base-revc.dts
> @@ -165,10 +165,28 @@ C1_L2: l2-cache1 {
> };
> };
>
> - memory@80000000 {
> + memory0: memory@80000000 {
> device_type = "memory";
> - reg = <0x00000000 0x80000000 0 0x80000000>,
> - <0x00000008 0x80000000 0 0x80000000>;
> + reg = <0x00 0x80000000 0x00 0x7c000000>;
> + };
> +
> + metadata0: metadata@c0000000 {
> + compatible = "arm,mte-tag-storage";
> + reg = <0x00 0xfc000000 0x00 0x3e00000>;
> + block-size = <0x1000>;
> + memory = <&memory0>;
> + };
> +
> + memory1: memory@880000000 {
> + device_type = "memory";
> + reg = <0x08 0x80000000 0x00 0x7c000000>;
> + };
> +
> + metadata1: metadata@8c0000000 {
> + compatible = "arm,mte-tag-storage";
> + reg = <0x08 0xfc000000 0x00 0x3e00000>;
> + block-size = <0x1000>;
> + memory = <&memory1>;
> };
>
Hi Alexandru,
AFAIK, the above memory configuration means that there are two region
of dram(0x80000000-0xfc000000 and 0x8_80000000-0x8_fc0000000) and this
is called PDD memory map.
Document[1] said there are some constraints of tag memory as below.
| The following constraints apply to the tag regions in DRAM:
| 1. The tag region cannot be interleaved with the data region.
| The tag region must also be above the data region within DRAM.
|
| 2.The tag region in the physical address space cannot straddle
| multiple regions of a memory map.
|
| PDD memory map is not allowed to have part of the tag region between
| 2GB-4GB and another part between 34GB-64GB.
I'm not sure if we can separate tag memory with the above
configuration. Or do I miss something?
[1] https://developer.arm.com/documentation/101569/0300/?lang=en
(Section 5.4.6.1)
Thanks,
Kuan-Ying Lee
> reserved-memory {
>
>
> Alexandru Elisei (37):
> mm: page_alloc: Rename gfp_to_alloc_flags_cma ->
> gfp_to_alloc_flags_fast
> arm64: mte: Rework naming for tag manipulation functions
> arm64: mte: Rename __GFP_ZEROTAGS to __GFP_TAGGED
> mm: Add MIGRATE_METADATA allocation policy
> mm: Add memory statistics for the MIGRATE_METADATA allocation
> policy
> mm: page_alloc: Allocate from movable pcp lists only if
> ALLOC_FROM_METADATA
> mm: page_alloc: Bypass pcp when freeing MIGRATE_METADATA pages
> mm: compaction: Account for free metadata pages in
> __compact_finished()
> mm: compaction: Handle metadata pages as source for direct
> compaction
> mm: compaction: Do not use MIGRATE_METADATA to replace pages with
> metadata
> mm: migrate/mempolicy: Allocate metadata-enabled destination page
> mm: gup: Don't allow longterm pinning of MIGRATE_METADATA pages
> arm64: mte: Reserve tag storage memory
> arm64: mte: Expose tag storage pages to the MIGRATE_METADATA
> freelist
> arm64: mte: Make tag storage depend on ARCH_KEEP_MEMBLOCK
> arm64: mte: Move tag storage to MIGRATE_MOVABLE when MTE is
> disabled
> arm64: mte: Disable dynamic tag storage management if HW KASAN is
> enabled
> arm64: mte: Check that tag storage blocks are in the same zone
> mm: page_alloc: Manage metadata storage on page allocation
> mm: compaction: Reserve metadata storage in compaction_alloc()
> mm: khugepaged: Handle metadata-enabled VMAs
> mm: shmem: Allocate metadata storage for in-memory filesystems
> mm: Teach vma_alloc_folio() about metadata-enabled VMAs
> mm: page_alloc: Teach alloc_contig_range() about MIGRATE_METADATA
> arm64: mte: Manage tag storage on page allocation
> arm64: mte: Perform CMOs for tag blocks on tagged page
> allocation/free
> arm64: mte: Reserve tag block for the zero page
> mm: sched: Introduce PF_MEMALLOC_ISOLATE
> mm: arm64: Define the PAGE_METADATA_NONE page protection
> mm: mprotect: arm64: Set PAGE_METADATA_NONE for mprotect(PROT_MTE)
> mm: arm64: Set PAGE_METADATA_NONE in set_pte_at() if missing
> metadata
> storage
> mm: Call arch_swap_prepare_to_restore() before arch_swap_restore()
> arm64: mte: swap/copypage: Handle tag restoring when missing tag
> storage
> arm64: mte: Handle fatal signal in reserve_metadata_storage()
> mm: hugepage: Handle PAGE_METADATA_NONE faults for huge pages
> KVM: arm64: Disable MTE is tag storage is enabled
> arm64: mte: Enable tag storage management
>
> arch/arm64/Kconfig | 13 +
> arch/arm64/include/asm/assembler.h | 10 +
> arch/arm64/include/asm/memory_metadata.h | 49 ++
> arch/arm64/include/asm/mte-def.h | 16 +-
> arch/arm64/include/asm/mte.h | 40 +-
> arch/arm64/include/asm/mte_tag_storage.h | 36 ++
> arch/arm64/include/asm/page.h | 5 +-
> arch/arm64/include/asm/pgtable-prot.h | 2 +
> arch/arm64/include/asm/pgtable.h | 33 +-
> arch/arm64/kernel/Makefile | 1 +
> arch/arm64/kernel/elfcore.c | 14 +-
> arch/arm64/kernel/hibernate.c | 46 +-
> arch/arm64/kernel/mte.c | 31 +-
> arch/arm64/kernel/mte_tag_storage.c | 667
> +++++++++++++++++++++++
> arch/arm64/kernel/setup.c | 7 +
> arch/arm64/kvm/arm.c | 6 +-
> arch/arm64/lib/mte.S | 30 +-
> arch/arm64/mm/copypage.c | 26 +
> arch/arm64/mm/fault.c | 35 +-
> arch/arm64/mm/mteswap.c | 113 +++-
> fs/proc/meminfo.c | 8 +
> fs/proc/page.c | 1 +
> include/asm-generic/Kbuild | 1 +
> include/asm-generic/memory_metadata.h | 50 ++
> include/linux/gfp.h | 10 +
> include/linux/gfp_types.h | 14 +-
> include/linux/huge_mm.h | 6 +
> include/linux/kernel-page-flags.h | 1 +
> include/linux/migrate_mode.h | 1 +
> include/linux/mm.h | 12 +-
> include/linux/mmzone.h | 26 +-
> include/linux/page-flags.h | 1 +
> include/linux/pgtable.h | 19 +
> include/linux/sched.h | 2 +-
> include/linux/sched/mm.h | 13 +
> include/linux/vm_event_item.h | 5 +
> include/linux/vmstat.h | 2 +
> include/trace/events/mmflags.h | 5 +-
> mm/Kconfig | 5 +
> mm/compaction.c | 52 +-
> mm/huge_memory.c | 109 ++++
> mm/internal.h | 7 +
> mm/khugepaged.c | 7 +
> mm/memory.c | 180 +++++-
> mm/mempolicy.c | 7 +
> mm/migrate.c | 6 +
> mm/mm_init.c | 23 +-
> mm/mprotect.c | 46 ++
> mm/page_alloc.c | 136 ++++-
> mm/page_isolation.c | 19 +-
> mm/page_owner.c | 3 +-
> mm/shmem.c | 14 +-
> mm/show_mem.c | 4 +
> mm/swapfile.c | 4 +
> mm/vmscan.c | 3 +
> mm/vmstat.c | 13 +-
> 56 files changed, 1834 insertions(+), 161 deletions(-)
> create mode 100644 arch/arm64/include/asm/memory_metadata.h
> create mode 100644 arch/arm64/include/asm/mte_tag_storage.h
> create mode 100644 arch/arm64/kernel/mte_tag_storage.c
> create mode 100644 include/asm-generic/memory_metadata.h
>