RE: [PATCH v8 1/8] mm/memfd: Introduce userspace inaccessible memfd

[Wang, Wei W]

On Thursday, September 22, 2022 5:11 AM, Andy Lutomirski wrote:
> To: Christopherson,, Sean <seanjc@xxxxxxxxxx>; David Hildenbrand
> <david@xxxxxxxxxx>
> Cc: Chao Peng <chao.p.peng@xxxxxxxxxxxxxxx>; kvm list
> <kvm@xxxxxxxxxxxxxxx>; Linux Kernel Mailing List
> <linux-kernel@xxxxxxxxxxxxxxx>; linux-mm@xxxxxxxxx;
> linux-fsdevel@xxxxxxxxxxxxxxx; Linux API <linux-api@xxxxxxxxxxxxxxx>;
> linux-doc@xxxxxxxxxxxxxxx; qemu-devel@xxxxxxxxxx; Paolo Bonzini
> <pbonzini@xxxxxxxxxx>; Jonathan Corbet <corbet@xxxxxxx>; Vitaly
> Kuznetsov <vkuznets@xxxxxxxxxx>; Wanpeng Li <wanpengli@xxxxxxxxxxx>;
> Jim Mattson <jmattson@xxxxxxxxxx>; Joerg Roedel <joro@xxxxxxxxxx>;
> Thomas Gleixner <tglx@xxxxxxxxxxxxx>; Ingo Molnar <mingo@xxxxxxxxxx>;
> Borislav Petkov <bp@xxxxxxxxx>; the arch/x86 maintainers <x86@xxxxxxxxxx>;
> H. Peter Anvin <hpa@xxxxxxxxx>; Hugh Dickins <hughd@xxxxxxxxxx>; Jeff
> Layton <jlayton@xxxxxxxxxx>; J . Bruce Fields <bfields@xxxxxxxxxxxx>; Andrew
> Morton <akpm@xxxxxxxxxxxxxxxxxxxx>; Shuah Khan <shuah@xxxxxxxxxx>;
> Mike Rapoport <rppt@xxxxxxxxxx>; Steven Price <steven.price@xxxxxxx>;
> Maciej S . Szmigiero <mail@xxxxxxxxxxxxxxxxxxxxx>; Vlastimil Babka
> <vbabka@xxxxxxx>; Vishal Annapurve <vannapurve@xxxxxxxxxx>; Yu Zhang
> <yu.c.zhang@xxxxxxxxxxxxxxx>; Kirill A. Shutemov
> <kirill.shutemov@xxxxxxxxxxxxxxx>; Nakajima, Jun <jun.nakajima@xxxxxxxxx>;
> Hansen, Dave <dave.hansen@xxxxxxxxx>; Andi Kleen <ak@xxxxxxxxxxxxxxx>;
> aarcange@xxxxxxxxxx; ddutile@xxxxxxxxxx; dhildenb@xxxxxxxxxx; Quentin
> Perret <qperret@xxxxxxxxxx>; Michael Roth <michael.roth@xxxxxxx>;
> Hocko, Michal <mhocko@xxxxxxxx>; Muchun Song
> <songmuchun@xxxxxxxxxxxxx>; Wang, Wei W <wei.w.wang@xxxxxxxxx>;
> Will Deacon <will@xxxxxxxxxx>; Marc Zyngier <maz@xxxxxxxxxx>; Fuad Tabba
> <tabba@xxxxxxxxxx>
> Subject: Re: [PATCH v8 1/8] mm/memfd: Introduce userspace inaccessible
> memfd
> 
> (please excuse any formatting disasters.  my internet went out as I was
> composing this, and i did my best to rescue it.)
> 
> On Mon, Sep 19, 2022, at 12:10 PM, Sean Christopherson wrote:
> > +Will, Marc and Fuad (apologies if I missed other pKVM folks)
> >
> > On Mon, Sep 19, 2022, David Hildenbrand wrote:
> >> On 15.09.22 16:29, Chao Peng wrote:
> >> > From: "Kirill A. Shutemov" <kirill.shutemov@xxxxxxxxxxxxxxx>
> >> >
> >> > KVM can use memfd-provided memory for guest memory. For normal
> >> > userspace accessible memory, KVM userspace (e.g. QEMU) mmaps the
> >> > memfd into its virtual address space and then tells KVM to use the
> >> > virtual address to setup the mapping in the secondary page table (e.g.
> EPT).
> >> >
> >> > With confidential computing technologies like Intel TDX, the
> >> > memfd-provided memory may be encrypted with special key for special
> >> > software domain (e.g. KVM guest) and is not expected to be directly
> >> > accessed by userspace. Precisely, userspace access to such
> >> > encrypted memory may lead to host crash so it should be prevented.
> >>
> >> Initially my thaught was that this whole inaccessible thing is TDX
> >> specific and there is no need to force that on other mechanisms.
> >> That's why I suggested to not expose this to user space but handle
> >> the notifier requirements internally.
> >>
> >> IIUC now, protected KVM has similar demands. Either access
> >> (read/write) of guest RAM would result in a fault and possibly crash
> >> the hypervisor (at least not the whole machine IIUC).
> >
> > Yep.  The missing piece for pKVM is the ability to convert from shared
> > to private while preserving the contents, e.g. to hand off a large
> > buffer (hundreds of MiB) for processing in the protected VM.  Thoughts
> > on this at the bottom.
> >
> >> > This patch introduces userspace inaccessible memfd (created with
> >> > MFD_INACCESSIBLE). Its memory is inaccessible from userspace
> >> > through ordinary MMU access (e.g. read/write/mmap) but can be
> >> > accessed via in-kernel interface so KVM can directly interact with
> >> > core-mm without the need to map the memory into KVM userspace.
> >>
> >> With secretmem we decided to not add such "concept switch" flags and
> >> instead use a dedicated syscall.
> >>
> >
> > I have no personal preference whatsoever between a flag and a
> > dedicated syscall, but a dedicated syscall does seem like it would
> > give the kernel a bit more flexibility.
> 
> The third option is a device node, e.g. /dev/kvm_secretmem or
> /dev/kvm_tdxmem or similar.  But if we need flags or other details in the
> future, maybe this isn't ideal.
> 
> >
> >> What about memfd_inaccessible()? Especially, sealing and hugetlb are
> >> not even supported and it might take a while to support either.
> >
> > Don't know about sealing, but hugetlb support for "inaccessible"
> > memory needs to come sooner than later.  "inaccessible" in quotes
> > because we might want to choose a less binary name, e.g.
> > "restricted"?.
> >
> > Regarding pKVM's use case, with the shim approach I believe this can
> > be done by allowing userspace mmap() the "hidden" memfd, but with a
> > ton of restrictions piled on top.
> >
> > My first thought was to make the uAPI a set of KVM ioctls so that KVM
> > could tightly tightly control usage without taking on too much
> > complexity in the kernel, but working through things, routing the
> > behavior through the shim itself might not be all that horrific.
> >
> > IIRC, we discarded the idea of allowing userspace to map the "private"
> > fd because
> > things got too complex, but with the shim it doesn't seem _that_ bad.
> 
> What's the exact use case?  Is it just to pre-populate the memory?

Add one more use case here. For TDX live migration support, on the destination side,
we map the private fd during migration to store the encrypted private memory data sent
from source, and at the end of migration, we unmap it and make it inaccessible before
resuming the TD to run.