Re: AMD SME encrpytion and PCI BAR pages to user space

[Tom Lendacky]

On 10/19/20 10:25 AM, Jason Gunthorpe wrote:
> Hi Tom,

Hi Jason,

> 
> We've found a bug where systems that have the AMD SME turned on are
> not able to run RDMA work loads. It seems the kernel is automatically
> encrypting VMA's pointing at PCI BAR memory created by
> io_remap_pfn_range() - adding a prot_decrypted() causes things to
> start working.
> 
> To me this is surprising, before I go adding random prot_decrypted()
> into the RDMA subsystem can you confirm this is actually how things
> are expected to work?

Yes, currently, the idea is that anything being done in user space is
mapped encrypted.

> 
> Is RDMA missing something? I don't see anything special in VFIO for
> instance and the two are very similar - does VFIO work with SME, eg
> DPDK or something unrelated to virtualization?

If user space is mapping un-encrypted memory, then, yes, it would seem
that there is a gap in the support where the pgprot_decrypted() would be
needed in order to override the protection map.

>      
> Is there a reason not to just add prot_decrypted() to
> io_remap_pfn_range()? Is there use cases where a caller actually wants
> encrypted io memory?

As long as you never have physical memory / ram being mapped in this path,
it seems that applying pgprot_decrypted() would be ok.

> 
> I saw your original patch series edited a few drivers this way, but
> not nearly enough. So I feel like I'm missing something.. Does vfio
> work with SME? I couldn't find any sign of it calling prot_decrypted()
> either?

I haven't tested SME with VFIO/DPDK.

> 
> (BTW, I don't have any AMD SME systems to test on here, I'm getting
>  this bug report from deployed system, running a distro kernel)

As a work around, if the system has support for TSME (transparent SME),
then that can be enabled (it is a BIOS option that the BIOS vendor would
have had to expose) to encrypt all of the system memory without requiring
SME support.

Thanks,
Tom

> 
> Thanks,
> Jason
>