Re: [PATCH] PCI/P2PDMA: Add AMD Zen 2 root complex to the list of allowed bridges

[Logan Gunthorpe]

On 2020-07-24 10:07 a.m., Alex Deucher wrote:
> On Thu, Jul 23, 2020 at 4:18 PM Alex Deucher <alexdeucher@xxxxxxxxx> wrote:
>>
>> On Thu, Jul 23, 2020 at 4:11 PM Logan Gunthorpe <logang@xxxxxxxxxxxx> wrote:
>>>
>>>
>>>
>>> On 2020-07-23 1:57 p.m., Bjorn Helgaas wrote:
>>>> [+cc Andrew, Armen, hpa]
>>>>
>>>> On Thu, Jul 23, 2020 at 02:01:17PM -0400, Alex Deucher wrote:
>>>>> On Thu, Jul 23, 2020 at 1:43 PM Logan Gunthorpe <logang@xxxxxxxxxxxx> wrote:
>>>>>>
>>>>>> The AMD Zen 2 root complex (Starship/Matisse) was tested for P2PDMA
>>>>>> transactions between root ports and found to work. Therefore add it
>>>>>> to the list.
>>>>>>
>>>>>> Signed-off-by: Logan Gunthorpe <logang@xxxxxxxxxxxx>
>>>>>> Cc: Bjorn Helgaas <bhelgaas@xxxxxxxxxx>
>>>>>> Cc: Christian KÃnig <christian.koenig@xxxxxxx>
>>>>>> Cc: Huang Rui <ray.huang@xxxxxxx>
>>>>>> Cc: Alex Deucher <alexdeucher@xxxxxxxxx>
>>>>>
>>>>> Starting with Zen, all AMD platforms support P2P for reads and writes.
>>>>
>>>> What's the plan for getting out of the cycle of "update this list for
>>>> every new chip"?  Any new _DSMs planned, for instance?
>>>
>>> Well there was an effort to add capabilities in the PCI spec to describe
>>> this but, as far as I know, they never got anywhere, and hardware still
>>> doesn't self describe with this.
>>>
>>>> A continuous trickle of updates like this is not really appealing.  So
>>>> far we have:
>>>>
>>>>   7d5b10fcb81e ("PCI/P2PDMA: Add AMD Zen Raven and Renoir Root Ports to whitelist")
>>>>   7b94b53db34f ("PCI/P2PDMA: Add Intel Sky Lake-E Root Ports B, C, D to the whitelist")
>>>>   bc123a515cb7 ("PCI/P2PDMA: Add Intel SkyLake-E to the whitelist")
>>>>   494d63b0d5d0 ("PCI/P2PDMA: Whitelist some Intel host bridges")
>>>>   0f97da831026 ("PCI/P2PDMA: Allow P2P DMA between any devices under AMD ZEN Root Complex")
>>>>
>>>> And that's just from the last year, not including this patch.
>>>
>>> Yes, it's not ideal. But most of these are adding old devices as people
>>> test and care about running on those platforms -- a lot of this is
>>> bootstrapping the list. I'd expect this to slow down a bit as by now we
>>> have hopefully got a lot of the existing platforms people care about.
>>> But we'd still probably expect to be adding a new Intel and AMD devices
>>> about once a year as they produce new hardware designs.
>>>
>>> Unless, the Intel and AMD folks know of a way to detect this, or even to
>>> query if a root complex is newer than a certain generation, I'm not sure
>>> what else we can do here.
>>
>> I started a thread internally to see if I can find a way.  FWIW,
>> pre-ZEN parts also support p2p DMA, but only for writes.  If I can get
>> a definitive list, maybe we could switch to a blacklist for the old
>> ones?
> 
> After talking with a few people internally, for AMD chips, it would
> probably be easiest to just whitelist based on the CPU family id for
> zen and newer (e.g., >= 0x17).

That seems sensible. I was trying to see if we could do something
similar for Intel, and just allow anything after Skylake. I found
this[1]. It seems they have been on family 6 for a long time, and I'm
not comfortable enabling the whole family. Their model numbers also
don't seem to increment in a favorable fashion, in that "small core"
atoms (which have different host bridges with very much unknown support)
have model numbers interspersed with regular "big core" CPUS.

So we might be stuck with the Intel white list for a while, but using
the AMD family number will at least cut the number of additions down a
fair amount. I can try to put a patch together in place of this one.

Logan

[1]
https://elixir.bootlin.com/linux/latest/source/arch/x86/include/asm/intel-family.h#L73