iommu dma mapping alignment requirements

From: Steve Wise
Date: Thu Dec 20 2007 - 12:41:05 EST


Hey Roland (and any iommu/ppc/dma experts out there):

I'm debugging a data corruption issue that happens on PPC64 systems running rdma on kernels where the iommu page size is 4KB yet the host page size is 64KB. This "feature" was added to the PPC64 code recently, and is in kernel.org from 2.6.23. So if the kernel is built with a 4KB page size, no problems. If the kernel is prior to 2.6.23 then 64KB page configs work too. Its just a problem when the iommu page size != host page size.

It appears that my problem boils down to a single host page of memory that is mapped for dma, and the dma address returned by dma_map_sg() is _not_ 64KB aligned. Here is an example:

app registers va 0x000000002d9a3000 len 12288
ib_umem_get() creates and maps a umem and chunk that looks like (dumping state from a registered user memory region):

umem len 12288 off 12288 pgsz 65536 shift 16
chunk 0: nmap 1 nents 1
sglist[0] page 0xc000000000930b08 off 0 len 65536 dma_addr 000000005bff4000 dma_len 65536


So the kernel maps 1 full page for this MR. But note that the dma address is 000000005bff4000 which is 4KB aligned, not 64KB aligned. I think this is causing grief to the RDMA HW.

My first question is: Is there an assumption or requirement in linux that dma_addressess should have the same alignment as the host address they are mapped to? IE the rdma core is mapping the entire 64KB page, but the mapping doesn't begin on a 64KB page boundary.

If this mapping is considered valid, then perhaps the rdma hw is at fault here. But I'm wondering if this is an PPC/iommu bug.

BTW: Here is what the Memory Region looks like to the HW:

TPT entry: stag idx 0x2e800 key 0xff state VAL type NSMR pdid 0x2
perms RW rem_inv_dis 0 addr_type VATO
bind_enable 1 pg_size 65536 qpid 0x0 pbl_addr 0x003c67c0
len 12288 va 000000002d9a3000 bind_cnt 0
PBL: 000000005bff4000



Any thoughts?

Steve.


--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/