RE: [PATCH] PCI: endpoint: Add DMA to Linux PCI EP Framework
From: Gustavo Pimentel
Date: Mon May 27 2019 - 05:13:04 EST
On Fri, May 24, 2019 at 20:42:43, Alan Mikhak <alan.mikhak@xxxxxxxxxx>
wrote:
Hi Alan,
> On Fri, May 24, 2019 at 1:59 AM Gustavo Pimentel
> <Gustavo.Pimentel@xxxxxxxxxxxx> wrote:
> >
> > Hi Alan,
> >
> > This patch implementation is very HW implementation dependent and
> > requires the DMA to exposed through PCIe BARs, which aren't always the
> > case. Besides, you are defining some control bits on
> > include/linux/pci-epc.h that may not have any meaning to other types of
> > DMA.
> >
> > I don't think this was what Kishon had in mind when he developed the
> > pcitest, but let see what Kishon was to say about it.
> >
> > I've developed a DMA driver for DWC PCI using Linux Kernel DMAengine API
> > and which I submitted some days ago.
> > By having a DMA driver which implemented using DMAengine API, means the
> > pcitest can use the DMAengine client API, which will be completely
> > generic to any other DMA implementation.
> >
> > My DMA driver for DWC PCI was done thinking on my use case, which is was
> > interacting from the Root Complex side with DMA IP implemented on the
> > Endpoint, which was exposed through PCI BARs. However, I think it would
> > be possible to reuse the same core code and instead of using the PCI-glue
> > to adapt it and be used easily on the Endpoint side and to be triggered
> > there.
> >
> > Gustavo
>
> Hi Gustavo,
>
> I saw your patches for the DMA driver for DWC PCI which added the EDDA
> driver to pci_endpoint_test.c on the host side. This suggested to me
> that the user would invoke pcitest on the host side to exercise your
> driver on the endpoint.
>
> However, I didn't see any changes to pci-epf-test.c to modify the
> pci_epf_test_write() and pci_epf_test_read() functions to use DMA
> instead of memcpy_toio() and memcpy_fromio(), respectively.
I didn't add the DMA engine client API to the pcitest yet. I was waiting
for the eDMA driver to be integrated on Kernel before doing any on
pcitest.
>
> I have four separate DMA requirements in mind as motivation for this patch.
>
> The following two requirements are implemented by this patch:
> Local DMA write or read transfer initiated by endpoint under user
> command from the host between system and endpoint memory buffers using
> the endpoint DMA controller with a local interrupt.
> 1) single block
> 2) linked-list mode
However, in most cases, you will not have the DMA block registers or
linked list memory accessible or defined on PCI BARs and based on I have
seen in your patch, the whole patch is based on that premise. The current
implementation is not generic and highly dependent on IP and design.
I strongly believe with some slight modifications it would be possible to
run the DMA engine controller driver with a platform glue-logic on the
Endpoint side, which could interact with pci_epf_test driver. This way
the pcitest would be always generic to all products.
>
> This patch anticipates two more requirements yet to be implemented in
> future patches:
> Remote DMA write or read transfer initiated by host between system and
> endpoint memory buffers using the endpoint DMA controller with a local
> interrupt to the endpoint processor and a remote interrupt to the host
> process.
> 1) single block
> 2) linked-list mode
That's what the submitted patch driver does. I didn't develop the
interaction with to pcitest, however, while developing DW eDMA IP driver,
I've sent in some an RFC patch containing dw-edma-test driver that
stimulates the DW eDMA.
>
> The descriptor format defined in pci-epc.h allows for pci-epf-test.c
> to be expanded over time to initiate more elaborate DMA tests to
> exercise other endpoint DMA scenarios.
Yes, but that's not the original issue been discussed here. But let's see
what Kishon as to say, In the end, he is the maintainer of this tool.
Regards,
Gustavo
>
> The following is a sample output of the pcitest usage for exercising
> DMA operations on the endpoint:
>
> $ pcitest -r -d
> READ ( 102400 bytes): OKAY
> $ pcitest -w -d
> WRITE ( 102400 bytes): OKAY
> $ pcitest -w -d -L
> WRITE ( 102400 bytes): OKAY
> $ pcitest -r -d -L
> READ ( 102400 bytes): OKAY
>
> The above are executed using DMA operations as opposed to the
> following which use memcpy_toio() and memcpy_fromio().
>
> $ pcitest -r
> READ ( 102400 bytes): OKAY
> $ pcitest -w
> WRITE ( 102400 bytes): OKAY
>
> Regards,
> Alan Mikhak
>
>
>
> >
> > -----Original Message-----
> > From: Alan Mikhak <alan.mikhak@xxxxxxxxxx>
> >
> > Sent: 23 de maio de 2019 23:24
> > To: linux-pci@xxxxxxxxxxxxxxx;
> > linux-kernel@xxxxxxxxxxxxxxx; kishon@xxxxxx; lorenzo.pieralisi@xxxxxxx;
> > arnd@xxxxxxxx; gregkh@xxxxxxxxxxxxxxxxxxx; jingoohan1@xxxxxxxxx;
> > gustavo.pimentel@xxxxxxxxxxxx; bhelgaas@xxxxxxxxxx;
> > wen.yang99@xxxxxxxxxx; kjlu@xxxxxxx; linux-riscv@xxxxxxxxxxxxxxxxxxx;
> > palmer@xxxxxxxxxx; paul.walmsley@xxxxxxxxxx
> > Cc: Alan Mikhak
> > <alan.mikhak@xxxxxxxxxx>
> > Subject: [PATCH] PCI: endpoint: Add DMA to Linux
> > PCI EP Framework
> >
> > This patch depends on patch the following patches:
> > [PATCH v2 1/2] tools: PCI: Fix broken pcitest compilation
> > [PATCH v2 2/2] tools: PCI: Fix compiler warning in pcitest
> >
> > The Linux PCI Endpoint Framework currently does not support initiation of
> > DMA read and write operations. This patch extends the Linux PCI Endpoint
> > Framework by adding support for the user of pcitest to inititate DMA
> > read and write operations via PCIe endpoint controller drivers. This
> > patch
> > provides the means but leaves it up to individual PCIe endpoint
> > controller
> > drivers to implement DMA support, if desired.
> >
> > This patch provides support for the pcitest user to instruct the endpoint
> > to initiate local DMA transfers consisting of single or linked-list
> > blocks
> > into endpoint buffers using the endpoint DMA controller. It anticipates
> > that future patches would add support for the pcitest user to instruct
> > the endpoint to participate in remote DMA transfers initiated from the
> > root complex into endpoint buffers using the endpoint DMA controller.
> >
> > This patch depends on the first two patches in its patchset to resolve
> > a pre-existing pcitest compilation error.
> >
> > * Add -d flag to pcitest command line options so user can specify
> > that a read or write command should execute using local DMA to be
> > initiated by endpoint.
> >
> > * Add -L flag to pcitest command line options so user can specify
> > that DMA operation should execute in linked-list mode.
> >
> > * Add struct pcitest_dma for pcitest to communicate DMA options
> > from host userspace to pci_endpoint_test driver in host kernel
> > via two new ioctls PCITEST_WRITE_DMA and PCITEST_READ_DMA.
> >
> > * Add command flags so pci_endpoint_test driver running on host
> > can communicate DMA read and write options across the PCI bus
> > to pci-epf-test driver running on endpoint.
> >
> > * Add struct pci_epc_dma so pci-epf-test driver can create DMA
> > read and write descriptors for single or linked-list DMA operations
> > and pass such descriptors to pci-epc-core via new functions
> > pci_epc_dma_read() and pci_epc_dma_write().
> >
> > * Add four new functions in pci-epf-test driver to implement
> > new DMA read and write tests by initiating local DMA transfers
> > in linked-list and single modes via PCIe endpoint controller
> > drivers: pci_epf_test_read_dma(), pci_epf_test_read_dma_list(),
> > pci_epf_test_write_dma(), and pci_epf_test_write_dma_list().
> >
> > * Add dma_read and dma_write functions to struct pci_epc_ops
> > so pci_epc_dma_read() and pci_epc_dma_write() functions can
> > pass DMA descriptors down the stack to pcie-designware-ep layer.
> >
> > * Add dma_read and dma_write functions to struct dw_pcie_ep_ops
> > so pcie-designware-ep layer can communicate DMA descriptors down
> > the stack to vendor PCIe endpoint controller drivers.
> >
> > * Add dma_base pointer to struct dw_pcie for vendor PCIe endpoint
> > controller driver to set if it implements DMA operations.
> >
> > * Add two common pcie-designware functions dw_pcie_writel_dma()
> > and dw_pcie_readl_dma() for use by vendor PCIe endpoint
> > controllers to access DMA registers via the dma_base pointer.
> >
> > Signed-off-by: Alan Mikhak <alan.mikhak@xxxxxxxxxx>
> > ---
> >
> > drivers/misc/pci_endpoint_test.c | 72 +++++++-
> > drivers/pci/controller/dwc/pcie-designware-ep.c | 22 +++
> > drivers/pci/controller/dwc/pcie-designware.h | 13 ++
> > drivers/pci/endpoint/functions/pci-epf-test.c | 211
> > +++++++++++++++++++++++-
> > drivers/pci/endpoint/pci-epc-core.c | 46 ++++++
> > include/linux/pci-epc.h | 45 +++++
> > include/uapi/linux/pcitest.h | 7 +
> > tools/pci/pcitest.c | 29 +++-
> > 8 files changed, 432 insertions(+), 13 deletions(-)
> >
> > diff --git a/drivers/misc/pci_endpoint_test.c
> > b/drivers/misc/pci_endpoint_test.c
> > index 7b015f2a1c6f..63b86d81a6b5 100644
> > --- a/drivers/misc/pci_endpoint_test.c
> > +++ b/drivers/misc/pci_endpoint_test.c
> > @@ -34,6 +34,7 @@
> > #include <linux/pci_regs.h>
> >
> > #include <uapi/linux/pcitest.h>
> > +#include <linux/uaccess.h>
> >
> > #define DRV_MODULE_NAME "pci-endpoint-test"
> >
> > @@ -51,6 +52,25 @@
> > #define COMMAND_READ BIT(3)
> > #define COMMAND_WRITE BIT(4)
> > #define COMMAND_COPY BIT(5)
> > +#define COMMAND_FLAG2 BIT(30)
> > +#define COMMAND_FLAG1 BIT(31)
> > +
> > +#define COMMAND_FLAGS (COMMAND_FLAG1 | \
> > + COMMAND_FLAG2)
> > +
> > +#define COMMAND_FLAG_NONE 0
> > +#define COMMAND_FLAG_DMA COMMAND_FLAG1
> > +#define COMMAND_FLAG_DMA_LIST COMMAND_FLAG2
> > +
> > +#define COMMAND_READ_DMA (COMMAND_READ | \
> > + COMMAND_FLAG_DMA)
> > +#define COMMAND_READ_DMA_LIST (COMMAND_READ_DMA | \
> > + COMMAND_FLAG_DMA_LIST)
> > +
> > +#define COMMAND_WRITE_DMA (COMMAND_WRITE | \
> > + COMMAND_FLAG_DMA)
> > +#define COMMAND_WRITE_DMA_LIST (COMMAND_WRITE_DMA | \
> > + COMMAND_FLAG_DMA_LIST)
> >
> > #define PCI_ENDPOINT_TEST_STATUS 0x8
> > #define STATUS_READ_SUCCESS BIT(0)
> > @@ -425,7 +445,9 @@ static bool pci_endpoint_test_copy(struct
> > pci_endpoint_test *test, size_t size)
> > return ret;
> > }
> >
> > -static bool pci_endpoint_test_write(struct pci_endpoint_test *test,
> > size_t size)
> > +static bool pci_endpoint_test_write(struct pci_endpoint_test *test,
> > + size_t size,
> > + u32 flags)
> > {
> > bool ret = false;
> > u32 reg;
> > @@ -480,7 +502,7 @@ static bool pci_endpoint_test_write(struct
> > pci_endpoint_test *test, size_t size)
> > pci_endpoint_test_writel(test, PCI_ENDPOINT_TEST_IRQ_TYPE, irq_type);
> > pci_endpoint_test_writel(test, PCI_ENDPOINT_TEST_IRQ_NUMBER, 1);
> > pci_endpoint_test_writel(test, PCI_ENDPOINT_TEST_COMMAND,
> > - COMMAND_READ);
> > + COMMAND_READ | flags);
> >
> > wait_for_completion(&test->irq_raised);
> >
> > @@ -494,7 +516,24 @@ static bool pci_endpoint_test_write(struct
> > pci_endpoint_test *test, size_t size)
> > return ret;
> > }
> >
> > -static bool pci_endpoint_test_read(struct pci_endpoint_test *test,
> > size_t size)
> > +static bool pci_endpoint_test_write_dma(struct pci_endpoint_test *test,
> > + unsigned long arg)
> > +{
> > + u32 flags = COMMAND_FLAG_DMA;
> > + struct pcitest_dma dma;
> > +
> > + if (copy_from_user(&dma, (void *)arg, sizeof(struct pcitest_dma)))
> > + return -EACCES;
> > +
> > + if (dma.list)
> > + flags |= COMMAND_FLAG_DMA_LIST;
> > +
> > + return pci_endpoint_test_write(test, dma.size, flags);
> > +}
> > +
> > +static bool pci_endpoint_test_read(struct pci_endpoint_test *test,
> > + size_t size,
> > + u32 flags)
> > {
> > bool ret = false;
> > void *addr;
> > @@ -542,7 +581,7 @@ static bool pci_endpoint_test_read(struct
> > pci_endpoint_test *test, size_t size)
> > pci_endpoint_test_writel(test, PCI_ENDPOINT_TEST_IRQ_TYPE, irq_type);
> > pci_endpoint_test_writel(test, PCI_ENDPOINT_TEST_IRQ_NUMBER, 1);
> > pci_endpoint_test_writel(test, PCI_ENDPOINT_TEST_COMMAND,
> > - COMMAND_WRITE);
> > + COMMAND_WRITE | flags);
> >
> > wait_for_completion(&test->irq_raised);
> >
> > @@ -555,6 +594,21 @@ static bool pci_endpoint_test_read(struct
> > pci_endpoint_test *test, size_t size)
> > return ret;
> > }
> >
> > +static bool pci_endpoint_test_read_dma(struct pci_endpoint_test *test,
> > + unsigned long arg)
> > +{
> > + u32 flags = COMMAND_FLAG_DMA;
> > + struct pcitest_dma dma;
> > +
> > + if (copy_from_user(&dma, (void *)arg, sizeof(struct pcitest_dma)))
> > + return -EACCES;
> > +
> > + if (dma.list)
> > + flags |= COMMAND_FLAG_DMA_LIST;
> > +
> > + return pci_endpoint_test_read(test, dma.size, flags);
> > +}
> > +
> > static bool pci_endpoint_test_set_irq(struct pci_endpoint_test *test,
> > int req_irq_type)
> > {
> > @@ -612,11 +666,17 @@ static long pci_endpoint_test_ioctl(struct file
> > *file, unsigned int cmd,
> > case PCITEST_MSIX:
> > ret = pci_endpoint_test_msi_irq(test, arg, cmd == PCITEST_MSIX);
> > break;
> > + case PCITEST_WRITE_DMA:
> > + ret = pci_endpoint_test_write_dma(test, arg);
> > + break;
> > + case PCITEST_READ_DMA:
> > + ret = pci_endpoint_test_read_dma(test, arg);
> > + break;
> > case PCITEST_WRITE:
> > - ret = pci_endpoint_test_write(test, arg);
> > + ret = pci_endpoint_test_write(test, arg, COMMAND_FLAG_NONE);
> > break;
> > case PCITEST_READ:
> > - ret = pci_endpoint_test_read(test, arg);
> > + ret = pci_endpoint_test_read(test, arg, COMMAND_FLAG_NONE);
> > break;
> > case PCITEST_COPY:
> > ret = pci_endpoint_test_copy(test, arg);
> > diff --git a/drivers/pci/controller/dwc/pcie-designware-ep.c
> > b/drivers/pci/controller/dwc/pcie-designware-ep.c
> > index 2bf5a35c0570..7e25c0f5edf1 100644
> > --- a/drivers/pci/controller/dwc/pcie-designware-ep.c
> > +++ b/drivers/pci/controller/dwc/pcie-designware-ep.c
> > @@ -366,6 +366,26 @@ dw_pcie_ep_get_features(struct pci_epc *epc, u8
> > func_no)
> > return ep->ops->get_features(ep);
> > }
> >
> > +static int dw_pcie_ep_dma_read(struct pci_epc *epc, struct pci_epc_dma
> > *dma)
> > +{
> > + struct dw_pcie_ep *ep = epc_get_drvdata(epc);
> > +
> > + if (!ep->ops->dma_read)
> > + return -EINVAL;
> > +
> > + return ep->ops->dma_read(ep, dma);
> > +}
> > +
> > +static int dw_pcie_ep_dma_write(struct pci_epc *epc, struct pci_epc_dma
> > *dma)
> > +{
> > + struct dw_pcie_ep *ep = epc_get_drvdata(epc);
> > +
> > + if (!ep->ops->dma_write)
> > + return -EINVAL;
> > +
> > + return ep->ops->dma_write(ep, dma);
> > +}
> > +
> > static const struct pci_epc_ops epc_ops = {
> > .write_header = dw_pcie_ep_write_header,
> > .set_bar = dw_pcie_ep_set_bar,
> > @@ -380,6 +400,8 @@ static const struct pci_epc_ops epc_ops = {
> > .start = dw_pcie_ep_start,
> > .stop = dw_pcie_ep_stop,
> > .get_features = dw_pcie_ep_get_features,
> > + .dma_read = dw_pcie_ep_dma_read,
> > + .dma_write = dw_pcie_ep_dma_write
> > };
> >
> > int dw_pcie_ep_raise_legacy_irq(struct dw_pcie_ep *ep, u8 func_no)
> > diff --git a/drivers/pci/controller/dwc/pcie-designware.h
> > b/drivers/pci/controller/dwc/pcie-designware.h
> > index b8993f2b78df..11d44ec8acc7 100644
> > --- a/drivers/pci/controller/dwc/pcie-designware.h
> > +++ b/drivers/pci/controller/dwc/pcie-designware.h
> > @@ -197,6 +197,8 @@ struct dw_pcie_ep_ops {
> > int (*raise_irq)(struct dw_pcie_ep *ep, u8 func_no,
> > enum pci_epc_irq_type type, u16 interrupt_num);
> > const struct pci_epc_features* (*get_features)(struct dw_pcie_ep *ep);
> > + int (*dma_read)(struct dw_pcie_ep *ep, struct pci_epc_dma *dma);
> > + int (*dma_write)(struct dw_pcie_ep *ep, struct pci_epc_dma *dma);
> > };
> >
> > struct dw_pcie_ep {
> > @@ -238,6 +240,7 @@ struct dw_pcie {
> > void __iomem *dbi_base2;
> > /* Used when iatu_unroll_enabled is true */
> > void __iomem *atu_base;
> > + void __iomem *dma_base;
> > u32 num_viewport;
> > u8 iatu_unroll_enabled;
> > struct pcie_port pp;
> > @@ -323,6 +326,16 @@ static inline u32 dw_pcie_readl_atu(struct dw_pcie
> > *pci, u32 reg)
> > return __dw_pcie_read_dbi(pci, pci->atu_base, reg, 0x4);
> > }
> >
> > +static inline void dw_pcie_writel_dma(struct dw_pcie *pci, u32 reg, u32
> > val)
> > +{
> > + __dw_pcie_write_dbi(pci, pci->dma_base, reg, 0x4, val);
> > +}
> > +
> > +static inline u32 dw_pcie_readl_dma(struct dw_pcie *pci, u32 reg)
> > +{
> > + return __dw_pcie_read_dbi(pci, pci->dma_base, reg, 0x4);
> > +}
> > +
> > static inline void dw_pcie_dbi_ro_wr_en(struct dw_pcie *pci)
> > {
> > u32 reg;
> > diff --git a/drivers/pci/endpoint/functions/pci-epf-test.c
> > b/drivers/pci/endpoint/functions/pci-epf-test.c
> > index 27806987e93b..3910073712e9 100644
> > --- a/drivers/pci/endpoint/functions/pci-epf-test.c
> > +++ b/drivers/pci/endpoint/functions/pci-epf-test.c
> > @@ -28,6 +28,25 @@
> > #define COMMAND_READ BIT(3)
> > #define COMMAND_WRITE BIT(4)
> > #define COMMAND_COPY BIT(5)
> > +#define COMMAND_FLAG2 BIT(30)
> > +#define COMMAND_FLAG1 BIT(31)
> > +
> > +#define COMMAND_FLAGS (COMMAND_FLAG1 | \
> > + COMMAND_FLAG2)
> > +
> > +#define COMMAND_FLAG_NONE 0
> > +#define COMMAND_FLAG_DMA COMMAND_FLAG1
> > +#define COMMAND_FLAG_DMA_LIST COMMAND_FLAG2
> > +
> > +#define COMMAND_READ_DMA (COMMAND_READ | \
> > + COMMAND_FLAG_DMA)
> > +#define COMMAND_READ_DMA_LIST (COMMAND_READ_DMA | \
> > + COMMAND_FLAG_DMA_LIST)
> > +
> > +#define COMMAND_WRITE_DMA (COMMAND_WRITE | \
> > + COMMAND_FLAG_DMA)
> > +#define COMMAND_WRITE_DMA_LIST (COMMAND_WRITE_DMA | \
> > + COMMAND_FLAG_DMA_LIST)
> >
> > #define STATUS_READ_SUCCESS BIT(0)
> > #define STATUS_READ_FAIL BIT(1)
> > @@ -187,6 +206,93 @@ static int pci_epf_test_read(struct pci_epf_test
> > *epf_test)
> > return ret;
> > }
> >
> > +static int pci_epf_test_read_dma(struct pci_epf_test *epf_test)
> > +{
> > + int ret = -ENOMEM;
> > + struct pci_epf *epf = epf_test->epf;
> > + struct device *dev = &epf->dev;
> > + struct pci_epc *epc = epf->epc;
> > + enum pci_barno test_reg_bar = epf_test->test_reg_bar;
> > + struct pci_epf_test_reg *reg = epf_test->reg[test_reg_bar];
> > + struct pci_epc_dma dma;
> > + u32 crc32;
> > + void *buf;
> > +
> > + buf = kzalloc(reg->size, GFP_KERNEL);
> > + if (buf) {
> > + dma.control = PCI_EPC_DMA_CONTROL_LIE;
> > + dma.size = reg->size;
> > + dma.sar = reg->src_addr;
> > + dma.dar = virt_to_phys(buf);
> > +
> > + ret = pci_epc_dma_read(epc, &dma);
> > + if (ret) {
> > + dev_err(dev, "pci_epc_dma_read %d\n", ret);
> > + } else {
> > + crc32 = crc32_le(~0, buf, reg->size);
> > + if (crc32 != reg->checksum)
> > + ret = -EIO;
> > + }
> > +
> > + kfree(buf);
> > + }
> > +
> > + return ret;
> > +}
> > +
> > +static int pci_epf_test_read_dma_list(struct pci_epf_test *epf_test)
> > +{
> > + int ret = -ENOMEM;
> > + struct pci_epf *epf = epf_test->epf;
> > + struct device *dev = &epf->dev;
> > + struct pci_epc *epc = epf->epc;
> > + enum pci_barno test_reg_bar = epf_test->test_reg_bar;
> > + struct pci_epf_test_reg *reg = epf_test->reg[test_reg_bar];
> > + struct pci_epc_dma *dma;
> > + u32 crc32;
> > + void *buf;
> > +
> > + dma = kcalloc(3, sizeof(*dma), GFP_KERNEL);
> > + if (dma) {
> > + buf = kzalloc(reg->size, GFP_KERNEL);
> > + if (buf) {
> > + int half_size = reg->size >> 1;
> > + phys_addr_t phys_addr = virt_to_phys(buf);
> > +
> > + dma[0].control = PCI_EPC_DMA_CONTROL_CB;
> > + dma[0].size = half_size ? half_size : 1;
> > + dma[0].sar = reg->src_addr;
> > + dma[0].dar = phys_addr;
> > +
> > + dma[1].control = PCI_EPC_DMA_CONTROL_CB |
> > + PCI_EPC_DMA_CONTROL_LIE;
> > + dma[1].size = reg->size - half_size;
> > + dma[1].sar = reg->src_addr + half_size;
> > + dma[1].dar = phys_addr + half_size;
> > +
> > + dma[2].control = PCI_EPC_DMA_CONTROL_EOL;
> > + dma[2].size = 0;
> > + dma[2].sar = virt_to_phys(dma);
> > + dma[2].dar = 0;
> > +
> > + ret = pci_epc_dma_read(epc, dma);
> > + if (ret) {
> > + dev_err(dev, "pci_epc_dma_read %d\n", ret);
> > + } else {
> > + crc32 = crc32_le(~0, buf, reg->size);
> > + if (crc32 != reg->checksum)
> > + ret = -EIO;
> > + }
> > +
> > + kfree(buf);
> > + }
> > +
> > + kfree(dma);
> > + }
> > +
> > + return ret;
> > +}
> > +
> > static int pci_epf_test_write(struct pci_epf_test *epf_test)
> > {
> > int ret;
> > @@ -244,6 +350,87 @@ static int pci_epf_test_write(struct pci_epf_test
> > *epf_test)
> > return ret;
> > }
> >
> > +static int pci_epf_test_write_dma(struct pci_epf_test *epf_test)
> > +{
> > + int ret = -ENOMEM;
> > + struct pci_epf *epf = epf_test->epf;
> > + struct device *dev = &epf->dev;
> > + struct pci_epc *epc = epf->epc;
> > + enum pci_barno test_reg_bar = epf_test->test_reg_bar;
> > + struct pci_epf_test_reg *reg = epf_test->reg[test_reg_bar];
> > + struct pci_epc_dma dma;
> > + void *buf;
> > +
> > + buf = kzalloc(reg->size, GFP_KERNEL);
> > + if (buf) {
> > + get_random_bytes(buf, reg->size);
> > + reg->checksum = crc32_le(~0, buf, reg->size);
> > +
> > + dma.control = PCI_EPC_DMA_CONTROL_LIE;
> > + dma.size = reg->size;
> > + dma.sar = virt_to_phys(buf);
> > + dma.dar = reg->dst_addr;
> > +
> > + ret = pci_epc_dma_write(epc, &dma);
> > + if (ret)
> > + dev_err(dev, "pci_epc_dma_write %d\n", ret);
> > +
> > + kfree(buf);
> > + }
> > +
> > + return ret;
> > +}
> > +
> > +static int pci_epf_test_write_dma_list(struct pci_epf_test *epf_test)
> > +{
> > + int ret = -ENOMEM;
> > + struct pci_epf *epf = epf_test->epf;
> > + struct device *dev = &epf->dev;
> > + struct pci_epc *epc = epf->epc;
> > + enum pci_barno test_reg_bar = epf_test->test_reg_bar;
> > + struct pci_epf_test_reg *reg = epf_test->reg[test_reg_bar];
> > + struct pci_epc_dma *dma;
> > + void *buf;
> > +
> > + dma = kcalloc(3, sizeof(*dma), GFP_KERNEL);
> > + if (dma) {
> > + buf = kzalloc(reg->size, GFP_KERNEL);
> > + if (buf) {
> > + int half_size = reg->size >> 1;
> > + phys_addr_t phys_addr = virt_to_phys(buf);
> > +
> > + get_random_bytes(buf, reg->size);
> > + reg->checksum = crc32_le(~0, buf, reg->size);
> > +
> > + dma[0].control = PCI_EPC_DMA_CONTROL_CB;
> > + dma[0].size = half_size ? half_size : 1;
> > + dma[0].sar = phys_addr;
> > + dma[0].dar = reg->dst_addr;
> > +
> > + dma[1].control = PCI_EPC_DMA_CONTROL_CB |
> > + PCI_EPC_DMA_CONTROL_LIE;
> > + dma[1].size = reg->size - half_size;
> > + dma[1].sar = phys_addr + half_size;
> > + dma[1].dar = reg->dst_addr + half_size;
> > +
> > + dma[2].control = PCI_EPC_DMA_CONTROL_EOL;
> > + dma[2].size = 0;
> > + dma[2].sar = virt_to_phys(dma);
> > + dma[2].dar = 0;
> > +
> > + ret = pci_epc_dma_write(epc, dma);
> > + if (ret)
> > + dev_err(dev, "pci_epc_dma_write %d\n", ret);
> > +
> > + kfree(buf);
> > + }
> > +
> > + kfree(dma);
> > + }
> > +
> > + return ret;
> > +}
> > +
> > static void pci_epf_test_raise_irq(struct pci_epf_test *epf_test, u8
> > irq_type,
> > u16 irq)
> > {
> > @@ -303,18 +490,34 @@ static void pci_epf_test_cmd_handler(struct
> > work_struct *work)
> > }
> >
> > if (command & COMMAND_WRITE) {
> > - ret = pci_epf_test_write(epf_test);
> > - if (ret)
> > - reg->status |= STATUS_WRITE_FAIL;
> > + command &= (COMMAND_WRITE | COMMAND_FLAGS);
> > + if (command == COMMAND_WRITE)
> > + ret = pci_epf_test_write(epf_test);
> > + else if (command == COMMAND_WRITE_DMA)
> > + ret = pci_epf_test_write_dma(epf_test);
> > + else if (command == COMMAND_WRITE_DMA_LIST)
> > + ret = pci_epf_test_write_dma_list(epf_test);
> > else
> > + ret = -EINVAL;
> > + if (!ret)
> > reg->status |= STATUS_WRITE_SUCCESS;
> > + else
> > + reg->status |= STATUS_WRITE_FAIL;
> > pci_epf_test_raise_irq(epf_test, reg->irq_type,
> > reg->irq_number);
> > goto reset_handler;
> > }
> >
> > if (command & COMMAND_READ) {
> > - ret = pci_epf_test_read(epf_test);
> > + command &= (COMMAND_READ | COMMAND_FLAGS);
> > + if (command == COMMAND_READ)
> > + ret = pci_epf_test_read(epf_test);
> > + else if (command == COMMAND_READ_DMA)
> > + ret = pci_epf_test_read_dma(epf_test);
> > + else if (command == COMMAND_READ_DMA_LIST)
> > + ret = pci_epf_test_read_dma_list(epf_test);
> > + else
> > + ret = -EINVAL;
> > if (!ret)
> > reg->status |= STATUS_READ_SUCCESS;
> > else
> > diff --git a/drivers/pci/endpoint/pci-epc-core.c
> > b/drivers/pci/endpoint/pci-epc-core.c
> > index e4712a0f249c..a57e501d4abc 100644
> > --- a/drivers/pci/endpoint/pci-epc-core.c
> > +++ b/drivers/pci/endpoint/pci-epc-core.c
> > @@ -107,6 +107,52 @@ unsigned int pci_epc_get_first_free_bar(const struct
> > pci_epc_features
> > EXPORT_SYMBOL_GPL(pci_epc_get_first_free_bar);
> >
> > /**
> > + * pci_epc_dma_write() - DMA a block of memory to remote address
> > + * @epc: the EPC device on which to perform DMA transfer
> > + * @dma: DMA descriptors array
> > + *
> > + * Write contents of local memory to remote memory by DMA.
> > + */
> > +int pci_epc_dma_write(struct pci_epc *epc, struct pci_epc_dma *dma)
> > +{
> > + int ret;
> > + unsigned long flags;
> > +
> > + if (IS_ERR(epc) || !epc->ops->dma_write)
> > + return -EINVAL;
> > +
> > + spin_lock_irqsave(&epc->lock, flags);
> > + ret = epc->ops->dma_write(epc, dma);
> > + spin_unlock_irqrestore(&epc->lock, flags);
> > +
> > + return ret;
> > +}
> > +EXPORT_SYMBOL_GPL(pci_epc_dma_write);
> > +
> > +/**
> > + * pci_epc_dma_read() - DMA a block of memory from remote address
> > + * @epc: the EPC device on which to perform DMA transfer
> > + * @dma: DMA descriptors array
> > + *
> > + * Read contents of remote memory into local memory by DMA.
> > + */
> > +int pci_epc_dma_read(struct pci_epc *epc, struct pci_epc_dma *dma)
> > +{
> > + int ret;
> > + unsigned long flags;
> > +
> > + if (IS_ERR(epc) || !epc->ops->dma_read)
> > + return -EINVAL;
> > +
> > + spin_lock_irqsave(&epc->lock, flags);
> > + ret = epc->ops->dma_read(epc, dma);
> > + spin_unlock_irqrestore(&epc->lock, flags);
> > +
> > + return ret;
> > +}
> > +EXPORT_SYMBOL_GPL(pci_epc_dma_read);
> > +
> > +/**
> > * pci_epc_get_features() - get the features supported by EPC
> > * @epc: the features supported by *this* EPC device will be returned
> > * @func_no: the features supported by the EPC device specific to the
> > diff --git a/include/linux/pci-epc.h b/include/linux/pci-epc.h
> > index f641badc2c61..d845f13d0baf 100644
> > --- a/include/linux/pci-epc.h
> > +++ b/include/linux/pci-epc.h
> > @@ -21,6 +21,45 @@ enum pci_epc_irq_type {
> > };
> >
> > /**
> > + * struct pci_epc_dma - descriptor for a DMA transfer element
> > + * @control: DMA channel control bits for read or write transfer
> > + * @size: size of DMA transfer element
> > + * @sar: source addrees for DMA transfer element
> > + * @dar: destination address for DMA transfer element
> > + */
> > +
> > +struct pci_epc_dma {
> > + u32 control;
> > + u32 size;
> > + u64 sar;
> > + u64 dar;
> > +};
> > +
> > +#define PCI_EPC_DMA_CONTROL_CB (BIT(0))
> > +#define PCI_EPC_DMA_CONTROL_TCB (BIT(1))
> > +#define PCI_EPC_DMA_CONTROL_LLP (BIT(2))
> > +#define PCI_EPC_DMA_CONTROL_LIE (BIT(3))
> > +#define PCI_EPC_DMA_CONTROL_RIE (BIT(4))
> > +#define PCI_EPC_DMA_CONTROL_CS (BIT(5) | BIT(6))
> > +#define PCI_EPC_DMA_CONTROL_CCS (BIT(8))
> > +#define PCI_EPC_DMA_CONTROL_LLE (BIT(9))
> > +#define PCI_EPC_DMA_CONTROL_FUNC (BIT(12) | BIT(13) | BIT(14) | \
> > + BIT(15) | BIT(16))
> > +#define PCI_EPC_DMA_CONTROL_NS_DST (BIT(23))
> > +#define PCI_EPC_DMA_CONTROL_NS_SRC (BIT(24))
> > +#define PCI_EPC_DMA_CONTROL_RO (BIT(25))
> > +#define PCI_EPC_DMA_CONTROL_TC (BIT(27) | BIT(28) | BIT(29))
> > +#define PCI_EPC_DMA_CONTROL_AT (BIT(30) | BIT(31))
> > +
> > +#define PCI_EPC_DMA_CONTROL_EOL (PCI_EPC_DMA_CONTROL_TCB | \
> > + PCI_EPC_DMA_CONTROL_LLP)
> > +
> > +#define PCI_EPC_DMA_CONTROL_LIST (PCI_EPC_DMA_CONTROL_CB | \
> > + PCI_EPC_DMA_CONTROL_EOL| \
> > + PCI_EPC_DMA_CONTROL_CCS | \
> > + PCI_EPC_DMA_CONTROL_LLE)
> > +
> > +/**
> > * struct pci_epc_ops - set of function pointers for performing EPC
> > operations
> > * @write_header: ops to populate configuration space header
> > * @set_bar: ops to configure the BAR
> > @@ -38,6 +77,8 @@ enum pci_epc_irq_type {
> > * @raise_irq: ops to raise a legacy, MSI or MSI-X interrupt
> > * @start: ops to start the PCI link
> > * @stop: ops to stop the PCI link
> > + * @dma_read: ops to read remote memory into local memory by DMA
> > + * @dma_write: ops to write local memory to remote memory by DMA
> > * @owner: the module owner containing the ops
> > */
> > struct pci_epc_ops {
> > @@ -61,6 +102,8 @@ struct pci_epc_ops {
> > void (*stop)(struct pci_epc *epc);
> > const struct pci_epc_features* (*get_features)(struct pci_epc *epc,
> > u8 func_no);
> > + int (*dma_read)(struct pci_epc *epc, struct pci_epc_dma *dma);
> > + int (*dma_write)(struct pci_epc *epc, struct pci_epc_dma *dma);
> > struct module *owner;
> > };
> >
> > @@ -152,6 +195,8 @@ void pci_epc_destroy(struct pci_epc *epc);
> > int pci_epc_add_epf(struct pci_epc *epc, struct pci_epf *epf);
> > void pci_epc_linkup(struct pci_epc *epc);
> > void pci_epc_remove_epf(struct pci_epc *epc, struct pci_epf *epf);
> > +int pci_epc_dma_read(struct pci_epc *epc, struct pci_epc_dma *dma);
> > +int pci_epc_dma_write(struct pci_epc *epc, struct pci_epc_dma *dma);
> > int pci_epc_write_header(struct pci_epc *epc, u8 func_no,
> > struct pci_epf_header *hdr);
> > int pci_epc_set_bar(struct pci_epc *epc, u8 func_no,
> > diff --git a/include/uapi/linux/pcitest.h b/include/uapi/linux/pcitest.h
> > index cbf422e56696..505f4a3811c2 100644
> > --- a/include/uapi/linux/pcitest.h
> > +++ b/include/uapi/linux/pcitest.h
> > @@ -10,11 +10,18 @@
> > #ifndef __UAPI_LINUX_PCITEST_H
> > #define __UAPI_LINUX_PCITEST_H
> >
> > +struct pcitest_dma {
> > + size_t size;
> > + bool list;
> > +};
> > +
> > #define PCITEST_BAR _IO('P', 0x1)
> > #define PCITEST_LEGACY_IRQ _IO('P', 0x2)
> > #define PCITEST_MSI _IOW('P', 0x3, int)
> > #define PCITEST_WRITE _IOW('P', 0x4, unsigned long)
> > +#define PCITEST_WRITE_DMA _IOW('P', 0x4, struct pcitest_dma)
> > #define PCITEST_READ _IOW('P', 0x5, unsigned long)
> > +#define PCITEST_READ_DMA _IOW('P', 0x5, struct pcitest_dma)
> > #define PCITEST_COPY _IOW('P', 0x6, unsigned long)
> > #define PCITEST_MSIX _IOW('P', 0x7, int)
> > #define PCITEST_SET_IRQTYPE _IOW('P', 0x8, int)
> > diff --git a/tools/pci/pcitest.c b/tools/pci/pcitest.c
> > index 6f1303104d84..66cd19acf18c 100644
> > --- a/tools/pci/pcitest.c
> > +++ b/tools/pci/pcitest.c
> > @@ -44,11 +44,14 @@ struct pci_test {
> > bool read;
> > bool write;
> > bool copy;
> > + bool dma;
> > + bool dma_list;
> > unsigned long size;
> > };
> >
> > static int run_test(struct pci_test *test)
> > {
> > + struct pcitest_dma dma;
> > int ret = -EINVAL;
> > int fd;
> >
> > @@ -113,7 +116,13 @@ static int run_test(struct pci_test *test)
> > }
> >
> > if (test->write) {
> > - ret = ioctl(fd, PCITEST_WRITE, test->size);
> > + if (test->dma) {
> > + dma.size = test->size;
> > + dma.list = test->dma_list;
> > + ret = ioctl(fd, PCITEST_WRITE_DMA, &dma);
> > + } else {
> > + ret = ioctl(fd, PCITEST_WRITE, test->size);
> > + }
> > fprintf(stdout, "WRITE (%7ld bytes):\t\t", test->size);
> > if (ret < 0)
> > fprintf(stdout, "TEST FAILED\n");
> > @@ -122,7 +131,13 @@ static int run_test(struct pci_test *test)
> > }
> >
> > if (test->read) {
> > - ret = ioctl(fd, PCITEST_READ, test->size);
> > + if (test->dma) {
> > + dma.size = test->size;
> > + dma.list = test->dma_list;
> > + ret = ioctl(fd, PCITEST_READ_DMA, &dma);
> > + } else {
> > + ret = ioctl(fd, PCITEST_READ, test->size);
> > + }
> > fprintf(stdout, "READ (%7ld bytes):\t\t", test->size);
> > if (ret < 0)
> > fprintf(stdout, "TEST FAILED\n");
> > @@ -163,7 +178,7 @@ int main(int argc, char **argv)
> > /* set default endpoint device */
> > test->device = "/dev/pci-endpoint-test.0";
> >
> > - while ((c = getopt(argc, argv, "D:b:m:x:i:Ilhrwcs:")) != EOF)
> > + while ((c = getopt(argc, argv, "D:b:m:x:i:IlhrwcdLs:")) != EOF)
> > switch (c) {
> > case 'D':
> > test->device = optarg;
> > @@ -204,6 +219,12 @@ int main(int argc, char **argv)
> > case 'c':
> > test->copy = true;
> > continue;
> > + case 'd':
> > + test->dma = true;
> > + continue;
> > + case 'L':
> > + test->dma_list = true;
> > + continue;
> > case 's':
> > test->size = strtoul(optarg, NULL, 0);
> > continue;
> > @@ -223,6 +244,8 @@ int main(int argc, char **argv)
> > "\t-r Read buffer test\n"
> > "\t-w Write buffer test\n"
> > "\t-c Copy buffer test\n"
> > + "\t-d DMA mode for read or write test\n"
> > + "\t-L Linked-List DMA flag for DMA mode\n"
> > "\t-s <size> Size of buffer {default: 100KB}\n"
> > "\t-h Print this help message\n",
> > argv[0]);
> > --
> > 2.7.4
> >