Re: [RFC V2] virtio: Add platform specific DMA API translation for virito devices

From: Michael S. Tsirkin
Date: Sun Jun 10 2018 - 23:28:32 EST


On Sun, Jun 10, 2018 at 07:39:09PM -0700, Ram Pai wrote:
> On Thu, Jun 07, 2018 at 07:28:35PM +0300, Michael S. Tsirkin wrote:
> > On Wed, Jun 06, 2018 at 10:23:06PM -0700, Christoph Hellwig wrote:
> > > On Thu, May 31, 2018 at 08:43:58PM +0300, Michael S. Tsirkin wrote:
> > > > Pls work on a long term solution. Short term needs can be served by
> > > > enabling the iommu platform in qemu.
> > >
> > > So, I spent some time looking at converting virtio to dma ops overrides,
> > > and the current virtio spec, and the sad through I have to tell is that
> > > both the spec and the Linux implementation are complete and utterly fucked
> > > up.
> >
> > Let me restate it: DMA API has support for a wide range of hardware, and
> > hardware based virtio implementations likely won't benefit from all of
> > it.
> >
> > And given virtio right now is optimized for specific workloads, improving
> > portability without regressing performance isn't easy.
> >
> > I think it's unsurprising since it started a strictly a guest/host
> > mechanism. People did implement offloads on specific platforms though,
> > and they are known to work. To improve portability even further,
> > we might need to make spec and code changes.
> >
> > I'm not really sympathetic to people complaining that they can't even
> > set a flag in qemu though. If that's the case the stack in question is
> > way too inflexible.
>
> We did consider your suggestion. But can't see how it will work.
> Maybe you can guide us here.
>
> In our case qemu has absolutely no idea if the VM will switch itself to
> secure mode or not. Its a dynamic decision made entirely by the VM
> through direct interaction with the hardware/firmware; no
> qemu/hypervisor involved.
>
> If the administrator, who invokes qemu, enables the flag, the DMA ops
> associated with the virito devices will be called, and hence will be
> able to do the right things. Yes we might incur performance hit due to
> the IOMMU translations, but lets ignore that for now; the functionality
> will work. Good till now.
>
> However if the administrator
> ignores/forgets/deliberatey-decides/is-constrained to NOT enable the
> flag, virtio will not be able to pass control to the DMA ops associated
> with the virtio devices. Which means, we have no opportunity to share
> the I/O buffers with the hypervisor/qemu.
>
> How do you suggest, we handle this case?

As step 1, ignore it as a user error.
Further you can for example add per-device quirks in virtio so it can be
switched to dma api. make extra decisions in platform code then.

> >
> >
> >
> > > Both in the flag naming and the implementation there is an implication
> > > of DMA API == IOMMU, which is fundamentally wrong.
> >
> > Maybe we need to extend the meaning of PLATFORM_IOMMU or rename it.
> >
> > It's possible that some setups will benefit from a more
> > fine-grained approach where some aspects of the DMA
> > API are bypassed, others aren't.
> >
> > This seems to be what was being asked for in this thread,
> > with comments claiming IOMMU flag adds too much overhead.
> >
> >
> > > The DMA API does a few different things:
> > >
> > > a) address translation
> > >
> > > This does include IOMMUs. But it also includes random offsets
> > > between PCI bars and system memory that we see on various
> > > platforms.
> >
> > I don't think you mean bars. That's unrelated to DMA.
> >
> > > Worse so some of these offsets might be based on
> > > banks, e.g. on the broadcom bmips platform. It also deals
> > > with bitmask in physical addresses related to memory encryption
> > > like AMD SEV. I'd be really curious how for example the
> > > Intel virtio based NIC is going to work on any of those
> > > plaforms.
> >
> > SEV guys report that they just set the iommu flag and then it all works.
>
> This is one of the fundamental difference between SEV architecture and
> the ultravisor architecture. In SEV, qemu is aware of SEV. In
> ultravisor architecture, only the VM that runs within qemu is aware of
> ultravisor; hypervisor/qemu/administrator are untrusted entities.

Spo one option is to teach qemu that it's on a platform with an
ultravisor, this might have more advantages.

> I hope, we can make virtio subsystem flexibe enough to support various
> security paradigms.

So if you are worried about qemu attacking guests, I see
more problems than just passing an incorrect iommu
flag.


> Apart from the above reason, Christoph and Ben point to so many other
> reasons to make it flexibe. So why not, make it happen?
>

I don't see a flexibility argument. I just don't think new platforms
should use workarounds that we put in place for old ones.


> > I guess if there's translation we can think of this as a kind of iommu.
> > Maybe we should rename PLATFORM_IOMMU to PLARTFORM_TRANSLATION?
> >
> > And apparently some people complain that just setting that flag makes
> > qemu check translation on each access with an unacceptable performance
> > overhead. Forcing same behaviour for everyone on general principles
> > even without the flag is unlikely to make them happy.
> >
> > > b) coherency
> > >
> > > On many architectures DMA is not cache coherent, and we need
> > > to invalidate and/or write back cache lines before doing
> > > DMA. Again, I wonder how this is every going to work with
> > > hardware based virtio implementations.
> >
> >
> > You mean dma_Xmb and friends?
> > There's a new feature VIRTIO_F_IO_BARRIER that's being proposed
> > for that.
> >
> >
> > > Even worse I think this
> > > is actually broken at least for VIVT event for virtualized
> > > implementations. E.g. a KVM guest is going to access memory
> > > using different virtual addresses than qemu, vhost might throw
> > > in another different address space.
> >
> > I don't really know what VIVT is. Could you help me please?
> >
> > > c) bounce buffering
> > >
> > > Many DMA implementations can not address all physical memory
> > > due to addressing limitations. In such cases we copy the
> > > DMA memory into a known addressable bounc buffer and DMA
> > > from there.
> >
> > Don't do it then?
> >
> >
> > > d) flushing write combining buffers or similar
> > >
> > > On some hardware platforms we need workarounds to e.g. read
> > > from a certain mmio address to make sure DMA can actually
> > > see memory written by the host.
> >
> > I guess it isn't an issue as long as WC isn't actually used.
> > It will become an issue when virtio spec adds some WC capability -
> > I suspect we can ignore this for now.
> >
> > >
> > > All of this is bypassed by virtio by default despite generally being
> > > platform issues, not particular to a given device.
> >
> > It's both a device and a platform issue. A PV device is often more like
> > another CPU than like a PCI device.
> >
> >
> >
> > --
> > MST
>
> --
> Ram Pai