Re: [PATCH v2] iommu/arm-smmu: Break insecure users by disabling bypass by default

[Tim Harvey]

On Fri, Oct 4, 2019 at 11:34 AM Robin Murphy <robin.murphy@xxxxxxx> wrote:
>
> On 04/10/2019 18:13, Tim Harvey wrote:
> [...]
> >>> No difference... still need 'arm-smmu.disable_bypass=n' to boot. Are
> >>> all four iommu-map props above supposed to be the same? Seems to me
> >>> they all point to the same thing which looks wrong.
> >>
> >> Hmm... :/
> >>
> >> Those mappings just set Stream ID == PCI RID (strictly each one should
> >> only need to cover the bus range assigned to that bridge, but it's not
> >> crucial) which is the same thing the driver assumes for the mmu-masters
> >> property, so either that's wrong and never could have worked anyway -
> >> have you tried VFIO on this platform? - or there are other devices also
> >> mastering through the SMMU that aren't described at all. Are you able to
> >> capture a boot log? The SMMU faults do encode information about the
> >> offending ID, and you can typically correlate their appearance
> >> reasonably well with endpoint drivers probing.
> >>
> >
> > Robin,
> >
> > VFIO is enabled in the kernel but I don't know anything about how to
> > test/use it:
> > $ grep VFIO .config
> > CONFIG_KVM_VFIO=y
> > CONFIG_VFIO_IOMMU_TYPE1=y
> > CONFIG_VFIO_VIRQFD=y
> > CONFIG_VFIO=y
> > # CONFIG_VFIO_NOIOMMU is not set
> > CONFIG_VFIO_PCI=y
> > CONFIG_VFIO_PCI_MMAP=y
> > CONFIG_VFIO_PCI_INTX=y
> > # CONFIG_VFIO_PLATFORM is not set
> > # CONFIG_VFIO_MDEV is not set
>
> No worries - since it's a networking-focused SoC I figured there was a
> chance you might be using DPDK or similar userspace drivers with the NIC
> VFs, but I was just casting around for a quick and easy baseline of
> whether the SMMU works at all (another way would be using Qemu to run a
> VM with one or more PCI devices assigned).
>
> > I do have a boot console yet I'm not seeing any smmu faults at all.
> > Perhaps I've mis-diagnosed the issue completely. To be clear when I
> > boot with arm-smmu.disable_bypass=y the serial console appears to not
> > accept input in userspace and with arm-smmu.disable_bypass=n I'm fine.
> > I'm using a buildroot initramfs rootfs for simplicity. The system
> > isn't hung as I originally expected as the LED heartbeat trigger
> > continues blinking... I just can't get console to accept input.
>
> Curiouser and curiouser... I'm inclined to suspect that the interrupt
> configuration might also be messed up, such that the SMMU is blocking
> traffic and jammed up due to pending faults, but you're not getting the
> IRQ delivered to find out. Does this patch help reveal anything?
>
> http://linux-arm.org/git?p=linux-rm.git;a=commitdiff;h=29ac3648b580920692c9b417b2fc606995826517
>
> (untested, but it's a direct port of the one I've used for SMMUv3 to
> diagnose something similar)

This shows:
arm-smmu 830000000000.smmu0: Unexpected global fault, this could be serious
arm-smmu 830000000000.smmu0:     GFSR 0x80000002, GFSYNR0 0x00000002,
GFSYNR1 0x00000140, GFSYNR2 0x00000000
arm-smmu 830000000000.smmu0: Unexpected global fault, this could be serious
arm-smmu 830000000000.smmu0:     GFSR 0x80000002, GFSYNR0 0x00000002,
GFSYNR1 0x00000010, GFSYNR2 0x00000000
arm-smmu 830000000000.smmu0: Unexpected global fault, this could be serious
arm-smmu 830000000000.smmu0:     GFSR 0x80000002, GFSYNR0 0x00000002,
GFSYNR1 0x00000010, GFSYNR2 0x00000000
arm-smmu 830000000000.smmu0: Unexpected global fault, this could be serious
arm-smmu 830000000000.smmu0:     GFSR 0x80000002, GFSYNR0 0x00000002,
GFSYNR1 0x00000010, GFSYNR2 0x00000000
arm-smmu 830000000000.smmu0: Unexpected global fault, this could be serious
arm-smmu 830000000000.smmu0:     GFSR 0x80000002, GFSYNR0 0x00000002,
GFSYNR1 0x00000010, GFSYNR2 0x00000000
arm-smmu 830000000000.smmu0: Unexpected global fault, this could be serious
arm-smmu 830000000000.smmu0:     GFSR 0x80000002, GFSYNR0 0x00000002,
GFSYNR1 0x00000010, GFSYNR2 0x00000000
arm-smmu 830000000000.smmu0: Unexpected global fault, this could be serious
arm-smmu 830000000000.smmu0:     GFSR 0x80000002, GFSYNR0 0x00000002,
GFSYNR1 0x00000010, GFSYNR2 0x00000000
arm-smmu 830000000000.smmu0: Unexpected global fault, this could be serious
arm-smmu 830000000000.smmu0:     GFSR 0x80000002, GFSYNR0 0x00000002,
GFSYNR1 0x00000010, GFSYNR2 0x00000000
arm-smmu 830000000000.smmu0: Unexpected global fault, this could be serious
arm-smmu 830000000000.smmu0:     GFSR 0x80000002, GFSYNR0 0x00000002,
GFSYNR1 0x00000010, GFSYNR2 0x00000000
arm-smmu 830000000000.smmu0: Unexpected global fault, this could be serious
arm-smmu 830000000000.smmu0:     GFSR 0x80000002, GFSYNR0 0x00000002,
GFSYNR1 0x00000010, GFSYNR2 0x00000000
...
arm-smmu 830000000000.smmu0: Unexpected global fault, this could be serious
arm-smmu 830000000000.smmu0:     GFSR 0x80000002, GFSYNR0 0x00000002,
GFSYNR1 0x00000010, GFSYNR2 0x00000000
^^^ these two repeat over and over

>
> That said, it's also puzzling that no other drivers are reporting DMA
> errors or timeouts either - is there any chance that some device is set
> running by the firmware/bootloader and not taken over by a kernel driver?
>

anything is possible - I'm using the Cavium 'BDK' as boot firmware to
configure the board which sits in from of arm trusted firmare and
bootloader.

Tim