RFC: Xilinx FPGA based accelerator kernel drivers

[Sonal Santan]

Hello,

Xilinx would like to begin upstreaming kernel drivers used with our Alveo FPGA accelerator cards. These drivers are part of Xilinx Runtime (XRT) open source stack and have been deployed by leading FaaS vendors and many enterprise customers.

The PCIe drivers are available for review on GitHub against drm-next tree--

https://github.com/sonals/XRTDrvUpstream

The ioctls are defined in the following two files:
https://github.com/sonals/XRTDrvUpstream/blob/drm-next/include/uapi/drm/xmgmt_drm.h
https://github.com/sonals/XRTDrvUpstream/blob/drm-next/include/uapi/drm/xocl_drm.h

The new drivers are located at: 
https://github.com/sonals/XRTDrvUpstream/tree/drm-next/drivers/gpu/drm/xocl

PLATFORM ARCHITECTURE

Alveo PCIe platforms have a static shell and a reconfigurable (dynamic) region. The shell is automatically loaded from PROM when host is booted and PCIe is enumerated by BIOS. Shell cannot be changed till next cold reboot. The shell has two physical functions: Management Physical Function and User Physical Function. The reconfigurable region contents are compiled by the user using SDx compiler tool chain like xocc which produces xclbin file.

XRT DRIVERS

XRT Linux kernel driver xclmgmt binds to mgmt pf. The driver is modular and organized into several platform drivers which handle the following functionality:
1.   ICAP programming (FPGA bitstream download with FPGA Mgr integration)
2.   Clock scaling
3.   Loading firmware container also called dsabin (embedded Microblaze firmware for ERT and XMC, optional clearing bitstream)
4.   In-band sensors: temp, voltage, power, etc.
5.   AXI Firewall management
6.   Access to Flash programmer
7.   Device reset and rescan
8.   Hardware mailbox
9.   Interrupt handling for AXI Firewall and mailbox
10. ECC handling

XRT Linux kernel driver xocl binds to user pf. The driver is modular and organized into several platform drivers which handle the following functionality:
1.   Device memory topology discovery and memory management 
2.   Buffer object abstraction and management for client process 
3.   XDMA MM PCIe DMA engine programming 
4.   Multi-process aware context management 
5.   Standardized compute unit execution management (optionally with help of ERT) for client processes 
6.   Interrupt handling for DMA, compute unit completion and mailbox

The drivers export ioctls and sysfs nodes for various services. xocl driver makes heavy use of DRM GEM features for device memory management, reference counting, mmap support and export/import. xocl also has a very simple scheduler called KDS which schedules compute units and interacts with hardware scheduler running ERT firmware. The scheduler interface is somewhat similar to GPUs with a custom command object format with an asynchronous done notification via POSIX poll.

More details of our architecture, software APIs, ioctl definitions, execution model, etc. is available as Sphinx documentation--
 
https://xilinx.github.io/XRT/2018.3/html/index.html

The complete runtime software stack (XRT) which includes out of tree kernel drivers, user space libraries, board utilities and firmware for the hardware scheduler is open source and available at https://github.com/Xilinx/XRT. Both PCIe based Alveo boards and Zynq Ultrascale+ MPSoC based embedded platforms are supported by XRT. We will work on upstreaming the driver for embedded platforms after PCIe drivers have been merged upstream. 

I am not subscribed to dri-devel so please cc me in the replies.

-Sonal