Re: [RFC PATCH Xilinx Alveo 0/6] Xilinx PCIe accelerator driver

From: Ronan KERYELL
Date: Fri Mar 29 2019 - 21:09:33 EST

I am adding linux-fpga@xxxxxxxxxxxxxxx, since this is why I missed this
thread in the first place...

>>>>> On Fri, 29 Mar 2019 14:56:17 +1000, Dave Airlie <airlied@xxxxxxxxx> said:

Hi Dave!

Dave> On Thu, 28 Mar 2019 at 10:14, Sonal Santan <sonals@xxxxxxxxxx> wrote:

>>> From: Daniel Vetter [mailto:daniel.vetter@xxxxxxxx]


>>> Note: There's no expectation for the fully optimizing compiler,
>>> and we're totally ok if there's an optimizing proprietary
>>> compiler and a basic open one (amd, and bunch of other
>>> companies all have such dual stacks running on top of drm
>>> kernel drivers). But a basic compiler that can convert basic
>>> kernels into machine code is expected.

>> Although the compiler is not open source the compilation flow
>> lets users examine output from various stages. For example if you
>> write your kernel in OpenCL/C/C++ you can view the RTL
>> (Verilog/VHDL) output produced by first stage of compilation.
>> Note that the compiler is really generating a custom circuit
>> given a high level input which in the last phase gets synthesized
>> into bitstream. Expert hardware designers can handcraft a circuit
>> in RTL and feed it to the compiler. Our FPGA tools let you view
>> the generated hardware design, the register map, etc. You can get
>> more information about a compiled design by running XRT tool like
>> xclbinutil on the generated file.

>> In essence compiling for FPGAs is quite different than compiling
>> for GPU/CPU/DSP. Interestingly FPGA compilers can run anywhere
>> from 30 mins to a few hours to compile a testcase.

Dave> So is there any open source userspace generator for what this
Dave> interface provides? Is the bitstream format that gets fed into
Dave> the FPGA proprietary and is it signed?

Short answer:

- a bitstream is an opaque content similar to various firmware handled
by Linux, EFI capsules, x86 microcode, WiFi modems, etc.

- there is no open-source generator for what the interface consume;

- I do not know if it is signed;

- it is probably similar to what Intel FPGA (not GPU) drivers provide
already inside the Linux kernel and I guess there is no pure
open-source way to generate their bit-stream either.

Long answer:

- processors, GPU and other digital circuits are designed from a lot of
elementary transistors, wires, capacitors, resistors... using some
very complex (and expensive) tools from some EDA companies but at the
end, after months of work, they come often with a "simple" public
interface, the... instruction set! So it is rather "easy" at the end
to generate some instructions with a compiler such as LLVM from a
description of this ISA or some reverse engineering. Note that even if
the ISA is public, it is very difficult to make another efficient
processor from scratch just from this ISA, so there is often no
concern about making this ISA public to develop the ecosystem ;

- FPGA are field-programmable gate arrays, made also from a lot of
elementary transistors, wires, capacitors, resistors... but organized
in billions of very low-level elementary gates, memory elements, DSP
blocks, I/O blocks, clock generators, specific
accelerators... directly exposed to the user and that can be
programmed according to a configuration memory (the bitstream) that
details how to connect each part, routing element, configuring each
elemental piece of hardware. So instead of just writing instructions
like on a CPU or a GPU, you need to configure each bit of the
architecture in such a way it does something interesting for
you. Concretely, you write some programs in RTL languages (Verilog,
VHDL) or higher-level (C/C++, OpenCL, SYCL...) and you use some very
complex (and expensive) tools from some EDA companies to generate the
bitstream implementing an equivalent circuit with the same
semantics. Since the architecture is so low level, there is a direct
mapping between the configuration memory (bitstream) and the hardware
architecture itself, so if it is public then it is easy to duplicate
the FPGA itself and to start a new FPGA company. That is unfortunately
something the existing FPGA companies do not want... ;-)

To summarize:

- on a CPU & GPU, the vendor used the expensive EDA tools once already
for you and provide the simpler ISA interface;

- on an FPGA, you have access to a pile of low-level hardware and it is
up to you to use the lengthy process of building your own computing
architecture using the heavy expensive very subtle EDA tools that will
run for hours or days to generate some good-enough placement for your

There is some public documentation on-line:

To have an idea of the elementary architecture:

Even on the configuration and the file format, but without any detailed semantics:

The Xilinx compiler xocc taking for example some LLVM IR and generating
some bitstream is not open-source and will probably never be for the
reasons above... :-(

Xilinx is open-sourcing all what can reasonably be open-sourced:

- the user-level and system run-time, including the OpenCL runtime: to handle the bitstreams generated by
some close-source tools

- the kernel device drivers which are already in but we want to upstream into the Linux
kernel to make life easier (this is the matter of this e-mail thread);

- to generate some real code in the most (modern and) open-source way,
there is an open-source framework to compile some SYCL C++ including
some Xilinx FPGA-specific extensions down to SPIR LLVM IR using
Clang/LLVM and to feed the close-source xocc tool with it

You can see starting from how
to start from C++ code, generate some SPIR LLVM IR and to feed xocc
and build a fat binary that will use the XRT runtime.

Some documentation in

There are other more official ways to generate bitstream (they are
called products instead of research projects like triSYCL :-) ).

We are also working on an other open-source SYCL compiler with Intel
to have a better common implementation and to upstream this into Clang/LLVM.

So for Xilinx FPGA, you can see the LLVM IR as the equivalent of PTX for
nVidia. But xocc is close-source for some more fundamental reasons: it
would expose all the details of the FPGA. I guess this is exactly the
same for Xilinx FPGA.

Note that probably most of the tool chains used to generate the
low-level firmware for the various CPU (microcode), GPU, etc. are
also close-source.

See you,
Ronan KERYELL, Xilinx Research Labs / San JosÃ, California.