[RFC] Proposal for FMC Subsystem Eradication

[Federico Vaga]

=======================
The fmc-bus Eradication
=======================

The *fmc-bus* Linux sub-system was developed under CERN supervision
around 2012. During these years, the fmc-bus has evolved and drivers
for particular FMC applications (FMC DEL, FMC ADC, FMC TDC, etc.)
have been based on it. The sub-system works fine and it is running in
the biggest particle accelerator on Earth without major problems.

Despite the fact that the fmc-bus works, the faults in its design
reveal, with the experience acquired, that its use is unduly limiting
and, actually, counter productive.

The purpose of this document is to explain the rationale behind
fmc-bus eradication from the Linux kernel mainline and to propose a
better alternative.

The fmc-bus
===========

The FMC Standard
----------------

The FMC standard (`VITA-57.1`_) describes FMC modules and carriers
from the mechanical and the electrical point of view. `VITA-57.1`_
also specifies the information stored in the (mandatory) EEPROM on the
FMC module.
On the other hand, `VITA-57.1`_ does not specify at all how FMC carriers and
its FMC modules communicate.

>From the software engineering point of view, the most interesting parts of
this standard are the following chapters:

5.5 I2C bus signal
  This chapter specifies that an I2C bus connecting an FMC carrier
  and its FMC modules is mandatory. Consequently, it specifies the FMC
  signals that are reserved for that purpose.

  By this chapter, an FMC module must contain an EEPROM connected to that
  I2C bus. The purpose of this EEPROM is to store
  *hardware definition information* using the
  `Platform Management FRU Information Storage Definition V1.0`_

  Optionally, it can support the base IPMI commands defined in the
  *PICMG 2.6* specification.

  So, from a software point of view, we are interested in reading the FRU
  information that provides useful data to identify the FMC modules, and
  to do so at a standardized I2C address and EEPROM programming interface.

5.6 Geographical address
  This chapter describes how to address the different I2C devices hosted
  in the FMC mezzanines that live in a single FMC carrier. As the
  standard specifies up to four mezzanines per I2C bus, it enforces the
  convention that the two least significant bits of the I2C address
  identify the slot.
  For example, the mandatory I2C EEPROM must be addressable at the 7-bit
  address b10100XX, the suffix XX=00,01,10,11 determining the geographical
  addressing of the module in the carrier. Other I2C devices follow the same
  schema.


The fmc-bus Design
------------------

The Linux fmc-bus was designed in 2012 with the purpose of adding
generic support for the FMC modules that were under development at CERN
at that time.
Everything was targeted to work with Linux kernels 2.6.24 and later.

The idea behind the Linux fmc-bus subsystem is to enumerate the FMC modules
and to provide a driver for them using the Linux bus mechanism. With this idea
in mind, the Linux fmc-bus uses the information contained in the FMC module
EEPROM to do the match against the available fmc-bus drivers. Then, the fmc-
bus
drivers will use a set of functions from the FMC bus to make use of
the resources available on the FMC carrier (which is plugged on another Linux
sub-system, for example: PCI or VME).

Flawed conceptual assumptions in the fmc-bus
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The main wrong assumption in the Linux fmc-bus is the idea that drivers
are bound to FMC modules through the carrier FPGA: this is not true.
Behind this lies another wrong assumption: that FMC modules are in a
one-to-one relationship with the FPGA code: this is wrong. The actual
relationship is one-to-many: one FMC module can be driven by many FPGA
code designs, which might be wildly varying. The real behavior of a card
(FMC carrier + FMC module[s]) is determined by the FPGA code that has been
loaded on the FPGA.

A simple example will clarify the concept.
We can have an FMC module with a simple 5 digital I/O channel. We all know 
that
such module is versatile, so we can have FPGA code that implements: two I2C
masters, an SPI master, a GPIO interface, a TDC, custom logic. In all these
cases the FMC module is the same but the FPGA code that drives it is 
different,
as well as the Linux driver that should drive it.

So, the fmc-bus design is wrong in trying to match devices and drivers using
the FRU information from the FMC module EEPROM. Instead, the real match must
be done between the devices instantiated internally in the FPGA code and their
drivers.

Eradication Proposal
====================

The fmc-bus *today* and in its *current status* is not just inadequate: it has
too many overlaps with other Linux components (it was not the case back
in 2012). It seems appropriate to completely replace it by an FMC device 
class.
Here is how to do it and how existing drivers (not in Linux mainline) should
be migrated to it.

FMC Device Class
----------------
I propose to retain all the functionality of the fmc-bus that should
remain (see below) in a kernel module implementing an FMC device class
that registers the presence of individual FMC modules.

The new FMC device class must be limited to the real FMC business logic. This
means that existing Linux kernel interfaces will be reused, instead of
implementing dedicated interfaces as the fmc-bus does.
There are four overlaps between the Linux fmc-bus and other Linux
sub-systems/drivers: fpga-manager, i2c sub-system, eeprom drivers and
irq management.

FPGA Manager
  The Linux fmc-bus asks the FMC carrier driver to export the necessary
  operations to program the FMC carrier FPGA. Currently, this is handled
  by the Linux kernel FPGA manager subsystem.
  So, the FMC carrier driver should create a device instance for the FPGA
  manager.
I2C Master Driver
  The Linux fmc-bus current implementation asks the FMC carrier drivers to
  implement the I2C operations in order to access the EEPROM. In theory this
  is not really necessary; Linux provides a nice I2C sub-system for this
  purpose. Only the particular I2C master instance needs to be registered by
  the FMC carrier driver. A driver for that master has to be provided only if
  missing in the stock Linux kernel.
EEPROM Driver
  The Linux fmc-bus uses the I2C programming interface of an FMC module EEPROM
  to extract its contents, in particular the module identification for driver
  Again, this is rendered superfluous by the Linux kernel I2C EEPROM
  drivers. There is, however, a difficulty with the particular EEPROM
  to use: which one is it? `VITA-57.1`_ is unclear about this.
  rule 5.69 about what kind of EEPROM is mandatory:

      Rule 5.69: The IO mezzanine module shall provide an onboard EEPROM,
      which shall interface with the I2C bus signals.

      Rule 5.74: Data fields provided by the IPMI resource on the mezzanine
      module shall include the minimum records defined in the Platform
      Management FRU Information Storage Definition V1.0.

  FRU Information Storage Definition V1.0:

      are SEEPROMs that use a '24C02'-compatible interface, and SEEPROMs that
      are compatible with the Dallas Semiconductor DS1624 Temperature
      Sensor/SEEPROM interface.

  to support 24C02 EEPROMs. Support of other EEPROMs should not be
  excluded, though, as `VITA-57.1`_ is admittedly open to misinterpretaion
  on this point.
IRQ Controller
  The Linux fmc-bus asks the FMC carrier to implement the operations to
  request, free and acknowledge interrupts. This is necessary because
  FMC modules plugged into an FMC carrier share the unique interrupt
  line or vector of the latter. However, this function does not belong
  in the fmc-bus. As the obvious need for such logic suggests,
  the FMC carrier contains interrupt routing logic; thus, an IRQ
  controller is the proper way to handle this.
This approach will significantly reduce the FMC scope to its true business.
Most of the existent code will be re-used and the future FMC carrier drivers
need only to glue toghether the device instances required for them to run.

Another duty left to the FMC class is to export an interface to extract the
information from the FMC module EEPROM.

Platform Device For FPGA Internal Devices
-----------------------------------------
As stated before, there is no such thing as an "FMC module driver", nor
a "device driver for the FPGA code".
The FPGA code is typically made of N ip-cores; groups of ip-cores may need
different drivers.
The simplest case to imagine is an FMC carrier with one FPGA and 4 FMC slots.
Within the same FPGA code we have different ip-cores to drive potentially 4
different FMC modules.
So, the only way I see it possible is by using platform drivers for the
*FPGA internal devices*. The description of the internal devices is not within
the scope of the FMC software layer.

.. _`Platform Management FRU Information Storage Definition V1.0`: https://
www.intel.com/content/www/us/en/servers/ipmi/information-storage-definition-