Re: [PATCH 00/23] Subject: [PATCH net-next 00/23] net/hpe: add HPE Slingshot NIC drivers

[Doug Ledford]

Top-posting an executive summary: HPE's ask is whether we should submit
this driver through the netdev community or the rdma community. HPE
prefers netdev, with RDMA as an aux driver, but needs the maintainers’
call. A minimal Ethernet driver is about 48k lines plus 36k lines of
mostly machine-generated headers (details below).

On 11/2/26 6:26 AM, Andrew Lunn wrote:
> On Sun, Nov 01, 2026 at 07:45:24PM -0800, Doug Ledford wrote:
>> On 11/1/26 5:35 PM, Andrew Lunn wrote:
>>> Can you add a single portal entry which matches everything? Or matches
>>> a single MAC address? KISS.

In order to get the hardware engine to process Ethernet frames, either
sending or receiving, you must create a portal table entry (PtlTE,
basically a receive filter with its own queues, but it also defines what
wire format this entry uses) with .ethernet = 1.  You can do things with
a single PtlTE.  We have a specific PtlTE for promisc, so that can be
used.  We also define a PtlTE for any unicast, multicast, all-multicast,
etc. addresses.  But, yes, you can start with just a single one.

I think it's worth noting here though that this does not imply small,
concise code.  Portals 4.0 API is an RDMA API (link to the API at the
end of this email).  The hardware itself is RDMA capable. It directly
supports sending Ethernet traffic via hardware like other Ethernet
devices, but the similarities pretty much stop there.

The Ethernet layer is implemented using special PtlTEs that set up the
hardware for processing Ethernet wire frames, both on send and receive.

That means in order to just get to the point of being able to send an
Ethernet frame on the wire, we must first set up the "Portals machinery"
let's call it.  There are command queues, event queues, two forms of
address mapping, traffic classes, communication profiles, a special
system service id reserved for Ethernet that must be allocated, accessor
functions that know how to build the commands necessary to write to the
command queues in order to get the hardware to send packets, accessors
for the event queues, and even more stuff that has to happen.  I get
what you are asking for, I'm just trying to make it clear that even KISS
in this case is going to be more than you are used to by a wide margin.

Once done, the device will be up and running with send and receive
Ethernet wire formats working.  I have stripped the ss1 driver (the
Slingshot Cassini core driver, everything else hangs off of it as an aux
device) down to the minimum to send frames over the wire.  I hadn't
stripped it to only do one address type.  To additionally strip it down
to just one PtlTE would only result in negligible savings.

So far, I've been discussing the core ss1 driver.  That is the machinery
that handles the Portals API of the card and implements the Ethernet
support.  It has two aux drivers that hang off of its main device.  They
are the sbl driver (Cassini1 link layer driver) and sl driver (Cassini2
and upcoming Cassini3 link layer driver).  In an earlier email, you made
this statement "all your PCS driver needs to do is poll for link. A
typical PCS driver is around 2K lines of code, and that actually does
more" encouraging me to strip it down further.  I think the difference
here is that Slingshot defines its own wire format, and its own high
performance link layer elements, and there was no off the shelf PHY we
could use.  So we don't have a PCS driver, nor could we use anything in
the phy subsystem as none of them support our link layer.  The sbl and
sl drivers implement the management of the PCS, PMA, PMD, FEC, and five
Slingshot specific wire features (32-byte minimum frames, no
inter-packet gap, link-level retransmission (LLR), lane degrade,
low-latency FEC) all in software because there is no firmware existing
to do so. That makes them a protocol implementation, so just getting to
link up requires significant code.

We have stripped the two link drivers of all optional items, but we left
what's necessary to actually drive the link layer as removing any of it
would make the links non-functional.  The minimal driver removes
management, reporting, virtualization and offload surfaces (sysfs,
debugfs, ethtool, netlink, configfs, SR-IOV, telemetry, PTP, GPU memory
paths) while retaining the complete link-layer implementation — LLR,
lane degrade, FEC admission/monitoring policy, HPC framing, and full
SerDes/PCS/MAC/AN bring-up.

That they are an implementation of our proprietary protocol is also why
I left them in the hpe directory and did not attempt to move them into
the phy subsystem or use phylink.  Should the netdev community decide it
is the right home for the driver, we can re-evaluate at that time.  The
existing phylink contract does not include the items our hardware needs,
for example the MAC interface only handles (speed/duplex/pause) and we
would need it to also support our Link Layer Retry, per-direction lane
degrade, and FEC policy.  For our 802.3 mode, we do support mapping our
items to the standard ethtool elements.  That makes these drivers
considerably larger than what I think you had envisioned they should be.
 These two drivers have been stripped to their minimum, so that ethtool
mapping will be gone.  To make the submission process easier, we can
limit the first round of driver submissions to just one of these two
drivers, but in order for the core ss1 driver not to effectively be dead
code, one of these two also must be submitted.

> 
> The point is you want a very simple driver which it is possible to
> review.

I want to back things up a step.  I think my cover letter was worded
poorly and I wasn't clear on what I was asking the netdev community for.

My first request was an answer to the question: Should this driver go
through the netdev community or should it go through the rdma community?
 I should have thrown RFC in the subject line.  So please think about
the responses I've made in this thread with that in mind. I've been
trying to bring out details that I thought the community would need in
order to answer that ask.

HPE would prefer to have it go through the netdev community.  We would
prefer this for two basic reasons: 1) It is capable of being used as a
plain Ethernet device, sending bare wire format Ethernet packets
compatible with any Ethernet switch and other vendor's hardware.  But it
does implement it via Portals technology and not the normal RX/TX
descriptor rings the netdev community is used to; 2) If it goes through
the rdma community then our Ethernet support would be as a thin shim
layer just implementing a virtual netdev on top of the Portals
infrastructure that would reside in the rdma stack, that would mean our
netdev would always require the rdma driver to be up and running before
it can be brought up.  This would invert the normal relationship between
any truly Ethernet capable device (like mlx5, cxgb4, etc.) and the rdma
device.  To my knowledge, the only devices that have this inverted
relationship today are ones that do not have true hardware Ethernet
support like InfiniBand, OPA, etc. and they all have to encapsulate
their Ethernet packets in some way to send them over the wire, whereas
we have actual Ethernet wire format support.

I submitted the driver in whole to be transparent about what this driver
in its final form might look like.  What our hardware is, how it works,
and how it differs from ordinary Ethernet NICs.  We will gladly break
the submission down into parts that are easier to consume, but this way
the community knows where we are headed eventually.  No doubt a final
version of the driver would not look the same, but the submission in
whole provides an idea.

The second request, assuming the netdev community decided that this
driver could/should go through here, was guidance on the sysfs usage.  I
had already removed the 1,925 telemetry items from sysfs, but there were
several hundred still in place.  I was just looking for any sort of
standard about what acceptable use is.  In the meantime, I've stripped
the remaining sysfs items too.  But I would still like guidance on when
sysfs is usable and not.

Those were my two asks with the initial submission, and they both still
remain.  So, just to be perfectly explicit, I'm asking the netdev
community to say: netdev or rdma community.  There are valid arguments
for both options.

If the netdev community wants this to come through them, then we can
start the process of breaking things down, etc.  I can start with SerDes
driver(s), then the Portals machinery, then the Ethernet support as a
series of chained submissions to be merged as one once approved.  From
there we can add the RDMA aux driver plumbing, link up the RDMA portion
of the driver, and then we can start adding the remaining components
such as ethtool, sysfs (potentially, subject to review), debugfs,
netlink, GPU RDMA support, etc.

If you need to see that minimal driver to answer my ask, I have a first
cut on hand now.  It has been built, but not run tested.  I would run
test it and then file a new submission to the list for review.

The recommended tree for submission at a minimal level:

Component    Submitted    Minimal      First  Why
             lines/files  lines/files  round
-----------  -----------  -----------  -----  --------------------------
ss1 core     57,378/95    32,824/64    yes    Portals machinery and the
                                              Ethernet support; the
                                              other drivers hang off it.
sbl (C1)     17,911/39    15,005/34    yes    Cassini1 link driver, the
                                              smallest, with the same
                                              structure as sl. One link
                                              driver is needed so ss1 is
                                              not dead code.
sl (C2/C3)   71,906/238   32,580/166   no     Cassini2/3 link driver,
                          19,015/1*           much larger. Same
                                              structure as sbl, so it
                                              can follow once the
                                              pattern has been reviewed.
headers      37,640/34    36,283/32    yes    Shared by ss1 and the link
                                              drivers; mostly
                                              machine-generated hardware
                                              definitions.
-----------  -----------  -----------  -----  --------------------------

Original submission:        184,835 lines / 406 files
First round (ss1 + sbl +
  headers):                  84,112 lines / 130 files
  of which code:             47,829 lines
  of which headers:          36,283 lines
Everything minimal (adds sl): 135,707 lines / 297 files
                             (116,692 without the ucode header)

Note: The sbl driver is Slingshot link version 1, sl is link version 2.
The sl driver is larger because not all features of the second
generation link are present in the first.

* - there also is a 19,015 line header comprised of generated ucode in
the sl driver.  That will get dropped when the binary it produces lands
in linux-firmware.

References:

This is the Portals 4.0 API, which is what our hardware is based on.
https://www.sandia.gov/research/publications/details/the-portals-4-0-network-programming-interface-2012-11-01/

The current version is 4.3 and can be found on OSTI.
https://www.osti.gov/biblio/1875218

-- 
Doug Ledford <doug.ledford@xxxxxxx>
    GPG KeyID: B826A3330E572FDD
    Key fingerprint = AE6B 1BDA 122B 23B4 265B  1274 B826 A333 0E57 2FDD

Attachment: OpenPGP_signature.asc
Description: OpenPGP digital signature