Re: [PATCH 7/7] mailbox: goog-mba: Introduce the goog-mba mailbox driver

[Doug Anderson]

Hi,

On Sun, Sep 20, 2026 at 3:46 PM Jassi Brar <jassisinghbrar@xxxxxxxxx> wrote:
>
> > The numbers here for interrupt latency are made up for my example and
> > I haven't personally measured them, but I think it's not completely
> > absurd to say that interrupt latency (on both the Linux and remote
> > sides) dominates the communication path.
> >
> Yes, the numbers do look biased. It takes 50us for remote to get the
> irq and act upon it before ACKing but it takes 100us for that ACK to
> get back.
> And the benefit will be hard to achieve - it involves three unrelated
> clk_prepare() requests done within 50us often enough. When the stars
> align you save 300us on a clk_prepare()
>
> It feels you are trying to optimize a non-issue. clk_prepare() is
> expected to be slow and anyways shouldn't be frequent enough from all
> devices to give noticeable benefit.

Fair enough. I've jumped into a pre-existing design. Let me see if I
can find old information or gather evidence myself. Then with real
data we can figure out what makes sense. OK, gathered some data...

FWIW, to explain things clearly, I have been simplifying by saying
that just clock prepare/unprepare goes over this channel. In reality,
there is much more traffic. On Pixel 10, the mailbox using queue mode
like this connects to the "CPM" (central power manager). Looking at
the device tree, we see the following things using this mailbox:
* One of the main clock controllers in the system.
* Most of the power domains in the system
* Devfreq controllers
* Thermal controllers
* A GPIO controller
* A reset controller
* An interrupt controller
* A RTC
* A pile of other stuff

So basically a whole crap-ton of resources are managed over this
mailbox. The team designing this Phone apparently decided that the
mailbox is one of the primary communication pipelines in the system.
Yes, everything that communicates over the mailbox needs to be able to
sleep, but that doesn't mean we shouldn't keep it fast if possible.

On an off-the-shelf Pixel 10, one can spy on the CPM mailbox like this:

echo 1 > /sys/kernel/tracing/events/goog_mba_ctrl/enable
echo 1 > /sys/kernel/tracing/tracing_on
echo "" > /sys/kernel/tracing/trace
cat /sys/kernel/tracing/trace_pipe | grep 'process_q_t\|send_data_q'

When I do that, messages spew by pretty much constantly showing just
how busy this mailbox is. I can see instances where the queue is
actually utilized like this:

cat /sys/kernel/tracing/trace_pipe | \
  grep 'process_q_t\|send_data_q' | \
  grep -C10 'eqs_completed=[^1]\|anding_msg=[^0]'

If I do that, it's quite easy to see the queue being used. If I use
the device actively, I get several hits per second. A few examples
traces (using sed to shorten them slightly):

6946.884142: send_q: {0xa0016969,0xf,0x2,0x0} tx_q_wr_ptr=4
6946.884496: send_q: {0xa0026969,0x7,0x1,0x0} tx_q_wr_ptr=5
6946.884851: q_txdone: tx_q_rd_ptr=4 reqs_completed=2 outstanding_msgs=0
6946.884880: q_txdone: tx_q_rd_ptr=5 reqs_completed=1 outstanding_msgs=0

Here you can see that it took 709 us to get the response to the first
message. ...but, luckily we didn't have to wait for that 709 us before
sending the second message. That means we still got both responses at
once. Note that things aren't always so slow. The next messages
through this mailbox only took 42 us.

6946.885000: send_q: {0xa0036969,0x7,0x1,0x0} tx_q_wr_ptr=6
6946.885042: q_txdone: tx_q_rd_ptr=6 reqs_completed=1 outstanding_msgs=0

Here's a pretty big usage of the queue. I assume the CPM was busy at the time:

7751.863328: send_q: {0xa0050708,0x1020001,0x0,0x0} tx_q_wr_ptr=7
7751.863619: send_q: {0xa0060708,0x1030001,0x0,0x0} tx_q_wr_ptr=0
7751.863850: send_q: {0xa0070708,0x1000001,0x0,0x0} tx_q_wr_ptr=1
7751.864123: send_q: {0xa0080708,0x1010001,0x0,0x0} tx_q_wr_ptr=2
7751.864329: q_txdone: tx_q_rd_ptr=7 reqs_completed=4 outstanding_msgs=0
7751.864338: q_txdone: tx_q_rd_ptr=0 reqs_completed=3 outstanding_msgs=0
7751.864347: q_txdone: tx_q_rd_ptr=1 reqs_completed=2 outstanding_msgs=0
7751.864378: q_txdone: tx_q_rd_ptr=2 reqs_completed=1 outstanding_msgs=0

A full millisecond before the first response, but luckily we got them
all at once.

...and this one is pretty interesting here:

7782.321344: send_q: {0xa0040708,0x30001,0x0,0x0} tx_q_wr_ptr=4
7782.321481: send_q: {0xa0150708,0x10001,0x0,0x0} tx_q_wr_ptr=5
7782.321769: send_q: {0xa0060708,0x1,0x0,0x0} tx_q_wr_ptr=6
7782.321799: q_txdone: tx_q_rd_ptr=4 reqs_completed=2 outstanding_msgs=1
7782.321813: q_txdone: tx_q_rd_ptr=5 reqs_completed=1 outstanding_msgs=1
7782.321842: q_txdone: tx_q_rd_ptr=6 reqs_completed=1 outstanding_msgs=0

This one is interesting because we can see that the remote side first
ACKed two of the three outstanding messages. Then a short time later
it managed to ACK the last message.

...and just to show the queue being used quite often, here are two
instances right in a row (it's not hard to see this happening):

8521.693559: send_q: {0xa0010708,0x1030001,0x0,0x0} tx_q_wr_ptr=5
8521.693649: send_q: {0xa0120708,0x10001,0x0,0x0} tx_q_wr_ptr=6
8521.693690: q_txdone: tx_q_rd_ptr=5 reqs_completed=2 outstanding_msgs=0
8521.693701: q_txdone: tx_q_rd_ptr=6 reqs_completed=1 outstanding_msgs=0
8521.693920: send_q: {0xa0030708,0x1,0x0,0x0} tx_q_wr_ptr=7
8521.693956: q_txdone: tx_q_rd_ptr=7 reqs_completed=1 outstanding_msgs=0
8521.694171: send_q: {0xa0040705,0x2,0x14,0x1} tx_q_wr_ptr=0
8521.694194: send_q: {0xa0150708,0x1000001,0x0,0x0} tx_q_wr_ptr=1
8521.694214: q_txdone: tx_q_rd_ptr=0 reqs_completed=2 outstanding_msgs=0
8521.694226: q_txdone: tx_q_rd_ptr=1 reqs_completed=1 outstanding_msgs=0

...and in case you want to see an even bigger use of the queue, here
are 6 queued up at once:

9223.248560: send_q: {0xa0060403,0x3050600,0x0,0x0} tx_q_wr_ptr=7
9223.248598: q_txdone: tx_q_rd_ptr=7 reqs_completed=1 outstanding_msgs=0
9223.248824: send_q: {0xa007000b,0x1,0x418,0x7} tx_q_wr_ptr=0
9223.248960: send_q: {0xa008000b,0x1,0x18,0x7} tx_q_wr_ptr=1
9223.249289: send_q: {0xa0090008,0xa1800,0x0,0x0} tx_q_wr_ptr=2
9223.249489: send_q: {0xa00a0008,0x1803,0x0,0x0} tx_q_wr_ptr=3
9223.249612: send_q: {0xa00b0008,0xa1900,0x0,0x0} tx_q_wr_ptr=4
9223.250046: send_q: {0xa01c0403,0x3050600,0xffffffff,0x0} tx_q_wr_ptr=5
9223.250232: q_txdone: tx_q_rd_ptr=0 reqs_completed=6 outstanding_msgs=0
9223.250236: q_txdone: tx_q_rd_ptr=1 reqs_completed=5 outstanding_msgs=0
9223.250237: q_txdone: tx_q_rd_ptr=2 reqs_completed=4 outstanding_msgs=0
9223.250239: q_txdone: tx_q_rd_ptr=3 reqs_completed=3 outstanding_msgs=0
9223.250240: q_txdone: tx_q_rd_ptr=4 reqs_completed=2 outstanding_msgs=0
9223.250241: q_txdone: tx_q_rd_ptr=5 reqs_completed=1 outstanding_msgs=0
9223.250406: send_q: {0xa00d0008,0x1903,0x0,0x0} tx_q_wr_ptr=6
9223.250437: q_txdone: tx_q_rd_ptr=6 reqs_completed=1 outstanding_msgs=0


This looks pretty clearly worth it. Have I convinced you? Is there
other data you'd like to see?


> If you do have some real numbers and think it is worth it on your
> platform, then maybe expose each doorbell/shm-slot as a generic
> channel. clk_mailbox will request a generic channel, do the request
> and free it. The same effect but without inventing a new api. I can
> share a draft if you want, but I suggest let's not make things
> complicated without proven benefit.

FWIW, the downstream code in Pixel does what I think you're
suggesting. I can confidently say that, while it doesn't require
changes to the mailbox core, it is much more convoluted and
complicated. It also bleeds into the device-tree representation, which
doesn't feel great.

Just to be concrete, I'll document how the downstream driver works. If
this isn't what you were thinking, please correct me.

Back to our simplified "clk_mailbox" driver. We'll say that our
"clk_mailbox" driver talks over a single mailbox to the remote
processor. Let's say each message is 4 words big. The message space is
32-words big. 32 / 4 = 8 which means this space is divided into 8
queue slots. Downstream represents each of these queue slots as a
generic channel. That means that, in the device tree, our
"clk_mailbox" driver looks looks like this:

mboxes = <&cpm_tx_mba 0>,
         <&cpm_tx_mba 1>,
         <&cpm_tx_mba 2>,
         <&cpm_tx_mba 3>,
         <&cpm_tx_mba 4>,
         <&cpm_tx_mba 5>,
         <&cpm_tx_mba 6>,
         <&cpm_tx_mba 7>;

Then the "clk_mailbox" driver is in charge of rotating through each of
the channels. First it writes to channel 0, then it writes to channel
1, etc. This works with no changes to the core, but...

1. IMO, it's ugly. Logically, this is one communication channel
between the processor running Linux and the remote processor. We
shouldn't represent it as 8 channels. It feels especially bad to leak
this into the device tree.

2. The "clk_mailbox" driver needs to re-implement queuing and can't
use the mailbox core's queue. This is because the remote side
absolutely requires strict adherance to the "rotation" protocol in
order for it to receive messages. It expects a message in slot 0, then
slot 1, then slot 2, etc. If we used the mailbox core's queue, this
would break the protocol.

3. It may involve code duplication in the future. Currently, the only
driver using "queue mode" is the "CPM", but other drivers could
conceivably use it since many of the "LGA MBA" blocks support queue
mode in hardware.


I'm also a little confused about the resistance. I don't feel like the
mailbox core change is that complicated. The diffstat shows 58
insertions and 17 deletions. 14 of those added lines are comments.
While we certainly don't want to add useless APIs, to me this truly
seems like the correct way to add the functionality. It also doesn't
seem absurd to me that some future mailbox controller out there will
also support queuing like this.

-Doug