[PATCH 0/5] jump_label: mitigate dynamic-debug induced IPI storms

[Jim Cromie]

This series adds batched toggling of jump_labels to address IPI storms
triggered by bulk toggles of dynamic-debug pr_debug()s.

patches 1-3 are for submission. 4-5 are for demonstration only.

The Scaling Problem:

In testing on 7.1-rc6 guests (virtme-ng) hosted on a stock Fedora
kernel, I measured the cost of toggling ~1,900 dyndbg callsites. The
current one-at-a-time patching approach creates a massive
synchronization bottleneck that scales linearly with both key count
and core count:

Baseline IPIs (Total CAL Interrupts)
Successive query pairs (+p ; -p ;) toggling ~3,792 keys per pair.
Pairs | -p 2 | -p 3 | -p 4 | -p 5 | Notes
-----------------------------------------------------------
  1   | 10,738 |  21,468 |  32,148 |  42,837 | ~11.3 IPI/key/CPU
  2   | 21,269 |  42,976 |  64,416 |  85,880 | Linear scaling (K*M)
  3   | 32,125 |  64,434 |  96,604 | 128,793 | 
  4   | 42,831 |  85,886 | 128,776 | 171,697 | 
  5   | 53,540 | 107,353 | 160,985 | 214,636 | 
  6   | 64,174 | 128,800 | 193,159 | 257,562 | > 250k IPI storm

On a 5-VCPU system, a sequence of 6 query pairs triggers over a 
quarter-million interrupts. This overhead is a direct result of 
invoking the full patching machinery for every individual key toggle.

The Solution:

While x86 already has a 256-entry queue for jump_label, 2 issues
currently block its use:

1. dyndbg uses the synchronous API; an IPI is issued for every static-key.
   patch-1 provides a fix for this with a _queued api extension.
   patch-3 uses the new api in dyndbg.

2. The queue flushes if current_patch_addr < previous_patch_addr.
   dyndbg has consecutive descriptors, but not consecutive patch_addrs.
   Using the API with just patch-1 yields ~1/2 the IPIs.
   Patch 2 sorts the patch_addrs, allowing the queue to fill to its 
   256-entry capacity before flushing, for a bigger win.

Results Summary:

Jump-Batch IPIs (Total CAL Interrupts)
Pairs | -p 2  | -p 3  | -p 4  | -p 5  |
---------------------------------------
  1   |  135  |  202  |  239  |  290  |
  2   |  145  |  256  |  347  |  445  |
  3   |  198  |  352  |  479  |  640  |
  4   |  246  |  445  |  630  |  840  |
  5   |  293  |  545  |  774  | 1,033 |
  6   |  341  |  642  |  915  | 1,222 |

Marginal Cost Comparison (IPIs added per '+p;-p;' query pair)
VCPUs | Baseline (m) | Jump-Batch (m) | Efficiency Gain
-------------------------------------------------------
  2   |    10,696    |      42.4      |      252x
  3   |    21,466    |      89.3      |      240x
  4   |    32,207    |     138.1      |      233x
  5   |    42,945    |     190.5      |      225x

Why not static_key_deferred?

static_key_deferred is a rate-limiter for single, high-frequency keys.
For a bulk update of 1,900 distinct keys, a deferred timer would still
fire 1,900 times, resulting in the same IPI storm spread across a
longer window.  This will make it harder to use, particularly for
tools which are fast enough to perceive the asynchrony.  In contrast,
jump-batch just amortizes the IPI cost, eliminating the need for
rate-throttling.

Why wasnt this done previously ?

The presence of the queue for x86 suggests it was done, minimally, for
the cases where the patch-addresses were ordered, and where
performance mattered more (ftrace etc).  In contrast, pr_debug()s are
pretty basic, for direct use by humans, who dont toggle them on and
off at high rates, and who are far more selective than "+p" the 2nd
time they use dyndbg.

Model Summary:

The baseline cost is driven by (Keys * CPUs), resulting in 43k IPIs 
per transaction on a 5-VCPU VM. The jump-batch solution reduces 
this to ~190 IPIs, moving the bottleneck from key density (K) 
to architectural rendezvous cost (M).

Detailed Data:

NB: most tests below used the " !module virtio* +p ; -p ; "
query-pair, relying on patch-5 for the "!module virtio*" parsing to
avoid flooding the guest/VM logs with messages.  This is less
important when both on;off are sent together, as console flooding
impedes submitting a separate off command.

Baseline:

jimc@frodo:~/projects/lx/linux.git$ for_bo_ -p 1 ../../dd-tools.rc dd_ipis
b0-dd:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1897 Delta-CAL (IPI): 0
b0-dd-drm:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1849 Delta-CAL (IPI): 0
b0-defcon:
 doing: dd_ipis
../../dd-tools.rc: line 79: /proc/dynamic_debug/control: No such file or directory
cat: /tmp/dd_err: No such file or directory
ERROR (0): 
wc: /proc/dynamic_debug/control: No such file or directory
 key-ct:  Delta-CAL (IPI): 0
b0-ddc:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1 Delta-CAL (IPI): 0
b0-dd-drm-all:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1849 Delta-CAL (IPI): 0
b0-dd-tym:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1897 Delta-CAL (IPI): 0
b0-dd-tyy:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1897 Delta-CAL (IPI): 0
b0-ksan:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1897 Delta-CAL (IPI): 0
b0-ksan-leak:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1897 Delta-CAL (IPI): 0
b0-ftrace:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1849 Delta-CAL (IPI): 0
jimc@frodo:~/projects/lx/linux.git$ for_bo_ -p 2 ../../dd-tools.rc dd_ipis
b0-dd:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1897 Delta-CAL (IPI): 10387
b0-dd-drm:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1849 Delta-CAL (IPI): 10111
b0-defcon:
 doing: dd_ipis
../../dd-tools.rc: line 79: /proc/dynamic_debug/control: No such file or directory
cat: /tmp/dd_err: No such file or directory
ERROR (0): 
wc: /proc/dynamic_debug/control: No such file or directory
 key-ct:  Delta-CAL (IPI): 161
b0-ddc:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1 Delta-CAL (IPI): 28
b0-dd-drm-all:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1849 Delta-CAL (IPI): 10162
b0-dd-tym:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1897 Delta-CAL (IPI): 10418
b0-dd-tyy:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1897 Delta-CAL (IPI): 10390
b0-ksan:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1897 Delta-CAL (IPI): 10379
b0-ksan-leak:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1897 Delta-CAL (IPI): 10467
b0-ftrace:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1849 Delta-CAL (IPI): 9812
jimc@frodo:~/projects/lx/linux.git$ for_bo_ -p 3 ../../dd-tools.rc dd_ipis
b0-dd:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1897 Delta-CAL (IPI): 20856
b0-dd-drm:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1849 Delta-CAL (IPI): 20277
b0-defcon:
 doing: dd_ipis
../../dd-tools.rc: line 79: /proc/dynamic_debug/control: No such file or directory
cat: /tmp/dd_err: No such file or directory
ERROR (0): 
wc: /proc/dynamic_debug/control: No such file or directory
 key-ct:  Delta-CAL (IPI): 163
b0-ddc:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1 Delta-CAL (IPI): 97
b0-dd-drm-all:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1849 Delta-CAL (IPI): 20259
b0-dd-tym:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1897 Delta-CAL (IPI): 20856
b0-dd-tyy:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1897 Delta-CAL (IPI): 20861
b0-ksan:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1897 Delta-CAL (IPI): 20818
b0-ksan-leak:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1897 Delta-CAL (IPI): 20848
b0-ftrace:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1849 Delta-CAL (IPI): 20275
jimc@frodo:~/projects/lx/linux.git$ for_bo_ -p 4 ../../dd-tools.rc dd_ipis
b0-dd:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1897 Delta-CAL (IPI): 31237
b0-dd-drm:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1849 Delta-CAL (IPI): 30349
b0-defcon:
 doing: dd_ipis
../../dd-tools.rc: line 79: /proc/dynamic_debug/control: No such file or directory
cat: /tmp/dd_err: No such file or directory
ERROR (0): 
wc: /proc/dynamic_debug/control: No such file or directory
 key-ct:  Delta-CAL (IPI): 171
b0-ddc:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1 Delta-CAL (IPI): 101
b0-dd-drm-all:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1849 Delta-CAL (IPI): 30367
b0-dd-tym:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1897 Delta-CAL (IPI): 31224
b0-dd-tyy:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1897 Delta-CAL (IPI): 31233
b0-ksan:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1897 Delta-CAL (IPI): 31223
b0-ksan-leak:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1897 Delta-CAL (IPI): 31219
b0-ftrace:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1849 Delta-CAL (IPI): 30346
jimc@frodo:~/projects/lx/linux.git$ for_bo_ -p 5 ../../dd-tools.rc dd_ipis
b0-dd:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1897 Delta-CAL (IPI): 41593
b0-dd-drm:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1849 Delta-CAL (IPI): 40447
b0-defcon:
 doing: dd_ipis
../../dd-tools.rc: line 79: /proc/dynamic_debug/control: No such file or directory
cat: /tmp/dd_err: No such file or directory
ERROR (0): 
wc: /proc/dynamic_debug/control: No such file or directory
 key-ct:  Delta-CAL (IPI): 174
b0-ddc:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1 Delta-CAL (IPI): 91
b0-dd-drm-all:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1849 Delta-CAL (IPI): 40451
b0-dd-tym:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1897 Delta-CAL (IPI): 41546
b0-dd-tyy:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1897 Delta-CAL (IPI): 41595
b0-ksan:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1897 Delta-CAL (IPI): 41622
b0-ksan-leak:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1897 Delta-CAL (IPI): 41607
b0-ftrace:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1849 Delta-CAL (IPI): 40442


Patched:

jimc@frodo:~/projects/lx/wk-A$ . ../farmit.sh 
jimc@frodo:~/projects/lx/wk-A$ for_bo_ -p 1 ../../dd-tools.rc dd_ipis
b0-dd:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1896 Delta-CAL (IPI): 0
b0-dd-drm:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1849 Delta-CAL (IPI): 0
b0-defcon:
 doing: dd_ipis
../../dd-tools.rc: line 79: /proc/dynamic_debug/control: No such file or directory
cat: /tmp/dd_err: No such file or directory
ERROR (0): 
wc: /proc/dynamic_debug/control: No such file or directory
 key-ct:  Delta-CAL (IPI): 0
b0-ddc:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1 Delta-CAL (IPI): 0
b0-dd-drm-all:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1849 Delta-CAL (IPI): 0
b0-dd-tym:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1896 Delta-CAL (IPI): 0
b0-dd-tyy:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1896 Delta-CAL (IPI): 0
b0-ksan:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1896 Delta-CAL (IPI): 0
b0-ksan-leak:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1896 Delta-CAL (IPI): 0
b0-ftrace:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1849 Delta-CAL (IPI): 0
jimc@frodo:~/projects/lx/wk-A$ for_bo_ -p 2 ../../dd-tools.rc dd_ipis
b0-dd:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1896 Delta-CAL (IPI): 131
b0-dd-drm:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1849 Delta-CAL (IPI): 127
b0-defcon:
 doing: dd_ipis
../../dd-tools.rc: line 79: /proc/dynamic_debug/control: No such file or directory
cat: /tmp/dd_err: No such file or directory
ERROR (0): 
wc: /proc/dynamic_debug/control: No such file or directory
 key-ct:  Delta-CAL (IPI): 130
b0-ddc:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1 Delta-CAL (IPI): 78
b0-dd-drm-all:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1849 Delta-CAL (IPI): 124
b0-dd-tym:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1896 Delta-CAL (IPI): 116
b0-dd-tyy:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1896 Delta-CAL (IPI): 134
b0-ksan:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1896 Delta-CAL (IPI): 141
b0-ksan-leak:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1896 Delta-CAL (IPI): 153
b0-ftrace:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1849 Delta-CAL (IPI): 124
jimc@frodo:~/projects/lx/wk-A$ for_bo_ -p 3 ../../dd-tools.rc dd_ipis
b0-dd:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1896 Delta-CAL (IPI): 359
b0-dd-drm:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1849 Delta-CAL (IPI): 200
b0-defcon:
 doing: dd_ipis
../../dd-tools.rc: line 79: /proc/dynamic_debug/control: No such file or directory
cat: /tmp/dd_err: No such file or directory
ERROR (0): 
wc: /proc/dynamic_debug/control: No such file or directory
 key-ct:  Delta-CAL (IPI): 175
b0-ddc:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1 Delta-CAL (IPI): 100
b0-dd-drm-all:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1849 Delta-CAL (IPI): 191
b0-dd-tym:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1896 Delta-CAL (IPI): 186
b0-dd-tyy:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1896 Delta-CAL (IPI): 186
b0-ksan:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1896 Delta-CAL (IPI): 171
b0-ksan-leak:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1896 Delta-CAL (IPI): 181
b0-ftrace:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1849 Delta-CAL (IPI): 182
jimc@frodo:~/projects/lx/wk-A$ for_bo_ -p 4 ../../dd-tools.rc dd_ipis
b0-dd:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1896 Delta-CAL (IPI): 235
b0-dd-drm:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1849 Delta-CAL (IPI): 225
b0-defcon:
 doing: dd_ipis
../../dd-tools.rc: line 79: /proc/dynamic_debug/control: No such file or directory
cat: /tmp/dd_err: No such file or directory
ERROR (0): 
wc: /proc/dynamic_debug/control: No such file or directory
 key-ct:  Delta-CAL (IPI): 173
b0-ddc:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1 Delta-CAL (IPI): 93
b0-dd-drm-all:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1849 Delta-CAL (IPI): 227
b0-dd-tym:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1896 Delta-CAL (IPI): 294
b0-dd-tyy:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1896 Delta-CAL (IPI): 250
b0-ksan:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1896 Delta-CAL (IPI): 273
b0-ksan-leak:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1896 Delta-CAL (IPI): 307
b0-ftrace:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1849 Delta-CAL (IPI): 266
jimc@frodo:~/projects/lx/wk-A$ for_bo_ -p 5 ../../dd-tools.rc dd_ipis
b0-dd:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1896 Delta-CAL (IPI): 277
b0-dd-drm:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1849 Delta-CAL (IPI): 265
b0-defcon:
 doing: dd_ipis
../../dd-tools.rc: line 79: /proc/dynamic_debug/control: No such file or directory
cat: /tmp/dd_err: No such file or directory
ERROR (0): 
wc: /proc/dynamic_debug/control: No such file or directory
 key-ct:  Delta-CAL (IPI): 189
b0-ddc:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1 Delta-CAL (IPI): 96
b0-dd-drm-all:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1849 Delta-CAL (IPI): 267
b0-dd-tym:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1896 Delta-CAL (IPI): 278
b0-dd-tyy:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1896 Delta-CAL (IPI): 272
b0-ksan:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1896 Delta-CAL (IPI): 285
b0-ksan-leak:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1896 Delta-CAL (IPI): 280
b0-ftrace:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1849 Delta-CAL (IPI): 265

LINEAR PROGRESSION

both progressions are linear, but patched has much lower slope and
intercept (y=mx+b)

Baseline:

jimc@frodo:~/projects/lx/linux.git$ cd b0-dd
jimc@frodo:~/projects/lx/linux.git/b0-dd$ vrun_ -p 2 ../../dd-tools.rc dd_linear
virtme-ng 1.40
doing: vng --name v7.0-rc6-71-g347c80eb3607 --user root --rwdir=/home/jimc/projects/lx/linux.git/b0-dd -a dynamic_debug.verbose=1 -p 4 -p 2 ../../dd-tools.rc dd_linear
 doing: dd_linear
SUCCESS: +p ; -p ;
 key-ct: 1897 Delta-CAL (IPI): 10738
SUCCESS: +p ; -p ; +p ; -p ;
 key-ct: 1897 Delta-CAL (IPI): 21269
SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ;
 key-ct: 1897 Delta-CAL (IPI): 32125
SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ;
 key-ct: 1897 Delta-CAL (IPI): 42831
SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ;
 key-ct: 1897 Delta-CAL (IPI): 53540
SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ;
 key-ct: 1897 Delta-CAL (IPI): 64174
jimc@frodo:~/projects/lx/linux.git/b0-dd$ vrun_ -p 3 ../../dd-tools.rc dd_linear
virtme-ng 1.40
doing: vng --name v7.0-rc6-71-g347c80eb3607 --user root --rwdir=/home/jimc/projects/lx/linux.git/b0-dd -a dynamic_debug.verbose=1 -p 4 -p 3 ../../dd-tools.rc dd_linear
 doing: dd_linear
SUCCESS: +p ; -p ;
 key-ct: 1897 Delta-CAL (IPI): 21468
SUCCESS: +p ; -p ; +p ; -p ;
 key-ct: 1897 Delta-CAL (IPI): 42976
SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ;
 key-ct: 1897 Delta-CAL (IPI): 64434
SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ;
 key-ct: 1897 Delta-CAL (IPI): 85886
SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ;
 key-ct: 1897 Delta-CAL (IPI): 107353
SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ;
 key-ct: 1897 Delta-CAL (IPI): 128800
jimc@frodo:~/projects/lx/linux.git/b0-dd$ vrun_ -p 4 ../../dd-tools.rc dd_linear
virtme-ng 1.40
doing: vng --name v7.0-rc6-71-g347c80eb3607 --user root --rwdir=/home/jimc/projects/lx/linux.git/b0-dd -a dynamic_debug.verbose=1 -p 4 -p 4 ../../dd-tools.rc dd_linear
 doing: dd_linear
SUCCESS: +p ; -p ;
 key-ct: 1897 Delta-CAL (IPI): 32148
SUCCESS: +p ; -p ; +p ; -p ;
 key-ct: 1897 Delta-CAL (IPI): 64416
SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ;
 key-ct: 1897 Delta-CAL (IPI): 96604
SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ;
 key-ct: 1897 Delta-CAL (IPI): 128776
SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ;
 key-ct: 1897 Delta-CAL (IPI): 160985
SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ;
 key-ct: 1897 Delta-CAL (IPI): 193159
jimc@frodo:~/projects/lx/linux.git/b0-dd$ vrun_ -p 5 ../../dd-tools.rc dd_linear
virtme-ng 1.40
doing: vng --name v7.0-rc6-71-g347c80eb3607 --user root --rwdir=/home/jimc/projects/lx/linux.git/b0-dd -a dynamic_debug.verbose=1 -p 4 -p 5 ../../dd-tools.rc dd_linear
 doing: dd_linear
SUCCESS: +p ; -p ;
 key-ct: 1897 Delta-CAL (IPI): 42837
SUCCESS: +p ; -p ; +p ; -p ;
 key-ct: 1897 Delta-CAL (IPI): 85880
SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ;
 key-ct: 1897 Delta-CAL (IPI): 128793
SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ;
 key-ct: 1897 Delta-CAL (IPI): 171697
SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ;
 key-ct: 1897 Delta-CAL (IPI): 214636
SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ;
 key-ct: 1897 Delta-CAL (IPI): 257562
jimc@frodo:~/projects/lx/linux.git/b0-dd$ 

PATCHED:

jimc@frodo:~/projects/lx/wk-A/b0-dd$ vrun_ -p 2 ../../dd-tools.rc dd_linear
virtme-ng 1.40
doing: vng --name v7.0-rc6-5-g6a8736c5bd10 --user root --rwdir=/home/jimc/projects/lx/wk-A/b0-dd -a dynamic_debug.verbose=1 -p 4 -p 2 ../../dd-tools.rc dd_linear
 doing: dd_linear
SUCCESS: +p ; -p ;
 key-ct: 1896 Delta-CAL (IPI): 135
SUCCESS: +p ; -p ; +p ; -p ;
 key-ct: 1896 Delta-CAL (IPI): 145
SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ;
 key-ct: 1896 Delta-CAL (IPI): 198
SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ;
 key-ct: 1896 Delta-CAL (IPI): 246
SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ;
 key-ct: 1896 Delta-CAL (IPI): 293
SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ;
 key-ct: 1896 Delta-CAL (IPI): 341
jimc@frodo:~/projects/lx/wk-A/b0-dd$ vrun_ -p 3 ../../dd-tools.rc dd_linear
virtme-ng 1.40
doing: vng --name v7.0-rc6-5-g6a8736c5bd10 --user root --rwdir=/home/jimc/projects/lx/wk-A/b0-dd -a dynamic_debug.verbose=1 -p 4 -p 3 ../../dd-tools.rc dd_linear
 doing: dd_linear
SUCCESS: +p ; -p ;
 key-ct: 1896 Delta-CAL (IPI): 202
SUCCESS: +p ; -p ; +p ; -p ;
 key-ct: 1896 Delta-CAL (IPI): 256
SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ;
 key-ct: 1896 Delta-CAL (IPI): 352
SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ;
 key-ct: 1896 Delta-CAL (IPI): 445
SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ;
 key-ct: 1896 Delta-CAL (IPI): 545
SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ;
 key-ct: 1896 Delta-CAL (IPI): 642
jimc@frodo:~/projects/lx/wk-A/b0-dd$ vrun_ -p 4 ../../dd-tools.rc dd_linear
virtme-ng 1.40
doing: vng --name v7.0-rc6-5-g6a8736c5bd10 --user root --rwdir=/home/jimc/projects/lx/wk-A/b0-dd -a dynamic_debug.verbose=1 -p 4 -p 4 ../../dd-tools.rc dd_linear
 doing: dd_linear
SUCCESS: +p ; -p ;
 key-ct: 1896 Delta-CAL (IPI): 239
SUCCESS: +p ; -p ; +p ; -p ;
 key-ct: 1896 Delta-CAL (IPI): 347
SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ;
 key-ct: 1896 Delta-CAL (IPI): 479
SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ;
 key-ct: 1896 Delta-CAL (IPI): 630
SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ;
 key-ct: 1896 Delta-CAL (IPI): 774
SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ;
 key-ct: 1896 Delta-CAL (IPI): 915
jimc@frodo:~/projects/lx/wk-A/b0-dd$ vrun_ -p 5 ../../dd-tools.rc dd_linear
virtme-ng 1.40
doing: vng --name v7.0-rc6-5-g6a8736c5bd10 --user root --rwdir=/home/jimc/projects/lx/wk-A/b0-dd -a dynamic_debug.verbose=1 -p 4 -p 5 ../../dd-tools.rc dd_linear
 doing: dd_linear
SUCCESS: +p ; -p ;
 key-ct: 1896 Delta-CAL (IPI): 290
SUCCESS: +p ; -p ; +p ; -p ;
 key-ct: 1896 Delta-CAL (IPI): 445
SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ;
 key-ct: 1896 Delta-CAL (IPI): 640
SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ;
 key-ct: 1896 Delta-CAL (IPI): 840
SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ;
 key-ct: 1896 Delta-CAL (IPI): 1033
SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ;
 key-ct: 1896 Delta-CAL (IPI): 1222
jimc@frodo:~/projects/lx/wk-A/b0-dd$ 

Acknowldegement:

I used google-gemini extensively to discuss and review this work, and
to read the detailed data, and prepare the summary tables and prose
above. It is patient with my pedantry.

Signed-off-by: Jim Cromie <jim.cromie@xxxxxxxxx>


Jim Cromie (5):
  jump_label: add queueing API for batched static key updates
  x86/alternative.c: sort text-pokes before flushing the queue
  dyndbg: use static-key queueing API in dynamic-debug
  dyndbg: factor ddebug_match_desc out from ddebug_change (for
    demonstration only)
  lib/dynamic_debug: add negation support to queries

 arch/x86/kernel/alternative.c |  42 +++++------
 include/linux/jump_label.h    |  18 +++++
 kernel/jump_label.c           | 104 +++++++++++++++++++------
 lib/dynamic_debug.c           | 138 ++++++++++++++++++++++------------
 4 files changed, 211 insertions(+), 91 deletions(-)


base-commit: 7aaa8047eafd0bd628065b15757d9b48c5f9c07d
-- 
2.53.0