Backport request: Revert "softirq: Let ksoftirqd do its job" (d15121be7485)

[Gyutae Bae]

Hi,

Please backport the following upstream commit to 6.1.y and 5.15.y:

  d15121be7485655129101f3960ae6add40204463
  Revert "softirq: Let ksoftirqd do its job"

On a production 5.15 PREEMPT_NONE node, long-running overlayfs
getdents64() calls prevented ksoftirqd from running.  RPS IPIs
continued to arrive, but NET_RX processing on that CPU stalled for
up to about 3 seconds.  Over 15 days, the node logged about 1,000
bonding ARP monitor failovers, with no link-state changes recorded
during that period.  The failovers stopped after the workload was
removed.

On a 5.15.27 PREEMPT_NONE test node, we used a livepatch to make
ksoftirqd_running() always return false.  This removes the same
softirq deferral checks as the revert on this build.  Under the same
load, we measured three 10-minute windows: before patching, with the
livepatch active, and after removing it.  The number of HTTP requests
taking more than 200 ms was 17, 0 and 11, respectively.  TCPTimeouts
increased by 25, 1 and 21.

We verified with patch --dry-run that the revert applies cleanly
to v6.1.190 and v5.15.223.  Both already contain the related RCU
fix 1dd1eff161bd.  6.6.y and later already contain the revert.

The original commit was intended to prevent user-task starvation
under a softirq flood.  The revert's changelog points to threaded
NAPI as an alternative, but it is disabled by default.

The revert was merged in v6.5 without a Cc: stable tag.
Syzbot has also reported bisections to this revert [1].
Paolo, Thomas, Eric: I would appreciate your thoughts on whether
this revert is suitable for 6.1.y and 5.15.y.

[1] https://lore.kernel.org/all/000000000000a249880603717c15@xxxxxxxxxx/

Thanks,
Gyutae Bae