Re: [RFC/PROPOSAL] Shifting the x.y.z Stable Tree to a Continuous, Signed Patch-Stream Model

[Artem S. Tashkinov]

On 5/24/26 10:56 AM, Greg Kroah-Hartman wrote:
> On Sun, May 24, 2026 at 01:38:55PM +0400, Artem S. Tashkinov wrote:
> > Hi all,
> >
> > The relentless cadence of critical vulnerability disclosures and public
> > exploits over the past month—including Copy Fail (CVE-2026-31431), Dirty
> > Frag (CVE-2026-43284/500), Fragnesia (CVE-2026-46300), and the ptrace exit
> > race (CVE-2026-46333)—has highlighted a severe structural bottleneck in how
> > we package and distribute stable backports.
>
> Who is "we"?
>
> And there's nothing really "new" here, these issues are all normal,
> remember, we resolve, on average, 13 CVEs a day, most much more severe
> than the ones that happened to get marketing names that you list here
> (and how many systems have untrusted users?)
>
> > When fatal logic flaws or memory corruptions strike core subsystems, our
> > current point-release model fractures. Spinning up whole new point releases
> > (7.0.4, 7.0.5, 7.0.7) in a matter of days just to address incomplete fixes,
> > subsystem regressions, or independent public disclosures (such as the recent
> > GRO managed-frag UAF exploit dropped directly to GitHub gists by
> > researchers) creates massive administrative fatigue for maintainers and
> > downstream teams alike.
>
> it takes just a minute to "spin up" a point release, what is difficult
> about that?  If needed, just let us know and we can easily do so.
>
> > Upstream has long maintained that the stable tree is effectively a
> > continuous stream of fixes, and that users should track the tip of the
> > stable branch rather than cherry-picking. It is time our release
> > infrastructure matches this reality.
> >
> > ### The Proposal
> >
> > I propose transitioning the stable tree (`linux-x.y.y`) away from
> > manual,discrete point-release tarballs (`x.y.z`). Instead, we should treat
> > the stable sub-version purely as an append-only, continuous, git-native
> > patch stream.
>
> That's what we do today, we just happen go "jump" on a weekly basis.
>
> > Major releases (e.g., 7.0, 7.1) remain the foundational code boundaries, but
> > sub-versions are eliminated as monolithic manual artifacts.
> >
> > ### The Implementation: How It Works
> >
> > To ensure downstream distributions, enterprise compliance engines, and
> > automated testing rings can still securely ingest code, we can replace the
> > manual tarball with a decoupled, automated asset pipeline:
> >
> > 1. **The Git-First Stream:** The stable branch (`linux-7.0.y`) remains the
> > single source of truth. Commits are pushed as soon as they pass stable
> > criteria and automated sanity testing.
>
> Again, that's what we do today.
>
> > 2. **The Signed Patch-Stream Archive:** Instead of packaging the entire 30M+
> > line source code tree into a new tarball for every quick fix, upstream
> > infrastructure maintains a rolling, cumulative patch sequence for the major
> > cycle:
> >
> > linux-7.0-stable.series = \sum (patch_1 + patch_2 + ... + patch_n)
> >
> > Every time a fix is merged to the stable branch, the patch is appended to a
> > publicly accessible, cryptographically signed manifest file
> > (`linux-7.0-stable-patches.tar.bz2` or a standard `series` file) alongside a
> > detached signature.
>
> Who would use/need such a thing?  What's wrong with the 2 systems we
> have today that this would somehow help out with?
>
> > 3. **Automated Snapshot Tags:** If the industry strictly requires an
> > immutable archive for compliance,
>
> What "compliance"?
>
> > point-release numbers can be replaced by
> > automated, time-stamped git tags and machine-generated source snapshots cut
> > on a strict, automated interval (e.g., every 48 hours), removing human
> > maintainers entirely from the release timing.
>
> That's probably not a good idea anyway.  Are you doing continous testing
> of the stable queue?  If so, great, just take from there today.
> Everyone adds patches on top of releases anyway, what's a few more if it
> happens to resolve specific issues for a day or so before a .y release
> can be cut?
>
> > ### Why This Benefits the Ecosystem
> >
> > * **Eliminates Churn and Latency:**
> >
> > When a patch introduces an edge-case regression or requires an immediate
> > follow-up (a common reason for rapid point-release sequences), maintainers
> > do not need to coordinate a whole new release event.
>
> No real "coordination" happens here.
>
> > The follow-up fix is simply patch $n+1$. Downstream CI pipelines
> > ingest it natively via standard git fetches.
>
> Again, we do that today.
>
> > * **Maintains Git-Native Debugging:**
> >
> > Debugging stable regressions via `git bisect` has always been patch-based,
> > not release-based. Since point releases are meant strictly for backported
> > bug fixes, removing the arbitrary `x.y.z` release tags changes nothing about
> > a developer's ability to isolate a regression. If anything, it prevents
> > downstream vendors from pulling out-of-order patches that complicate
> > bisection across distros.
>
> Who bisects across distros?
>
> > * **Eases Downstream Automation:**
> >
> > Modern tracking distributions (Arch, Fedora snapshotting, etc.) can switch
> > to trunk-based intake, automatically building from the signed tip.
>
> Have you asked them if they need/want this?
>
> > For enterprise distributions (RHEL, Ubuntu LTS) where constant kernel
> > packaging and reboots are untenable,
>
> Why are reboots for these systems untenable?  Why not fix that root
> problem instead?
>
> > a fluid patch stream allows vendor
> > security teams to more rapidly feed live-patching infrastructure (`kpatch`,
> > `kgraft`), applying critical CVE fixes directly to runtime memory without
> > changing the base package version.
>
> They can do that today, and do do that today.  So again, what distro
> needs this?
>
> > * **Bridges the Compliance Gap:**
> >
> > Embedded, automotive, or medical compliance pipelines
> > that legally require a static, verifiable code artifact can validate their
> > software against the base major release tarball ($7.0.0$) plus the
> > cryptographically signed, append-only stable patch series manifest.
>
> Do they really need that?  Again, they can have that today, nothing new
> here.
>
> > The manual compilation, testing, and cutting of sub-version tarballs is an
> > administrative artifact of the late 1990s.
>
> Weekly releases is not an artivact of the 1990s :)
>
> > Shifting to an explicit, signed
> > patch-stream architecture acknowledges the velocity of modern vulnerability
> > research, strips away artificial latency, and frees our stable maintainers
> > to focus on code quality rather than release management overhead.
>
> Again, we have that today, on a weekly basis.
>
> greg k-h

Hi Greg,

I understand completely that from an upstream maintainer perspective, 
the current pipeline feels seamless. Running a script to cut a tag takes 
a minute, and git-stable is inherently fluid. But looking at the 
ecosystem from the downstream and infrastructure side, the view is a bit 
different.

Here is where the current model introduces friction that a patch-stream 
or automated snapshot approach could solve:

### 1. The Compounding Downstream Tax

When you spin up a rapid sequence like 7.0.4, 7.0.5, and 7.0.7 to 
address the immediate fallout of complex CVEs, it takes a minute for 
upstream, but it triggers a massive domino effect downstream. Hundreds 
of distribution mirrors sync gigabytes of redundant source data, package 
maintainers rewrite specs, and automated build farms recreate full 
packages. When a fix is incomplete or causes an immediate regression 
requiring a follow-up version 48 hours later, that entire global compute 
and human cycle repeats for what amounts to a few lines of diff.

### 2. The Infrastructure Reality of Reboots

You mentioned that systems should just be fixed so reboots aren't 
untenable. In an ideal architectural world, yes. But in production 
reality, orchestrating reboots across thousands of live cloud nodes 
running high-availability workloads carries immense risk and scheduling 
overhead.

Enterprise distros absolutely do use live-patching (`kpatch`/`kgraft`) 
to mitigate this, but tracking a shifting landscape of discrete 
point-release tarballs complicates their internal backport verification. 
A continuous, signed patch-stream would give vendor security teams a 
clean, linear ledger of upstream-approved deltas to feed directly into 
live-patch compilation engines without the noise of full tree packaging.

### 3. Archive Bloat on the Mirror Network

Every stable point release requires a new ~130MB `.tar.xz` full source 
archive. Multiplying that across multiple active LTS branches, dozens of 
point releases, and hundreds of worldwide mirrors results in massive 
data duplication just to distribute a cumulative handful of text diffs. 
The ecosystem is essentially re-shipping the entire ocean every time we 
need to add a cup of water.

### Why the Proposal Matters

When independent researchers drop working exploit primitives directly to 
public gists (like the recent GRO managed-frag UAF), the turnaround time 
to protected production systems needs to be near-zero.

The proposal wasn't meant to imply that the stable team isn't working 
fast enough—you guys are incredibly fast. It was an observation that the 
*delivery mechanism* (the discrete, human-timed tarball) forces 
downstream consumers to choose between packaging fatigue or artificial 
patch latency. Shifting the sub-version boundary to a machine-managed, 
append-only signed patch stream bridges that gap.

I know you have a workflow that works for **you**, and I respect the 
hell out of the massive volume of fixes you ship daily. I just wanted to 
share the perspective of how that weekly "jump" ripples out to the 
people who have to deploy it.

### Truly Massive Distro Churn Issue

There's also a very important downstream packaging issue that I think 
the current stable-kernel release model underestimates.

Debian and Ubuntu already diverge from the upstream stable versioning 
model in a useful way: they package kernels around their own ABI 
versioning. In practice, this means they can often ship fixes without 
forcing users to install an entirely new `/lib/modules/<kernel-version>` 
tree every time an upstream stable micro-release appears. A new 
ABI/package line is only needed when the kernel ABI they expose to 
packaged or out-of-tree modules changes.

That model is a major downstream advantage. It avoids needless module 
churn, avoids retaining multiple near-identical kernel trees, and makes 
security/regression updates cheaper to ship. ABI-breaking changes do 
happen, but in stable kernels they are rare enough that treating them as 
exceptional events is perfectly reasonable.

My proposal takes this idea further.

Stable kernel releases frequently contain important fixes for 
regressions introduced either in `.0` releases or in earlier stable 
updates. But the current release cadence means that downstream 
distributions often have to react before the next stable point release 
exists. They either wait, leaving users exposed to known regressions, or 
they cherry-pick individual commits from the stable queue. That creates 
avoidable work: maintainers have to select fixes, update packaging 
metadata, rebuild, test, and publish a distro-specific kernel update 
that is supposed to correspond to a “stable” upstream series.

Fedora’s 7.0 cycle is a good example of the downstream churn this 
creates: multiple downstream updates may be needed for what is logically 
the same upstream stable series, simply because important fixes land 
between formal stable releases.

Under the model I am proposing, stable would be treated less like a 
sequence of isolated point releases and more like a continuously 
updated, signed stable branch. Distributions could pin a specific signed 
stable commit and trigger a rebuild from that point, instead of 
maintaining their own ad hoc selection of fixes while waiting for the 
next point release.

This would not remove the need for distro testing or for avoiding 
known-bad commits. But in the common case it would simplify the workflow 
enormously: downstreams would consume the stable branch directly, 
rebuild from a known commit, and get all accepted stable fixes together. 
Only in rare cases would they need to temporarily blacklist or revert a 
problematic stable commit.

That is still much easier than every distribution independently deciding 
which urgent fixes to cherry-pick from the stable queue.

The end result would be less downstream packaging churn, faster delivery 
of regression fixes, fewer distro-specific stable-kernel deltas, and a 
model that better reflects how many downstreams already consume stable 
kernels in practice.

Best regards,
Artem