Re: [RFC/PROPOSAL] Shifting the x.y.z Stable Tree to a Continuous, Signed Patch-Stream Model

[Theodore Tso]

On Sun, May 24, 2026 at 12:56:54PM +0200, Greg Kroah-Hartman wrote:
> > point-release numbers can be replaced by
> > automated, time-stamped git tags and machine-generated source snapshots cut
> > on a strict, automated interval (e.g., every 48 hours), removing human
> > maintainers entirely from the release timing.
> 
> That's probably not a good idea anyway.  Are you doing continous testing
> of the stable queue?  If so, great, just take from there today.
> Everyone adds patches on top of releases anyway, what's a few more if it
> happens to resolve specific issues for a day or so before a .y release
> can be cut?

It's already the case that not all maintainers have the time to test
the stable queue, and it's not clear that current testing of the
weekly release is all that great.  I have seen stabilty regressions
where an xfstest running against ext4 will cause the kernel to crash
with the 6.1 and 6.6 LTS kernels.  It took me several days to figure
out the 6.1 regression, and I still haven't had time to look into the
6.6 regression, because my day job (which is not ext4, but herding
cats for an AI infrastructure project --- it's amazing how many fellow
developers I met at LSF/MM are actually doing AI infrastructure
projects for $WORK, and not kernel development as their primary job
responsibilities.)

So even the weekly cadence is starting to creek a bit from a quality
perspective.  I can't even *imagine* what a continuous, automated, "it
builds, ship it!" would do to the quality of the stable kernel series.

		     	  	      	  - Ted

P.S.  If someone is interesting in helping to test ext4 and xfs stable
kernel patches, talk to me.  There is partial automation to test
updates to the stable-rc trees, but I've never had time to automate
the rest of the test regression analysis combined with the automated
"which patches need to be backed out to avoid the regression / kernel
crash".  There had been a few companies contributing fractions of
engineers to do XFS stable maintainenace, all of those resources have
been withdrawn by their respective companies in the past year.




> 
> > ### Why This Benefits the Ecosystem
> > 
> > * **Eliminates Churn and Latency:**
> > 
> > When a patch introduces an edge-case regression or requires an immediate
> > follow-up (a common reason for rapid point-release sequences), maintainers
> > do not need to coordinate a whole new release event.
> 
> No real "coordination" happens here.
> 
> > The follow-up fix is simply patch $n+1$. Downstream CI pipelines
> > ingest it natively via standard git fetches.
> 
> Again, we do that today.
> 
> > * **Maintains Git-Native Debugging:**
> > 
> > Debugging stable regressions via `git bisect` has always been patch-based,
> > not release-based. Since point releases are meant strictly for backported
> > bug fixes, removing the arbitrary `x.y.z` release tags changes nothing about
> > a developer's ability to isolate a regression. If anything, it prevents
> > downstream vendors from pulling out-of-order patches that complicate
> > bisection across distros.
> 
> Who bisects across distros?
> 
> > * **Eases Downstream Automation:**
> > 
> > Modern tracking distributions (Arch, Fedora snapshotting, etc.) can switch
> > to trunk-based intake, automatically building from the signed tip.
> 
> Have you asked them if they need/want this?
> 
> > For enterprise distributions (RHEL, Ubuntu LTS) where constant kernel
> > packaging and reboots are untenable,
> 
> Why are reboots for these systems untenable?  Why not fix that root
> problem instead?
> 
> > a fluid patch stream allows vendor
> > security teams to more rapidly feed live-patching infrastructure (`kpatch`,
> > `kgraft`), applying critical CVE fixes directly to runtime memory without
> > changing the base package version.
> 
> They can do that today, and do do that today.  So again, what distro
> needs this?
> 
> > * **Bridges the Compliance Gap:**
> > 
> > Embedded, automotive, or medical compliance pipelines
> > that legally require a static, verifiable code artifact can validate their
> > software against the base major release tarball ($7.0.0$) plus the
> > cryptographically signed, append-only stable patch series manifest.
> 
> Do they really need that?  Again, they can have that today, nothing new
> here.
> 
> > The manual compilation, testing, and cutting of sub-version tarballs is an
> > administrative artifact of the late 1990s.
> 
> Weekly releases is not an artivact of the 1990s :)
> 
> > Shifting to an explicit, signed
> > patch-stream architecture acknowledges the velocity of modern vulnerability
> > research, strips away artificial latency, and frees our stable maintainers
> > to focus on code quality rather than release management overhead.
> 
> Again, we have that today, on a weekly basis.
> 
> greg k-h
>