Re: [REGRESSION] Linux kernel 6.12.75 fails to compile with -Werror=implicit-function-declaration

[Brett A C Sheffield]

On 2026-03-05 15:21, Sasha Levin wrote:
> On Thu, Mar 05, 2026 at 05:40:09PM +0000, Brett A C Sheffield wrote:

> >The current stable process is introducing bugs. Bugs that never existed in
> >mainline.
> 
> Releasing yesterday's tree was (my) human error: I don't have as much
> automation and scripting as Greg does, so many of the steps I've taken were
> manual and prone to errors. I'm working on improving this workflow on my end.

This raises two questions:

1) why do you not have the same tools? This whole process is highly automated,
both at your end, and ours. Little changes break scripts and cause much hassle
that could be avoided.  I had to debug my scripts before I could even get
started (my fault, my bug) because of a difference in how you sent the emails.
Others had similar problems by the sound of it.  Please, Greg, make these tools
and the process public and make sure whoever is cutting the release knows how to
use them.

2) why is so much scripting involved in cutting the final release? This should be
the exact kernel we just tested with our Tested-by lines and your signoff tacked
on.  Last minute changes break things.

> This, however, wasn't an issue with our process, which is why I'm curious which
> bugs you're referring to?
> 
> >The 3 kernels released today were tested by no one before release.
> 
> Right - the 3 kernels released today simply dropped a commit that caused a
> built failure. We sometimes do that to address simple build or functionality
> breakages (this happened with v6.19.2 and v6.19.5 too).
> 
> I don't disagree that there's a risk in doing so, but the risk is fairly minor,
> and doing a quick release allows users to get important fixes without waiting
> another cycle.
> 
> We could discuss a policy change here, but could you show that doing these
> quick releases introduced regressions?
> 
> If not, why are we changing something that works?

It isn't working, as we've just demonstrated.

It's not your fault, Sasha. It's a failure (lack) of process. You were trying to
release seven kernels, for the first time, without using the same tools as
usual.

The problem was that you (following the broken process), released kernels,
after modifying them, without getting anyone to retest.

In the rest of the software world testing happens after changes and before
release, not in the middle.

That's (mainly) what I want to change. No more last minute dropped and added
patches.

Tagging RCs would help at our end too, rather than relying on extracting
metadata from email headers, and trying to figure out which force-pushed commit
matches RCn at any given point in time.

> >The seven kernels yesterday were similarly tested by no one before release.
> >We weren't given the opportunity.
> 
> Could you explain this point please? There were quite a few folks who provided
> their Tested-by...

Yes, I was one of them, although you omitted mine from the hastily pushed 6.6
and 6.12 kernels ;-)

Taking 6.6 as an example because I have the numbers on the whiteboard beside me:

RC1 had 375 patches.
RC2 had 956.
These are not the same kernel.

Then we had another RC2 force-pushed over the top of the original RC2 confusing
the hell out of me after two long kernel bisects when I found my tree was out of
sync and wasting a bunch more time as I retested because I thought I must have
made a mistake.

So for 6.6 and 6.12 I tested 3 kernels each, none of which were what was actually
released.

And that turned out to be broken. Which we could have told you, if we'd had an
RC to test.

The re-testing happened anyway, after release. I re-tested 6.6 and 6.12 and they
seemed fine. Peter retested and found it was broken. Better for that to have
been an RC3. Less work for you cutting extra releases, and less public
embarrassment for an already tarnished stable kernel process.

This whole release cycle was a complete mess, not because of one person, but
because of poor process and missing tools.


Brett