Re: [REGRESSION] Linux kernel 6.12.75 fails to compile with -Werror=implicit-function-declaration

[Sasha Levin]

On Thu, Mar 05, 2026 at 10:57:49PM +0100, Brett A C Sheffield wrote:
> On 2026-03-05 15:21, Sasha Levin wrote:
> > On Thu, Mar 05, 2026 at 05:40:09PM +0000, Brett A C Sheffield wrote:
>
> > >The current stable process is introducing bugs. Bugs that never existed in
> > >mainline.
> >
> > Releasing yesterday's tree was (my) human error: I don't have as much
> > automation and scripting as Greg does, so many of the steps I've taken were
> > manual and prone to errors. I'm working on improving this workflow on my end.
>
> This raises two questions:
>
> 1) why do you not have the same tools? This whole process is highly automated,
> both at your end, and ours. Little changes break scripts and cause much hassle
> that could be avoided.  I had to debug my scripts before I could even get
> started (my fault, my bug) because of a difference in how you sent the emails.
> Others had similar problems by the sound of it.  Please, Greg, make these tools
> and the process public and make sure whoever is cutting the release knows how to
> use them.

Greg's scripts are public :)

We have very different development environments. We use different tools, our
workflows are different, and we (normally) handle different parts of the
process. Sure, at times it creates issues (like here, when I'm trying to tackle
some of Greg's work), but sometimes it also helps us catch issues that one of
us would have missed. The latter part usually happens behind the scenes when no
one notices.

I suppose that if I were to do releases more often we'd likely end up with the
same (or similar) set of scripts around this. We're not there.

> 2) why is so much scripting involved in cutting the final release? This should be
> the exact kernel we just tested with our Tested-by lines and your signoff tacked
> on.  Last minute changes break things.

The stable kernel is managed by a quilt set of branches in the background.
Every release of the stable tree (for forever?) has been generated that day
from the quilt queue and released. There is no notion of "exact kernel" because
the tree, until it's released, is ephermal.

What happened in this case, is that I dropped the offending commit from the
quilt queue and regenerated the trees, and pushed out -rc2. I'm still trying to
learn why the change to the quilt queue wasn't pushed out, but when the trees
were later regenrated from the queue, it still had the offending commit.

There was no irresponsible last second change as you're trying to suggest.

> > This, however, wasn't an issue with our process, which is why I'm curious which
> > bugs you're referring to?
> >
> > >The 3 kernels released today were tested by no one before release.
> >
> > Right - the 3 kernels released today simply dropped a commit that caused a
> > built failure. We sometimes do that to address simple build or functionality
> > breakages (this happened with v6.19.2 and v6.19.5 too).
> >
> > I don't disagree that there's a risk in doing so, but the risk is fairly minor,
> > and doing a quick release allows users to get important fixes without waiting
> > another cycle.
> >
> > We could discuss a policy change here, but could you show that doing these
> > quick releases introduced regressions?
> >
> > If not, why are we changing something that works?
>
> It isn't working, as we've just demonstrated.

So this is narrowly about the 3 releases from earlier today, where there were
concerns that these small releases without -rcs prior are causing regressions.
I'm trying to understand which regressions were caused by this type of
releases.

> It's not your fault, Sasha. It's a failure (lack) of process. You were trying to
> release seven kernels, for the first time, without using the same tools as
> usual.
>
> The problem was that you (following the broken process), released kernels,
> after modifying them, without getting anyone to retest.
>
> In the rest of the software world testing happens after changes and before
> release, not in the middle.
>
> That's (mainly) what I want to change. No more last minute dropped and added
> patches.
>
> Tagging RCs would help at our end too, rather than relying on extracting
> metadata from email headers, and trying to figure out which force-pushed commit
> matches RCn at any given point in time.
>
> > >The seven kernels yesterday were similarly tested by no one before release.
> > >We weren't given the opportunity.
> >
> > Could you explain this point please? There were quite a few folks who provided
> > their Tested-by...
>
> Yes, I was one of them, although you omitted mine from the hastily pushed 6.6
> and 6.12 kernels ;-)
>
> Taking 6.6 as an example because I have the numbers on the whiteboard beside me:
>
> RC1 had 375 patches.
> RC2 had 956.
> These are not the same kernel.

It's not :)

This one was a fun git bug to figure out.

-rc2, however, had the same amount of time to be tested, and the test priod for
-rc2 was Monday-Wednesday rather than the weeked for -rc1.

-rc1 was in no place to be released as is, so what would have been my other
choice? I can't release a tree with two thirds of it's commits missing, so I go
ahead and do an -rc2 with more commits, and I allow time for testing.

> Then we had another RC2 force-pushed over the top of the original RC2 confusing
> the hell out of me after two long kernel bisects when I found my tree was out of
> sync and wasting a bunch more time as I retested because I thought I must have
> made a mistake.

This is on me. I decided to push -rc2 because the issue was pointed out so
quickly I felt I could sneak it in.

--
Thanks,
Sasha