Re: linus-next: improving functional testing for to-be-merged pull requests

[Sasha Levin]

On Mon, Oct 21, 2024 at 11:48:34PM -0700, Christoph Hellwig wrote:
> On Mon, Oct 21, 2024 at 09:54:53PM -0700, Kees Cook wrote:
> > For example, for a given PR, the bot can report:
> >
> > - Were the patches CCed to a mailing list?
> > - A histogram of how long the patches were in next (to show bake times)
> > - Are any patches associated with test failures? (0day and many other
> > CIs are already running tests against -next; parse those reports)
> >
> > We could have a real pre-submit checker! :)
>
> That would be very useful.  Items 1 and 2 should be trivial, 3 would
> require a bit of work but would still be very useful.

So I've ended up writing something hacky that does (1) and (2) above, and
ran it on the existing (already merged) PRs on the mailing list to give
it a spin.

The script tripped on the very first PR it looked at:
https://lore.kernel.org/all/20241021171728.274997-1-pbonzini@xxxxxxxxxx/

And in particular, this commit: afa9b48f327c ("KVM: arm64: Shave a few
bytes from the EL2 idmap code")

(sorry, not trying to pick on anyone/anything, just an example...)

The commit can't be found on lore.kernel.org, it was never in -next, and
yet Linus pulled it promptly without questioning anything.

So yes, we could argue that a bot sending out this information would
have informed Linus of this issue, but somehow I suspect that Linus is
not an idiot and he has enough scripts in place already to alert him of
these issues, which he chooses to ignore in some cases.

--
Thanks,
Sasha