Re: linus-next: improving functional testing for to-be-merged pull requests

[Sasha Levin]

On Tue, Oct 29, 2024 at 09:10:25AM +0100, Thorsten Leemhuis wrote:
> On 28.10.24 23:46, Sasha Levin wrote:
> > On Mon, Oct 21, 2024 at 11:48:34PM -0700, Christoph Hellwig wrote:
> > > On Mon, Oct 21, 2024 at 09:54:53PM -0700, Kees Cook wrote:
> > > > For example, for a given PR, the bot can report:
> > > >
> > > > - Were the patches CCed to a mailing list?
> > > > - A histogram of how long the patches were in next (to show bake times)
> > > > - Are any patches associated with test failures? (0day and many other
> > > > CIs are already running tests against -next; parse those reports)
> > > >
> > > > We could have a real pre-submit checker! :)
> > >
> > > That would be very useful.  Items 1 and 2 should be trivial, 3 would
> > > require a bit of work but would still be very useful.
> >
> > If you've been following so far, there is a bot that is capable of doing
> > most of the above
> > (https://git.kernel.org/pub/scm/linux/kernel/git/sashal/next-
> > analysis.git/).
> >
> > Here's a histogram that describes v6.12-rc4..v6.12-rc5 as far as how
> > long commits spent in -next:
>
> I took a quick look at that tree and histo.sh that lead to a few
> questions here the code had no obvious answers to (or I missed them due
> to the "quick" aspect):
>
> * How does histo.sh handle changes where the commit-id changed between
> the first time in -next and their merge into Linus' tree (while the
> patch itself did not change)? For example due to a rebase or workflows
> where the commit-id changes regularly, such as those used by the
> bluetooth tree (for -fixes, as it queues them in their -next branch
> first) or the -mm tree (for most of it iirc -- this made things hard in
> a script of mine that looks up the arrival in -next)?

The "database" the scripts use stores 3 things:

 - commit ID
 - git patch-id
 - subject line

We try to match by either of the three. It means that maintainers can
rebase, change the subject, or even change the patch slightly, but as
long as one of the above stays the same we treat the commit the same.

> * Do those lore scripts detect if a committer adjusted the subject of a
> patch that has been on lore?

Yes, they also look up by patch-id, so if only the subject was adjusted
then we will still find the commit.

> * How do the scripts handle patches that changed a lot while they were
> in -next? I know of one subsystem that regularly drops whole patch-sets
> from their trees included in -next to replace them with newer versions
> of said patch-sets -- and then the timer maybe should restart.

The timer should just restart, right? If we uploaded patches that look
different from older ones, then their timer starts from 0 again.

> > This is where I think the value of linus-next comes during the -rc
> > cycles: the (89 + 21) commits that haven't gone through the -next
> > workflow before being pulled.
> >
> > I'm not looking to delay the process and
> > add latency, I'm looking to plug a hole where code would flow directly
> > to Linus's tree bypassing -next.
>
> Overall after all the discussions in this thread I still fail to see why
> we need a new tree for that. Why not make pending-fixes a bit more
> prominent while motivating maintainers to have proper -fixes branches
> included there?

Because that will add latency: my understanding that we don't want to
necessarily add another day or two between when fixes are ready and the
time it would take to get them through linux-next.

--
Thanks,
Sasha