Re: AUTOSEL process
From: Sasha Levin
Date: Sat Mar 11 2023 - 15:59:39 EST
On Sat, Mar 11, 2023 at 09:19:54PM +0100, Willy Tarreau wrote:
On Sat, Mar 11, 2023 at 11:46:05AM -0800, Eric Biggers wrote:
(And please note, the key word here is *confidence*. We all agree that it's
never possible to be absolutely 100% sure whether a commit is appropriate for
stable or not. That's a red herring.
In fact even developers themselves sometimes don't know, and even when they
know, sometimes they know after committing it. Many times we've found that
a bug was accidently resolved by a small change. Just for this it's important
to support a post-merge analysis.
And I would assume, or at least hope, that the neural network thing being used
for AUTOSEL outputs a confidence rating and not just a yes/no answer. If it
actually just outputs yes/no, well how is anyone supposed to know that and fix
that, given that it does not seem to be an open source project?)
Honestly I don't know. I ran a few experiments with natural language
processors such as GPT-3 on commit messages which contained human-readable
instructions, and asking "what am I expected to do with these patches", and
seeing the bot respond "you should backport them to this version, change
this and that in that version, and preliminary take that patch". It
summarized extremely well the instructions delivered by the developer,
which is awesome, but was not able to provide any form of confidence
level. I don't know what Sasha uses but wouldn't be surprised it shares
some such mechanisms and that it might not always be easy to get such a
confidence level. But I could be wrong.
It's actually pretty stupid: it uses the existence of ~10k of the most
common words in commit messages + metrics from cqmetrics
(github.com/dspinellis/cqmetrics) as input.
Although I get a score, which is already set pretty high, confidence is
really non-existant here: at the end it depends mostly on the writing
style of said commit author more than anything.
--
Thanks,
Sasha