Re: [LSF/MM TOPIC] FS, MM, and stable trees

From: Sasha Levin
Date: Thu Feb 14 2019 - 20:50:25 EST


On Wed, Feb 13, 2019 at 12:14:35PM -0800, James Bottomley wrote:
On Wed, 2019-02-13 at 20:52 +0100, Greg KH wrote:
On Wed, Feb 13, 2019 at 02:25:12PM -0500, Sasha Levin wrote:
> On Wed, Feb 13, 2019 at 10:18:03AM +0100, Greg KH wrote:
> > On Wed, Feb 13, 2019 at 11:01:25AM +0200, Amir Goldstein wrote:
> > > Best effort testing in timely manner is good, but a good way to
> > > improve confidence in stable kernel releases is a publicly
> > > available list of tests that the release went through.
> >
> > We have that, you aren't noticing them...
>
> This is one of the biggest things I want to address: there is a
> disconnect between the stable kernel testing story and the tests
> the fs/ and mm/ folks expect to see here.
>
> On one had, the stable kernel folks see these kernels go through
> entire suites of testing by multiple individuals and organizations,
> receiving way more coverage than any of Linus's releases.
>
> On the other hand, things like LTP and selftests tend to barely
> scratch the surface of our mm/ and fs/ code, and the maintainers of
> these subsystems do not see LTP-like suites as something that adds
> significant value and ignore them. Instead, they have a
> (convoluted) set of testing they do with different tools and
> configurations that qualifies their code as being "tested".
>
> So really, it sounds like a low hanging fruit: we don't really need
> to write much more testing code code nor do we have to refactor
> existing test suites. We just need to make sure the right tests are
> running on stable kernels. I really want to clarify what each
> subsystem sees as "sufficient" (and have that documented
> somewhere).

kernel.ci and 0-day and Linaro are starting to add the fs and mm
tests to their test suites to address these issues (I think 0-day
already has many of them). So this is happening, but not quite
obvious. I know I keep asking Linaro about this :(

0day has xfstests at least, but it's opt-in only (you have to request
that it be run on your trees). When I did it for the SCSI tree, I had
to email Fenguangg directly, there wasn't any other way of getting it.

It's very tricky to do even if someone would just run it. I worked with
the xfs folks for quite a while to gather the various configs they want
to use, and to establish the baseline for a few of the stable trees
(some tests are know to fail, etc).

So just running xfstests "blindly" doesn't add much value beyond ltp I
think.

--
Thanks,
Sasha