Re: LTS testing with latest kselftests - some failures

[Luis R. Rodriguez]

On Fri, Jun 16, 2017 at 09:29:52PM +0200, Greg Kroah-Hartman wrote:
> On Fri, Jun 16, 2017 at 06:46:51PM +0200, Luis R. Rodriguez wrote:
> > Kees, please review 47e0bbb7fa98 below.
> > Brian, please review be4a1326d12c below.
> > 
> > On Thu, Jun 15, 2017 at 11:26:53PM +0530, Sumit Semwal wrote:
> > > Hello Greg, Shuah,
> > > 
> > > While testing 4.4.y and 4.9.y LTS kernels with latest kselftest,
> > 
> > To be clear it seems like you are taking the latest upstream ksefltest and run
> > it against older stable kernels. Furthermore you seem to only run the shell
> > script tests but are using older kselftests drivers? Is this all correct?
> > Otherwise it is unclear how you are running into the issues below.
> > 
> > Does 0-day so the same? I thought 0-day takes just the kselftest from each tree
> > submitted. That *seemed* to me like the way it was designed. Shuah ?
> > 
> > What's the name of *this* testing effort BTW? Is this part of the overall
> > kselftest ? Or is this something Linaro does for LTS kernels ? If there
> > is a name to your effort can you document it here so that others are aware:
> 
> It's a "test LTS kernels to make sure Greg didn't break anything" type
> of testing effort that Linaro is helping out with.

OK so its "standard" :)

> This could also be called, "it's about time someone did this..." :)

Good to know!

> > > we found a couple more test failures due to test-kernel mismatch:
> > > 
> > > 1. firmware tests: - linux 4.5 [1] and 4.10 [2] added a few updates to
> > > tests, and related updates to lib/test_firmware.c to improve the
> > > tests. Stable-4.4 misses these patches to lib/test_firmware.c. Stable
> > > 4.9 misses the second update.
> > 
> > <-- snip, skipped 2. and 3. -->
> > 
> > > For all the 3 listed above, we will try and update the tests to gracefully exit.
> > 
> > Hmm, this actually raises a good kselftest question:
> > 
> > I *though* kselftests were running tests on par with the kernels, so we would
> > *not* take latest upstream kselftests to test against older kernels. Is this
> > incorrect?
> 
> That is incorrect.  Your test should always degrade gracefully if the
> feature is not present in the kernel under test.

OK perfect, now I know to look for knobs in the shell tests to ensure this
doesn't happen again.

Some of the knobs however are for extending tests for
existing APIs in older kernels, the async and custom fallback one are an
example.  There are a series of test cases later added which could help
test LTS kernels. Would Linaro pick these test driver enhancements to help
increase coverage of tests? Or is it not worth it? If its worth it then
what I was curious was how to help make this easier for this process to
bloom.

> If the test is for a
> bug that was fixed, then that fix should also go to a stable kernel
> release.

Indeed, that was perfectly clear.

  Luis