Re: [PATCH v2] doc: kselftest: update for clarity on running kselftests in CI rings

[shuah]

On 8/29/19 11:30 AM, Brendan Higgins wrote:
> On Mon, Aug 26, 2019 at 5:37 PM Shuah Khan <skhan@xxxxxxxxxxxxxxxxxxx> wrote:
> > Update to add clarity and recommendations on running newer kselftests
> > on older kernels vs. matching the kernel and kselftest revisions.
> >
> > The recommendation is "Match kernel revision and kselftest."
> >
> > Signed-off-by: Shuah Khan <skhan@xxxxxxxxxxxxxxxxxxx>
>
> Reviewed-by: Brendan Higgins <brendanhiggins@xxxxxxxxxx>
>
> One minor comment below.
>
> > ---
> > Changes since v1: Fixed "WARNING: Title underline too short."
> >
> >   Documentation/dev-tools/kselftest.rst | 25 +++++++++++++++++++++++++
> >   1 file changed, 25 insertions(+)
> >
> > diff --git a/Documentation/dev-tools/kselftest.rst b/Documentation/dev-tools/kselftest.rst
> > index 25604904fa6e..308506c5e8fa 100644
> > --- a/Documentation/dev-tools/kselftest.rst
> > +++ b/Documentation/dev-tools/kselftest.rst
> > @@ -12,6 +12,31 @@ write new tests using the framework on Kselftest wiki:
> >
> >   https://kselftest.wiki.kernel.org/
> >
> > +Recommendations on running kselftests in Continuous Integration test rings
> > +==========================================================================
> > +
> > +It is recommended that users run Kselftest from the same release. Running
> > +newer Kselftest on older kernels isn't recommended for the following
> > +reasons:
> > +
> > +- Kselftest from mainline and linux-next might not be stable enough to run
> > +  on stable kernels.
> > +- Kselftests detect feature dependencies at run-time and skip tests if a
> > +  feature and/or configuration they test aren't enabled. Running newer
> > +  tests on older kernels could result in a few too many skipped/failed
> > +  conditions. It becomes difficult to evaluate the results.
> > +- Newer tests provide better coverage. However, users should make a judgement
> > +  call on coverage vs. run to run consistency and being able to compare
> > +  run to run results on older kernels.
> > +
> > +Recommendations:
>
> I think this should be the "note" directive. You already have a list
> of recommendations above; this seems like a TLDR section, or an
> important note or tip.
>
> > +
> > +Match kernel revision and kselftest. Especially important for LTS and
> > +Stable kernel Continuous Integration test rings.
> > +
> > +Hot-plug tests
> > +==============
> > +
> >   On some systems, hot-plug tests could hang forever waiting for cpu and
> >   memory to be ready to be offlined. A special hot-plug target is created
> >   to run the full range of hot-plug tests. In default mode, hot-plug tests run
> > --
> > 2.20.1

Thanks Brendan. I am holding off on this patch for now. I will have
to make some updates. I will fold your comment in when I send an
updated patch.

thanks,
-- Shuah