Re: [PATCH 2/2] ftrace/selftest: absence of modules/programs should trigger unsupported errors

[Masami Hiramatsu]

On Fri, 7 Feb 2020 08:27:13 +0000 (GMT)
Alan Maguire <alan.maguire@xxxxxxxxxx> wrote:

> On Fri, 7 Feb 2020, Masami Hiramatsu wrote:
> 
> > Hi Alan,
> > 
> > On Thu,  6 Feb 2020 15:09:20 +0000
> > Alan Maguire <alan.maguire@xxxxxxxxxx> wrote:
> > 
> > > In a number of cases, the ftrace tests check for the presence of
> > > ftrace testing-related modules (ftrace-direct, trace-printk) and
> > > programs (checkbashisms), returning exit_unresolved if these
> > > are not found.  The problem is, exit_unresolved causes execution
> > > of ftracetest to return an error, when really our tests are
> > > failing due to not having the requisite kernel configuration/tools
> > > present, which is I think more of an unsupported error condition.
> > > With these fixed, we see no unresolved test cases and ftracetest
> > > returns success ("ok" when run via kselftest).
> > 
> > If your problem is to pass the test even if you don't test the
> > feature, please change the ftracetest itself instead of replacing
> > unresolved with unsupported. Those notice different situation.
> > 
> > unresolved - Testcase can not find some tools or helper drivers
> >              which are required for this testcase.
> > 
> > unsupported - Kernel does not have tested feature because of
> >               the version or the configuration.
> > 
> > Obviously the unresolved is a test environment issue. No test-module
> > doesn't mean no feature to be tested.
> > Could you tell me the reason why you can't install those required
> > tools and modules on the test environment?
> > 
> 
> Sure! In my case, I'm testing a distro production kernel,
> where I can't control the CONFIG variable settings.  In
> this case, ideally I'd like the tests to return success
> if no problems with ftrace were detected, even if some
> of the tests could not be run due to missing modules
> and programs.

OK, for modules, we need to find another way to solve the issue.
But how about checkbashisms? you can download and build it.

https://sources.debian.org/src/devscripts/2.20.2/

For the modules, you might be able to build it from kernel
source code as out-of-tree modules, or not?
(hmm, how do the other test handle it...?)

>  As you suggest above (unless I'm
> misunderstanding), this could be accomplished by modifying
> ftracetest itself.  Would doing something like what is done
> for UNSUPPORTED_RESULT (defaults to 0, but can be set to
> 1 via --fail-unsupported, such that ftracetest returns
> 1 if we encounter unsupported results) make sense for
> the unresolved case too?

Yes, but at first could you try to setup your testing environment?
If you are officially testing your distro kernel, the distro
might need to be tested with full-set of testcases.

If not (like you are testing kernel for fun :)), you can just
make your custom set of testcases. (just remove those test files)

Thank you,


-- 
Masami Hiramatsu <mhiramat@xxxxxxxxxx>