Re: [PATCH 5/6] selftests: Preserve subtarget failures in all/install

[Ricardo B. Marlière]

On Wed Apr 15, 2026 at 12:40 PM -03, Shuah Khan wrote:
> On 4/15/26 07:58, Mark Brown wrote:
>> On Fri, Mar 20, 2026 at 03:29:20PM -0300, Ricardo B. Marlière wrote:
>>> Track failures explicitly in the top-level selftests all/install loops.
>>>
>>> The current code multiplies `ret` by each sub-make exit status. For
>>> example, with `TARGETS=net`, the implicit `net/lib` dependency runs after
>>> `net`, so a failed `net` build can be followed by a successful `net/lib`
>>> build and reset the final result to success.
>>>
>>> Set `ret` to 1 on any non-zero sub-make exit code and keep it sticky, so
>>> the top-level make returns failure when any selected selftest target
>>> fails.
>> 
>> This patch, which is now in mainline as 7e47389142b8, is breaking a
>> bunch of CI systems - at least KernelCI, our Arm internal CI and my
>> personal stuff.  It causes the equivalent of FORCE_TARGETS behaviour in
>> the top level Makefile, the prior behaviour where the exit status of the
>> top level Makefile ignores failures from individual directories is
>> desirable since by default we try to build almost all the selftests but
>> between quality issues and build time dependencies it's very common for
>> at least one of them to fail.  With this commit unless the user has
>> configured a more restricted set of selftests it would be surprising if
>> we manage to get a successful build and install.
>> 
>> As well as being a poor default due to the very high likelyhood of build
>> failures this also has the undesirable effect of causing a build failure
>> in one selftest to cause the whole install target to fail, meaning that
>> the build failure is escallated to a complete lost of coverge for all
>> selftests in common CI usage.
>> 
>> This wasn't showing up in my -next build tests since I set FORCE_TARGETS
>> and explicitly choose a restricted set of kselftests which actually
>> build with my system and configuration.  It was less obvious than it
>> should have been with the other systems since they did not expect there
>> to be a complete failure to generate a kselftest tarball and variously
>> masked the error or reported it in a manner that looked like an
>> infrastructure issue.
>
> I didn't see it when I did test on linux-next and my repo. I did install
> to catch problems.
>
> Sorry for not catching this. We can drop this patch.
>
>> 
>> It would be really nice to get to the point where we can reasonably do
>> this but we're simply not there at the current time.  At the moment if
>> people want to see build failures reported at the top level that really
>> needs to be opt in, we have FORCE_TARGETS for that.
>
> Good point - I will go look and see if we document this in kselftest doc
> and add it.

It's not documented. It would have solved my issue, sorry for
overlooking this!

>
> Mark, would you like to a revert for this?
>
> thanks,
> -- Shuah