Introduce a new test to identify regressions causing devices to go
missing on the system.
For each bus and class on the system the test checks the number of
devices present against a reference file, which needs to have been
generated by the program at a previous point on a known-good kernel, and
if there are missing devices they are reported.
Signed-off-by: Nícolas F. R. A. Prado <nfraprado@xxxxxxxxxxxxx>
---
Hi,
Key points about this test:
* Goal: Identify regressions causing devices to go missing on the system
* Focus:
* Ease of maintenance: the reference file is generated programatically
* Minimum of false-positives: the script makes as few assumptions as possible
about the stability of device identifiers to ensure renames/refactors don't
trigger false-positives
* How it works: For each bus and class on the system the test checks the number
of devices present against a reference file, which needs to have been
generated by the program at a previous point on a known-good kernel, and if
there are missing devices they are reported.
* Comparison to other tests: It might be possible(*) to replace the discoverable
devices test [1] with this. The benefits of this test is that it's easier
to setup and maintain and has wider coverage of devices.
Additional detail:
* Having more devices on the running system than the reference does not cause a
failure, but a warning is printed in that case to suggest that the reference
be updated.
* Missing devices are detected per bus/class based on the number of devices.
When the test fails, the known metadata for each of the expected and detected
devices is printed and some simple similitarity comparison is done to suggest
the devices that are the most likely to be missing.
* The proposed place to store the generated reference files is the
'platform-test-parameters' repository in KernelCI [2].