Testing Quality Call notes - 2024-07-11

[Laura Nao]

Hello,

KernelCI is hosting a bi-weekly call on Thursday to discuss improvements to
existing upstream tests, the development of new tests to increase kernel
testing coverage, and the enablement of these tests in KernelCI. In recent
months, we at Collabora have focused on various kernel areas, assessing the
tests already available upstream and contributing patches to make them
easily runnable in CIs.

Below is a list of the tests we've been working on and their latest status
updates, as discussed in the last meeting held on 2024-07-11:

*USB/PCI devices kselftest*

- Upstream test to detect unprobed devices on discoverable buses:
  https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=dacf1d7a78bf8a131346c47bfba7fe1f3ff44beb
- Updated KernelCI PRs according to feedback, now waiting for the first
  test results: https://github.com/kernelci/kernelci-core/pull/2577 and
  https://github.com/kernelci/kernelci-pipeline/pull/642

*Error log test*

- Proposing new kselftest to report device log errors:
  https://lore.kernel.org/all/20240423-dev-err-log-selftest-v1-0-690c1741d68b@xxxxxxxxxxxxx/
- Series got Acked-By from Greg, going to be picked up by Shuah soon
- Feedback from Tim Bird: this series follows an unusual model where
  tests can only fail but never pass, as no test case is generated unless
  there is an error. It takes an unusual approach to detect regressions and
  fixes. The autogenerated test case names are not very descriptive.

*Suspend/resume in cpufreq kselftest*

- Enabling suspend/resume test within the cpufreq kselftest in KernelCI -
- Sent patch upstream for adding RTC wakeup alarm in the cpufreq
  kselftest:
  https://lore.kernel.org/all/2e667d-668ff800-1-22d70300@133606496/
- Received a review from Rafael J. Wysocki, who suggested using the
  rtcwake utility instead of the sysfs entry

*Boot time test*

- Drafted initial implementation with two scripts, a config fragment and
  a bootconfig file
- One script generates a YAML file containing initial timestamps for
  relevant boot events, parsed from the trace file (run once)
- The other script is the actual test, which takes the generated YAML
  file and a delta in seconds as arguments. The script then parses the
  current trace file and checks if any timestamp deviates from the
  reference timestamps in the YAML file by more than the specified delta.
- Tracking only a few functions at the moment (populate_rootfs,
  unpack_to_rootfs, run_init_process). Next steps: refine bootconfig file
  to include more tracepoints (potentially initcalls too?). Useful
  tracepoints should be discussed upstream.
- Will present this at LPC 2024 (embedded and IoT MC)

*Support for benchmark data in KTAP*

- Tim Bird is working on adding performance data to KTAP output, which
  can be used in tests to detect slowdowns
- The idea is to keep reference values and criteria separate from the
  test itself
- There is a need to store per-platform files with previous times for
  comparison
- Will need to figure out where these files can be stored so they can be
  shared and used by different people and systems. Potential options: KCIDB
  or https://github.com/kernelci/platform-test-parameters
- Submitted a proposal for LPC 2024 
- Other related topics for discussion at LPC 2024 include: how to avoid
  device tree overhead in the boot process and boot phases (time-critical
  vs non-critical)

*TAP conformance in kselftests*

- Focusing on standardizing the way kernel's testing modules report
  results
- Discussion ongoing upstream over patches converting tests to TAP:
  https://lore.kernel.org/all/fb305513-580a-4bac-a078-fe0170a6ffa2@xxxxxxxxxxxxxxxxxxx
  and
  https://lore.kernel.org/all/6d82fa16-ed2e-41f1-a466-c752032b6f68@xxxxxxxxxxxxxxxxxxx

Please reply to this thread if you'd like to join the call or discuss any
of the topics further. We look forward to collaborating with the community
to improve upstream tests and expand coverage to more areas of interest
within the kernel.

Best regards,

Laura Nao