Re: [PATCH net-next] Documentation: networking: Add a test plan for ethtool pause validation

[Andrew Lunn]

> From what I get from Andrew and you, the test definitions is lacking in
> details w.r.t the start conditions, the return code handling as well.
> 
> Having that in the doc is going to be too verbose, but in some python test
> that could really be great.

My experience with such documents is that they are verbose, very much
cut/paste repeated text, with just one parameter changed between test
variants, and a slight different expected result. And i personally
tend to make a lot of cut/paste errors :-(

Return code handling is also important, so i think it should be
stated. It is also potentially the most complex part of it. It happens
in all the corner cases which are going to get the least testing when
implementing the tests. They only really get exercised when the tests
are thrown at oddball hardware by developers trying to validate their
hardware.

There are also cases where the hardware reports it can do something,
and then returns EOPNOTSUPP when asked to do it, which is a bug. So
the tests becomes a sort of decision tree. The hardware reports it can
only do symmetric pause. We go down the branch that makes sure
symmetric pause passes, but asymmetric reports EOPNOTSUPP, etc.

There are a lot of combinations/permutations, and limited resources,
so it could be we cannot actually test everything. We might need to
accept EOPNOTSUPP at any stage as a pass, simply because we don't have
the resources to test all the corner cases.

I would normally actually concentrate on the corner cases, because
developers should have tested the happy path cases themselves.
However, for pause, we often see the happy path being very wrong, so
we should spend more time testing the happy path, and less on the
corner cases.

And that is probably true for all the testing we are doing at the
moment, because we are concentrating on subjects where developers
often get the basics wrong.

     Andrew