Re: [PATCH] hwmon: pmbus: Make reg check and clear faults functions return errors

From: Andrew Jeffery
Date: Wed Sep 06 2017 - 19:32:48 EST

Next message: Linus Torvalds: "Re: [GIT] Networking"
Previous message: Ard Biesheuvel: "Re: HAVE_EFFICIENT_UNALIGNED_ACCESS on ARM32 (was: Alignment issues in zImage with Linux 4.12, LZ4 and GCC5.3)"
In reply to: Guenter Roeck: "Re: [PATCH] hwmon: pmbus: Make reg check and clear faults functions return errors"
Next in thread: Guenter Roeck: "Re: [PATCH] hwmon: pmbus: Make reg check and clear faults functions return errors"
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]

On Wed, 2017-09-06 at 15:51 -0700, Guenter Roeck wrote:
> On Wed, Sep 06, 2017 at 10:23:37AM +1000, Andrew Jeffery wrote:
> > On Tue, 2017-09-05 at 10:00 -0700, Guenter Roeck wrote:
> > > On Tue, Sep 05, 2017 at 05:01:32PM +1000, Andrew Jeffery wrote:
> > > > Some functions exposed by pmbus core conflated errors that occurred when
> > > > setting the page to access with errors that occurred when accessing
> > > > registers in a page. In some cases, this caused legitimate errors to be
> > > > hidden under the guise of the register not being supported.
> > > >Â
> > >Â
> > > I'll have to look into this. If I recall correctly, the reason for not returning
> > > errors from the "clear faults" command was that some early chips did not support
> > > it, or did not support it on all pages. Those chips would now always fail.
> >Â
> > Yes, that is a concern.
> >Â
> > However, shouldn't the lack of support for CLEAR_FAULTS be a described
> > property of the chip or page?
> >Â
> > Regardless, the issue here is the PAGE command is sometimes failing
> > (more below). This means we won't have issued a CLEAR_FAULTS against
> > the page even if the page supports CLEAR_FAULTS. That to me is
> > something that should be propagated back up the call chain, as faults
> > that can be cleared will not have been.
> >Â
>Â
> We could also consider
> - retry

I guess that leads to the usual retries problem: How many? Oneshot? More?

My expectation is that oneshot would be enough, but we'd still need to consider
what to do if that wasn't successful.

Does the retry policy need to be in the kernel? I guess if it's not we would
need all operations in the path to the failure to be idempotent.

> - Add a marker indicating that faults (specifically command errors)
> Â are unreliable after clearing faults failed

What kind of marker were you thinking? Something in dmesg? If it's anything
else we'd probably want something to respond to the condition, which nothing
would do currently.

>Â
> If we can't reliably clear faults, all bets are off.

Yeah, it's a messy problem. However even if we resolve our issues in hardware
(i.e. it's discovered that it is a design failure or something), I don't think
we should drop a resolution to the problem highlighted by this patch.

>Â
> > >Â
> > > On a higher level, I am not sure if it is a good idea to return an error
> > > from a function intended to clear faults (and nothing else). That seems
> > > counterproductive.
> > >Â
> > > Is this a problem you have actually observed ?Â
> >Â
> > Unfortunately yes. I was trying to reproduce some issues that we are
> > seeing in userspace but wasn't having much luck (getting -EIO on
> > reading a hwmon attribute). I ended up instrumenting our I2C bus
> > driver, and found that the problem was more prevalent than what the
> > read failures in userspace were indicating. The paths that were
> > triggering these unreported conditions all traced through the check
> > register and clear fault functions, which on analysis were swallowing
> > the error.
> >Â
> > > If so, what is the chip ?
> >Â
> > Well, the chip that I'm *communicating* with is the Maxim MAX31785
> > Intelligent Fan Controller. As to whether that's what is *causing* the
> > PAGE command to fail is still up in the air. I would not be surprised
> > if we have other issues in the hardware design.
> >Â
> > I haven't sent a follow-up to the series introducing the MAX31785
> > driver because I've been chasing down communication issues. I felt it
> > was important to understand whether the device has quirks that need to
> > be worked around in the driver, or if our hardware design has bigger
> > problems that should be handled in other ways. I've been in touch with
> > Maxim who have asserted that some of the problems we're seeing cannot
> > be caused by their chip.
> >Â
>Â
> Guess I need to dig up my eval board and see if I can reproduce the problem.
> Seems you are saying that the problem is always seen when issuing a sequence
> of "clear faults" commands on multiple pages ?

Yeah. We're also seeing bad behaviour under other command sequences as well,
which lead to this hack of a work-around patch[1].

I'd be very interested in the results of testing against the eval board. I
don't have access to one and it seems Maxim have discontinued them.

[1] https://patchwork.kernel.org/patch/9876083/

>Â
> The MAX31785 is microcode based, so I would not be entirely surprised if
> it sometimes fails to reply if a sequence of <set page, clear faults>
> commands is executed.

Have you seen this behaviour in other microcode-based chips?

>Â
> If possible, you might try reducing the i2c clock. I have not seen that with
> Maxim chips, but some other PMBus chips don't operate reliably at 400 KHz.

It's already running at 100kHz, which is the maximum clock rate the device is
specified for. I've even underclocked the bus to 75kHz and still observed
issues. Again, I wouldn't be surprised if the problem is something other than
the Maxim chip, the bus that we have it on has a complex topology. I'm working
with hardware people internally to try to isolate the problem.

Cheers,

Andrew

>Â
> Guenter
>Â
> > >Â
> > > > > > Signed-off-by: Andrew Jeffery <andrew@xxxxxxxx>
> > > >Â
> > > > ---
> > > > ÂDocumentation/hwmon/pmbus-coreÂÂÂ|ÂÂ12 ++--
> > > > Âdrivers/hwmon/pmbus/pmbus.hÂÂÂÂÂÂ|ÂÂÂ6 +-
> > > > Âdrivers/hwmon/pmbus/pmbus_core.c | 115 +++++++++++++++++++++++++++++----------
> > > > Â3 files changed, 95 insertions(+), 38 deletions(-)
> > > >Â
> > > > diff --git a/Documentation/hwmon/pmbus-core b/Documentation/hwmon/pmbus-core
> > > > index 8ed10e9ddfb5..3e9f41bb756f 100644
> > > > --- a/Documentation/hwmon/pmbus-core
> > > > +++ b/Documentation/hwmon/pmbus-core
> > > > @@ -218,17 +218,17 @@ Specifically, it provides the following information.
> > > > ÂÂÂThis function calls the device specific write_byte function if defined.
> > > > ÂÂÂTherefore, it must _not_ be called from that function.
> > > > Â
> > > > -ÂÂbool pmbus_check_byte_register(struct i2c_client *client, int page, int reg);
> > > > +ÂÂint pmbus_check_byte_register(struct i2c_client *client, int page, int reg);
> > > > Â
> > > > -ÂÂCheck if byte register exists. Return true if the register exists, false
> > > > -ÂÂotherwise.
> > > > +ÂÂCheck if byte register exists. Returns 1 if the register exists, 0 if it does
> > > > +ÂÂnot, and less than zero on an unexpected error.
> > > > ÂÂÂThis function calls the device specific write_byte function if defined to
> > > > ÂÂÂobtain the chip status. Therefore, it must _not_ be called from that function.
> > > > Â
> > > > -ÂÂbool pmbus_check_word_register(struct i2c_client *client, int page, int reg);
> > > > +ÂÂint pmbus_check_word_register(struct i2c_client *client, int page, int reg);
> > > > Â
> > > > -ÂÂCheck if word register exists. Return true if the register exists, false
> > > > -ÂÂotherwise.
> > > > +ÂÂCheck if word register exists. Returns 1 if the register exists, 0 if it does
> > > > +ÂÂnot, and less than zero on an unexpected error.
> > > > ÂÂÂThis function calls the device specific write_byte function if defined to
> > > > ÂÂÂobtain the chip status. Therefore, it must _not_ be called from that function.
> > > > Â
> > > > diff --git a/drivers/hwmon/pmbus/pmbus.h b/drivers/hwmon/pmbus/pmbus.h
> > > > index bfcb13bae34b..c53032a04a6f 100644
> > > > --- a/drivers/hwmon/pmbus/pmbus.h
> > > > +++ b/drivers/hwmon/pmbus/pmbus.h
> > > > @@ -413,9 +413,9 @@ int pmbus_write_byte_data(struct i2c_client *client, int page, u8 reg,
> > > > > > Â ÂÂu8 value);
> > > >Â
> > > > Âint pmbus_update_byte_data(struct i2c_client *client, int page, u8 reg,
> > > > > > Â ÂÂÂu8 mask, u8 value);
> > > >Â
> > > > -void pmbus_clear_faults(struct i2c_client *client);
> > > > -bool pmbus_check_byte_register(struct i2c_client *client, int page, int reg);
> > > > -bool pmbus_check_word_register(struct i2c_client *client, int page, int reg);
> > > > +int pmbus_clear_faults(struct i2c_client *client);
> > > > +int pmbus_check_byte_register(struct i2c_client *client, int page, int reg);
> > > > +int pmbus_check_word_register(struct i2c_client *client, int page, int reg);
> > > > Âint pmbus_do_probe(struct i2c_client *client, const struct i2c_device_id *id,
> > > > > > Â ÂÂÂstruct pmbus_driver_info *info);
> > > >Â
> > > > Âint pmbus_do_remove(struct i2c_client *client);
> > > > diff --git a/drivers/hwmon/pmbus/pmbus_core.c b/drivers/hwmon/pmbus/pmbus_core.c
> > > > index f1eff6b6c798..153700e35431 100644
> > > > --- a/drivers/hwmon/pmbus/pmbus_core.c
> > > > +++ b/drivers/hwmon/pmbus/pmbus_core.c
> > > > @@ -304,18 +304,24 @@ static int _pmbus_read_byte_data(struct i2c_client *client, int page, int reg)
> > > > > > Â return pmbus_read_byte_data(client, page, reg);
> > > >Â
> > > > Â}
> > > > Â
> > > > -static void pmbus_clear_fault_page(struct i2c_client *client, int page)
> > > > +static int pmbus_clear_fault_page(struct i2c_client *client, int page)
> > > > Â{
> > > > > > > > > > > > - _pmbus_write_byte(client, page, PMBUS_CLEAR_FAULTS);
> > > > > > + return _pmbus_write_byte(client, page, PMBUS_CLEAR_FAULTS);
> > > >Â
> > > > Â}
> > > > Â
> > > > -void pmbus_clear_faults(struct i2c_client *client)
> > > > +int pmbus_clear_faults(struct i2c_client *client)
> > > > Â{
> > > > > > > > > > > > Â struct pmbus_data *data = i2c_get_clientdata(client);
> > > > > > > > > > > > + int rv;
> > > > > > Â int i;
> > > >Â
> > > > Â
> > > > > > > > > > > > - for (i = 0; i < data->info->pages; i++)
> > > > > > > > > > > > - pmbus_clear_fault_page(client, i);
> > > > > > > > > > > > + for (i = 0; i < data->info->pages; i++) {
> > > > > > > > > > > > + rv = pmbus_clear_fault_page(client, i);
> > > > > > > > > > > > + if (rv)
> > > > > > > > > > > > + return rv;
> > > > > > + }
> > > >Â
> > > > +
> > > > > > + return 0;
> > > >Â
> > > > Â}
> > > > ÂEXPORT_SYMBOL_GPL(pmbus_clear_faults);
> > > > Â
> > > > @@ -333,28 +339,45 @@ static int pmbus_check_status_cml(struct i2c_client *client)
> > > > > > Â return 0;
> > > >Â
> > > > Â}
> > > > Â
> > > > -static bool pmbus_check_register(struct i2c_client *client,
> > > > +static int pmbus_check_register(struct i2c_client *client,
> > > > > > > > > > > > Â Âint (*func)(struct i2c_client *client,
> > > > > > > > > > > > Â ÂÂÂÂÂint page, int reg),
> > > > > > Â Âint page, int reg)
> > > >Â
> > > > Â{
> > > > > > > > > > > > + struct pmbus_data *data;
> > > > > > > > > > > > + int check;
> > > > > > > > > > > > Â int rv;
> > > > > > - struct pmbus_data *data = i2c_get_clientdata(client);
> > > >Â
> > > > Â
> > > > > > > > > > > > - rv = func(client, page, reg);
> > > > > > > > > > > > - if (rv >= 0 && !(data->flags & PMBUS_SKIP_STATUS_CHECK))
> > > > > > > > > > > > - rv = pmbus_check_status_cml(client);
> > > > > > > > > > > > - pmbus_clear_fault_page(client, -1);
> > > > > > > > > > > > - return rv >= 0;
> > > > > > + data = i2c_get_clientdata(client);
> > > >Â
> > > > +
> > > > > > > > > > > > + /*
> > > > > > > > > > > > + Â* pmbus_set_page() guards transactions on the requested page matching
> > > > > > > > > > > > + Â* the current page. This may be done in the execution of func(), but
> > > > > > > > > > > > + Â* at that point a set-page error is conflated with accessing a
> > > > > > > > > > > > + Â* non-existent register.
> > > > > > > > > > > > + Â*/
> > > > > > > > > > > > + rv = pmbus_set_page(client, page);
> > > > > > > > > > > > + if (rv < 0)
> > > > > > + return rv;
> > > >Â
> > > > +
> > > > > > > > > > > > + check = func(client, page, reg);
> > > > > > > > > > > > + if (check >= 0 && !(data->flags & PMBUS_SKIP_STATUS_CHECK))
> > > > > > + check = pmbus_check_status_cml(client);
> > > >Â
> > > > +
> > > > > > > > > > > > + rv = pmbus_clear_fault_page(client, -1);
> > > > > > > > > > > > + if (rv < 0)
> > > > > > + return rv;
> > > >Â
> > > > +
> > > > > > + return check >= 0;
> > > >Â
> > > > Â}
> > > > Â
> > > > -bool pmbus_check_byte_register(struct i2c_client *client, int page, int reg)
> > > > +int pmbus_check_byte_register(struct i2c_client *client, int page, int reg)
> > > > Â{
> > > > > > Â return pmbus_check_register(client, _pmbus_read_byte_data, page, reg);
> > > >Â
> > > > Â}
> > > > ÂEXPORT_SYMBOL_GPL(pmbus_check_byte_register);
> > > > Â
> > > > -bool pmbus_check_word_register(struct i2c_client *client, int page, int reg)
> > > > +int pmbus_check_word_register(struct i2c_client *client, int page, int reg)
> > > > Â{
> > > > > > Â return pmbus_check_register(client, _pmbus_read_word_data, page, reg);
> > > >Â
> > > > Â}
> > > > @@ -390,7 +413,7 @@ static struct pmbus_data *pmbus_update_device(struct device *dev)
> > > > Â
> > > > > > > > > > > > Â mutex_lock(&data->update_lock);
> > > > > > > > > > > > Â if (time_after(jiffies, data->last_updated + HZ) || !data->valid) {
> > > > > > > > > > > > - int i, j;
> > > > > > + int i, j, ret;
> > > >Â
> > > > Â
> > > > > > > > > > > > Â for (i = 0; i < info->pages; i++) {
> > > > > > Â data->status[PB_STATUS_BASE + i]
> > > >Â
> > > > @@ -424,7 +447,13 @@ static struct pmbus_data *pmbus_update_device(struct device *dev)
> > > > > > > > > > > > Â ÂÂÂÂsensor->page,
> > > > > > > > > > > > Â ÂÂÂÂsensor->reg);
> > > > > > > > > > > > Â }
> > > > > > - pmbus_clear_faults(client);
> > > >Â
> > > > +
> > > > > > > > > > > > + ret = pmbus_clear_faults(client);
> > > > > > > > > > > > + if (ret < 0) {
> > > > > > > > > > > > + mutex_unlock(&data->update_lock);
> > > > > > > > > > > > + return ERR_PTR(ret);
> > > > > > + }
> > > >Â
> > > > +
> > > > > > > > > > > > Â data->last_updated = jiffies;
> > > > > > > > > > > > Â data->valid = 1;
> > > > > > Â }
> > > >Â
> > > > @@ -754,6 +783,9 @@ static ssize_t pmbus_show_boolean(struct device *dev,
> > > > > > > > > > > > Â struct pmbus_data *data = pmbus_update_device(dev);
> > > > > > Â int val;
> > > >Â
> > > > Â
> > > > > > > > > > > > + if (IS_ERR(data))
> > > > > > + return PTR_ERR(data);
> > > >Â
> > > > +
> > > > > > > > > > > > Â val = pmbus_get_boolean(data, boolean, attr->index);
> > > > > > > > > > > > Â if (val < 0)
> > > > > > Â return val;
> > > >Â
> > > > @@ -766,6 +798,9 @@ static ssize_t pmbus_show_sensor(struct device *dev,
> > > > > > > > > > > > Â struct pmbus_data *data = pmbus_update_device(dev);
> > > > > > Â struct pmbus_sensor *sensor = to_pmbus_sensor(devattr);
> > > >Â
> > > > Â
> > > > > > > > > > > > + if (IS_ERR(data))
> > > > > > + return PTR_ERR(data);
> > > >Â
> > > > +
> > > > > > > > > > > > Â if (sensor->data < 0)
> > > > > > Â return sensor->data;
> > > >Â
> > > > Â
> > > > @@ -995,7 +1030,11 @@ static int pmbus_add_limit_attrs(struct i2c_client *client,
> > > > > > Â struct pmbus_sensor *curr;
> > > >Â
> > > > Â
> > > > > > > > > > > > Â for (i = 0; i < nlimit; i++) {
> > > > > > > > > > > > - if (pmbus_check_word_register(client, page, l->reg)) {
> > > > > > > > > > > > + ret = pmbus_check_word_register(client, page, l->reg);
> > > > > > > > > > > > + if (ret < 0)
> > > > > > + return ret;
> > > >Â
> > > > +
> > > > > > > > > > > > + if (ret) {
> > > > > > > > > > > > Â curr = pmbus_add_sensor(data, name, l->attr, index,
> > > > > > > > > > > > Â page, l->reg, attr->class,
> > > > > > Â attr->update || l->update,
> > > >Â
> > > > @@ -1041,6 +1080,8 @@ static int pmbus_add_sensor_attrs_one(struct i2c_client *client,
> > > > > > > > > > > > Â if (!base)
> > > > > > > > > > > > Â return -ENOMEM;
> > > > > > > > > > > > Â if (attr->sfunc) {
> > > > > > + int check;
> > > >Â
> > > > +
> > > > > > > > > > > > Â ret = pmbus_add_limit_attrs(client, data, info, name,
> > > > > > > > > > > > Â ÂÂÂÂindex, page, base, attr);
> > > > > > Â if (ret < 0)
> > > >Â
> > > > @@ -1050,9 +1091,13 @@ static int pmbus_add_sensor_attrs_one(struct i2c_client *client,
> > > > > > > > > > > > Â Â* alarm attributes, if there is a global alarm bit, and if
> > > > > > > > > > > > Â Â* the generic status register for this page is accessible.
> > > > > > > > > > > > Â Â*/
> > > > > > > > > > > > - if (!ret && attr->gbit &&
> > > > > > > > > > > > - ÂÂÂÂpmbus_check_byte_register(client, page,
> > > > > > - ÂÂÂÂÂÂdata->status_register)) {
> > > >Â
> > > > +
> > > > > > > > > > > > + check = pmbus_check_byte_register(client, page,
> > > > > > > > > > > > + ÂÂdata->status_register);
> > > > > > > > > > > > + if (check < 0)
> > > > > > + return check;
> > > >Â
> > > > +
> > > > > > > > > > > > + if (!ret && attr->gbit && check) {
> > > > > > > > > > > > Â ret = pmbus_add_boolean(data, name, "alarm", index,
> > > > > > > > > > > > Â NULL, NULL,
> > > > > > Â PB_STATUS_BASE + page,
> > > >Â
> > > > @@ -1604,8 +1649,12 @@ static int pmbus_add_fan_attributes(struct i2c_client *client,
> > > > > > > > > > > > Â if (!(info->func[page] & pmbus_fan_flags[f]))
> > > > > > Â break;
> > > >Â
> > > > Â
> > > > > > > > > > > > - if (!pmbus_check_word_register(client, page,
> > > > > > > > > > > > - ÂÂÂÂÂÂÂpmbus_fan_registers[f]))
> > > > > > > > > > > > + ret = pmbus_check_word_register(client, page,
> > > > > > > > > > > > + ÂÂÂÂÂÂÂpmbus_fan_registers[f]);
> > > > > > > > > > > > + if (ret < 0)
> > > > > > + return ret;
> > > >Â
> > > > +
> > > > > > > > > > > > + if (!ret)
> > > > > > Â break;
> > > >Â
> > > > Â
> > > > > > Â /*
> > > >Â
> > > > @@ -1628,9 +1677,13 @@ static int pmbus_add_fan_attributes(struct i2c_client *client,
> > > > > > > > > > > > Â Â* Each fan status register covers multiple fans,
> > > > > > > > > > > > Â Â* so we have to do some magic.
> > > > > > > > > > > > Â Â*/
> > > > > > > > > > > > - if ((info->func[page] & pmbus_fan_status_flags[f]) &&
> > > > > > > > > > > > - ÂÂÂÂpmbus_check_byte_register(client,
> > > > > > > > > > > > - page, pmbus_fan_status_registers[f])) {
> > > > > > > > > > > > + ret =ÂÂpmbus_check_byte_register(client, page,
> > > > > > > > > > > > + pmbus_fan_status_registers[f]);
> > > > > > > > > > > > + if (ret < 0)
> > > > > > + return ret;
> > > >Â
> > > > +
> > > > > > > > > > > > + if ((info->func[page] & pmbus_fan_status_flags[f])
> > > > > > > > > > > > + && ret) {
> > > > > > Â int base;
> > > >Â
> > > > Â
> > > > > > > > Â if (f > 1) /* fan 3, 4 */
> > > >Â
> > > > @@ -1696,10 +1749,13 @@ static int pmbus_identify_common(struct i2c_client *client,
> > > > > > Â Âstruct pmbus_data *data, int page)
> > > >Â
> > > > Â{
> > > > > > > > > > > > Â int vout_mode = -1;
> > > > > > + int rv;
> > > >Â
> > > > Â
> > > > > > > > > > > > - if (pmbus_check_byte_register(client, page, PMBUS_VOUT_MODE))
> > > > > > > > > > > > + rv = pmbus_check_byte_register(client, page, PMBUS_VOUT_MODE);
> > > > > > > > > > > > + if (rv == 1)
> > > > > > > > > > > > Â vout_mode = _pmbus_read_byte_data(client, page,
> > > > > > Â ÂÂPMBUS_VOUT_MODE);
> > > >Â
> > > > +
> > > > > > > > > > > > Â if (vout_mode >= 0 && vout_mode != 0xff) {
> > > > > > > > > > > > Â /*
> > > > > > Â Â* Not all chips support the VOUT_MODE command,
> > > >Â
> > > > @@ -1725,8 +1781,7 @@ static int pmbus_identify_common(struct i2c_client *client,
> > > > > > > > > > > > Â }
> > > > > > Â }
> > > >Â
> > > > Â
> > > > > > > > > > > > - pmbus_clear_fault_page(client, page);
> > > > > > > > > > > > - return 0;
> > > > > > + return pmbus_clear_fault_page(client, page);
> > > >Â
> > > > Â}
> > > > Â
> > > > Âstatic int pmbus_init_common(struct i2c_client *client, struct pmbus_data *data,
> > > > @@ -1756,7 +1811,9 @@ static int pmbus_init_common(struct i2c_client *client, struct pmbus_data *data,
> > > > > > > > > > > > Â if (ret >= 0 && (ret & PB_CAPABILITY_ERROR_CHECK))
> > > > > > Â client->flags |= I2C_CLIENT_PEC;
> > > >Â
> > > > Â
> > > > > > > > > > > > - pmbus_clear_faults(client);
> > > > > > > > > > > > + ret = pmbus_clear_faults(client);
> > > > > > > > > > > > + if (ret < 0)
> > > > > > + return ret;
> > > >Â
> > > > Â
> > > > > > > > > > > > Â if (info->identify) {
> > > > > > Â ret = (*info->identify)(client, info);
> > > >Â
> > > > --Â
> > > > 2.11.0
> > > >Â
>Â
>Â

Attachment: signature.asc
Description: This is a digitally signed message part

Next message: Linus Torvalds: "Re: [GIT] Networking"
Previous message: Ard Biesheuvel: "Re: HAVE_EFFICIENT_UNALIGNED_ACCESS on ARM32 (was: Alignment issues in zImage with Linux 4.12, LZ4 and GCC5.3)"
In reply to: Guenter Roeck: "Re: [PATCH] hwmon: pmbus: Make reg check and clear faults functions return errors"
Next in thread: Guenter Roeck: "Re: [PATCH] hwmon: pmbus: Make reg check and clear faults functions return errors"
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]