Re: [PATCH v12 09/16] iio: afe: rescale: fix accuracy for small fractional scales

[Andy Shevchenko]

On Sat, Jan 8, 2022 at 10:53 PM Liam Beguin <liambeguin@xxxxxxxxx> wrote:
>
> The approximation caused by integer divisions can be costly on smaller
> scale values since the decimal part is significant compared to the
> integer part. Switch to an IIO_VAL_INT_PLUS_NANO scale type in such
> cases to maintain accuracy.

...

> +               tmp = 1 << *val2;

Unfortunately, potential UB is still here. I think you missed a subtle
detail in the implementation of BIT()/BIT_ULL(). Let's put it here:

  #define BIT(nr) (UL(1) << (nr))

where

  #define UL(x) (_UL(x))
  #define _UL(x) (_AC(x, UL))

For non-assembler case

  #define __AC(X,Y) (X##Y)
  #define _AC(X,Y) __AC(X,Y)

Now you may easily see the difference.

...

> +               rem2 = *val % (int)tmp;
> +               *val = *val / (int)tmp;
> +
> +               *val2 = rem / (int)tmp;

Hmm... You divide s64 by 10^9, which means that the maximum value can
be ~10^10 / 2 (because 2^64-1 ~= 10^19), but this _might_ be bigger
than 'int' can hold. Can you confirm that tmp can't be so big?

-- 
With Best Regards,
Andy Shevchenko