[PATCH 1/2] lib/math: rational: avoid overflow when comparing semiconvergents

[961222243021 SHYAM J]

The comparison of a semiconvergent with the previous convergent doubles
its final term t. When t exceeds ULONG_MAX / 2, this multiplication wraps
and can incorrectly select the previous convergent.

For example, approximating 1/ULONG_MAX with a maximum numerator of 1 and
a maximum denominator of ULONG_MAX - 1 returns 0/1 instead of the closer
1/(ULONG_MAX - 1). The same problem occurs with 32-bit unsigned longs.

If t exceeds ULONG_MAX / 2, its mathematical double is greater than any
representable value of a. Select the semiconvergent before evaluating the
multiplication in that case, preserving the existing tie comparison.

Fixes: 323dd2c3ed06 ("lib/math/rational.c: fix possible incorrect
result from rational fractions helper")
Assisted-by: LLM
---
 lib/math/rational.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/lib/math/rational.c b/lib/math/rational.c
index d2c34e629..dbd24dabc 100644
--- a/lib/math/rational.c
+++ b/lib/math/rational.c
@@ -90,8 +90,10 @@ void rational_best_approximation(
                        /* This tests if the semi-convergent is closer
than the previous
                         * convergent.  If d1 is zero there is no
previous convergent as this
                         * is the 1st iteration, so always choose the
semi-convergent.
+                        * Avoid overflowing the multiplication of t by two.
                         */
-                       if (!d1 || 2u * t > a || (2u * t == a && d0 *
dp > d1 * d)) {
+                       if (!d1 || t > ULONG_MAX / 2 || 2u * t > a ||
+                           (2u * t == a && d0 * dp > d1 * d)) {
                                n1 = n0 + t * n1;
                                d1 = d0 + t * d1;
                        }
--
2.51.1