Re: [OFFTOPIC] Rounding problem

Andrew Morton (andrewm@uow.edu.au)
Wed, 24 Feb 1999 01:56:28 +1100

Andrew Logan wrote:
>
> Hi all,
>
> I've found a worrying rounding error which is manifested by the
> following simple test program.
> [ snip ]

Wow. One I can answer..

Here's the degenerate example:

double d1 = 2.4, d2 = 1000, d3 = 2400;

main()
{
if (d1 * d2 == d3)
printf("just lucky\n");

printf("%d\n", (long)(d1 * d2));
}

You'll find that the floating point representation of one or both of 2.4
and 1000 isn't exact, so the product of them comes to a tiny, tiny bit
less that 2400. The Pentium's hardware float to long conversion then
rounds down to 2399.

Note that this program doesn't say "just lucky". This is why floats
should never be compared for equality.

It works with 'cc -msoft-float'. Just lucky.

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.rutgers.edu
Please read the FAQ at http://www.tux.org/lkml/