Hi Steve,
On Mon, 2019-02-11 at 10:24 -0800, Steve Longerbeam wrote:
[...]
Looking more closely at these coefficients now, I see you are right,That just controls the saturation. The result after the matrix
they are the BT.601 YUV full-range coefficients (Y range 0 to 1, U and V
range -0.5 to 0.5). Well, not even that -- the coefficients are not
being scaled to the limited ranges, but the 0.5 offset (128) _is_ being
added to U/V, but no offset for Y. So it is even more messed up.
Your corrected coefficients and offsets look correct to me: Y
coefficients scaled to (235 - 16) / 255 and U/V coefficients scaled to
(240 - 16)Â / 255, and add the offsets for both Y and U/V.
But what about this "SAT_MODE" field in the IC task parameter memory?
multiplication is either saturated to [0..255] or to [16..235]/[16..240]
when converting from the internal representation to the 8 bit output.
According to the manual the hardware will automatically convert theWhere did you get that from? "The final calculation result is limited
written coefficients to the correct limited ranges.
according to the SAT_MODE parameter and rounded to 8 bits." I see no
mention of coefficients being modified.
I see there is aÂ"sat" field defined in the struct but is not beingSAT_MODE should be set for conversions to YUV limited range so that the
set in the tables.
So what should we do, define the full range coefficients, and make use
of SAT_MODE h/w feature, or scale/offset the coefficients ourselves and
not use SAT_MODE? I'm inclined to do the former.
coefficients can be rounded to the closest value.
Otherwise we'd have to
round towards zero, possibly with a larger error, to make sure the
results are inside the valid ranges.