Re: [RFC] How to add hardware rotation, scaling etc to a DRM/KMS driver

[Hans de Goede]

Hi Daniel,

On 1/20/22 12:15, Daniel Palmer wrote:
> Hi Daniel,
> 
> On Thu, 20 Jan 2022 at 01:30, Daniel Vetter <daniel@xxxxxxxx> wrote:
>>> I got the feeling that maybe I should just provide an interface to the
>>> blitter from userspace and userspace should be doing the rotation. I'd
>>> like to do it in the kernel so stuff like SDL1 apps just work but
>>> maybe that isn't possible?
>>
>> panel orientation property is for that stuff:
>> fbcon will head this and rotate in sw,
> 
> This is working. On boot I get Tux rotated correctly etc.
> 
>> as should any competent compositor
>> in userspace (but some might not, it depends).
> 
> That's the problem I guess. I don't have one. SDL1 apps like prboom
> use the fbdev emulation as-is so they render upside down[0].
> I have 16MB of local storage and 128MB of RAM so I don't think I'll
> manage to get the standard userland bits onto it.
> 
> I wanted to do the rotation in the kernel so I didn't have to hack up SDL1.

Right, doing the rotation in the kernel to make this all transparent
was my first idea / wish too. Unfortunately that just doesn't really
work well. Most display-blocks have multiple layers, for things like
hw-rendering a mouse cursor, video overlays etc. I guess this is mostly
exposed through the DRM/kms interfaces, but I believe fbdev also
export some of this.

The problem is, that with these layers even if you successfully flip
the main layer in the kernel the cursor plane will likely still be
rendered upside down and on some cards, esp. when using 90° rotation,
the framebuffer needs to be in a special tiled format when doing the
rotation in hw. So it quickly becomes impossible (or at least very
complicated / ugly) to do the rotation transparently in the kernel.

For Fedora we have moved all SDL1 apps over to using the SDL1
compatibility wrapper around SDL2:

https://github.com/laibsdl-org/sdl12-compat

And SDL2 has a drm/kms backend. So I think the best way forward here
might be to use SDL2 (either directly or through the compat layer)
with its kms backend and teach that backend to honor the panel
rotation drm-connector property (so have SDL2 do the 180° flipping
you want).

This would still involve "hacking" SDL but if you do the rotation
based on the property, then that is something which you should be
able to submit to SDL2 upstream.

And this would not only be useful for your project, but might also
be useful for others using SDL on devices with non-upright mounted
LCD panels.

Regards,

Hans