Doing a new port of glibc is actually fairly easy (and like the kernel,
getting easier as more are done). Like you say, each new port solves a few
more weird cases, to the extent that you can almost just mix and match the
bits you need from other architectures these days.
Once that's working you should be able to get your Red Hat disk, `rpm
--rebuild' everything and hey presto, more-or-less instant user space.
X11 may or may not be more difficult. If your video hardware is standard
(like you have a PCI backplane and use off-the-shelf cards) it's probably
pretty simple. If you're faced with something like a Sun and have to reverse
engineer the board then write a driver from scratch it'll be a bit more of an
undertaking.
p.