If the memory can't be accessed by the CPU then it shouldn't be mapped
into a PTE in the first place. The fact you made userspace faults
(only) work is nifty but still an ugly hack to get around the fact you
shouldn't be mapping in the first place.
We already have ZONE_DEVICE/DEVICE_PRIVATE to handle exactly this
scenario. "memory" that cannot be touched by the CPU but can still be
specially accessed by enlightened components.
guest_memfd, and more broadly memfd based instead of VMA based, memory
mapping in KVM is a similar outcome to DEVICE_PRIVATE.
I think you need to stay in the world of not mapping the memory, one
way or another.