If a device driver wants "DMAable" memory, and thus does a ZONE_DMA
allocation, and we've moved all its memory from ZONE_DMA to ZONE_NORMAL
(as I think you're proposing doing for PPC64 (and ia64?)), then the
allocation will fail.
I would not presume proposing anything for PPC64 and left everything the way it is. The arch people can control if they want ZONE DMA or not and the default is to leave things as is. If PPC64 wants to go ZONE_DMAless then the arch code needs to be modified not refer to ZONE_DMA anymore and if that works then we can switch CONFIG_ZONE_DMA off for PPC64. See the patches in mm for examples how other arches have done it.
So are we saying that no driver code should be calling with GFP_DMA
(a quick grep turns up 148 instances under driver/), that if they do
they should only work on specific architectures (some instances were
s390-only drivers)? If so, should we not be removing the definiton of
GFP_DMA itself if ZONE_DMA is config'ed out, so that it fails at
compile time, rather than runtime?
This was covered at length before. Removing all GFP_DMA references would require extensive #ifdefs. The limited patch in mm is only neutering GFP_DMA for arches that do not need it. If an arch has removed its definition of CONFIG_ZONE_DMA then __GFP_DMA will be ignored in the page allocator.