This is where I see the need for "CPU sets". I.e. as a replacement/modification to the CPU affinity mechanism basically adding an extra level of abstraction to make it easier to use for implementing the type of isolation that people seem to want. I say this because, strictly speaking and as you imply, the current affinity mechanism is sufficient to provide that isolation BUT it would be a huge pain to implement.
The way I see it you just replace the task's affinity mask with a pointer to its "CPU set" which contains the affinity mask shared by tasks belonging to that set (and this is used by try_to_wake_up() and the load balancing mechanism to do their stuff instead of the per task affinity mask). Then when you want to do something like take a CPU away from one group of tasks and give it to another group of tasks it's just a matter of changing the affinity masks in the sets instead of visiting every one of the tasks individually and changing their masks.
There should be no need to explicitly move tasks off the "lost" CPU after such a change as it should/could be done next time that they go through try_to_wake_up() and/or finish a time slice. Moving a task from one CPU set to another would be a similar process to the current change of affinity mask.
There would, of course, need to be some restriction on the movement of CPUs from one set to another so that you don't end up with an empty set with live tasks, etc.
A possible problem is that there may be users whose use of the current affinity mechanism would be broken by such a change. A compile time choice between the current mechanism and a set based mechanism would be a possible solution. Of course, this proposed modification wouldn't make any sense with less than 3 CPUs.
PS Once CPU sets were implemented like this, configurable CPU schedulers (such as (blatant plug :-)) ZAPHOD) could have "per CPU set" configurations, CKRM could do its (CPU management stuff) stuff within a CPU set, etc.
Tricky in that it needs to be decided what the class hierarchy definitions and whether to CKRM cpu scheduling within each cpuset and what the exact definition of share then is ?
The tricky stuff comes in from the fact that CKRM assumes a system wide definition of a class and a system wide "calculation" of shares.
Doesn't sound insurmountable or particularly tricky :-).
Peter