Generally, the penalties for getting this stuff wrong are very very high:
orders of magnitude slowdowns in the right situations. Which I suspect
will make any system-wide knob ultimately unsuccessful.
The ideal way of getting this *right* is to change every application in the
world to get smart about using sync_page_range() and/or posix_fadvise(),
then to add a set of command-line options to each application in the world
so the user can control its pagecache handling.
Obviously that isn't practical. But what _could_ be done is to put these
pagecache smarts into glibc's read() and write() code. So the user can do:
MAX_PAGECACHE=4M MAX_DIRTY_PAGECACHE=2M rsync foo bar
This will provide pagecache control for pretty much every application. It
has limitations (fork+exec behaviour??) but will be useful.
A kernel-based solution might use new rlimits, but would not be as flexible-
or successful as a libc-based one, I suspect.