[BUG] Linux VM use-once mechanisms don't work (test case withnumbers included)

[Eric St-Laurent]

Linux VM use-once mechanisms don't seem to work.  Simple scenario like
streaming a file much greater than physical RAM size should be
identified to avoid trashing the page cache with useless data.

I know the VM cannot predict the future or assume anything about the
user's intent.  But this workload is simple and common, it should be
detected and better handled.

Test case:

Linux 2.6.20-16-lowlatency SMP PREEMPT x86_64 (also tried on 2.6.23-rc1)

- A file of 1/3 the RAM size is created, mapped and frequently accessed
(4 times).
- The test is run multiple times (4 total) to time it's execution.
- After the first run, other runs take much less time, because the file
is cached.
- A previously created file, 4 times the size of the RAM, is read or
copied.
- The test is re-run (2 times) to time it's execution.

To test:

$ make
# ./use-once-test.sh

Some big files will be created in your /tmp. They don't get erased after
the test to speedup multiple runs.

Results:

- The test execution time greatly increase after reading or copying the
large file.
- Frequently used data got kick out of the page cache and replaced with
useless read once data.
- Both the read only and copy (read + write) cases don't work.

I believe this clearly illustrate the slowdowns I experience after I
copy large files around my system.  All applications on my desktop are
jerky for some moments after that.  Watching a DVD is another example.

Base test:

1st run: 0m8.958s
2nd run: 0m3.442s
3rd run: 0m3.452s
4th run: 0m3.443s

Reading a large file test:

1st run: 0m8.997s
2nd run: 0m3.522s
`/tmp/large_file' -> `/dev/null'
3rd run: 0m8.999s    <<< page cache trashed
4th run: 0m3.440s

Copying (using cp) a large file test:

1st run: 0m8.979s
2nd run: 0m3.442s
`/tmp/large_file' -> `/tmp/large_file.copy'
3rd run: 0m13.814s    <<< page cache trashed
4th run: 0m3.455s

Copying (using fadvise_cp) a large file test:

1st run: 0m9.018s
2nd run: 0m3.444s
Copying large file...
3rd run: 0m14.024s    <<< page cache trashed
4th run: 0m3.449s

Copying (using splice-cp) a large file test:

1st run: 0m8.977s
2nd run: 0m3.442s
Copying large file...
3rd run: 0m14.118s    <<< page cache trashed
4th run: 0m3.456s

Possible solutions:

Various patches to fix the use-once mechanisms were discussed in the
past.  Some more that 6 years ago and some more recently.

http://lwn.net/2001/0726/a/2q.php3
http://lkml.org/lkml/2005/5/3/6
http://lkml.org/lkml/2006/7/17/192
http://lkml.org/lkml/2007/7/9/340
http://lkml.org/lkml/2007/7/21/219 (*1)

(*1) I have tested Peter's patch with some success.  It fix the read
case, but no the copy case.  Results: http://lkml.org/lkml/2007/7/24/527

Test programs and batch files are attached.


- Eric

#include <fcntl.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>

int main(int argc, char *argv[])
{
	int in;
	int out;
	int pagesize; 
	void *buf;
	off_t pos;

	if (argc != 3) {
		printf("Usage: %s <src> <dest>\n", argv[0]); 
		return EXIT_FAILURE;
	}

	in = open(argv[1], O_RDONLY, 0);
	out = open(argv[2], O_CREAT | O_WRONLY | O_TRUNC, 0666);

	posix_fadvise(in, 0, 0, POSIX_FADV_SEQUENTIAL);
	posix_fadvise(out, 0, 0, POSIX_FADV_SEQUENTIAL);

	pagesize = getpagesize();
	buf = malloc(pagesize);

	pos = 0;

	for (;;) {
		ssize_t count;

		count = read(in, buf, pagesize);
		if (!count || count == -1)
			break;

		write(out, buf, count);

		/* right usage pattern? */
		posix_fadvise(in, pos, count, POSIX_FADV_NOREUSE);
		posix_fadvise(out, pos, count, POSIX_FADV_NOREUSE);

		pos += count;
	}

	free(buf);
	close(in);
	close(out);

	return EXIT_SUCCESS;
}
all:
	gcc fadvise_cp.c -o fadvise_cp
	gcc working_set_simul.c -o working_set_simul

Attachment: use-once-test.sh
Description: application/shellscript

#include <fcntl.h>
#include <memory.h>
#include <stdio.h>
#include <stdlib.h>
#include <sys/mman.h>
#include <unistd.h> 

int main(int argc, char *argv[])
{
	int fd;
	off_t size;
	char *mapping;
	unsigned r;
	unsigned i;

	if (argc != 2) {
		printf("Usage: %s <file>\n", argv[0]); 
		return EXIT_FAILURE;
	}

	fd = open(argv[1], O_RDONLY, 0);
	size = lseek(fd, 0, SEEK_END); 

	mapping = mmap(NULL, size, PROT_READ, MAP_PRIVATE, fd, 0);

	/* access (read) the file a couple of times*/
	for (r = 0; r < 4; r++) {
		for (i = 0; i < size; i++) {
			char t = mapping[i];
		}
	}

	munmap(mapping, size);
	close(fd);

	return EXIT_SUCCESS;
}