> Date: Mon, 17 May 1999 10:54:04 +0200 (MEST)
> From: Rogier Wolff <R.E.Wolff@BitWizard.nl>
> To: Pavel Machek <pavel@bug.ucw.cz>
> Cc: jorgegv@icai.upco.es, linux-kernel@vger.rutgers.edu
> Subject: Re: Deletion of big files...
>
> Pavel Machek wrote:
> > PS: I could imagine Irix being faster not because asynchrony, but
> > because their fs is more effective for this concrete issue.
>
> Linux is horribly slow in this regard. Rob van Nieuwkerk reports 15Mb
> per second of "deletion". You can almost fill a modern disk at that
> speed!
>
> Somehow, rm-ing a large file involves much more IO than what can be
> accounted for in a quick calculation:
>
> To free 15Mb of disk space, Linux needs to write 15k bits to
> disk. That is 2k bytes. We should be able to do better than 2k per
> second....
>
> Indirect and double indirect (*) blocks might need reading. To do 15Mb
> per second, we need to read 15k worth of indirect block pointers. That
> is 60k per second. We should be able to do better than that....
>
> I don't know what it is doing.
>
> Hmm. Maybe I do....
>
> If we read a 1k indirect block, and then have to seek for the next 1k
> block of indirect pointers, then that might explain things: 10ms seek
> time, 6 ms avg rotational latency -> 16ms per 1k block -> 60 blocks
> per second.
When I tried removing large files on filesystem with 1k blocks there is
indeed a lot of seek activity visible on the disk.
>
> This would mean that we'd need to group the indirect blocks more than
> we do now in writing large files.
>
> To test if this theory is close, a prediction:
>
> Delete performance will be almost 16 times better on a filesystem with
> 4k blocksize than with a 1k blocksize.
>
Actually I made such a test a few weeks ago.
I made 2 500MB filesystems on a SCSI disk, one with 1kB block size
and the another with 4kB block size.
I must admit the test is not 100% scientific, since the filesystems were
in different locations on the same disk, so there might be some
differences in seek time or transfer speed depending on physical location
on the disk (I just did not believe it would make much difference, so I
have stopped these tests).
The test was like this :
I created 4 files 100MB each on one filesystem,
and 4 identical files on another filesystem (using dd from /dev/zero,
ibs=1k, obs=1024k);
Then I timed rm command;
For files on filesystem with 1k block size, removing one file took about
7 seconds elapsed time; for files on filesystem with 4k block size
it took 0.1 to 0.2 second, that is more than 30 times faster.
When removing files from filesystem with 1k block size, vmstat showed
about 60 operations per second in bi column and 6..10 operations
in bo columns every 6..8 seconds:
[root@RHL52box /root]# vmstat 2 18
procs memory swap io system cpu
r b w swpd free buff cache si so bi bo in cs us sy id
0 0 0 60 2508 49876 1436 0 0 2 7 104 5 0 0 100
0 0 0 60 2508 49876 1436 0 0 0 0 102 9 0 0 100
0 0 0 60 2508 49876 1436 0 0 0 0 102 7 0 0 100
0 0 0 60 2508 49876 1436 0 0 0 0 104 7 0 0 99
1 0 0 60 2384 49880 1448 0 0 32 3 134 69 0 1 99
1 0 0 60 2384 49880 1448 0 0 60 0 162 125 0 2 98
1 0 0 60 2380 49884 1448 0 0 62 1 164 130 0 1 99
1 0 0 60 2380 49884 1448 0 0 60 0 163 244 0 1 99
0 1 0 60 2376 49888 1448 0 0 60 0 162 125 0 0 100
0 1 0 60 2372 49892 1448 0 0 58 9 168 123 0 1 99
1 0 0 60 2368 49896 1448 0 0 61 0 163 127 0 1 99
0 1 0 60 2368 49896 1448 0 0 61 6 169 246 1 1 98
0 1 0 60 2364 49900 1448 0 0 60 0 162 125 0 2 98
0 1 0 60 2360 49904 1448 0 0 62 0 163 129 0 0 100
0 1 0 60 2360 49904 1448 0 0 59 10 171 241 0 1 99
0 1 0 60 2356 49908 1448 0 0 61 0 170 132 0 1 99
0 1 0 60 2356 49908 1448 0 0 58 6 165 122 0 0 100
1 0 0 60 2356 49908 1448 0 0 60 0 163 127 0 1 99
When removing files from filesystem with 4k block size, vmstat showed output
like this:
[root@RHL52box /root]# vmstat 2 12
procs memory swap io system cpu
r b w swpd free buff cache si so bi bo in cs us sy id
0 0 0 60 2452 49908 1448 0 0 2 7 104 5 0 0 100
0 0 0 60 2452 49908 1448 0 0 0 0 102 7 0 0 100
0 0 0 60 2452 49908 1448 0 0 0 0 102 7 0 0 100
0 0 0 60 2444 49916 1448 0 0 45 0 153 571 1 6 93
0 0 0 60 2444 49916 1448 0 0 0 0 103 7 0 0 100
0 0 0 60 2444 49916 1448 0 0 0 3 104 8 0 0 100
The tests were performed on an idle system with RedHat Linux 5.2 with
updates for 2.2 kernels and 2.2.5 kernel with no patches.
>
> I just tested my "fresh out of the box" system (1k blocksize), and it
> deletes almost 30Mb per second. Creating the file was done at 7Mb per
> second (there is something not completely right with that disk: It
> should be able to do double that....)
>
> Regards,
>
> Roger Wolff.
>
>
> (*) Double indirect are soooo rare, that they don't figure into the
> performance calculation....
>
>
I would suggest that filesystems containg large files be created with
block size larger than 1k. In my tests involving 500MB filesystems and
100MB files the time difference for rm command was more that 30-fold
between filesystem with 4kB block and one with 1kB block size.
If you deal with even larger files, you might want to test performance
of filesystems with even larger block size.
Best regards,
Wojtek
--------------------
Wojtek Pilorz
Wojtek.Pilorz@bdk.pl
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.rutgers.edu
Please read the FAQ at http://www.tux.org/lkml/