Re: large directory handling speed

Alexander V. Lukyanov (lav@yars.free.net)
Mon, 31 May 1999 19:44:07 +0400


On Mon, May 31, 1999 at 05:05:46PM +0200, Jamie Lokier wrote:
> Alexander V. Lukyanov wrote:
> > Cyclic search looks good. At least it would make better the common
> > case of readdir/stat. But what if a stupid program sort file name list
> > before stat's :) (fortunately it is not common)
>
> For small/medium ext2 directories and cold caches, the fastest way to
> stat all the entries is to sort by inode number, d_ino. This orders the
> disk I/O optimally, and the saving from that is greater than the loss
> due to directory search time.

I was talking about vfat... But I doubt kernel can order stat's in any
way, that can be done in program though. But then the program becomes
optimized for one particular filesystem, which is not good.

> I haven't tested this on very large directories -- but I'd expect the
> quadratic directory search time to be more significant than I/O time
> for ext2, fat etc.
>
> The cyclic search suggestion
>
> - wouldn't help for randomly ordered directories,
> but wouldn't make things any worse either
> (on ext2, this tends to be directories containing other directories)
>
> - would help a lot for directories whose entries are mostly in inode order
> (on ext2, this tends to be most directories containing files)

I think cyclic search would not do anything at all for ext2 and vfat
(good or bad) on average - that is when order of access does not
correlate with order of directory entries.

Both the cyclic search and search from beginning have their worst
cases and best cases (equal). But ciclic search would speed up certain
common patterns of access, e.g. readdir/stat of ls/find/mc etc.

But for ext2 it is not so critical because it stores directory in a
more effective form so reading it is faster. And soom it will have
b-tree directories that would eliminate the whole problem. So it is
more of a problem for vfat/iso9660 and other ineffective filesystems.

Alexander.

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.rutgers.edu
Please read the FAQ at http://www.tux.org/lkml/