Re: [PATCH][RFC] Rearranging files to improve disk performanc

[Török Edvin]

On 9/25/06, Phillip Susi <psusi@xxxxxxxxxx> wrote:
> This ability is already available at least for ext2/3 in the old defrag
> package, which can take a list of files and priorities to assign to
> them, and it will pack them in the order given at the start of the disk.

My idea was to use existing filesystem calls to move files around. That is why
the basic idea is to have online defragmentation.
If it is possible to use the code from the old ext2/3 defrag to create
an online defragmentation program, excellent. But I'd like a general
solution, that would work
/could be made to work with all filesystems in the future.
That is why I asked for comments on my suggestion of a (real) simple
filesystem rearranger (or defragmenter if you want to call it that
way).

> As for your idea, if there is going to be online defrag support in the
> kernel ( and yes, this is a form of defragmenting ) I'd rather see the
> ability to move files around deterministically rather than just give a
> hint and pray, similar to how windows handles online defragmenting.

I agree with this, and maybe sysfs is a bad choice for what I'd like
to accomplish. Maybe a netlink based api would be better, as we would
have a more obvious way of reporting errors?

Anyway, regardless of the user to kernel communication, IMHO it should
be established _what_ will userspace communicate to kernel, and
viceversa (i.e. an API).


1. Determine current file location on disk.
We already have this: FIBMAP ioctl.

2. For determining where to put files, we need to know the free extents.
'debugreiserfs -m <partition>' already accomplishes this, but it
should be made more general.

3. Ascertaining the optimal location for files.
Can be done entirely in userspace, once you have the information from
steps 1 and 2.

4. Notifying userspace on writes to disk/pending writes to disk.
Can be done via inotify/dnotify. This is needed, so that userspace
knows to refresh its idea of
free extents, and possibly the defrag strategy.

5. Once a defrag strategy has been determined, tell the kernel to
reserve those blocks
for our files. (preallocation?) This can be useful if you are
defragmenting a system under load, and can't afford to shut down
processes that write to the disk. And also refreshing the fs bitmap
would be expensive.

6. It should be easy to reclaim blocks preallocated in step 5. (in the
(unlikely?) event the defrag program crashes/needs to be restarted, or
if defrag is aborted).

7. An API to move files around, or at least chunks of it.
This is the critical part. If space has been preallocated in step 5,
it should be as easy as copy the file to new place/ delete from old
place. (relocate_inode(inode,newlocation)).

8. Have the ability to submit multiple move request simultaneously to
take advantage of ncq/tcq/write-read merges,...

9. Take advantage of blocks freed up after moving files. Thus if
multiple requests are issued, they must not operate on the same block
(obviously).

10. Status reports on the success of preallocation, file moves,...
especially the reason why an operation was not possible.


I welcome any comments/suggestions on the above. Feel free to improve
on it, or to rewrite the defrag-api-ideas from scratch.
I would really appreciate if somebody could point out problems with
certain assumptions I made above, and of course welcome any criticism
too.


Best regards,
Edwin
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/