Re: albods are not a clean set of orthogonal primitives (was Re: File

Wanderer (wanderer@everywhere.net)
Thu, 24 Jun 1999 10:10:57 -0700


Hans Reiser wrote:

> I feel that you are not orthogonalizing the desired feature into a set
> of fully independent orthogonal primitives.
>
> Compression is one example of this. Compression is a fully orthogonal
> issue. Why tie it to any other functionality?
>
> Placing the filesystem into user libraries exo-kernel style is another
> orthogonal issue. I don't want to start an exo-kernel implementation
> right now, especially not without doing it completely and systematically
> for all of the filesystem.
>
> Transferring directories and virtual files can be solved in several
> simple and effective ways, we should pick one, and systematically
> implement it for all virtual files. I like the one in which when you
> transfer files you access the a special view of the FS:
> /filters-off-and-portable-format-only-visible/pathname/foo (or rather some
> such equivalent short named thing) to do the transfer. Tar can use this
> too. Someday we ought to use it for symlinks and file holes also (they
> are just virtual files too.)
>
> I think that there should be only one interface to the set primitive,
> and that streams are hideous for at least the reason that they create a
> second kind of set with a second kind of interface.
>
> What I understand here is that you are proposing a second kind of set
> with a second kind of interface, so that albod ignorant programs can
> transfer an albod without having to understand it. The problem that
> leaves unsolved is that albod ignorant programs cannot access its
> components because they don't understand it, and that a second
> interfaces is per se bad. I think you feel that albod ignorant programs
> don't need to access it. Here I think you are wrong. Richard correctly
> pointed out that there is no reason to change emacs to handle any of
> this, it can all be done by the FS. He is right, so long as we keep
> things clean.
>
> you would have us make it feasible to ignorantly transfer a
> directory/albod, at the cost of us not ignorantly using a
> directory/albod. I think the use of views makes it feasible to get
> both.
>
> Finally, I don't see any need for the word application anywhere in the
> name of any of these orthogonal primitives we are creating. What we are
> creating has no need to be more application specific than a directory is
> application specific. But then, maybe you are one of those folks who
> thinks it is clear what is OS and what is application. I am not one.
>
> In summary, each of the following has merit independent of the others:
>
> overloading names so that directories and files can have the same names
>
> filters to convert directories into various formats with rdf as one
> default so that they may be edited as flat files, and tar as another so
> that they may be transferred. Note that, as Richard pointed out, the
> format you want to use for editing with emacs is probably not the format
> you want to use for transferring. The format for editing is especially
> not tar. With filters come bundled some convention for inserting
> certain standard filters into the namespace. (dirname/..rdf, and
> dirname/..tar, and dirname/..cat might be good. Note that dirname/..rdf
> supports write, but dirname/..cat is a read-only file. Actually, if I
> am the filter author, or somebody I pay is, then only dirname/..rdf
> would have support for write, because it would be more work to do more,
> and I am lazy/cheap. Writing dirname/..rdf would allow specifying a
> non-random ordering of the elements in the directory, otherwise reading
> dirname/..rdf defaults to whatever order is convenient to the FS).
>
> file body inheritance
>
> stat data inheritance
>
> symlinks (already implemented)
>
> Note that I am unsure whether filters should also be used to implement
> file body inheritance, stat data inheritance, and symlinks, and feel
> vulnerable to good arguments on the topic.
>
> Together, they would allow all the functionality afforded by streams
> that Jeremy needs.. Yet because they are kept fully orthogonal, they
> will be useful for much more than applications asking for streams
> functionality. Especially compression.
>
> Now I think it is time to get this implemented. I am going to try to
> hire somebody or somebodies for this this week. As it gets implemented
> a lot of the details will fall into place, I don't really want to design
> this more before I have a person assigned to do the work. If Acy does
> filters, I'll focus my guys on the other features, otherwise we'll
> implement filters too.
>
> Hans
>

I don't think filters is a good idea. They just keep multiplying out of
control. Kernel implementations are supposed to be simple, elegant
solutions to specific issues. The problem here is how to provide 'clustering'
of information without sacrificing accessibility. Clustering of
information includes both the file data, directory data, and meta-data.

My original proposal addressed most of these issues. First, non-specialized
applications would see special Meta-Files as ordinary files. There would be
no problem with FTP, cp, etc. There could be a problem with randomly
using commands that could alter a file, but that is why we have permissions.

I had already gone through the PRELOAD user space iteration, but there is
one big problem. The basic reason that I proposed this as a kernel issue was
the
problem of being able to execute an image that is contained within a
Meta-File.
Nearly all other features are user space issues.

Ignoring this thread's misinformed flaming ... I've considered (I think) the
shortfalls of the other implementations (e.g., MacOS) when I decided that
any Meta-File implementation MUST look like a standard file system
hierarchy to current applications. This permits all users and/or tools to
delve
into the meta-file and alter the content as needed. Protected by permissions.

I've begun seriously looking at implementation of basic meta-aware
patches. What I'm planning is essentially zero impact to existing systems
that do not use meta-files. Any meta-files placed on such a system will simply

be files. Meta-aware systems will register meta-file formats as is currently
done
for file system types. My 'binder' style would be one potential meta-file
format. VFS would access meta-file handlers only when the a file is tagged
as a meta-file. The meta-file handler then verifies the file format and
accepts the meta-file and adds it to a special VFS super block for meta-files.

At this point, the meta-handler simply is attached via the standard VFS
vectors to 'do the right thing' with standard VFS calls to handle the
meta-file itself. This preserves all the kernel handling of dcache, buffer,
mmap, and bin formats with no change (or very little at least).

In the binder meta-format (others may differ) a directory entry would have
additional data type/data format information ala mime. It would retain
user and group information as an embedded application may need some
euid/egid handling. I see no need for time information as this is all derived
and set in the file system entry. I do want a mark of some sort to indicate
that a binder has been changed from the original so we don't unintentionally
propagate modified versions (since we make FTPing so easy :) ). Effective
permissions for an entry are derived by merging actual FS permissions
with those contained in the meta-file (probably an OR, but I need to think
about it more). Symlinks would be permitted (perhaps limited), but hard
links would not.

In summary, meta-files look like directories outside the kernel. Inside the
kernel, special handlers, styled after VFS, handle the conversion between
VFS file view and Meta directory view. File flags tell VFS when a
meta-handler is to be invoked. These file flags must be explicitly set via
'chmod' (or equivalent). Linux file systems can save the flags, other FSes
must generate the flags as needed. Entries in a meta-file appear as standard
files (byte streams) to existing applications. Meta-aware applications would
have additional searching and type handling available. They would also have
file name mapping applied. Two file flags indicate - Not a Meta-File;
Meta-File,
Meta-Application; and Meta-Special (for growth).

I'm not aware of the special W2K issues referenced in some messages, but
allowing differing meta-file handlers should provide a base to build upon.
(I did a quick dejanews search, but didn't see anything relevant.) Everything

else I've read in these discussions seem to be addressed by this approach.
Such as ... multiple forks, special re-maps, nesting of meta-files, extended
type information, and legacy applications. Special filters and such are user
space issues - kernel provides the container, user space pours contents into
it. Did I miss something?

--------
[Don't wander much these days ... I'm jmills@portland.quik.com]

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.rutgers.edu
Please read the FAQ at http://www.tux.org/lkml/