Re: Compression filter for Loopback device

[Paulo Marques]

On Thu, 2004-07-22 at 20:27, Lei Yang wrote:
> Hi all,
> 
> Is there anything like 'losetup' that allows choosing encryption
> algorithm for a loopback device that can be used on compression
> algorithms? Or in other words, when the data passes through loopback
> device to a real storage device, it can be filtered and the filter is
> compression instead of encryption; when kernel ask for data in a storage
> device that is mounted to a loopback device with compression, it will be
> filtered again -- decompressed.
> 
> If there is not a ready-to-use method for this, is there any way I can
> implement the idea?

There is cloop. The only problem with cloop is that it is read-only.

If you want read-write, there is no solution AFAIK(*).

I did start working on something like that a while ago. I even
registered for a project on sourceforge: 

http://sourceforge.net/projects/zloop/

I stopped working on it because:

1 - I didn't have the time

2 - There are some nasty issues with this concept:

    - The image file would ideally shrink and grow according to the 
achieved compression ratio. In the worst case it would have to grow to
more than the size of the "block device" (if you wrote a bunch of
already compressed files to the device, for instance), because it has to
keep some metadata. This reduces the scenarios where this sort of
compressed loopback device could be used.

    - Respect the sequence of writes to the block device is tricky. This
is important because you have to guarantee that a journaled filesystem
on top of the block device will assure data integrity. This is worst
than on a normal loopback because you have to make sure the "block
device metadata" is also written at appropriate times. I actually
conceived an algorithm that could acomplish this with little overhead,
but never got to implement it.

    - The block device doesn't understand anything about files. This is
an advantage because it will compress the filesystem metadata
transparently, but it is bad because it compresses "unused" blocks of
data. This could probably be avoided with a patch I saw floating around
a while ago that zero'ed delete ext2 files. Zero'ed blocks didn't accupy
any space at all in my compressed scheme, only metadata (only 2 bytes
per block).

3 - There didn't seem to be much interest from the commmunity in
something like this.


If interest rises now, I'll probably have the time to resume the project
where I left off.

I did a proof of concept using a nbd server. This way I could test
everything in user space.

With this NBD server I tested the compression ratios that my scheme
could achieve, and they were much better than those achieved by cramfs,
and close to tar.gz ratios. This I wasn't expecting, but it was a nice
surprise :)

Anyway, any feedback, suggestions, ideas on this will be greatly
appreciated.

-- 
Paulo Marques - www.grupopie.com
"In a world without walls and fences who needs windows and gates?"



(*) There is JFFS2 that you can mount on top of a mtd-block device. I
never tried it personally, because it seemed a bit of a cludge, and it
didn't get the shrink and grow effect that I wanted from the compressed
loop-back. There is also an old ext2 "compression extension" that seemed
not to be mantained for a long time, last time I checked.





-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/