Re: [ANNOUNCE] Reiser5: Data Tiering. Burst Buffers. Speedup synchronous modifications

[Edward Shishkin]

On 05/30/2020 02:32 PM, jose.r.r@xxxxxxxxxxx wrote:
> On Mon, May 25, 2020 at 6:08 PM Edward Shishkin <edward.shishkin@xxxxxxxxx> wrote:
> >                    Reiser5: Data Tiering. Burst Buffers
> >                     Speedup synchronous modifications
> >
> >
> >                Dumping peaks of IO load to a proxy device
> >
> >
> > Now you can add a small high-performance block device to your large
> > logical volume composed of relatively slow commodity disks and get
> > an impression that the whole your volume has throughput which is as
> > high, as the one of that "proxy" device!
> >
> > This is based on a simple observation that in real life IO load is
> > going by peaks, and the idea is to dump those peaks to a high-
> > performance "proxy" device. Usually you have enough time between peaks
> > to flush the proxy device, that is, to migrate the "hot data" from the
> > proxy device to slow media in background mode, so that your proxy
> > device is always ready to accept a new portion of "peaks".
> >
> > Such technique, which is also known as "Burst Buffers", initially
> > appeared in the area of HPC. Despite this fact, it is also important
> > for usual applications. In particular, it allows to speedup the ones,
> > which perform so-called "atomic updates".
> >
> >
> >                  Speedup "atomic updates" in user-space
> >
> >
> > There is a whole class of applications with high requirements to data
> > integrity. Such applications (typically data bases) want to be sure
> > that any data modifications either complete, or they don't. And they
> > don't appear as partially occurred. Some applications has weaker
> > requirements: with some restrictions they accept also partially
> > occurred modifications.
> >
> > Atomic updates in user space are performed via a sequence of 3 steps.
> > Suppose you need to modify data of some file "foo" in an atomic way.
> > For this you need to:
> >
> > 1. write a new temporary file "foo.tmp" with modified data
> > 2. issue fsync(2) against "foo.tmp"
> > 3. rename "foo.tmp" to "foo".
> >
> > At step 1 the file system populates page cache with new data
> > At step 2 the file system allocates disk addresses for all logical
> > blocks of the file foo.tmp and writes that file to disk. At step 3 all
> > blocks containing old data get released.
> >
> > Note that steps 2 and 3 become a reason of essential performance drop
> > on slow media. The situation gets improved, when all dirty data are
> > written to a dedicated high-performance proxy-disk, which exactly
> > happens in a file system with Burst Buffers support.
> >
> >
> >             Speedup all synchronous modifications (TODO)
> >                 Burst Buffers and transaction manager
> >
> >
> > Not only dirty data pages, but also dirty meta-data pages can be
> > dumped to the proxy-device, so that step (3) above also won't
> > contribute to the performance drop.
> >
> > Moreover, not only new logical data blocks can be dumped to the proxy
> > disk. All dirty data pages, including ones, which already have
> > location on the main (slow) storage can also be relocated to the proxy
> > disk, thus, speeding up synchronous modification of files in _all_
> > cases (not only in atomic updates via write-fsync-rename sequence
> > described above).
> >
> > Indeed, let's remind that any modified page is always written to disk
> > in a context of committing some transaction. Depending on the commit
> > strategy (there are 2 ones "relocate" and "overwrite"), for each such
> > modified dirty page there are only 2 possibility:
> >
> > a) to be written right away to a new location,
> > b) to be written first to a temporary location (journal), then to be
> >      written back to permanent location.
> >
> > With Burst buffers support in the case (a) the file system writes
> > dirty page right away to the proxy device. Then user should take care
> > to migrate it back to the permanent storage (see section "Flushing
> > proxy devise" below). In the case (b) the modified copy will be
> > written to the proxy device (wandering logs), then at checkpoint time
> > (playing a transaction) reiser4 transaction manager will write it to
> > the permanent location (on commodity disks). In this case user doesn't
> > need to worry on flushing proxy device, however, the procedure of
> > commit takes more time, as user should also wait for "checkpoint
> > completion".
> >
> > So from the standpoint of performance "write-anywhere" transaction
> > model (reiser4 mount option "txmod=wa") is more preferable then
> > journalling model (txmod=journal), or even hybrid model (txmod=hybrid)
> >
> >
> >               Predictable and non-predictable migration
> >                         Meta-data migration
> >
> >
> > As we already mentioned, not only dirty data pages, but also dirty
> > meta-data pages can be dumped to the proxy-device. Note, however, that
> > not predictable meta-data migration is not possible because of
> > chicken-eggish problem. Indeed, non-predictable migration means that
> > nobody knows, on what device of your logical volume a stripe of data
> > will be relocated in the future. Such migration requires to record
> > location of data stripes. Now note, that such records is always a part
> > of meta-data. Hence, you are now able to migrate meta-data in
> > non-predictable way.
> >
> > However, it is perfectly possible to distribute/migrate meta-data in a
> > predictable way (it will be supported in so-called "symmetric" logical
> > volumes - currently not implemented). Classic example of predictable
> > migration is RAID arrays (once you add, or remove a device to/from the
> > array, all data blocks migrate in predictable way during rebalancing).
> > If relocation is predictable, then it is not need to record locations
> > of data stripes - it can always be calculated.
> >
> > Thus, non-predictable migration is applicable to data only.
> >
> >
> >                     Definition of data tiering.
> >                Using proxy device to store hot data (TODO)
> >
> >
> > Now we can precisely define tiering as (meta-)data relocation in
> > accordance with some strategy (automatic, or user-defined), so that
> > every relocated unit always gets location on another device-component
> > of the logical volume.
> >
> > During such relocation block number B1 on device D1 gets released,
> > first address component is changed to D2, second component is changed
> > to 0 (which indicates not allocated block number), then the file
> > system allocates block number B2 on device D2:
> >
> >                   (D1, B1) -> (D2, 0) -> (D2, B2)
> >
> > Note that tiering is not defined for simple volumes (i.e. volumes,
> > consisting only of one device). Blocks relocation within one device
> > is always in a competence of a file system (to be precisely, of block
> > allocator.
> >
> > Burst buffers is just one of strategies, in accordance with which all
> > new logical blocks (optionally, all dirty pages) always get location
> > on a dedicated proxy device. As we have figured out, Burst Buffers is
> > useful for HPC applications, as well as for usual applications
> > executing fsync(2) frequently.
> >
> > There are other data tiering strategies, which can be useful for other
> > class of applications. All of them can be easily implemented in
> > Reiser5.
> >
> > For example, you can use proxy device to store hot data only. With
> > such strategy new logical blocks (which are always "cold") will always
> > go to the main storage (in contrast with Burst Buffers, where new
> > logical blocks first get written to the proxy disk). Once in a while
> > you need to scan your volume in order to push colder data out, and
> > pull hotter data in the proxy disk. Reiser5 contains a common
> > interface for this. It is possible to maintain per-file, or even per-
> > blocks-extent "temperature" of data (e.g. as a generation counter),
> > but we still don't have more or less satisfactory algorithms to
> > determine "critical temperature" for pushing data in/out proxy disk.
> >
> >
> >           Getting started with proxy disk over logical volume
> >
> >
> > Just follow the administration guide:
> > https://reiser4.wiki.kernel.org/index.php/Proxy_Device_Administration
> Re:
> > WARNING: THE STUFF IS NOT STABLE! Don't store important data on
> > Reiser5 logical volumes till beta-stability announcement.
> Will you be releasing reiser4 Software Format Release Number 4.0.2 for Linux kernel 5.6 patch?

I've uploaded it.
Please, find here:

https://sourceforge.net/projects/reiser4/files/reiser4-for-linux-5.x/

Thanks,
Edward

>  From personal experience SFRN 4.0.2 is stable and all my data, local and cloud virtual machines instances, as well as computing for the last six(6)+ years is in that format/environment. Although I have not tried an Debian based installation with this second iteration of SFRN 5 I have no use for the kernel/reiser4progs until they play well with Debian installer, python, etc..
>
> Best Professional Regards.