Re: [PATCH 0/2] Add support for ZSTD-compressed kernel

[Nick Terrell]

On 10/10/17, 2:56 PM, "hpa@xxxxxxxxx" <hpa@xxxxxxxxx> wrote:
>On October 10, 2017 2:22:42 PM PDT, Nick Terrell <terrelln@xxxxxx> wrote:
>>Hi,
>>
>>This patch set adds support for a ZSTD-compressed kernel and ramdisk
>>images in the kernel boot process. It only integrates the support with
>>x86, though the first patch is generic to all architectures.
>>
>>Zstandard requires slightly more memory during the kernel decompression
>>on x86 (192 KB vs 64 KB), and the memory usage is independent of the
>>window size.
>>
>>Zstandard requires memory proprortional to the window size used during
>>compression for decompressing the ramdisk image, since streaming mode
>>is
>>used. Newer versions of zstd (1.3.2+) list the window size of a file
>>with `zstd -lv <file>'. The absolute maximum amount of memory required
>>is just over 8 MB.
>>
>>Thanks,
>>Nick Terrell
>>
>>Nick Terrell (2):
>>  lib: Add support for ZSTD-compressed kernel
>>  x86: Add support for ZSTD-compressed kernel
>>
>> Documentation/x86/boot.txt        |   6 +-
>> arch/x86/Kconfig                  |   1 +
>> arch/x86/boot/compressed/Makefile |   5 +-
>> arch/x86/boot/compressed/misc.c   |   4 +
>> arch/x86/boot/header.S            |   8 +-
>> arch/x86/include/asm/boot.h       |   6 +-
>> include/linux/decompress/unzstd.h |  26 +++
>> init/Kconfig                      |  14 +-
>> lib/Kconfig                       |   4 +
>> lib/Makefile                      |   1 +
>> lib/decompress.c                  |   5 +
>>lib/decompress_unzstd.c           | 341
>>++++++++++++++++++++++++++++++++++++++
>> lib/xxhash.c                      |  21 ++-
>> lib/zstd/decompress.c             |   2 +
>> lib/zstd/fse_decompress.c         |   9 +-
>> scripts/Makefile.lib              |  15 ++
>> usr/Kconfig                       |  22 +++
>> 17 files changed, 465 insertions(+), 25 deletions(-)
>> create mode 100644 include/linux/decompress/unzstd.h
>> create mode 100644 lib/decompress_unzstd.c
>
>And, pray tell, what are the actual results? What is the trade-off of kernel size versus decompression performance versus the other algorithms that we already support? Adding algorithms for their own sake is a bad thing not a good thing.

Sorry I neglected to include the benchmarks I've run so far. I've included
them below, and will add them to the next version's cover letter.

Comparing the command line tools on a kernel image that is 68970616 B large:

| Algorithm | Compression Ratio | Decompression MB/s |
|-----------|-------------------|--------------------|
| zstd      |              4.42 |              436.5 |
| gzip      |              3.72 |              134.1 |
| xz        |              4.83 |               53.1 |
| lz4       |              3.18 |             1682.2 |
| lzo       |              3.36 |              389.6 |
| bzip2     |              4.03 |               33.3 |

This isn't actually benchmarks of the kernel decompression, but I expect
the relative results to be the same. It compresses the second best and
decompresses the second fastest.

I know that this isn't a real benchmark of the kernel decompression. I
still need to figure out how to time the kernel decompression. If you have
any suggestions let me know. Otherwise, I'll get back to you when I've
figured out how to run the benchmark.

Best,
Nick Terrell