[ANN] Squashfs tools 4.3 released

[Phillip Lougher]

Hi,

I'm pleased to announce the release of Squashfs tools 4.3.
This is the first release in over 3 years, and there are substantial
improvements to stability, new compression options and compressors,
speed optimisations, new options for Mksquashfs/Unsquashfs, and fixes
for CVE-2012-4024 and CVE-2012-4025.

The new release can be downloaded here:
http://sourceforge.net/projects/squashfs/files/latest/download?source=files

Changes in Squashfs-tools 4.3
-----------------------------

1. Stability improvements.  Better checking of user input for out of
   range/invalid values.  Better handling of corrupted Squashfs filesystems
   (Mksquashfs append mode, and Unsquashfs).  Better handling of buffer
   overflow/underflow.

2. GZIP compressor now supports compression options, allowing different
   compression levels to be used.

3. Rewritten LZO compressor with compression options, allowing different
   LZO algorithms and different compression levels to be used.

4. New LZ4 compressor (note not yet in mainline kernel)

5. Better default memory usage for Mksquashfs.  Mksquashfs by default now
   uses 25% of physical memory.

6. Duplicate checking in Mksquashfs further optimised.  With certain
   "problem filesystems" greater than 2x performance improvement.
   Filesystems with a lot of duplicates should see at least 10-20% speed
   improvement.

7. The -stat option in Unsquashfs now displays the compression options
   used to generate the original filesystem.  Previously -stat only displayed
   the compression algorithm used.

8. The file being compressed/uncompressed in Mksquashfs/Unsquashfs is now
   displayed if CTRL-\ (SIGQUIT from keyboard) typed.

9. The status of the internal queues/caches in Mksquashfs/Unsquashfs is
   now displayed if CTRL-\ (SIGQUIT from keyboard) is typed twice within
   one second.  Normally only useful for "power users", but it can be
   used to discover if there's any bottlenecks affecting performance
   (the bottleneck will normally be the compressors/fragment compressors).

10. Miscellaneous new options for Mksquashfs/Unsquashfs to fine tune behaviour.

11. Fixes for CVE-2012-4024 and CVE-2012-4025.

Compatiblity
------------

Mksquashfs 4.3 generates 4.0 filesystems.  These filesystems are fully
compatible/interchangable with filesystems generated by Mksquashfs 4.0 and are
mountable on 2.6.29 and later kernels.

Compressors
-----------

New compression options and compressors are now supported.

The new options and compressors are:

1. gzip
	  -Xcompression-level <compression-level>
		<compression-level> should be 1 .. 9 (default 9)
	  -Xwindow-size <window-size>
		<window-size> should be 8 .. 15 (default 15)
	  -Xstrategy strategy1,strategy2,...,strategyN
		Compress using strategy1,strategy2,...,strategyN in turn
		and choose the best compression.
		Available strategies: default, filtered, huffman_only,
		run_length_encoded and fixed

2. lzo
	  -Xalgorithm <algorithm>
		Where <algorithm> is one of:
			lzo1x_1
			lzo1x_1_11
			lzo1x_1_12
			lzo1x_1_15
			lzo1x_999 (default)
	  -Xcompression-level <compression-level>
		<compression-level> should be 1 .. 9 (default 8)
		Only applies to lzo1x_999 algorithm

3. lz4
	  -Xhc
		Compress using LZ4 High Compression

The compression specific options are, obviously, specific to the compressor
in question, and you should read the compressor documentation and check
their web sites to understand their behaviour.

In general the defaults used by Mksquashfs for each compressor are optimised
to give the best performance for each compressor, where what constitutes
best depends on the compressor.  For gzip/xz best means highest compression
(trying multiple filters/strategies can improve compression, but this is
extremely expensive computationally, and hence, not suitable for the defaults),
for LZO/LZ4 best means a tradeoff between compression and (de)-compression
overhead (LZO/LZ4 by definition are intended for weaker processors).

New Mksquashfs options
----------------------

1. -mem <size>

  Set the amount of memory used by Mksquashfs to <size> bytes.  G/M and K
  post-fixes are supported.

  By default Mksquashfs uses 25% of the physical memory.  Increasing
  this with the -mem option can increase performance (note it does not have
  any effect on compression).  Reducing it can prevent thrashing if the
  system is busy and there is not 25% of physical memory free (again, note
  it does not have any effect on compression).

2. -exit-on-error

  By default Mksquashfs treats certain errors as benign, if these
  errors occur Mksquashfs prints the error on the console but continues.
  These errors are typically failure to read a file from the source filesystem.
  This is deliberate, in many cases users prefer Mksquashfs to flag
  the error but continue rather than abort what may be hours of compression.

  But there are times where failure to read any file is considered critical,
  and users (especially in the case of automated scripts where the
  errors output to the console may be missed) prefer Mksquashfs to exit.

  The new -exit-on-error option can be used in this scenario.  This option
  makes Mksquashfs treat all benign errors as fatal.

3. -progress

  By default if -info is specified, the progress bar is disabled as it gets
  in the way.  Occasionally you might want the progress bar enabled whilst
  -info is enabled.  This option forces Mksquashfs to output the progress
  bar when -info is specified.

4. -Xhelp

  Display the usage text for the currently selected compressor.

New Unsquashfs options
----------------------

1. -u[ser-xattrs]

  Only write user xattrs.  This forces Unsquashfs to ignore system xattrs.
  This is useful when Unsquashing a filesystem as a non-root user, and the
  filesystem contains system xattrs which are only writable by root.

Major bugs fixed
----------------

1. If Mksquashfs ran out of space in the destination filesystem, this
   would not cause Mksquashfs to immediately abort, and Mksquashfs would
   continue to process the source filesystem.  Mksquashfs now immediately
   aborts on out of space in the destination filesystem.

2. Unsquashfs ignored the maximum number of open files limit, and if that
   was lower than the default limit for Linux, it would run out of file
   descriptors.  Unsquashfs now limits the number of open files to the
   limit currently in force (e.g. specified by setrlimit).

3. If huge numbers of dynamic pseudo files were specified, Mksquashfs
   could exceed the maximum number of open files limit.  This was because
   Mksquashfs created all the dynamic file processes up front before
   commencing source filesystem reading and compression.  Mksquashfs
   now creates the dynamic file processes on demand whilst reading
   and compressing the source filesystem, thus limiting the number of
   dynamic pseudo file processes in existence at any one time.

4. When outputting Unsquashfs used to set the permissions of directories
   as it recursively descended.  This in hindsight had an obvious oversight,
   if a directory had only read permission (or was otherwise restricted), then
   Unsquashfs would fail to write its contents when descending into it.  Fixed
   by setting directory permissions as Unsquashfs recursively unwinds.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/