A few words on lzip compressor

(update on 2016-01-28: please note that this post is outdated and no longer adequately expresses my attitude towards the two competing formats)

Some of you may already have noticed that sys-apps/ed and sys-fs/ddrescue packages started pulling in lzip archiver. «Is this some new fancy archiver?» you may ask. The answer is «no. It’s been around for a very long time, and it never got any real interest.»

You can read some of the background story in New Options in the World of File Compression Linux Gazette article. Long story short, lzip was created before xz as a response to the limitations of .lzma format used by lzma-utils. However, it never got any real attention and when xz-utils was released as a direct successor to lzma-utils it became practically redundant. And the two projects co-existed silently until lately…

Over the past five years, Antonio Diaz Diaz, lzip’s author, and a few project supporters were trying to convince the community that the lzip format is superior to xz. However, they were never able to provide any convincing arguments to the community, and while xz gained popularity lzip stayed in the shadow. And it was used mostly by the projects Diaz was member of.

It seems that he has finally decided that advocacy will not help his pet project in gaining popularity. Instead, he decided to take advantage of his administrator position in the mentioned GNU projects and discontinue providing non-.lz tarballs. As he says, «surely every user of ddrescue would like to know about lzip […]».

So, Gentoo user, would you like to know about lzip? Let’s try to get a few fair points here.

First of all, it should be noted that the two competing projects are two different implementations of the same compression algorithm — LZMA2 (correct on 2016-01-28: Antonio Diaz Diaz pointed out that LZMA2 is purely an additional container on top of LZMA; so: lzip is container on top of LZMA, while xz is container on top of LZMA2 which is a container on top of LZMA), and they use incompatible file formats. Therefore, both can achieve the same compression ratio (and similar speed) but using either of them requires users to download the appropriate tool.

xz is gaining traction lately. The initial doubt period seems to be over, and growing number of projects is adapting the format. This includes both use of the compression library and distribution of sources in .tar.xz. An xz coder is implemented within the kernel and it can be used as a compressor for filesystems. Practically saying, it’s inevitable for Gentoo users since it is used to compress some of the larger sources (like kernel sources).

Lzip is a side project with minor interest. Most of our users were unaware of its existence until they were forced to install it in order to unpack some other package. In fact, since it is used in particularly small packages, the gain from using it (compared to gzip) is even smaller than size of lzip tarball. Not to mention if we compare it to xz that most of our users have installed already.

As analyzed by Ohloh, xz-utils «has a well established, mature codebase» though it is mostly maintained by a single person. lzip is developed by a single person with no officially known contributors, and no public source code repository. The author claims that lzip is mature and will be continually developed but those are only promises, compared to the community growing around xz.

Feature-wise, xz supports more fine-grained configuration of LZMA2 and additional filters (alike 7-Zip). Lzip has a recovery tool and a few extra promises.

xz-utils are written in C, with the compression library and basic utilities being public-domain. Lzip is C++, and GPLv3 (but there’s also a limited public-domain C version). xz-utils use clean autotools, lzip — custom configure script and Makefile.

To sum up: both tools are quite similar and have no strong advantages over the other. However, the popularity of xz makes it a better choice most of the time, while using lzip mostly forces users to install an extra tool for no real benefit. The continuous support and development in xz-utils is guaranteed by the community, while in lzip it’s just author’s promise.

This post would end here if not for the late events. Now lzip gained an important disadvantage: its author is simply unprofessional. While many other projects start shipping .xz compressed tarballs following its rise in popularity, Diaz is grasping at straws and abusing his position trying to force people to use his pet project instead. He has clearly more concern about the popularity of lzip than about the friendliness to users of the other projects he is administering.

8 thoughts on “A few words on lzip compressor”

  1. lzip licencing (GPL v3) is a very strong argument in favor of lzip (for any software-freedom-concerned user).

  2. xz-5.1 has threaded de-/compression, which will boost its popularity even more, so the “battle” will be over soon.

    1. Honestly, it seems no different than pxz, and seems to have the same poor defaults. More specifically, it seems that xz splits the input into uneven blocks, leaving most of the compression to the first thread.

      Of course, this can be solved by using a different block size. But it’s no step forward from what we had a few years back…

      1. My point was more along the lines that, if the default tool (xz) does parallel de-/compression by default, the xz format will become even more popular.

    2. there’s plzip as part of the lzip family since quite a while. on debian, it can be used transparenly.

  3. I was very surprised that there isn’t something in the GNU project guidelines that prevent this. I thought for sure there was a requirement that projects release a tar.gz (or whatever the bare minimum is these days) along with whatever other options they provide.

  4. It would be interesting to know your current opinion on this subject. Are all the advantages listed by Diaz real (protection against corruption, recovery tools, and so on…) ?

    1. It’s a good compressor. I like the simplicity of the file format, compared to the bloat found in xz (I’m using a “minimal” version of xz-utils on Gentoo these days, and occasionally seeing problems because random test suites expect more filters). Sometimes it also yields marginally better compression than xz. Perversely, new versions of xz-utils support unpacking .lz archives, so eventually you would be able to use it without worrying that people won’t be able to unpack it.

Leave a Reply

Your email address will not be published.