Josh Saddler

So, a couple days in, and I'm still trying to (re)install Gentoo. More on that in a bit. First, let's talk about speed.

It's hard to tell whether or not my new SSDs are really a speedy improvement over the old software RAID1 array of magnetic HDDs. Normally, a bare-bones commandline system feels much faster than an aging graphical desktop, even on the same hardware.

I notice that compile times are slightly faster, though I've also been using tmpfs for Portage and the usual tmp file locations, so putting it all on RAM will lead to a significant speedup anyway.

Boot times are indeed quite zippy; the longest wait is for my media HDD to finish mounting -- it's on ReiserFS, which is known to have very slow mounts.

Now, let's talk filesystems.

The critical showstopper that's made me reinstall two times (and counting) is ext4. So far, ext4 has completely corrupted a whole drive (/var and /usr/portage) and made the other drive (/ and /boot) almost unbootable.

ext4 has eaten my data, hosed my system, and ruined my life.

No amount of fscking has fixed /var and /usr/portage, both on the second SSD. Did you know that you shouldn't let fsck try to resize broken inodes? apparently the resize behavior is known to be broken in the latest versions. It's known to corrupt filesystems. I didn't know that, either. I'm sorry, but what part of "production-ready" applies to ext4? Yeah, it's a new kid on the block, but it's moved out of the "experimental" status into the kernel.

That does not make it ready for your system. The first and second Gentoo installs largely didn't work because (I think) there might have been an invalid mount option. Or something could not be found. Or a superblock was missing. Or the moon was wrong. #$#^#&@ shitty unintelligible error messages. (Here's a tip, developers: don't put every possible thing that could have gone wrong into an error message, then repeat that message for every different error.)

My mount options seemed to be good after double-checking the manpage and around the internet, including kernel.org. Here was my original fstab, from when I had only one partition for / (no separate /boot):

/dev/sda1  /     ext4  noatime,data=writeback,commit=60,nobarrier  0 1
/dev/sdb1  /var  ext4  noatime,data=writeback,commit=60,nobarrier  0 1
/dev/sdb2  /usr/portage  ext4  noatime,data=writeback,commit=60,nobarrier 0 1

Livin' on the edge here. I figured I wouldn't need a separate /boot partition on my first drive, so I lumped it all into one. I did that back in 2005 and 2006 with no problems, right? Right. The rest of the options were designed to maximize SSD performance.

Unfortunately, I couldn't get the system to boot. Made it past grub, the kernel loaded, but when it came time to mount /, it couldn't mount the filesystem rw. No amount of changing options worked -- adding rw to grub.conf, to the fstab options, nothing.

So I figured it must be my one-partition setup, and wiped my disks. Reinstalled again, this time adding a /boot partition on sda. Same ext4 options for /boot as for the other partitions. Rebooted and . . . nope, same errors. Now I'm also seeing a message about a possible bad option or other variable, which I can only assume was in fstab, thanks to the aforementioned shitty nonspecific error messages.

Hit up Google. Not much help. I again backed off on some of the ext4 options, tried playing with Grub parameters, but got the same results. The filesystems mostly weren't mounting, and when a few of them did, it was all readonly.

Sigh. Time to reinstall again. Set up a similar fstab, but this time I changed an ext4 option for /boot to data=ordered, based on this blog post. Reboot and . . . hey, it works. /boot gets mounted. Nothing else does, but it's a start.

I quickly booted back into the LiveCD, changed the other fstab entries to data=ordered, and reboot again. This time, the system seems to boot just fine . . . until it tries to mount /var and /usr/portage from the second SSD. *bzzt*, these cannot be mounted! Something's gone wrong. One more reboot, just for luck, then . . . *bzzt*, now there are filesystem errors! Fsck wants to fix them, so I let it run. Except it completely hoses both partitions. They seem to be so badly scrambled that even running mkfs.ext4 on them from the liveCD results in errors, some of which seem to be emitted from the libata system, which makes me wonder if now the SSD itself has also been corrupted.

I'll have to completely reinitialize and repartition that disk, now. Thanks, ext4. Thanks for hosing my data. Up yours, ext4.

I'm done trying to figure out why ext4 doesn't work. I don't care that it's supposed to be a fast file system for SSDs. I don't care that it's 40 times faster than ReiserFS to mount at bootup. I don't care. ext4 has lost my data three times now. I think my fingers are sufficiently burned to know that "the oven is hot; don't touch."

Up yours, ext4. I'm going back to ReiserFS. At least it works. It's never failed me in more than four years.

Update: On top of the initial ext4 errors, fsck problems, and mount issues, the Mobi drive was also going bad. Now the motherboard BIOS can't see it, regardless of which SATA port or cable I plug in. So just a day or so after trying out the device, when it was initially working for the first install (though the filesystem was throwing ext4 errors, at least /var and /usr/portage worked okay), and it finally finished failing. F***. I contacted the seller to request an RMA; I have a feeling that I'll end up having to go through the manufacturer, which will take a long time. Meanwhile, I'm without a workstation for an indefinite time, so I've set my devaway on dev.gentoo.org. I did find a couple other reports on the internets that say that their Mobis also died shortly after they arrived, so maybe there was a batch of bad drives.

But don't get me wrong, the Mobi drive dying doesn't absolve ext4 of any guilt. The ext4 filesystem still completely f**ked itself repeatedly on the system drive, the UltraDrive ME. It still refuses to do what it's told to do. But rather than continue to investigate related LaunchPad bugs on mounting ext4 rw and fsck errors, I'm going to move back to ReiserFS for the UltraDrive, and just live with longer boots. The RMA process will take awhile, so I may have to reinstall everything on a single drive and just avoid syncing Portage for awhile.

On a good note, OCZ (the company that makes the Vertex, an identical drive with an Indilinx controller), has been experimenting with a homegrown beta firmware that lets the drive do online garbage collection in the background. This is important for keeping the performance of the drive as fresh as when it was first used, even after it gets filled up with files and repeated (re)writes. The firmware is still in testing, but I'm hopeful that it'll make it out the door soon. Hopefully the same firmware features will find their way to my Super Talent drive -- and hopefully the TRIM command will also be implemented in the firmware.

Of course, the only Linux filesystem I know of that supports TRIM is ext4 . . .