[LINUX TECH] Experience with btrfs?

2014-05-28 19:08:22 UTC | /dev/null

[re_nortex]

What piqued my interest in btrfs was the article, Bitrot and atomic COWs: Inside “next-gen” filesystems. The "graphic gone wrong" with the JPG shown in the article is a vivid example of a flipped bit.

I encountered such a thing a few years ago while still using the ext3 filesystem and an IDE (remember those?) drive. Here's what happened. I extracted a largish tarball to build QT-4 from source. All was going well until the compile failed on a C++ file since it yelped about a missing header. I examined the source file and saw this:

#include <uniste.h>

Yes, one bit was flipped (should be unistd.h, of course). I edited the errant file and happily finished the compile. Curious as to what went amiss, I re-extracted the tarball into a partition on another drive (also IDE but on a different controller) and it was correct. That was readily visible and I caught it but I can't help but wonder how much silent data corruption has gone on through the decades on my various systems.

[Bloody Sam Roberts]

Fine print: Or at least on my own machines. On customer systems, I'm a coward. :)

Ha! I know what you mean.

[ShadowAce]

I don’t have any handy. It was an option on Fedora 20.

[zeugma]

If it’s just your local system and a couple TB, ext4 still fits a good mold.

That was my first thought. Zfs is freaking cool because of the huge sizes it can support. If you're going to have an exobyte of storage, zfs is a great option. I haven't really read enough about btrfs to get a good feel for what it's purpose is. For me, at least until I get to 100TB, I figure ext4 will work fine. I haven't have any problem with it at all. I'm not seeing the bit rot people are talking about.

It seems to work well, and I don't reboot often enough to have even hit a forced check through my file system.

 

[cynwoody]

Yes, one bit was flipped (should be unistd.h, of course). I edited the errant file and happily finished the compile. Curious as to what went amiss, I re-extracted the tarball into a partition on another drive (also IDE but on a different controller) and it was correct. That was readily visible and I caught it but I can't help but wonder how much silent data corruption has gone on through the decades on my various systems.

I got to wondering how noticeable flipped bits in binaries might be. So, I wrote a little Ruby program to flip a random bit in the cat command and see if the corrupted binary still runs:

#!/usr/bin/env ruby
require 'tmpdir'
require 'open3'
$outPath = "#{Dir.tmpdir}/catmandu"
$data = File.new(`which cat`.chomp, "rb").read
$bits = $data.length * 8
$me = IO.read __FILE__
puts "The cat command is #{$bits} bits and #{$data.length} bytes long."

def flipBit
  byte, bit = (rand*$bits).to_i.divmod 8
  d = $data.clone
  d[byte] = (d[byte].ord ^ (1<<bit)).chr
  File.open($outPath, 'wb') do |f|
    f.write d
    f.chmod 0700
  end
  return byte, bit
end

def runTest trial
  byte, bit = flipBit
  begin
    check, err, status = Open3.capture3("#{$outPath} #{__FILE__}")
  rescue => e
  end
  if check == $me && status.to_i == 0 && e == nil
    result = 'pass'
    trouble = ''
  else
    result = 'fail'
    err = e if e
    err = err.to_s.force_encoding("ASCII-8bit").tr("\n", "/")[0, 60]
    trouble = " status = #{status}: #{err}"
  end
  puts "#{'%4s' % trial}: #{result} : byte #{byte} #{bit}#{trouble}"
  result
end

if __FILE__ == $0
  trials = (ARGV[0] || 50).to_i
  failures = 0
  trials.times do |trial|
    failures += 1 if runTest(trial) == 'fail'
  end
  puts "#{failures} out of #{trials} failed."
end

I found a much lower error rate than I expected. In one run of 1000 trials, it failed only 38 times. There was a little excitement, however. About half-way through, one of the failures took the form of an output loop, requiring me to kill the corrupted child process to allow the program to complete. Most of the failures are stuff like seg faults and load errors due to symbols not found or faulty binary file format.

I have a hunch bit rot in source code and script files probably shows up more readily than in most binaries. Although a flipped bit in a comment will probably not cause a failure, a similar error elsewhere is likely to cause a syntax error or some sort of not-found condition, as in your example.

[re_nortex]

I wrote a little Ruby program to flip a random bit in the cat command and see if the corrupted binary still runs...

Great stuff and very nice, readable code (as expected from the denizens of FReeperdom). Here's the tail end of one my runs, just for grins:

2494: fail : byte 6880 7 status = pid 17827 SIGSEGV (signal 11): 
2495: pass : byte 16316 0
2496: pass : byte 35390 7
2497: pass : byte 32143 6
2498: fail : byte 4651 1 status = pid 17843 SIGSEGV (signal 11): 
2499: pass : byte 1744 2
174 out of 2500 failed.

And yes, there were some runs where it wedged and needed a surekill. I ain't gonna try it on the kernel though!

My experience with the flipped bit in QT goes back further than I thought. It was March 2006 with QT-4.1.1 and I even documented the drive I was using at the time: a Maxtor 6Y060L0. And the kernel was 2.16.15.2 on a FIC FB11 mainboard. How's that for being anal? :) Considering how long ago it was and the creaky old hardware, just about anything may have caused that one-time-only bitflip.

Seriesly, whenever something goes amiss, I keep a log of such things just for archival purposes so that if an oddity recurs, it won't be such a surprise the second time around.