The devil is always in the details.  Software systems can be quite complex and assumptions about how they work can get you in trouble.  On my previously mentioned TV server, this week, I noticed some nearly unwatchable shows.  My initial thought was that I needed to realign my antenna, but I noticed that the shows were all recorded at the same time and I was watching another.  Maybe they were bandwidth starved?  The worst stutters were during scenes with lots of motion.  Now, in my previous setup, I know I was not able to watch anything while recording 4 HD shows.  At least I could record 4 HD show simultaneously.  The old setup was only one disk, though.  Now I’m running a one disk ZFS RAIDZ.  I absolutely know RAID5 type setups don’t perform like RAID0 setups (and that RAIDZ isn’t exactly RAID5), but early generalizations I’d read lead me to expect RAID5 type performance.  I didn’t investigate further and made assumptions based on my understanding of the technology.  Boy was I wrong.  Take a look at the numbers from bonnie++:

Version  1.96       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
dagobah.boonthe 16G    96  99 131609  34 104142  31   277  99 271779  41 116.4  14
Latency               253ms    7136ms    7453ms   36211us     731ms     785ms
Version  1.96       ------Sequential Create------ --------Random Create--------
dagobah.boontheekul -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
 files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
 16 22615  96 +++++ +++ 17543  97 20137  91 +++++ +++ 18455  98
Latency             12940us    9652us     227us   18233us     153us     377us

Those are crazy numbers.  So, I turned to Google and did some more reading.  It turns out a RAIDZ’s write performance is much worse than I expected.  This quote (well, quote of a quote) is very good:

"Now we come to the crucial decision ZFS has made for raidz and
raidz2: in raidz and raidz2, the data block is striped across all of
the disks. Instead of a model where a parity stripe is a bunch of data
blocks, each with an independent checksum, ZFS stripes a single data
block (and its parity), with a single checksum, across all the disks
(or as many of them as necessary).

This is a rational implementation decision, but when combined with the
need to verify checksums, it has an important consequence: in ZFS,
reads always involve all disks, because ZFS always must verify the
data block's checksum, which requires reading all of the data block,
which is spread across all of the drives. This is unlike normal RAID-5
or RAID-6, in which a small enough read will only touch one drive, and
means that adding more disks to a ZFS raidz pool does not increase how
many random reads you can do per second.

(A normal RAID-5 or RAID-6 array has a (theoretical) random read IO
capacity equal to the sum of the random IO operations rate of each of
the disks in the array, and so adding another disk adds its IOPs per
second to your read capacity. A ZFS raidz or raidz2 pool instead has a
capacity equal to the slowest disk's IOPs per second, and adding
another disk does nothing to help. Effectively a raidz ZFS gives you a
single disk's read IOPs per second rate.)"

This was on a blog of a SUN engineer (although a post from a few years
ago), unfortunately I don't have the link, I actually had to go
through my posting history on the Ars Technica forum to even find this
quote in the first place. If the situation has changed and the above
quote no longer holds true, it would be nice if someone more
knowledgeable on the performance implications could elaborate what
kind of performance is to be expected on a raidz system  

- Sincerely,
Dan Naumov

Wow.  In that same thread and in another I found, someone posted some benchmarking results they had done.  They are very interesting.  Follow these links:

http://virtual.tehinterweb.net/livejournal/2009-06-22_zfs_diskperf/zfs-diskperf-contig-write.png

http://virtual.tehinterweb.net/livejournal/2009-06-22_zfs_diskperf/zfs-diskperf-5MB-readwrite.png

http://virtual.tehinterweb.net/livejournal/2009-06-22_zfs_diskperf/zfs-diskperf-1MB-readwrite.png

I was quite surprised.  Further reading has lead me to rethink my setup.  For this TV/media server I need the ability to read and write simultaneously at high rates.  Write speed is more important in that I’ll likely be recording more shows at any given time than I’m watching, but I’ll still need to be able to stream a couple of HD shows at the same time.  Heck, my 4 tuners haven’t been enough on a couple of occasions.  So, I’m going to have to sacrifice space for speed.  I don’t know how I’m going to do the data shuffle, but I’m considering picking up another pair of 1TB drives and a PCI-Express SATA controller (only one free one left on the mainboard).  That’ll help.  Then I guess I’ll build the pool from mirrored pairs of 1TB drives.  My read/write performance should improve, but the addition of 2TB more drives won’t give me any more space.  It will be interesting to see what kind of numbers I get out of it.

Update: I reconfigured the pool from a 4 disk RAIDZ to a pool of 2 2 disk mirrors.  Bonnie++ results:

Version  1.96       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
dagobah.boonthe 16G    89  99 95149  25 87337  23   289  99 270826  31 214.8  28
Latency               354ms   11106ms   10391ms   40391us    3882ms     464ms
Version  1.96       ------Sequential Create------ --------Random Create--------
dagobah.boontheekul -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
 files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
 16 20303  95 +++++ +++  8108  99 19027  97 11837  99  4149  99
Latency             13531us     265us     478us   39328us     308us    5922us

Update: I got in 2 more 1TB disks (and a HighPoint 2310, as I’ve used up all 6 on-board SATA ports).  I added the 2 new disks (via the HighPoint, no other rearranging of the drives) as another mirror in the pool.  Now the capacity is back up to what it was as a RAIDZ.  Bonnie++ results:

Version  1.96       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
dagobah.boonthe 16G   112  99 145807  39 124058  33   289  99 372925  46 274.9  22
Latency               331ms    3891ms    7095ms   31678us    2229ms     457ms
Version  1.96       ------Sequential Create------ --------Random Create--------
dagobah.boontheekul -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
 files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
 16 22492  94 +++++ +++ 19562  97 16765  97 18918  99  6837  99
Latency             18093us    1113us     185us   37416us     203us     386us

It’s hard to make any conclusions from this.  The numbers for the pool of mirrors are completely different than what I expected.  In addition, I can say that interactive performance with similar workloads to what caused me problems previously have much improved.  Just from the reconfiguration of the existing drives, I was able to record 4 HD streams simultaneously while watching another and no apparent stuttering or the like in any recording or the playback.  Adding the 2 additional disks as another mirror in the pool had apparent impact in the bonnie++ numbers and brought my usable space back up to previous levels, but I’m still rather surprised at the bonnie++ numbers.

Today (well, technically yesterday), I got my first dose of nitro therapy (went to the range) in almost 3 years.  I can’t believe it’s been so long.  You know you’re getting old when 3 years goes by and it doesn’t feel that long.

I’m on the hunt for a new carry gun and I’m highly focused on .45acp as being my new round round of choice.  That certainly narrows the field of choices quite a bit and my natural first interest is the venerable 1911.  With 100 years of history, there are a lot of options.  Through a lot of reading and question asking of my local expert I started narrowing down options I thought I might prefer, but I was missing a key component in my evaluation — I’d never fired a 1911 before, or even a .45.  So, today we went up to Range USA to rent one.  I brought along a trusty old Makarov to get back into the shooting groove.  I was certainly a little rusty, but I quickly re-acclimated to the Makarov.  Range USA had a couple of 1911’s and I chose the Kimber Stainless Pro Carry II (with Crimson Trace grips, although I didn’t realize it before making the pick).  It’s a closer match to the options and size I’m considering than the full size Springfield they also had.

What did I think?  Well, I’m not sure.  Recoil and muzzle flip weren’t really a problem, but I certainly shot different with it than the Makarov.  It alleviated my concerns about an aluminum frame.  Reliability wasn’t a problem, but I wouldn’t think an unreliable gun is good for a rental anyway.  The sites were horrible — not that the sites on my Makarov are any better.  Actually, before coming out here to Memphis, I had no idea that the plain sites on my Makarov were that bad (although I’d read it many times on forums).  They seemed fine to me, but back in Oklahoma I always shot outdoors and during the day.  Indoors with poor lighting is a very different thing and makes a huge difference.  The bottom line is that I’ll still be pursuing the 1911 and now I’ve got more of an idea what I’m getting into.

I knew it, but I guess I didn’t really know it. I do now — tires really make a difference. A week ago, I bought a new set of Michelin Primacy MXV4’s for my car. If I recall, this is the 3rd set of tires I’ve put on in the 100,000 miles I’ve put on it (in 10 years). I’ve needed them for a while, but I kept putting it off, because I knew they’d be a special order. Nobody stocks the size tires I need. As this October was the wettest October in Memphis history (at least since they started record keeping in the 1800’s), I finally got sick of slipping and sliding like I was on ice and called Sears up to get them to order the tires. They came in after a few days and I got them put on over lunch on Friday a week ago. The difference is dramatically different. The road noise, ride, and grip have all improved more than I expected. The most impressive change is in road noise. I knew my car was loud, but I just assumed it was because I’d pulled out the headliner and that it’s 20 or 21 years old. Sure, I realized there was some tire noise, but I didn’t think it was such a large contributor. Now, it’s like I’m driving a new car.

11/8/2009 – Slight update: Gas mileage increased as well. With one tank behind me I’d say 5%.

11/11/2009 – Further update, I would have been in an accident yesterday if it weren’t for the new tires.  Cruising home from work and suddenly everything stopped.  Lots of cars sliding and pulling onto the shoulder.  I absolutely know the old tires would have let go, but I didn’t even get a chirp with these.

As I’ve mentioned, I’ve been very interested in ZFS lately. Well this week, the hypothetical happened to me and ZFS was there to make it a non-event. I was checking up on my newest ZFS’d filesystem (a pair of 1TB drives in a ZFS mirror). I ran ‘zpool status’ to see how the health of the pool was and it reported that one drive had had a checksum error. I’ve been processing gigs of log files from it, so one block error isn’t much of a pattern. I ran a ‘zpool scrub’ to check everything out (it was clean), and I’ll certainly watch that drive a little closer. ZFS made it simple and seamless.

I’ve been doing a hard drive shuffle in an attempt to utilize ZFS on FreeBSD some more in one set of my systems.  Well, I decided that I wanted a direct dedicated network between one pair of boxes, but the catch is that there are no more expansion slots available to plug in another network card.  That got me looking at the onboard FireWire port.  I had read about the capability of runnning IP over FireWire and so I looked into it some more.  Things looked good, so I set forth with that as my plan — I’d run a FireWire network between the two machines.  Well today I got the boxes together, plugged-in the FireWire cable and configured the virtual fwip interface on both.  What’s the typical first test? ping.  BAM!  Immediate kernel panic on the pinger (running FreeBSD 8.0-RC1 amd64).  Ugh, what did I do wrong?  My on-the-fly plan B was to try the fwe interface instead.  It’s not as recommended, as it doesn’t implement a standard like the fwip interface does, but both boxes are going to be FreeBSD anyway, why not try it?  It didn’t have any problems.  I need to do some more network speed testing, but so far so good.  I’d really like to be able to up the MTU, as I plan to use the interface primarily for NFS, but it doesn’t look like it’s possible.  Tonight I did some digging to see if anyone else has encountered the same problem with the fwip interface and I found a bug report:

http://www.freebsd.org/cgi/query-pr.cgi?pr=139162

Well, at least it wasn’t me.  It’s the only reference I’ve found to this problem, so it’s probably low-priority for 8.0-RELEASE, but who knows.  I’m just thankful that there is a viable work-around.

So, I did the unpredictable, impossible thing — I cut my cable TV.  Yes.  I did it.  You’d think I’d be in withdrawal.  Well, the fates are working against me.  The ones in Hollywood anyway.  They decided to put some good shows on the regular networks (all made possible by OTA HD).  A lot of them.  It’s ridiculous.  I have more shows set to record than I ever have.  It’s crazy.  Anyway, saving $600 a year on TV services is a great bonus.  My top new show: Modern Family.  Although, I haven’t gotten through all the new shows yet.

Recently, I started moving from BeyondTV to SageTV for my PVR needs.  Maybe I’ll hash out all the reasons in a future post, but in the end it leaves me with my old BeyondTV “server” hardware free to move on to other purposes.  As such, I’m building out FreeBSD on it for it’s next task.  I decided to build out with gjournal from the start.  I’ve build a few boxes with gjournal after the fact and I hated having the extra slices.  Besides, I quickly run out of available slices.  So this time I wanted to build it out from scratch with gjournal such that I could have the journal in the same filesystem AND I decided to go with GPT as it’s the next big thing.  Well, to accomplish this I used Fix-it mode on the installer DVD and for the first time in a dozen years of FreeBSD usage, I installed without sysinstall.  I’d never contemplated not using sysinstall, but having done some reading on ZFSroot installations people were doing I used some similar procedures.  I must admit, I was surprised how simple it was and there’s a good bet this is going to be the way I do installs from now on (until gjournal and/or ZFS are available in sysinstall).  It worked beautifully and in a single attempt.  Disks are just too big now for fsck.  I lived through one 800GB UFS2 filesystem having to be fsck’d when someone decided to turn off a system on me and it’s just way too slow (and I was lucky to have enough RAM).  I’ve got probably 9TB of hard drive space now (6TB in 1TB drives; 3 of them in 1 system) and there’s now way I could fsck many of them if something happened.  Gjournal and/or ZFS are the answers in FreeBSD land.  Manual install is the way to get them going from the beginning.

We had a pleasant surprise in the form of a phone call last week.  It was our bank calling to let us know we qualify for a special refinancing program.  Immediately it sounds like a scam, but the account manager said he’d get a package sent out to us to review.  Sure enough, a few days later we got a package in the mail that very much appears to be from our bank.  I wasn’t the one that talked to the agent, so I gave them a call back later to try and find out more.  With the interest rate drops that happened over the last year I was chomping at the bit to try a refinance to get our rate lowered, but I absolutely knew the value of our house had dropped too much.  You can walk into the model home here in the neighborhood and ask them to build a house 500 square foot larger than ours for $30k less than we paid for this one two years ago.  We’re upside down on the mortgage.  So, I just put out of my mind the option of refinancing.  Well, it turns out (I think I read everything we have from the bank and I don’t recall this in any disclosure) that our loan is only serviced by our bank.  The money actually came from Freddie Mac.  As Freddie Mac has been bailed out by the government, they’ve required some programs be started to shore up their mortgage base.  We qualify for one of these programs.  It’s designed for people like us that are upside down on the mortgage to get a lower rate.  So, if everything goes through we’ll get a rate almost a point lower.  It’s not the 4.5% that would have been possible when the rates really dropped, but every bit helps.  We weren’t having any problems paying before, so we’ll be able to pay it off a few years earlier.  Yay.  Well, I shouldn’t count the chickens before they hatch, but so far so good.

Well, since my last post on the subject, more homes have sprung up and several more are on the to-be-built list.  Despite the horrible economy  homes are selling here.  I can’t say it’s better for the entire Memphis market, but at least here we seem to have the critical mass.  On my street alone, there are six new houses.  Three are complete, and three are still under construction.  Two are already occupied.  One was sold, but I guess something happened.  I met a nice lady yesterday that said she’s buying it (I hope it works out).  Another, directly across the street from us, now has a for sale sign on it even before it’s completed.  There are two more lots listed as sold and slated for construction a bit further down the street and three others throughout the neighborhood.  At this rate we might have a complete neighborhood in year or so.  I’ve seen quite a bit of what I assume is real estate traffic coming through, and hopefully sales follow.

So, I finally got an iPhone. I know it’s a few years behind the time, but the 3GS and 3.0 OS finally made it seem like a complete product. Also, the 32GB model seemed like it was the first to really be big enough as a convergence device. Additionally, the App Store “ecosystem” looked like it had reached the size needed to sustain itself. So, I dumped my no-longer-on-a-contract T-mobile account over to AT&T. So far so good. It’s nice to have a reasonable Internet browser all the time. I’m an NPR addict, so having instant access to their podcasts is awesome. Weather radar in your hand is amazing. I’ve got enough space to hold an enormous number of apps, music, podcasts, pictures, movies, and TV shows to remain computerless for a good chunk of time. I’ve even managed to connect to my office VPN, but I haven’t bought an SSH app to do my normal thing with yet (I live on a terminal). I even found this blog app, so maybe I will get back to posting again.