So we got a good deal on a 65 inch plasma at hhgregg.  They included a Blu-ray player as part of the package.  I’d thought to myself that someday when I got a Blu-ray player it’d probably be a PS3, but I guess not.  I ran and picked up the BBC’s Planet Earth box set to have some content.  I’d seen it on Discovery when it was first released and was amazed.  I knew I had to have it in HD someday.  We’ve now rented several movies on Blu-ray and my verdict is that it can be quite amazing (like Planet Earth), but mostly not so much.  This has to be a big part of the low adoption rate.  I’ve got to rent some more  to see if I’ve just made bad picks, but so far — I’m not that impressed.

We rented Transformers: Revenge of the Fallen on iTunes last night.  Two words: pure crap.  I can’t believe they put something this bad out and that it made as much money as it did.  Ridiculous.

The devil is always in the details.  Software systems can be quite complex and assumptions about how they work can get you in trouble.  On my previously mentioned TV server, this week, I noticed some nearly unwatchable shows.  My initial thought was that I needed to realign my antenna, but I noticed that the shows were all recorded at the same time and I was watching another.  Maybe they were bandwidth starved?  The worst stutters were during scenes with lots of motion.  Now, in my previous setup, I know I was not able to watch anything while recording 4 HD shows.  At least I could record 4 HD show simultaneously.  The old setup was only one disk, though.  Now I’m running a one disk ZFS RAIDZ.  I absolutely know RAID5 type setups don’t perform like RAID0 setups (and that RAIDZ isn’t exactly RAID5), but early generalizations I’d read lead me to expect RAID5 type performance.  I didn’t investigate further and made assumptions based on my understanding of the technology.  Boy was I wrong.  Take a look at the numbers from bonnie++:

Version  1.96       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
dagobah.boonthe 16G    96  99 131609  34 104142  31   277  99 271779  41 116.4  14
Latency               253ms    7136ms    7453ms   36211us     731ms     785ms
Version  1.96       ------Sequential Create------ --------Random Create--------
dagobah.boontheekul -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
 files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
 16 22615  96 +++++ +++ 17543  97 20137  91 +++++ +++ 18455  98
Latency             12940us    9652us     227us   18233us     153us     377us

Those are crazy numbers.  So, I turned to Google and did some more reading.  It turns out a RAIDZ’s write performance is much worse than I expected.  This quote (well, quote of a quote) is very good:

"Now we come to the crucial decision ZFS has made for raidz and
raidz2: in raidz and raidz2, the data block is striped across all of
the disks. Instead of a model where a parity stripe is a bunch of data
blocks, each with an independent checksum, ZFS stripes a single data
block (and its parity), with a single checksum, across all the disks
(or as many of them as necessary).

This is a rational implementation decision, but when combined with the
need to verify checksums, it has an important consequence: in ZFS,
reads always involve all disks, because ZFS always must verify the
data block's checksum, which requires reading all of the data block,
which is spread across all of the drives. This is unlike normal RAID-5
or RAID-6, in which a small enough read will only touch one drive, and
means that adding more disks to a ZFS raidz pool does not increase how
many random reads you can do per second.

(A normal RAID-5 or RAID-6 array has a (theoretical) random read IO
capacity equal to the sum of the random IO operations rate of each of
the disks in the array, and so adding another disk adds its IOPs per
second to your read capacity. A ZFS raidz or raidz2 pool instead has a
capacity equal to the slowest disk's IOPs per second, and adding
another disk does nothing to help. Effectively a raidz ZFS gives you a
single disk's read IOPs per second rate.)"

This was on a blog of a SUN engineer (although a post from a few years
ago), unfortunately I don't have the link, I actually had to go
through my posting history on the Ars Technica forum to even find this
quote in the first place. If the situation has changed and the above
quote no longer holds true, it would be nice if someone more
knowledgeable on the performance implications could elaborate what
kind of performance is to be expected on a raidz system  

- Sincerely,
Dan Naumov

Wow.  In that same thread and in another I found, someone posted some benchmarking results they had done.  They are very interesting.  Follow these links:

http://virtual.tehinterweb.net/livejournal/2009-06-22_zfs_diskperf/zfs-diskperf-contig-write.png

http://virtual.tehinterweb.net/livejournal/2009-06-22_zfs_diskperf/zfs-diskperf-5MB-readwrite.png

http://virtual.tehinterweb.net/livejournal/2009-06-22_zfs_diskperf/zfs-diskperf-1MB-readwrite.png

I was quite surprised.  Further reading has lead me to rethink my setup.  For this TV/media server I need the ability to read and write simultaneously at high rates.  Write speed is more important in that I’ll likely be recording more shows at any given time than I’m watching, but I’ll still need to be able to stream a couple of HD shows at the same time.  Heck, my 4 tuners haven’t been enough on a couple of occasions.  So, I’m going to have to sacrifice space for speed.  I don’t know how I’m going to do the data shuffle, but I’m considering picking up another pair of 1TB drives and a PCI-Express SATA controller (only one free one left on the mainboard).  That’ll help.  Then I guess I’ll build the pool from mirrored pairs of 1TB drives.  My read/write performance should improve, but the addition of 2TB more drives won’t give me any more space.  It will be interesting to see what kind of numbers I get out of it.

Update: I reconfigured the pool from a 4 disk RAIDZ to a pool of 2 2 disk mirrors.  Bonnie++ results:

Version  1.96       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
dagobah.boonthe 16G    89  99 95149  25 87337  23   289  99 270826  31 214.8  28
Latency               354ms   11106ms   10391ms   40391us    3882ms     464ms
Version  1.96       ------Sequential Create------ --------Random Create--------
dagobah.boontheekul -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
 files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
 16 20303  95 +++++ +++  8108  99 19027  97 11837  99  4149  99
Latency             13531us     265us     478us   39328us     308us    5922us

Update: I got in 2 more 1TB disks (and a HighPoint 2310, as I’ve used up all 6 on-board SATA ports).  I added the 2 new disks (via the HighPoint, no other rearranging of the drives) as another mirror in the pool.  Now the capacity is back up to what it was as a RAIDZ.  Bonnie++ results:

Version  1.96       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
dagobah.boonthe 16G   112  99 145807  39 124058  33   289  99 372925  46 274.9  22
Latency               331ms    3891ms    7095ms   31678us    2229ms     457ms
Version  1.96       ------Sequential Create------ --------Random Create--------
dagobah.boontheekul -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
 files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
 16 22492  94 +++++ +++ 19562  97 16765  97 18918  99  6837  99
Latency             18093us    1113us     185us   37416us     203us     386us

It’s hard to make any conclusions from this.  The numbers for the pool of mirrors are completely different than what I expected.  In addition, I can say that interactive performance with similar workloads to what caused me problems previously have much improved.  Just from the reconfiguration of the existing drives, I was able to record 4 HD streams simultaneously while watching another and no apparent stuttering or the like in any recording or the playback.  Adding the 2 additional disks as another mirror in the pool had apparent impact in the bonnie++ numbers and brought my usable space back up to previous levels, but I’m still rather surprised at the bonnie++ numbers.

As I’ve mentioned, I’ve been very interested in ZFS lately. Well this week, the hypothetical happened to me and ZFS was there to make it a non-event. I was checking up on my newest ZFS’d filesystem (a pair of 1TB drives in a ZFS mirror). I ran ‘zpool status’ to see how the health of the pool was and it reported that one drive had had a checksum error. I’ve been processing gigs of log files from it, so one block error isn’t much of a pattern. I ran a ‘zpool scrub’ to check everything out (it was clean), and I’ll certainly watch that drive a little closer. ZFS made it simple and seamless.

I’ve been doing a hard drive shuffle in an attempt to utilize ZFS on FreeBSD some more in one set of my systems.  Well, I decided that I wanted a direct dedicated network between one pair of boxes, but the catch is that there are no more expansion slots available to plug in another network card.  That got me looking at the onboard FireWire port.  I had read about the capability of runnning IP over FireWire and so I looked into it some more.  Things looked good, so I set forth with that as my plan — I’d run a FireWire network between the two machines.  Well today I got the boxes together, plugged-in the FireWire cable and configured the virtual fwip interface on both.  What’s the typical first test? ping.  BAM!  Immediate kernel panic on the pinger (running FreeBSD 8.0-RC1 amd64).  Ugh, what did I do wrong?  My on-the-fly plan B was to try the fwe interface instead.  It’s not as recommended, as it doesn’t implement a standard like the fwip interface does, but both boxes are going to be FreeBSD anyway, why not try it?  It didn’t have any problems.  I need to do some more network speed testing, but so far so good.  I’d really like to be able to up the MTU, as I plan to use the interface primarily for NFS, but it doesn’t look like it’s possible.  Tonight I did some digging to see if anyone else has encountered the same problem with the fwip interface and I found a bug report:

http://www.freebsd.org/cgi/query-pr.cgi?pr=139162

Well, at least it wasn’t me.  It’s the only reference I’ve found to this problem, so it’s probably low-priority for 8.0-RELEASE, but who knows.  I’m just thankful that there is a viable work-around.

Recently, I started moving from BeyondTV to SageTV for my PVR needs.  Maybe I’ll hash out all the reasons in a future post, but in the end it leaves me with my old BeyondTV “server” hardware free to move on to other purposes.  As such, I’m building out FreeBSD on it for it’s next task.  I decided to build out with gjournal from the start.  I’ve build a few boxes with gjournal after the fact and I hated having the extra slices.  Besides, I quickly run out of available slices.  So this time I wanted to build it out from scratch with gjournal such that I could have the journal in the same filesystem AND I decided to go with GPT as it’s the next big thing.  Well, to accomplish this I used Fix-it mode on the installer DVD and for the first time in a dozen years of FreeBSD usage, I installed without sysinstall.  I’d never contemplated not using sysinstall, but having done some reading on ZFSroot installations people were doing I used some similar procedures.  I must admit, I was surprised how simple it was and there’s a good bet this is going to be the way I do installs from now on (until gjournal and/or ZFS are available in sysinstall).  It worked beautifully and in a single attempt.  Disks are just too big now for fsck.  I lived through one 800GB UFS2 filesystem having to be fsck’d when someone decided to turn off a system on me and it’s just way too slow (and I was lucky to have enough RAM).  I’ve got probably 9TB of hard drive space now (6TB in 1TB drives; 3 of them in 1 system) and there’s now way I could fsck many of them if something happened.  Gjournal and/or ZFS are the answers in FreeBSD land.  Manual install is the way to get them going from the beginning.

So, I finally got an iPhone. I know it’s a few years behind the time, but the 3GS and 3.0 OS finally made it seem like a complete product. Also, the 32GB model seemed like it was the first to really be big enough as a convergence device. Additionally, the App Store “ecosystem” looked like it had reached the size needed to sustain itself. So, I dumped my no-longer-on-a-contract T-mobile account over to AT&T. So far so good. It’s nice to have a reasonable Internet browser all the time. I’m an NPR addict, so having instant access to their podcasts is awesome. Weather radar in your hand is amazing. I’ve got enough space to hold an enormous number of apps, music, podcasts, pictures, movies, and TV shows to remain computerless for a good chunk of time. I’ve even managed to connect to my office VPN, but I haven’t bought an SSH app to do my normal thing with yet (I live on a terminal). I even found this blog app, so maybe I will get back to posting again.

Well, I never got around to finishing the job of cleaning up after the last major software update and yet another has come along. This looked pretty important so I did. I wonder how long it will take for me to figure out what else broke. Hopefully, it didn’t make it impossible for me to track it all down.

As anyone that works with software knows, upgrades don’t go as smoothly as you’d like.  Well, my latest here with the site has been the biggest bugger.  I don’t know how long it’ll take me to sort it out.  I don’t seem to have lost any data (yet), but I have missed a lot of organization that I’ll have to re-do.  That sucks.  So, things will be out of whack for a while…

The latest OpenBSD Hackathon has produced some truly amazing results. Sure, every Calgary based Hackathon has produced something cool or amazing, but this one has resulted in 3 things that are home runs to me:

FFS2 – no, it is not done yet, but it is in a testable state and big disk/partition support has been lacking and this will bring OpenBSD back into the fold for my consideration as a fileserver. I’ve got one of those 750GB disks in my TV, you can imagine how much data I want to be able to store on the network, and I’d prefer it to be logically contiguous.

Bluetooth – I can just imagine a bluetooth access point with all the magic of PF, plus the simplicity of OpenBSD.

PF performance optimizations – the sheer amount of performance increase they’ve achieved in this Hackathon is nothing short of amazing. PF was pretty dang fast before, but now …. wow. Should be able to use lower power boxes for the same throughput in most of the situations I’m imagining.

Big thumbs up!