We rented Transformers: Revenge of the Fallen on iTunes last night. Two words: pure crap. I can’t believe they put something this bad out and that it made as much money as it did. Ridiculous.
The devil is always in the details. Software systems can be quite complex and assumptions about how they work can get you in trouble. On my previously mentioned TV server, this week, I noticed some nearly unwatchable shows. My initial thought was that I needed to realign my antenna, but I noticed that the shows were all recorded at the same time and I was watching another. Maybe they were bandwidth starved? The worst stutters were during scenes with lots of motion. Now, in my previous setup, I know I was not able to watch anything while recording 4 HD shows. At least I could record 4 HD show simultaneously. The old setup was only one disk, though. Now I’m running a one disk ZFS RAIDZ. I absolutely know RAID5 type setups don’t perform like RAID0 setups (and that RAIDZ isn’t exactly RAID5), but early generalizations I’d read lead me to expect RAID5 type performance. I didn’t investigate further and made assumptions based on my understanding of the technology. Boy was I wrong. Take a look at the numbers from bonnie++:
Version 1.96 ------Sequential Output------ --Sequential Input- --Random- Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP dagobah.boonthe 16G 96 99 131609 34 104142 31 277 99 271779 41 116.4 14 Latency 253ms 7136ms 7453ms 36211us 731ms 785ms Version 1.96 ------Sequential Create------ --------Random Create-------- dagobah.boontheekul -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 22615 96 +++++ +++ 17543 97 20137 91 +++++ +++ 18455 98 Latency 12940us 9652us 227us 18233us 153us 377us
Those are crazy numbers. So, I turned to Google and did some more reading. It turns out a RAIDZ’s write performance is much worse than I expected. This quote (well, quote of a quote) is very good:
"Now we come to the crucial decision ZFS has made for raidz and raidz2: in raidz and raidz2, the data block is striped across all of the disks. Instead of a model where a parity stripe is a bunch of data blocks, each with an independent checksum, ZFS stripes a single data block (and its parity), with a single checksum, across all the disks (or as many of them as necessary). This is a rational implementation decision, but when combined with the need to verify checksums, it has an important consequence: in ZFS, reads always involve all disks, because ZFS always must verify the data block's checksum, which requires reading all of the data block, which is spread across all of the drives. This is unlike normal RAID-5 or RAID-6, in which a small enough read will only touch one drive, and means that adding more disks to a ZFS raidz pool does not increase how many random reads you can do per second. (A normal RAID-5 or RAID-6 array has a (theoretical) random read IO capacity equal to the sum of the random IO operations rate of each of the disks in the array, and so adding another disk adds its IOPs per second to your read capacity. A ZFS raidz or raidz2 pool instead has a capacity equal to the slowest disk's IOPs per second, and adding another disk does nothing to help. Effectively a raidz ZFS gives you a single disk's read IOPs per second rate.)" This was on a blog of a SUN engineer (although a post from a few years ago), unfortunately I don't have the link, I actually had to go through my posting history on the Ars Technica forum to even find this quote in the first place. If the situation has changed and the above quote no longer holds true, it would be nice if someone more knowledgeable on the performance implications could elaborate what kind of performance is to be expected on a raidz system- Sincerely, Dan Naumov
Wow. In that same thread and in another I found, someone posted some benchmarking results they had done. They are very interesting. Follow these links:
http://virtual.tehinterweb.net/livejournal/2009-06-22_zfs_diskperf/zfs-diskperf-contig-write.png
http://virtual.tehinterweb.net/livejournal/2009-06-22_zfs_diskperf/zfs-diskperf-5MB-readwrite.png
http://virtual.tehinterweb.net/livejournal/2009-06-22_zfs_diskperf/zfs-diskperf-1MB-readwrite.png
I was quite surprised. Further reading has lead me to rethink my setup. For this TV/media server I need the ability to read and write simultaneously at high rates. Write speed is more important in that I’ll likely be recording more shows at any given time than I’m watching, but I’ll still need to be able to stream a couple of HD shows at the same time. Heck, my 4 tuners haven’t been enough on a couple of occasions. So, I’m going to have to sacrifice space for speed. I don’t know how I’m going to do the data shuffle, but I’m considering picking up another pair of 1TB drives and a PCI-Express SATA controller (only one free one left on the mainboard). That’ll help. Then I guess I’ll build the pool from mirrored pairs of 1TB drives. My read/write performance should improve, but the addition of 2TB more drives won’t give me any more space. It will be interesting to see what kind of numbers I get out of it.
Update: I reconfigured the pool from a 4 disk RAIDZ to a pool of 2 2 disk mirrors. Bonnie++ results:
Version 1.96 ------Sequential Output------ --Sequential Input- --Random- Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP dagobah.boonthe 16G 89 99 95149 25 87337 23 289 99 270826 31 214.8 28 Latency 354ms 11106ms 10391ms 40391us 3882ms 464ms Version 1.96 ------Sequential Create------ --------Random Create-------- dagobah.boontheekul -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 20303 95 +++++ +++ 8108 99 19027 97 11837 99 4149 99 Latency 13531us 265us 478us 39328us 308us 5922us
Update: I got in 2 more 1TB disks (and a HighPoint 2310, as I’ve used up all 6 on-board SATA ports). I added the 2 new disks (via the HighPoint, no other rearranging of the drives) as another mirror in the pool. Now the capacity is back up to what it was as a RAIDZ. Bonnie++ results:
Version 1.96 ------Sequential Output------ --Sequential Input- --Random- Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP dagobah.boonthe 16G 112 99 145807 39 124058 33 289 99 372925 46 274.9 22 Latency 331ms 3891ms 7095ms 31678us 2229ms 457ms Version 1.96 ------Sequential Create------ --------Random Create-------- dagobah.boontheekul -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 22492 94 +++++ +++ 19562 97 16765 97 18918 99 6837 99 Latency 18093us 1113us 185us 37416us 203us 386us
It’s hard to make any conclusions from this. The numbers for the pool of mirrors are completely different than what I expected. In addition, I can say that interactive performance with similar workloads to what caused me problems previously have much improved. Just from the reconfiguration of the existing drives, I was able to record 4 HD streams simultaneously while watching another and no apparent stuttering or the like in any recording or the playback. Adding the 2 additional disks as another mirror in the pool had apparent impact in the bonnie++ numbers and brought my usable space back up to previous levels, but I’m still rather surprised at the bonnie++ numbers.
As I’ve mentioned, I’ve been very interested in ZFS lately. Well this week, the hypothetical happened to me and ZFS was there to make it a non-event. I was checking up on my newest ZFS’d filesystem (a pair of 1TB drives in a ZFS mirror). I ran ‘zpool status’ to see how the health of the pool was and it reported that one drive had had a checksum error. I’ve been processing gigs of log files from it, so one block error isn’t much of a pattern. I ran a ‘zpool scrub’ to check everything out (it was clean), and I’ll certainly watch that drive a little closer. ZFS made it simple and seamless.
I’ve been doing a hard drive shuffle in an attempt to utilize ZFS on FreeBSD some more in one set of my systems. Well, I decided that I wanted a direct dedicated network between one pair of boxes, but the catch is that there are no more expansion slots available to plug in another network card. That got me looking at the onboard FireWire port. I had read about the capability of runnning IP over FireWire and so I looked into it some more. Things looked good, so I set forth with that as my plan — I’d run a FireWire network between the two machines. Well today I got the boxes together, plugged-in the FireWire cable and configured the virtual fwip interface on both. What’s the typical first test? ping. BAM! Immediate kernel panic on the pinger (running FreeBSD 8.0-RC1 amd64). Ugh, what did I do wrong? My on-the-fly plan B was to try the fwe interface instead. It’s not as recommended, as it doesn’t implement a standard like the fwip interface does, but both boxes are going to be FreeBSD anyway, why not try it? It didn’t have any problems. I need to do some more network speed testing, but so far so good. I’d really like to be able to up the MTU, as I plan to use the interface primarily for NFS, but it doesn’t look like it’s possible. Tonight I did some digging to see if anyone else has encountered the same problem with the fwip interface and I found a bug report:
http://www.freebsd.org/cgi/query-pr.cgi?pr=139162
Well, at least it wasn’t me. It’s the only reference I’ve found to this problem, so it’s probably low-priority for 8.0-RELEASE, but who knows. I’m just thankful that there is a viable work-around.
Recently, I started moving from BeyondTV to SageTV for my PVR needs. Maybe I’ll hash out all the reasons in a future post, but in the end it leaves me with my old BeyondTV “server” hardware free to move on to other purposes. As such, I’m building out FreeBSD on it for it’s next task. I decided to build out with gjournal from the start. I’ve build a few boxes with gjournal after the fact and I hated having the extra slices. Besides, I quickly run out of available slices. So this time I wanted to build it out from scratch with gjournal such that I could have the journal in the same filesystem AND I decided to go with GPT as it’s the next big thing. Well, to accomplish this I used Fix-it mode on the installer DVD and for the first time in a dozen years of FreeBSD usage, I installed without sysinstall. I’d never contemplated not using sysinstall, but having done some reading on ZFSroot installations people were doing I used some similar procedures. I must admit, I was surprised how simple it was and there’s a good bet this is going to be the way I do installs from now on (until gjournal and/or ZFS are available in sysinstall). It worked beautifully and in a single attempt. Disks are just too big now for fsck. I lived through one 800GB UFS2 filesystem having to be fsck’d when someone decided to turn off a system on me and it’s just way too slow (and I was lucky to have enough RAM). I’ve got probably 9TB of hard drive space now (6TB in 1TB drives; 3 of them in 1 system) and there’s now way I could fsck many of them if something happened. Gjournal and/or ZFS are the answers in FreeBSD land. Manual install is the way to get them going from the beginning.
So, I finally got an iPhone. I know it’s a few years behind the time, but the 3GS and 3.0 OS finally made it seem like a complete product. Also, the 32GB model seemed like it was the first to really be big enough as a convergence device. Additionally, the App Store “ecosystem” looked like it had reached the size needed to sustain itself. So, I dumped my no-longer-on-a-contract T-mobile account over to AT&T. So far so good. It’s nice to have a reasonable Internet browser all the time. I’m an NPR addict, so having instant access to their podcasts is awesome. Weather radar in your hand is amazing. I’ve got enough space to hold an enormous number of apps, music, podcasts, pictures, movies, and TV shows to remain computerless for a good chunk of time. I’ve even managed to connect to my office VPN, but I haven’t bought an SSH app to do my normal thing with yet (I live on a terminal). I even found this blog app, so maybe I will get back to posting again.
Well, I never got around to finishing the job of cleaning up after the last major software update and yet another has come along. This looked pretty important so I did. I wonder how long it will take for me to figure out what else broke. Hopefully, it didn’t make it impossible for me to track it all down.
As anyone that works with software knows, upgrades don’t go as smoothly as you’d like. Well, my latest here with the site has been the biggest bugger. I don’t know how long it’ll take me to sort it out. I don’t seem to have lost any data (yet), but I have missed a lot of organization that I’ll have to re-do. That sucks. So, things will be out of whack for a while…
The latest OpenBSD Hackathon has produced some truly amazing results. Sure, every Calgary based Hackathon has produced something cool or amazing, but this one has resulted in 3 things that are home runs to me:
FFS2 – no, it is not done yet, but it is in a testable state and big disk/partition support has been lacking and this will bring OpenBSD back into the fold for my consideration as a fileserver. I’ve got one of those 750GB disks in my TV, you can imagine how much data I want to be able to store on the network, and I’d prefer it to be logically contiguous.
Bluetooth – I can just imagine a bluetooth access point with all the magic of PF, plus the simplicity of OpenBSD.
PF performance optimizations – the sheer amount of performance increase they’ve achieved in this Hackathon is nothing short of amazing. PF was pretty dang fast before, but now …. wow. Should be able to use lower power boxes for the same throughput in most of the situations I’m imagining.
Big thumbs up!
I just saw the exciting announcement: ZFS has been imported into the FreeBSD tree. Awesome work by pjd@ . It’s amazing what he’s done basically by himself. I’m not brave enough to run -CURRENT, but I’ll certainly be trying out the 7.0 Release. By that time someone might have added the ACL support so we can have fully featured Samba shares. I’m excited to try it out and see where this goes.
http://lists.freebsd.org/pipermail/freebsd-current/2007-April/070544.html
