How to Set the Block Size of Software RAID

How to Set the Block Size of Software RAID

Now here is a subject that most folks couldn’t care less about, which is OK until you need to find out how to do it. When you do need to know, guess what? You can’t find out.

I know because I have looked for this information for years, talked to Apple, and no one seems to really know the definitive answer. This article is going to change that. (You can thank me later.)

I have used software raids for years, for both Mac OS and Linux. Hardware RAIDs are always best, but sometimes you need a software RAID. The Mac Pro has four internal bays, so you need to figure out how you want to use them. The Mac OS lets you choose between RAID 1 (mirror) and RAID 0 (striped). The stripe distributes data across all drives and results in an increase of speed. If a drive fails in a mirror, you can replace it and rebuild your RAID and your data survives. If a drive fails in a striped RAID, all your data is lost. (It’s easy to remember the numbering system. RAID 0 is a stripe, which means if any drive fails you have ZERO chance of recovery. Raid 1 is a mirror and gives you ONE chance if a drive fails.) All hard drives fail, it’s just a question of what will happen to your data when they do fail.

There is another type of RAID, RAID 5. It is optimal because it stripes your data and still lets you recover if one drive fails. RAID 5 is only available on a Mac with a RAID hardware controller and RAID box. FYI RAIDS 2, 3, and 4 are no longer in use.

For years I just made 2 mirror RAIDs inside my Mac, reducing the number of available drives from 4 to 2, but then I had some backup in case of failure. This practice has saved my bippy two or three times. One interesting way of setting up a RAID on a Mac is pair each of 2 drives as mirrors, then pair the resulting 2 RAIDs as a stripe. So you end up with just one drive. Haven’t tried it, so I can’t recommend it. I also tried making 2 drives in to a mirror, then 2 drives into a stripe. I made sure I backed up the stripe somewhere each day with rsync. You can try Carbonite, but it is slow. For internal network backups try Data Backup 3.

With video you need speed. Larry Jordan mentions that with the new multi cam of FCPX you have to have a RAID. For us that means RAID 0, which is by far the fastest. Remember though, the more drives in a stripe, the more prone you are to failure. A three drive striped RAID is three times less reliable than one drive. Since I have a DroboPro and backup every day, I am not too worried about it. If you don’t have a reliable backup system, don’t even think about a striped RAID.

You set up the RAID on a Mac with Disk Utility. I’m not going into all the ins and out on how to do it, you can find out from the Apple instructions.  What Apple doesn’t tell you about setting up is how to know the optimal block size to speed up your particular task. It is under the options button. The block size tells the controller how much data to write to one block. If you have a large block size but are only writing little bits of data you are wasting space and slowing your drive. You have 5 choices from 16K to 256K, but which one to pick?

How to set the block size of a software RAID

If you are doing database work that writes data in small chunks, 32K may give you the best results. Video files are much larger so a bigger block size is better, but how big? That is has always been my question, and should be yours too if you are setting up a software RAID. The only way to find out is set up your RAID then test it yourself. One method you can use is to utilize AJA System Test, a free download from AJA.

How to Set Block Size of Software RAID I have tested and retested several configurations. I used the HD Video Frame Size with various File Sizes. I would do several tests with a block size of 32, then go back to Disk Utility, destroy the RAID, set up a new RAID with a block size of 64k. Then I ran several tests with AJA. Then I repeated with 128k and 256k block sizes. You can easily do the same (it takes a few hours), then you’ll know for yourself what works for you.

My results were interesting. For comparison one internal hard drive by itself ran at about 94/100 MB/s. My USB 2.0 external drive tests about 29/38 MB/s. My DroboPro running under Firewire 800 (iSCSI broken temporarily) runs at about 32/34 MB/s. A 2 drive internal RAID ran at about 240/250 MB/s. On a 3 drive RAID average speeds resulting from changing block sizes ranged from 280 to 440 MB/s.

For me the best testing results came when I set the block size to 128K, not 256K.

I got consistently slower results from choosing 256K. I did the testing with an empty drive, so my speeds were faster than a RAID that is full. A real world example with a disk that is moderately full runs at about 340 MB/s in and out when the block size is set to 128K. That is a heck of a lot faster than a single drive.

BTW only buy black hard drives, not green drives. Green drives don’t do anything for the environment, they just run slower. We don’t care about electricity usage, we only care about speed. If you are worried about the environment spend the time you save by using faster drives and go outside and plant a tree.

One last caveat, once a striped RAID gets about 90% full the speed slows significantly, so start looking for a new home for your data. If you want to do some testing the next time you set up your RAID please let me know your results in a comment.

 

Share this!

Subscribe to our RSS feed. Tweet this! StumbleUpon Reddit Digg This! Bookmark on Delicious Share on Facebook
  1. WILL BARTLETT
    February 26, 2014 at 6:03 pm

    Hey, thanks for posting. I stumbled on this article in my google search results when looking for best practices for creating a raid set. I’ve just set up a 3 Slice RAID on my Mac Pro using 3 640GB HDDs. I used 128K blocks as your article suggested, since I’m working largely with video and post production files.

    Thanks again!
    Will

    • Swainhart
      February 26, 2014 at 7:14 pm

      You are welcome, this worked fine for me for many years.

  2. RAID 10 on OS X – Striped set of Mirrors

  3. March 12, 2015 at 5:27 am

    Hello, perhaps you can include this, which is a piece of information all blogs on this subject fail to address.

    Yes video files are larger = larger block size however, you need to also consider all those tiny cache and database files any NLE produces. So if you’re storing them together with your video files, you can’t go too big or small of a block size, ideally, if your NLE allows, store only video/audio files on one drive, and NLE project/cache/database files on another, and set your block size accordingly. PS the default block size in MacOS is just 512bytes

  4. Rita C.
    April 12, 2016 at 8:01 am

    Thanks for this, I hope I can get a hold on this further question I got:

    When and why hack the option “Automatically rebuild RAID Mirror Sets”? I am performing data transfer through Silverstack Pomfort by the way. Thanks!

    • Christopher Swainhart
      April 15, 2016 at 3:21 pm

      Not really sure on this. We have moved to Pegasus RAIDs and they take care of all of this stuff.

Leave a reply

*

Your email address will not be published. Required fields are marked *

*