Once in a while I need to do some I/O benchmarking. Either when creating a baseline for a before and after performance tuning comparison, or just because I want to see how fast my new SSD/M.2 really is. The tool I find myself going to over the years has been iozone. I like it because it easily does a myriad of tests that I don’t need to create or think about how to simulate myself, and it’s easily installed on any linux distro. It also outputs the test data into a format that is easily imported into a spreadsheet. I recently added a couple SSDs to my workstation, and finally got around to benchmarking them.

IOzone has a slew of pre-built tests you can run, but I typically just do 5 (read, random read, write, random write, and stride read). From the iozone help:

Within each test you can also specify the record size, and file size to be written. This is my usual test syntax:

… and so on as it goes through all of the test scenarios. At the end, I’m given a nicely formatted spreadsheet that’s easy to make graphs from:

Looking at my read stats, at the peak, I’m getting just over 20 GB/sec. However, the average seems to be about 12-14 GB/sec. Not too shabby! (I’ve got a few SSDs in a RAID1 array). Full test details are in a spreadsheet on github.

Having recently been exposed to some SSD tweaking at work, I thought I’d do the same with my home PCs.  Prior to this weekend, I’ve just had four 1TB SATA drives in a RAID 5 configuration for a disk setup.  Performance has always been satisfactory, but with recent SSD prices coming down quite a bit, my late 2008 MacBook Pro feeling it’s age, it was time for an upgrade.  Also, it’s been a while since I’ve invested in some new gear for myself, so why not now? ;).

For the laptop, a 256 GB Samsung 840 Pro SSD.  For the tower, the 128 GB version.

These are higher grade SSDs, with quite a bit higher IOP rating than average.  Generally speaking, the larger the hard drive, the more IOPs you will get also.  The laptop would be using the entire drive for all data, while the tower would be using the SSD for just the OS.  My laptop runs OSX Mavericks, and the tower CentOS 6.5.  For now, we’ll just talk about the CentOS box.

Tuning the filesystems – optimizing data structure for SSD use, eliminating journal access time writes, temp filesystems to RAM, enabling TRIM

I’ve split up the OS into five filesystems that will sit on the SSD. Root, var, usr, boot, and ssd. The /ssd filesystem is just set aside for anything that I might need quick disk for.

The advantage of SSDs is read-performance, and they do wear when used, so ideally you want the files that are frequently read on the SSDs, but reduce the writes and file changes as much as you can to save wear and increase the SSDs life span. For this reason I’ve used the “noatime” and “nodiratime” mount options on the SSD ext4 filesystems. Since ext4 is a journaling filesystem, linux will record information about when files were created, last modified, as well as when it was last accessed. Turning this feature off not only reduces the writes to the SSD, it also can improve performance on often accessed and frequently changing files. Since no important application logs sit on these filesystems, and /var/log/messages includes timestamps within the file, I can safely turn this off. My tower /etc/fstab looks like this:

Since swap can be frequently accessed, I’ve opted to not put that on the SSD. Also, /export and /home contain important data that needs to be revivable if I ever lose a couple disks. SSDs are not good for archival use; once they wear out, you won’t be able to scrape the drive for any useable data. Once it goes, your data is gone with it.

I’ve also made some other tweaks on the filesystem, making /tmp and /var/tmp use RAM instead of disk for file storage. This will speed up temp file access by quite a bit. I have 32GB of RAM on the system, so I’m not crunched for memory.

I’ll also need to enable TRIM support. There’s a good article here that talks about enabling it properly. I’ve enabled TRIM for LVM, and have placed a simple script in the weekly cron to handle the discarding of blocks on the SSD.

Change the kernel I/O scheduler

The I/O scheduler optimizes disk operations for speed. The default scheduler is CFQ (Completely Fair Queuing), which is optimized for the mechanical rotational latency of regular HDDs. Since that doesn’t apply to SSDs, consider using the deadline scheduler instead which prioritizes reads over writes. This change will help take advantage of the SSD performance. You can read more about the I/O scheduler options here. Further to that, RedHat-based systems also have the tuned-adm command that can set pre-defined performance profiles based on your hardware and optimization purpose. (See man tuned-adm for more.)

To enable this change scheduler change permanently, you’ll need to modify GRUB to pass the “elevator=deadline” parameter on the kernel boot line.

Tuning virtual memory – Reduce swapping

Although I’ve placed my swap partition on the HDDs, swap file tuning is still an important tuneable to look at. Swap space is hard drive space that is used when RAM is exhausted and the current proccesses need more memory. The system will also move data to the swap file from RAM if it hasn’t been accessed in a while. This preemptively free’s up RAM for other processes. This is incredibly slow compared to RAM, and really should only be used as a fall-back when your system is running out of RAM. If your system is frequently swapping, go buy more RAM and use the swap file as a temporary work around.

In the kernel, there’s a setting that controls the degree to which the system swaps. Since I have a lot of RAM to use, I only want the system to swap when it’s absolutely necessary. A high value in the swappiness setting aggressively swaps processes into physical memory when they’re not active. A low value avoids swapping processes for as long as possible. So I’ve set the vm.swappiness parameter to zero. You can read more about this tuning here.

I’ve added this parameter to /etc/sysctl.conf, and made it active with sysctl -p. This will ensure this is persistent through a reboot.

Moving Firefox cache to RAM

This is a nice tweak that tells firefox to use RAM instead of disk for it’s cache. This should speed up browsing frequently visited sites. To enable it, go into the about:config, set browser.cache.disk.enable to false, set browser.cache.memory.enable to true, and browser.cache.memory.capacity to the number of KB you want to assign. I’ve used -1 to let Firefox dynamically determine my cache size depending on how much RAM the system has.

You can use the about:cache entry to see your newly changed cache information!

Results

Now for the results. I’ve used iozone as my benchmarking tool because it performs a series of tests quite easily, and graphs the output nicely for me in excel. Attached are the results of the read report statistics. As you can see, considering OS cache amongst a few other things blur the true speed differences a little bit, the SSD has roughly triple the performance in read speed vs the four HDDs in RAID5.