Calgary RHCE

A linux and open source technology blog.

Connect

Troubleshooting performance of VDO and NFS

By Leave a Comment

In setting up a local virtualization environment a little while back, I thought I’d try the recently GA’d VDO capabilities in the RHEL 7.5 kernel. These include data compression and de-duplication natively in the linux kernel (through a kernel module). This was Red Hat’s efforts behind the Permabit acquisition. Considering a virtualization data store is a prime candidate for a de-duplication use-case, I was anxious to reclaim some of my storage budget 🙂 . I was also curious to see what the extra overhead was like (if any), and understand the general performance characteristics of VDO.

I found VDO quite easy to setup, I followed this guide basically verbatim. Given my NFS virtualization back-end stores mostly the same OS images, I was happy to see excellent deduplication stats on my VDO device:

After a few months and several VMs spawned later, I noticed some slowness whenever I was doing high IO work like database updates and copying several gigs of data to disks all at once. (When my Satellite server downloads 50+ GB for a new content repo, and index’s it for example). My other VMs would notice a bit of a slowdown during this time. Given I hadn’t done much for tuning in this environment, it was probably time to look into it. I’ve also been debating upgrading my home lab to 10G networking and this seemed to line up with what I was seeing for storage performance. I thought I finally was being bottlenecked by the network, given I’ve got an SSD array in my NFS server, with a 4-port 1GB NIC in LACP. But before I went crazy buying 10G networking gear, I looked at tuning what I had.

The official documentation is fairly good at explaining the performance characteristics of VDO, and what you might want to tune. I also went into NFS server/client tuning as I hadn’t done much for this either. Given there’s a few things at play here (disk hardware performance, network performance, VDO optimization, NFS optimization) I quickly went down a few rabbit holes and realised I needed to do some basic benchmarking and baselining so I could understand which areas in this stack were actually performing well, and which ones were candidates for more tuning. In addition to the VDO tuning docs, here’s what I used for reference:

Firstly, it’s important to troubleshoot things in isolation, and use a benchmarking method that’s complimentary to isolation as well. I used the iperf3 utility for network benchmarking and fio utility for disk benchmarking. With this I’d do a series of sequential read, sequential write, random read, and random read and write tests, and these would be done both on local disk filesystems and over the network filesystem. For reference, here’s the fio commands:

After reading the above guides and doing some basic investigating and benchmarking, this is what I ended up tuning first:

Overall the networking looked alright. I saw some dropped packets, but I’ve been doing a fair amount of cable pulling, stop/starting hosts, and VPN up/down. The NICs on NFS server and clients weren’t using their full ring buffer, so I changed this. I don’t think this was much of a candidate for the dropped packets, but this tuning couldn’t hurt.

I also increased the default number of NFS threads on the NFS server. Considering I’ve got 15+ VMs, each VM looks to use 2-3 nfs threads depending on number of disks, tuning the default of 8 to 20 should help for concurrent disk activity:

Similarly, the VDO device I created only had 1 thread allocated in several places, so I upped these as well and doubled the cache size:

After these changes, I still wasn’t seeing any significant performance change. I was getting fairly abysmal speeds even on a local SSD filesystem on the NFS server, not even going over the network. I started to isolate this, and started to suspect a hardware/SSD tuning related issue. After updating my DL360p 420i storage controller firmware, making sure the RAID controller cache was disabled, SSD smart path was on, it started to dawn on me. Previously, these six SSD drives had been used in a RAID5 configuration that saw a ton of heavy disk IO. I had dedicated this host to an OpenStack environment and had done several builds hammering these disks. RAID5 is not an optimal SSD configuration, parity calculation is expensive, and this would add a ton of disk IO and disk wear that wouldn’t be present in a RAID0 or RAID10 configuration. Essentially, I’ve got worn SSDs. I needed to do a secure erase of these SSDs to return their cells to as close to original factory condition as one could get. These disks are approx 3 years old and haven’t had this done yet.

After doing an enhanced secure erase, I saw my local disk storage speeds come back up about 5 fold. This was more in line with the newer SSDs in other servers. Doing disk tests over the network saw the same speed increase. So it looks like my problem was entirely hardware related :). As I’m now turning on all the VMs, I’m seeing a much quicker response when doing the high IO activities. There’s more than 16 NFS threads consistently in use now and I’m monitoring to see the change in VDO related performance. I need to research a utility to get accurate VDO stats, I think this likely will be with a PCP module. But at first glance, with not much concrete data to go on yet, I *think* the VDO tuning has helped as well.

While I learned a bit about VDO and NFS performance tuning, it looks like I might need to spend that 10G networking budget on new SSDs instead. There’s diminishing life left in these.

Calgary OpenStack and other Meetup.com groups

By Leave a Comment

I’m a little late on this post, but last week we had an excellent presentation on OpenStack installers. Specifically, TripleO. Chris J from Red Hat gave us a good outline of TripleO’s capabilities, and kept it pretty upstream. For the presentation slide deck, see here. The group tries to keep it as vendor neutral as possible which is nice. There were a lot of questions and feedback, this is probably a topic worth going into in a little more detail, or perhaps doing some hackfest demo’ing!

Triple O by Chris J.

There’s been quite an uptick in our local OpenStack meetup group lately, it’s been fun to see and get involved with. Both this group, as well as our Red Hat User Group has brought in some high profile and very talented open source folks who’ve presented and shared their knowledge with us.

The OpenStack meetup groups across Canada have come together and created a Slack channel for easier communication and sharing among its members. If you’re interested in joining the channel, drop a comment or send me an email for an invite!

Network teaming in RHEL7

By Leave a Comment

If you’ve adopted or are just starting to read up on the new features included in Red Hat Enterprise Linux 7, you may have come across the new networking feature called teaming. It essentially is a replacement for bonding that offers more modularity, increased link monitoring features, higher network performance, and easier management of interfaces.

Habitually, network interface configuration is something I’m used to setting up once, and forgetting about it. I don’t usually re-visit my networking configuration unless some hardware changes, or I need to re-IP a system. With the advancement of SDN, this might change in the not-so-distant future, so I thought I’d give teaming a try. And hey, if it offers even marginally greater performance, why not get the most out of my OS?

I first started with reading a comparison of Network Bonding to Network Teaming, and the new network teaming daemon “teamd”, with it’s concept of “runners”. Network teaming has basically assigned a daemon to a link aggregate that allows management and monitoring of that interface through the daemon. I guess this is similar to systemd modularizing init scripts, we’ve got a daemon wrapper that will manage the config files for us in a programmatic way. After going through the docs, I found it pretty easy to get this setup and running. If you’re lazy (read efficient) like me, there’s adequate example configs in /usr/share/doc/teamd*/example-ifcfgs/ that are easily modified (there’s one for LACP).

On one of my hypervisors, I’ve got 2 NICs, an LACP bond (will be replaced by an LACP team), and a bridge device. My new interface configs look like this:

The only real differences between bonding and teaming as far the config files are concerned is that there’s new device types called “TeamPort”, and “Team”, and then inside the team config there’s a JSON formatted entry that replaces the traditional bonding options. In here you define the runner (or method of link management) you want for your slave ports, the link monitoring tool, and which interfaces are slaves (ports) of the team. You can use the same bridge options with the team as you did the bond, that doesn’t change. One important rule to note though, bringing up the team interface won’t also bring up the slave interfaces, and in order to add slaves to a team, the slaves need to be in a link down state prior to addition to the team. This is the reason my ONBOOT=no for the NICs, and when systemd brings up the network, it will add the downed slaves to the team, then link up the slaves, then link up the team. I guess this is the modularity intent, let the slaves be managed on their own, then manage the team based on the slave behaviour.

After modifying my files as above, restarting the network service, waiting a few seconds for LACP negotiation to occur, my team is up and running. You can use the new teamdctl command to query and control the interfaces:

Nice! That was less daunting than I thought it might be. The team driver also has good integration with NetworkManager, should you chose to manage your team interfaces with the GUI.

Some others have done comparison benchmarking of bonding vs teaming. Although probably quite marginal for my use-case, the benchmarking shows additional bandwidth throughput, with reduced CPU load.