Calgary RHCE

A linux and open source technology blog.

Connect

Managing Configuration Management w/ Puppet

By

Over the course of the past year or so, Puppet has taken off in popularity amongst developers and administrators. It’s becoming the de-facto tool for repeatable state and drift management, and part of the reason for the uptake in popularity of the DevOps culture. As with most techology that grows quickly and organically, it often looks much different a year into implementation than it did day one.  Folks come away with many lessons learned and new ideas that they would implement if they had the chance to do it all over again.

I’ve been fortunate to be part of two puppet implementations so far in my career and thankfully have been working alongside some intelligent and insightful folks in the process. In one implementation in particular, we started a Puppet Task Force that served to educate the many developers who were operating in silos with the intent of helping them integrate their code and development structure with other teams. No easy task. I’ve come across this challenge several times now and continously find myself leaning on this one puppet article & slidedeck by Craig Dunn to help us all understand how to architect puppet from a high-level view. It’s been very well received each time I share it, and its content is very insightful and useful.

Craig does a much better job articulating the concepts than I do, so I’ll just link you to him. If you’re struggling with managing your puppet modules either in sheer size, overlap, or integration with other developers, I urge you to take a look at Craig’s work.

Designing Puppet – Roles and Profiles
Designing Puppet – Roles/Profiles Pattern (slidedeck)

Reliable, resilient storage with GlusterFS

By 2 Comments

A need came up lately for some inexpensive resilient storage that was easily expandable, and that spanned multiple datacentres.  Having recently been playing with GlusterFS and Swift in our OpenStack trial, I was quick to point out that these were strong use-cases for a GlusterFS architecture. We needed something right away, and something that also wasn’t terribly expensive, which Gluster caters to quite well.  Typically we would purchase a SAN technology for this, but recently having some discussions on cost and business agility, we decided against that and opted to try a popular open source alternative that’s been gaining both momentum and adoption at the enterprise level.

I began researching the latest best-practise architecture for GlusterFS.  It’s been about a year since I’ve given it a deep investigation, so I started re-reading their documentation.  I was pleasantly surprised to see that quite a bit more work had been done on the documentation since I last checked-in with the project.  They’ve recently moved a lot of documentation into github, and redesigned their website as well.  I found some good information on their website here, but found the most useful information in github here.

Our storage needs for this project were very simple.  We need resilient, highly available storage that can expand quickly, but don’t need a ton of IOPs.  Databases and other intense I/O applications are out of scope of this project, so what we’re mostly looking for out of this architecture is basic file storage for things like:

  • ISOs
  • Config file backups
  • Restored file location
  • Static web files
  • FTP/Dropbox/OwnCloud file storage

After perusing through Gluster.org‘s documentation, I came across the architecture that would best suit our needs.  Distributed replicated volumes.  We wanted two servers in each datacentre to have a local copy of this data, and also have two copies stored in another datacentre for resiliency.  This way we can suffer a server loss, a server loss in each datacentre, or an entire datacentre loss and still have our data available.  The documentation even gives you the command syntax to create this architecture; perfect!  I wanted to change the architecture slightly, but that wasn’t a big deal.  There’s enough detail in the documentation that I was able to understand how to do this.

  1. I racked 2 CentOS 6.5 servers in each datacentre (for a total of 4. I had four Dell R420s that were decomissioned from another project – commodity hardware)
  2. Installed gluster-server on them all via the glusterfs-repo
  3. Made sure their DNS was resolvable for both forward and reverse
  4. Chopped up their RAID10 1TB hard disk with LVM leaving approx 80GB for the OS, and the rest for /gfs/brick1

…and I was ready to run their provided command – slightly modified of course.  Since I wanted the data to essentially be replicated across all 4 nodes, I changed the replica count from 2 to 4.  This means if I ever need to expand, I’ll need to do it 4 nodes at a time.  I then ran this command on one of the cluster members to create the gluster volume named ‘gfs’:

For a visual reference, this is what this architecture looks like:

GlusterFS - Distributed Replicated Volume
GlusterFS – Distributed Replicated Volume

That’s it! I could now install the glusterfs and glusterfs-fuse client software on any node that needed to access this volume, and mount it with

After succesfully mounting the volume, I did a few file writing tests with bash for loops, and dd, just to test the speed and see that any created files were replicating.  I then did a few failover tests by shutting down an interface on a node, rebooting a node completely, and shutting down two nodes in a datacentre.  Since I am using the native glusterfs client to connect to the volume, the failover and self-healing features are automatically handled unbeknownst to the user.  After the 42 second timeout, services failed over nicely, and files were replicated across active hosts.  When the downed nodes came back up and joined the volume again, the files they missed were automatically copied over.  Perfect!  And just like that – I was done.

I was surprised at how simple this was to architect and setup.  When I need additional disk space, I’ll rack another 4 nodes into the architecture, and expand it by using the gluster volume add-brick command documented here.

In another article, I plan to do some tweaking to the 42 second TCP timeout – Gluster provides documentation on all the options and tuneables you can set.  It’s also worth looking at using an IP-based load balancing technology.  (Either an F5, or CTDB).  This should increase the service availability even further.  In yet another article, I would also like to explore the new geo-replication technology that spans WANs and the internet.  For now, I have a 1TB highly available and resilient file storage volume to play with.

 

Integrating kickstart w/ CentOS 6 DVD ISO installation

By Leave a Comment

In our DevOps environment, we’ve got a lot of developers who regularly build VMs.  Sometimes they’re built locally on their workstations, or sometimes in the data center when they’re ready to formally move code.  Admittedly, the IT VM provisioning process can be slow at times and we often see them get frustrated and take matters into their own hands.  While this self-build method gets them their VM faster, they don’t always get IT’s supported “base image”.  This can lead into integration and support problems later on when they’re trying to move this code into production that was developed against a non-standard OS.  What to do?

OpenStack, or even OpenShift, is the long-term solution to this problem.  However, until our OpenStack environment is stood up, we will need something else to address it right now.  What if we could provide them an ISO that has our standard build integrated into it?  Well!  We are going to do just that.  Read on…

Lets make our own ISO file with our standard kickstart image build into it. We’ll change the ISO boot menu options to only allow one install selection, and let the developers get to work! You will need the CentOS install images, and a workstation running an RPM anaconda based distribution.  I used CentOS 6.5 for both.

Set up your build environment:

You will need the createrepo, isomd5sum, and genisoimage packages:

Setup a mount point for the DVD ISO, and mount it:

Next, make a working directory for what will be the new ISO image:

Copy the mounted DVD ISO files into the working directory:

You’ll need a comps.xml file. This file describes the package groups included in the ISO. You can grab one from a CentOS repo:

Create the kickstart file. You can use system-config-kickstart to build a new one, or copy one from an already installed server’s /root/anaconda-ks.cfg file. If you don’t have system-config-kickstart, use yum to install it. The RedHat knowledgebase provides a list & explanation of all the available kickstart options. You should create the kickstart file as /export/media/centos_build/kickstart/ks/ks.cfg:

Now, add any packages you’d like to include on the ISO. If you’ve got in-house RPMs, or are sourcing them from somewhere else (epel, fedora, cfengine, puppetlabs, etc.), place the .RPM file in the kickstart/CentOS/ directory. In my case, I’m going to include two in-house developed CFEngine packages. The first package installs the CFEngine agent, and the second package bootstraps CFEngine. My goal is to have this ISO & kickstart file to automatically provision the VM into the centralized configuration management tool –> CFEngine. CFEngine will then take care of adding all the admin accounts, setting up NTP, SSH, etc. (This is the real advantage with creating this custom ISO!)

In your kickstart file, be sure to have these package names listed in your %packages section. This is also a good time to add/remove any other packages you want. My %packages section will look like this:

Now that we have our kickstart file finished, lets modify the ISO boot menu to use the kickstart file by default:

My file looks like this. Be sure not to typo your kickstart file directory, otherwise it won’t work:

Now, if you’ve added packages into /export/media/centos_build/kickstart/CentOS/ directory, we will need to update the comps.xml file to reflect your changes. First, make a backup copy of comps.xml. Open the file for editing with your favourite XML editor. I’d suggest using an editor that can collapse groups of XML code as the comps.xml file is quite long. I use sublime text for this:

After the last <group>, but before the first <category> tag, add your new group entry. Mine looks like this:

sublime

You’ll need to add a new category as well. I made my entry at the end of all the <category> tags, but before the </comps> tag. This category is going to reference the previous group you just created, so be careful not to typo or use different case.  The field you need to get accurate here is groupid. Mine looks like this:

sublime2

Use an XML syntax checker to make sure you didn’t make any errors. xmllint is a good one:

Building the ISO:
We’ll now need to re-initialize the repo list:

And build the ISO:

Now my boot menu looks like this and will automatically use my ks.cfg file for installation:

bootmenu

And although I didn’t provide this option in the boot menu, if you boot the graphical installer, you should see a new section for the packages you specified:

graphicalinstall

And that’s it!  Put the ISO up on a file store or web server, and be satisfied that the developers are using a supported base OS install!