Calgary RHCE

A linux and open source technology blog.

Connect

Demystifying automation and Puppet

By Leave a Comment

Automation has grown from its traditional roots of simple scripting into a critical cog of present day IT infrastructure. Today we have full fledged software suites based on automation. We have myriads of start-ups who base their entire business model on it. Every CIO is including it in their execution strategy, and you can’t seem to find a conference that doesn’t have a trendy buzzword filled keynote dedicated to the topic. Oh how far we’ve come! What IT person or company isn’t looking at automation these days?

But with all the trendiness and hype comes a lot of opinion and confusion. Marketing and business development departments muddy the technical details. Software developers consume it differently than sys-admins. Automation requires more business process change than technology change, management wants it implemented yesterday, and the technology feature-set is evolving at a massive rate. How’s one expected to keep up with it all? Hopefully this post can help you do that. (I realise the hypocrisy in presenting an opinion on other opinions 😉 )

One of the leaders in the IT automation market is Puppet Labs and their configuration management language, Puppet. I’ve put together a slide-deck and most recently presented at our local Red Hat User Group meeting (meetup.com, LinkedIn) on the subject. Specifically, “What is Puppet? And challenges/gotchas experienced with real-world implementations”. I’ll take you through where I believe Puppet fits in in the enterprise, what it will and will not do for you, and some things to consider when adopting it yourself. This knowledge has been gathered from my past experience adopting it for previous and current employers. Included in this presentation is a slide on making the transition from traditional IT to private cloud server provisioning. I’ll dig into this deeper in another post.

For now, here you go. I’d welcome any feedback or comments you may have!

RHUG Puppet Path To Private Hybrid Cloud
RHUG Puppet Path To Private Hybrid Cloud

Managing Configuration Management w/ Puppet

By

Over the course of the past year or so, Puppet has taken off in popularity amongst developers and administrators. It’s becoming the de-facto tool for repeatable state and drift management, and part of the reason for the uptake in popularity of the DevOps culture. As with most techology that grows quickly and organically, it often looks much different a year into implementation than it did day one.  Folks come away with many lessons learned and new ideas that they would implement if they had the chance to do it all over again.

I’ve been fortunate to be part of two puppet implementations so far in my career and thankfully have been working alongside some intelligent and insightful folks in the process. In one implementation in particular, we started a Puppet Task Force that served to educate the many developers who were operating in silos with the intent of helping them integrate their code and development structure with other teams. No easy task. I’ve come across this challenge several times now and continously find myself leaning on this one puppet article & slidedeck by Craig Dunn to help us all understand how to architect puppet from a high-level view. It’s been very well received each time I share it, and its content is very insightful and useful.

Craig does a much better job articulating the concepts than I do, so I’ll just link you to him. If you’re struggling with managing your puppet modules either in sheer size, overlap, or integration with other developers, I urge you to take a look at Craig’s work.

Designing Puppet – Roles and Profiles
Designing Puppet – Roles/Profiles Pattern (slidedeck)

Reliable, resilient storage with GlusterFS

By 2 Comments

A need came up lately for some inexpensive resilient storage that was easily expandable, and that spanned multiple datacentres.  Having recently been playing with GlusterFS and Swift in our OpenStack trial, I was quick to point out that these were strong use-cases for a GlusterFS architecture. We needed something right away, and something that also wasn’t terribly expensive, which Gluster caters to quite well.  Typically we would purchase a SAN technology for this, but recently having some discussions on cost and business agility, we decided against that and opted to try a popular open source alternative that’s been gaining both momentum and adoption at the enterprise level.

I began researching the latest best-practise architecture for GlusterFS.  It’s been about a year since I’ve given it a deep investigation, so I started re-reading their documentation.  I was pleasantly surprised to see that quite a bit more work had been done on the documentation since I last checked-in with the project.  They’ve recently moved a lot of documentation into github, and redesigned their website as well.  I found some good information on their website here, but found the most useful information in github here.

Our storage needs for this project were very simple.  We need resilient, highly available storage that can expand quickly, but don’t need a ton of IOPs.  Databases and other intense I/O applications are out of scope of this project, so what we’re mostly looking for out of this architecture is basic file storage for things like:

  • ISOs
  • Config file backups
  • Restored file location
  • Static web files
  • FTP/Dropbox/OwnCloud file storage

After perusing through Gluster.org‘s documentation, I came across the architecture that would best suit our needs.  Distributed replicated volumes.  We wanted two servers in each datacentre to have a local copy of this data, and also have two copies stored in another datacentre for resiliency.  This way we can suffer a server loss, a server loss in each datacentre, or an entire datacentre loss and still have our data available.  The documentation even gives you the command syntax to create this architecture; perfect!  I wanted to change the architecture slightly, but that wasn’t a big deal.  There’s enough detail in the documentation that I was able to understand how to do this.

  1. I racked 2 CentOS 6.5 servers in each datacentre (for a total of 4. I had four Dell R420s that were decomissioned from another project – commodity hardware)
  2. Installed gluster-server on them all via the glusterfs-repo
  3. Made sure their DNS was resolvable for both forward and reverse
  4. Chopped up their RAID10 1TB hard disk with LVM leaving approx 80GB for the OS, and the rest for /gfs/brick1

…and I was ready to run their provided command – slightly modified of course.  Since I wanted the data to essentially be replicated across all 4 nodes, I changed the replica count from 2 to 4.  This means if I ever need to expand, I’ll need to do it 4 nodes at a time.  I then ran this command on one of the cluster members to create the gluster volume named ‘gfs’:

For a visual reference, this is what this architecture looks like:

GlusterFS - Distributed Replicated Volume
GlusterFS – Distributed Replicated Volume

That’s it! I could now install the glusterfs and glusterfs-fuse client software on any node that needed to access this volume, and mount it with

After succesfully mounting the volume, I did a few file writing tests with bash for loops, and dd, just to test the speed and see that any created files were replicating.  I then did a few failover tests by shutting down an interface on a node, rebooting a node completely, and shutting down two nodes in a datacentre.  Since I am using the native glusterfs client to connect to the volume, the failover and self-healing features are automatically handled unbeknownst to the user.  After the 42 second timeout, services failed over nicely, and files were replicated across active hosts.  When the downed nodes came back up and joined the volume again, the files they missed were automatically copied over.  Perfect!  And just like that – I was done.

I was surprised at how simple this was to architect and setup.  When I need additional disk space, I’ll rack another 4 nodes into the architecture, and expand it by using the gluster volume add-brick command documented here.

In another article, I plan to do some tweaking to the 42 second TCP timeout – Gluster provides documentation on all the options and tuneables you can set.  It’s also worth looking at using an IP-based load balancing technology.  (Either an F5, or CTDB).  This should increase the service availability even further.  In yet another article, I would also like to explore the new geo-replication technology that spans WANs and the internet.  For now, I have a 1TB highly available and resilient file storage volume to play with.