Calgary RHCE

A linux and open source technology blog.

Connect

Getting started with OpenStack

By 2 Comments

OpenStack logo

So you’d like to get started with OpenStack eh? (Sorry, that’s my Canadian coming through). You may have been thinking about it for a while, evaluating the landscape, the maturity, evaluating your tolerance for new technology, and waiting for the right time to adopt. Being on the verge of adopting a new and disruptive technology is an intimidating place to be. Lots of questions arise: Where to start? What to evaluate? Who to partner with? What gaps to address? What should the architecture look like? What is the project purpose? Is my staff well prepared? What training should we get? Is our infrastructure capable of supporting it? How to handle long term tasks like patching, scaling, or lifecycle management? These aren’t easy questions to answer, and the answers usually come from experience; which you don’t have. What do you do? I’m going to start the first in a series of posts addressing some of these questions and hopefully adding some clarity to OpenStack adoption. First, some of the basics.

What is OpenStack?

OpenStack is essentially the culmination of commoditized infrastructure, and years of frustration with inflexible, closed, proprietary data center infrastructure components. (That’s a bold statement, let me expand on that.) As hardware virtualization took off in the mid to late 2000’s we began to see the potential, and reap the benefit, of increased application deployment speed. No longer being tied to the physical server procurement process opened the industry’s eyes to server flexibility and utilization. Virtualization was born from the idea of vertical density, ie. how many OS’s can we cram onto this server to maximize its use? As we became accustomed to “spinning up a VM”, these tasks started to feel trivial, frequent, and burdensome. In true sysadmin fashion, we looked to automate some of these tasks, but we didn’t get very far. Existing virtualization tools were designed with focus on performance, high availability, and ease of management. But not automated management. As our virtual environments became more utilized and integrated, we started to see delays in the process of “spinning up a VM”. Many checks and balances were added, and soon the time to provision virtual resources started to creep closer and closer to that of its physical cousin. Sound familiar? (I also wrote a post on this last year). It seemed like we couldn’t even buy an adequate API, let alone integrate it with something else.

At the same time, applications started to adapt to the new concept of commoditized hardware. Server infrastructure began to be treated more like consumables. Scaling out started to be as important as scaling up, and the need to do that quickly was paramount.With these two challenges in mind, the idea of programmable infrastructure spawned, and the idiom infrastructure as code was born. The OpenStack project is essentially that – programmable infrastructure. Services native to a datacenter make up the project, and each is designed with a fully featured API.

Why should I care?

The drive behind virtualization was cost optimization and increased utilization of existing resources. The drive behind OpenStack is also cost optimization, but not of the traditional capital expenditure kind. OpenStack serves to reduce the amount of time your IT staff waits for data center resources. It also increases the flexibility and agility of your datacenter resources so that they may quickly be created and re-purposed in quite literally a matter of seconds. It enables your business to adjust incredibly quickly to the constant change we see in our industry. And lastly, it enables you to take advantage of new cloud capabilities in application and service architecture. Quickly scale up, scale down, integrate automation tooling, or create applications with higher resilience and additional capability, so that you are ready to adjust to the next business priority or opportunity.

What’s the difference between virtualization and OpenStack?

Others have described the difference better than I could have, so I’ll link their work. But I’ll add this; traditionally, hardware was assigned to an application. Then hardware was assigned to virtual datacenters or clusters. If you wanted to re-purpose hypervisors, add storage, or provision networks you would incur a lengthy migration, balancing, or configuration task. And none of these services came with a capable API. OpenStack takes the virtualization concept of a software-defined OS component and blows it out of the water. Now every piece of infrastructure is software defined, so you can “spin up” much more than just a VM.

How do I get started?

I thought you’d never ask! 😉 This is a loaded question with many correct answers to it, but we’ll want to start with sorting out a couple basics. Firstly, you need to decide the workload intent of your OpenStack environment. Specifically, I mean what type of application workloads do you expect to put there, and how many instances or VMs do you want in the environment. Secondly, you’ll want to decide the performance and redundancy you expect out of those workloads. Deciding on these two things will help give you a good idea of where to start for physical node count, physical node resources, and the node type or role each physical device should have. Also, this will help you decide what you should be using for an OpenStack installer. Since the environment architecture is a wildly variable conversation, I’m going to link some helpful resources to assist you in your sizing decisions. This decision is worth engaging an OpenStack provider in, depending on your intent of the environment. Some folks choose a small environment with a flagship app to deploy with it, others deploy general use OpenStack resources for mass consumption.

 

Next topic I will outline some of the options you have for installers, why you might pick one, and then I’ll choose an environment size to go through and do an example deploy. Have I missed anything important in the above topics? Got anything to add? I’d love to hear your experience.

Ansible-pull and kickstart, for one-touch server provisioning

By Leave a Comment

Recently I’ve been learning and using Ansible as my configuration management tool. It came recommended by several colleagues, recently had an O’Reilly book published, and went through an aquisition. Safe to say its momentum and adoption is on a high… and so far, I’m loving using it. I find it vastly easier to setup and use than Puppet, Chef, CFengine, or SaltStack. I bought the O’Reilly book, and had a playbook configuring a server in about 10 minutes… udderly quick. (Sorry, bad pun if you looked at the book cover). Most of my config management experience is with CFengine and Puppet, yet I found some interesting contrasts so far:

  • I found Ansible far easier to consume and get running right away. Literally after 10 minutes I had a basic playbook pushing /etc/motd to my server. There was no setup of a Puppet master or CFengine master server required, and I didn’t need to download any modules. It uses already encrypted SSH for traffic, so no SSL work was required either.
  • The syntax reads very easy, and the learning curve to understanding and writing a module (or a play in this case) was very short. I remember spending a few days with Puppet and CFengine before I surface-level understood modules and classes. Ansible was 10 minutes. I’ll use a phrase coined by a colleague, and one that also happens to be in the book…. Ansible reads like executable documentation.
  • No agent required, everything runs over SSH, so that alleviates management and maintenance of a local agent.
  • A formal Ansible server wasn’t required, although a remote client to be managed needs to talk to something that has the Ansible code. (This could be a formal server, or alternatively a git repo, or any server with repo access. How the client gets its code seems to be very flexible)

Pretty impressive so far. Then I started to think… well running a play or a series of plays (called a playbook – which are essentially configuration tasks or a group of tasks) looks to be a push methodology, and performed as a one-off task. What about drift management? (Puppet and CFengine excel at this). And how can I get a VM created, bootstrapped, and configured by ansible in a one-touch provisioning method?

Enter ansible-pull. Just like it reads, instead of initiating a push from the server to the remote client, a new client can request its config with ansible-pull. In fact, if you load-balance your config source (server, repo) this can theoretically scale infinitely. So, I investigated how to set this up, and this is what I came up with. Ansible-pull initiated by kickstart, a git repo with my ansible code, and a systemd unit file to turn the first-boot ansible-pull into a service that automatically configures my server, I just need to enable it with systemctl.

First, you’ll need some ansible code, and then a place to put it (Creating code is outside the scope of this post, but I’ll share mine for reference). I’ve got a gitlab server at home, so I’ve synced my code into a new repository named “ansible”. Having your code at the root of the repo helps make it easy for ansible-pull.

GitLab Screenshot
GitLab Screenshot

Ansible has a concept of an inventory file, so what I’ve done here is copied my site.yml file into localhost.localdomain.yml as I’ve told ansible-pull to look for that. In my home lab, VMs come up unnamed to begin with, then once they’re networked and configured I rename them into a more permanent home. This can easily be changed, but this is what I’ve done thus far. To make it easy to fetch the code, in GitLab I’ve created the user ansible, uploaded a public SSH key, and given the user access rights to the repository. Then, my CentOS 7 kickstart file %post section looks like this:

Above:

  • I make sure ansible and git are installed and the latest versions with the yum commands
  • I add the user, and configure it to use sudo without a password and tty
  • Then I unroll it’s .ssh/ directory w/ SSH keys and pre-populated known_hosts file, and also populate the ansible inventory file
  • Then I grab the script I’ve created that runs ansible-pull, and will be the executable of my systemd unit file
  • I grab the custom systemd unit file
  • Do a daemon-reload of systemd to trigger reading in my new unit file
  • Enable the new ansible service I’ve created

Now once kickstart is completed, the ansible-pull script will get run via systemd auto-starting the ansible-config-me service. My unit file and script look like this:

In the [Service] section, Type=oneshot tells systemd that the process will be short-lived. This is is useful for for scripts that do a single job, and then exit.

To take advantage of the pre-populated SSH keys and access of the ansible user, I run ansible-pull as user ansible, and tell it to pull the master branch, download the code and run it from the deploy directory, use the hosts inventory file which contains the new hosts’ hostname,  give it my git repo, SSH key, and then set the purge flag to delete the local deploy directory and code within it after it’s done. Since I also just want this script run once, I disable the service in systemd.

There you have it! A one-touch provisioning process. And as for the drift management piece, I read most folks are cron’ing this ansible-pull. However, an official drift management feature is coming in the near-future for Ansible Tower.

Oh, and for interest sake, this is the ansible playbook task I’m running:

 

Additional items to work on for an even sleeker provisioning process:

  • Tweaking my virsh-install script to provide a hostname, and have the OS configure it with ansible
  • Finding an appropriately safe, yet public place to pull this ansible code from, so I can use it with cloud providers
  • Removing the local ansible user and using LDAP instead for increased security and control

Automating Server Provisioning; the Transition to Private Hybrid Cloud

By 1 Comment

In every place I’ve worked, the server provisioning process was largely identical. There were of course subtle differences depending on vendor products, business processes, etc. but typically every customer’s server provisioning artifacts are made up of the same components. To provision a host, you’ll need IP information, DNS entry, VM container, a kickstart/jumpstart method, pre-provisioned storage/networks, and all of the various application level components on top of the OS (monitoring agents, backups, the application itself, etc.). These systems or processes usually had been in place for more than a few years, and would be considered the “Traditional IT” method of provisioning servers.

Since most of these components are a few to several years old, it’s accurate to say that not all of them were designed with true automation in mind. Some components had automation capability, but generally these existed in isolation and had manual kick-off processes or dependencies associated to them. In one particular shop, our home-grown asset management tool was completely manual. We could build an OS in an automated fashion, but had to wait for manual asset management tasks to be completed post-build before we could hand it over to a developer. In another shop, our IP address management tool didn’t have an API. We followed many manual steps to reserve and setup IP/DNS before we could kick off an automated OS build. In all of these server provisioning scenarios, a developer would be waiting anywhere from 4-10 hours (rush request) to 2 – 3.5 weeks before they could get their server. The quality of server OS going out the door was quite high, however it was hardly quick, never fully-automated, and “rush” requests were the only way developers could get an OS in a reasonable amount of time.

See the below graphic outlining each provisioning component. I’ve highlighted the areas of automation concern:

Server Provisioning Artifacts
Server Provisioning Artifacts

We spent a ton of time analyzing how we could improve this process. We eliminated steps where feasible, automated what we could, created efficiencies, etc.  After much scrutiny and completing all the “low-hanging-fruit”, we realized we weren’t ever going to be able to get down to the 15-20 minutes timeframe that developers were expecting (and were getting from other sources such as Amazon). We realised our limiting factors were the very tools themselves that we were using in each component. If our IPAM tool didn’t have an available API, we would never be able to programmatically address (read automate) our IP requirements. Similarly, if we couldn’t programmatically tie into our asset management tool, we’d be forever stuck with manual tasks to enter the data. In order to automate the entire process of server provisioning, we were in a “rip and replace” situation on some of our core server provisioning components. Ouch. What a sobering realization. As you can expect, ripping and replacing components wasn’t a very popular idea with the business, as there were large investments in each component and they were very useful and functional to other areas of the business. What was IT to do?

Enter infrastructure-as-a-service (IaaS), and more specifically, OpenStack. OpenStack was designed with these infrastructure limitations in mind. All of the common components in the traditional IT provisioning workflow were implemented in OpenStack with a capable API. In fact, the API is the only way to interact with these components. No more being locked-in with a vendor’s short sighted roadmap. Also, being an open-source project, if the API capability wasn’t there that you expected, you could engage the community to include your feature, or develop it yourself.

Once we discovered IaaS, we were quickly looking at how we could get it in house, and fast. As we learned about OpenStack and private/public clouds, we started to see where this was going to work for us, and where it was not. We discovered that most of the rush systems the developers were looking for were for testing/developing new capabilities, and were tied to time-to-market activities or other time-sensitive new business constraints. Most of the business critical systems were still being satisfied by the traditional provisioning method. We decided to PoC OpenStack for this developer test bed use-case, and look at a workload promotion process to the traditional IT environment once the business decided these new capabilities were worth pursuing seriously. We’ve ended up with a provisioning architecture like this:

Server Provisioning Artifacts
Server Provisioning Artifacts

We realised the traditional IT provisioning method was going to be around for quite some time for legacy systems. We also realised that we did have some workloads that would suit an IaaS method, and that we still had developers going to Amazon for bleeding edge development we couldn’t bring in house just yet. This left us with three different methods of server provisioning, so we decided upon Red Hat’s cloud management tool to orchestrate each provisioning workflow. This will allow us to perform consistent policy and process enforcement, as well as automate similar tasks, across all provisioning workflows. While we’re still working on integrating several of the automation components of each workflow, we soon will have sufficient ways to automate server provisioning for use-cases that need quick responses, while still satisfying the traditional IT legacy use-cases that our business will be running for quite some time. This will leave us in a private hybrid cloud state, with the agility to scale up or down any provisioning workflow we need as our business needs evolve.