Previously, I went through a couple OpenStack topics on installers, and deploying an undercloud as part of a virtual OpenStack deployment. Today I’ll walk through the overcloud deployment, and hopefully by the end of this post you will have had enough details to get your own deployment up and running. This particular environment is for Red Hat OpenStack Platform 8 (Liberty), but the same steps will apply to Mitaka as well. There isn’t a ton of change in the deployment methods from Liberty to Mitaka, the templates here will apply to either.

Having previously worked through undercloud deployment, you should have a working and configured undercloud that’s ready to further configure the overcloud and how it is to be deployed. In the Director Installation and Usage guide, we’ll be starting from Step 4.7. You’ll want to obtain the overcloud images either directly from access.redhat.com, or from the rhosp-director-images RPMs. (It might be handy to have that guide open and follow along here. I won’t post every command from that guide here, but will do the areas that might not be so clear).

After obtaining the images and uploading them to the undercloud, you should see a similar list of available images to be used in deployment:

These images displayed here, as well as the ones copied to the /httpboot directory serve a couple purposes. The /httpboot images are introspection images, or what the undercloud host will copy to the overcloud nodes as their hardware is being inspected (or introspected as it’s also called). What this introspection process does is it takes an audit of the hardware you’re giving the overcloud, and it puts it in an inventory in the undercloud, so the undercloud knows a little about the nodes it’s about to provision. This is used to help classify baremetal servers into node types (compute, controller, etc.). The bm-deploy* images are used during firstboot (a PXE boot) for the baremetal hosts you’ve set in the overcloud. These baremetal images are used to get the overcloud node(s) to a basic networked OS, then it copies the overcloud-full images to disk, which the node then uses to permanently boot from in the traditional server OS booting process. I know it’s a little convoluted process, but each image is purposeful and this functionality is basically a replication of the PXE boot process you may be familiar with from a cobbler server, or other PXE boot environment.

During the undercloud installation process, neutron network entries used for provisioning get created. This is commonly known as the “control plane” network, of which handles the provisioning traffic for nodes, as well as all of the API and Management traffic between the nodes once the cluster is up and running. In a sincle-nic-vlan deployment (the one we will be doing), this physical control plane network will have this individual API and Management traffic broken out into different subnets and VLANs for separation.

After completing Step 4.8 in assigning a nameserver to this neutron network, we’ve completed the undercloud configuration and are ready to move on to configuring the overcloud.

Chapter 5 starts off with configuring the basic overcloud requirements. I’ve previously decided that we’re going to be doing the “Medium Overcloud with High Availability and Ceph Storage” deployment scenario. You can check out other deployment scenarios in Chatper 3.1 in the same guide. The deployment scenario we’re going with has 3 controllers, 3 computes, and 3 ceph storage nodes. I think this will give you good exposure to all the important aspects of a truly production ready OpenStack deployment. This will give us true high availability with 3 controllers. It also gives us flexibility in compute capacity to create different availability zones, or tiers of compute offerings with exposure to live migration, as well as instance HA for automatic failover (configured separately after deployment.) Also, this gives you good exposure to ceph software defined storage, configured in a clustered environment, with a reasonable amount of storage to create instances and volumes without restriction and in a way you can explore its performance. The only thing you might want to do differently with this deployment is separate the traffic out further to dedicated NICs or bonds for increased resiliency, or add more compute/storage nodes for capacity.

Beginning with Step 5.1, we need to register the overcloud nodes with the undercloud. This basically pre-loads the undercloud inventory (ironic) with the overcloud nodes and their specs. The file used to do this is instackenv.json, and it is in JSON format. Previously in the undercloud deployment post, you should have created the overcloud nodes as VMs. For reference, here’s what you should create:

• 3x Controller VM: 1 vCPU, 6 GB RAM, 40 GB disk, 2 NICs (first NIC is on the provisioning network, second is on the linux bridge)
• 3x Compue VM: 4 vCPU, 6 GB RAM, 40 GB disk, 1 NIC (provisioning network)
• 3x Storage VM: 1 vCPU, 6 GB RAM, 40 GB disk (OS), 8 GB disk (simulate ceph journal SSD), 100 GB disk (ceph osd), 1 NIC (provisioning network)

In the instackenv.json file, we’re going to include basic information about the above overcloud VMs. Here’s the first and last entry of mine:

The first node is a compute node, and the last node is a storage node. (There’s the other 7 nodes in between these two, but I’ve truncated it for brevity). This file expects the MAC address of the provisioning NIC, and since we’re using the pxe_ssh driver in the virtual lab, we’ll be populating the pm_user, pm_password, and pm_addr fields to accommodate. If you’re using KVM for your virtual environment (recommended), this is what you’d use for deployment. If you’re using VMware, you’ll want to use the fake_pxe driver. More info on both of these drivers are in Appendix B.6, and B.7 of this deployment guide. The reason I say the pxe_ssh driver is preferred is because all the power management needed to be done using the fake_pxe driver with VMware needs to be done manually. (Rebooting of nodes during provisioning is automatically handled by ironic and the iDRAC/iLO/UCS/pxe_ssh drivers. You’ll need to manually boot and reboot VMs if you’re using the VMware driver, so the KVM option on a new deployment is much easier to use to get familiar with the provisioning process first).

Back to the instackenv.json file, the pm_password field expects the private SSH key of the pm_user, not it’s password. It expects a “\n” character at the end of the first line, and the beginning of the last to parse the key correctly. Also, you’ll want to setup password-less SSH access from your undercloud (director) node, to the hypervisor host as the pxe_ssh driver is essentially a wrapper to virsh, which handles the power management of your KVM VMs. The pm_addr in my case here is my hypervisor host IP. Before importing this file, you’ll want to check that you can successfully SSH from the undercloud to the hypervisor without a password as both the root and stack users. Use the ssh-copy-id tool for this.

Once that’s done, proceed with the import of the instackenv.json file, complete assigning the boot images to all nodes, and verify ironic sees all expected nodes in it’s inventory. If the import fails, it will throw an error to the console. Common things to check are making sure the MACs set are ones from the provisioning NIC for each node, as well as the password-less SSH access working. The undercloud node here will verify the presence of the VMs with virsh during this import. A successful node listing should display all 9 of your nodes:

The next step is to inspect (introspect) these nodes hardware details. You’ve already populated the major details it needs, so this step is mostly to discover additional root disks, and for automated node classification if you’re importing 100s of nodes at a time. This step isn’t mandatory, but I’d recommend doing an introspection and watching your VM console just to become familiar with the boot process.

The next (and critical) step is to tag these nodes with the overcloud profile you want them deployed as. So compute, controller, ceph-storage, etc. In our case, we’ll tag the three compute nodes with compute, the controllers with controller, and the ceph storage with ceph-storage. Pretty straight forward right? This is often an overlooked step and causes many issues come deployment time if your nodes are tagged in a different way than what you’ve told the undercloud to deploy. When deploying, remember that the nodes available to you during a deploy are the ones in the ironic inventory so in the future if you decide to change the 3 ceph-storage nodes to 3 additional compute nodes, you’ll need to come back here and re-tag them as such.

My first three ironic nodes are compute nodes, my second three are controllers, and my third three are storage. So I’ll be running 9 ironic commands, one against each node to tag them with their profile:

… you get the idea. After this is node tagging is done, you can check each node or list the tagged profiles all at once:

This shows my first node is tagged as compute, and the summary of all nodes says I’ve got 3 of each profile tagged. Awesome! Ironic now knows about all the overcloud nodes we want to provision, and it knows which role each should have. Now for probably the most complicated part – heat templates.

This is probably the most confusing part of an overcloud deployment, is how to setup the heat templates so that the nodes get deployed in the architecture that you want them to. In this section (now in Chapter 6 of the guide), I’ll just give you the templates we’re going to use, and briefly explain why. As you get some practice deploying a few environments and seeing the configuration first hand, these templates will start to make more sense. For me this was a little information overload reading all the options I could deploy with, it made understanding the one I need difficult to identify at first.

The general rule of thumb with an overcloud deployment is network isolation is a good thing. We want to separate the API traffic and data traffic to it’s own networks, and VLANs. This is true regardless of the size of your cloud, the only difference is how much farther you go in separating your traffic. So our first decision is that we’re going to use network isolation. The second decision we’re going to make is that we’re going to use the heat templates that setup for a single-nic-vlans deployment. Because we’re in a virtual environment, this option makes the most sense. There’s a myriad of other templates you can use depending on your network architecture, another popular one is multiple-nics, and bond-with-vlans. Those templates are for if you’ve got 6 NICS on your physical servers, or if you’re bonding 2 NICs for the provisioning network, and 2 more for the API and Mangement traffic in an LACP, linux, or openvswitch bond. For our use, copy the single-nic-vlans templates:

I’ve uploaded my yaml templates to my github repo, so you should be able to use these verbatim assuming you’ve setup your VMs and undercloud exactly as I have. I’ll dive into some of the notable details though. Keep in mind that these are .yaml files, and indenting is important.

network-environment.yaml:
In this file, you’ll want to make sure that the nic-configs templates path in the resource_registry section references where you’ve placed those template files. Example:

Also, in addition to assigning subnets to each network, take care to note the default routes of your external network, control plane network, and what the IP of your undercloud server is. These need to be reachable and set correctly, otherwise you’ll have networking problems in your overcloud at deployment time, and the deployment will fail. Since my external network has it’s own router gateway, and my control plane network (the virtual provisioning network) also has it’s own gateway, I’ve set these parameters accordingly, and set the EC2MetadataIP to the IP of my undercloud which is one IP above (Depending on your virtual environment setup, these values may differ slightly. It’s possible to have control plane and undercloud IP gateway the same – again depending on your virtual environment):

storage-environment.yaml:
In this file, we’re going to say we want to deploy using Ceph, and will take most of the defaults already set, except we’re going to add in an ExtraConfig section to assign a journal disk, and osd disk for Ceph. Example:

Now, of the single-nic-with-vlans files now in your nic-configs directory, we really only need to edit one file – the controller.yaml file. This is because this is the only host which has an extra external NIC, the rest of the templates already jive with the provisioning NIC being nic1. So I’ve edited the controller.yaml file as such:

controller.yaml:

Here we tell heat that this network interface type is an OVS Bridge (ovs_bridge). And that it’s going to be using the control plane network details, with nic1 as a member interface. In that ovs_bridge, we include all of the API networks and their VLANs. This is the config that isolates all the traffic over the provisioning NIC. Since I’ve attached a second NIC as an external nic, I’ve created a new interface entry with the external network details, and commented out that inclusion in the provisioning NIC.

Done! The last thing that’s good to do is set a timezone for the overcloud, you can use timezone.yaml for this:

Now that we’ve got all our yaml files created the way we want, it’s time to deploy the overcloud. We’re up to Chapter 7 now, step 7.3 to be specific. We’re going to create a deploy command that should deploy our overcloud for us. In this deploy command, we’re going to pass all the environment template files we’ve created so that heat understands how to deploy our overcloud. We’ll also set a couple of other useful flags. This command is usually best to do inside a bash script, as you may end up running it multiple times, and you’ll also need it for any future scaling events you’ll want to do with your overcloud. Here’s my deploy command:

We’re including the network isolation template to indicate we want to isolate the networks. Also included is our network and storage templates, the timezone, and the node count and node profile of each of our overcloud nodes. Additionally we set an NTP server (very important for the time to be correct among all nodes), and the tunnel types for neutron. I like to include the validation flags as well, as they’ll fail the deploy right away if it notices any misconfigs in your yaml files, which makes it quicker to fix. Also, the timeout flag specifies how long to wait before timing out the create if it gets stuck on a task. This way if you do hit an error, you’re not waiting hours for the deploy to report back that it failed.

And go!

Now assuming you’ve followed all the steps correctly, you should end up with your 9 node deployed into an OpenStack cluster, and get a successful deployment message like the above. Nova should show you your 9 nodes now deployed:

And browsing to your public endpoint should give you Horizon (you can find admin login credentials in the overcloudrc file):

After logging in, you should see the 3 available compute nodes, and their 300GB ceph cluster available to them:

Congratulations! You’ve stood up your first OpenStack environment. If you get stuck at any step along the way, comment below. I’ll point you in the right direction to get it deployed.