Getting Started with RedHat’s OpenStack Preview

I’ve been intrigued by OpenStack for some time now, and have tried setting up test beds on numerous occasions. These tests inevitably failed to get off the ground, but I kept coming back to try again. Most of my difficulties have been brought on by poor documentation, and a lack of cohesive software packaging. Many OpenStack guides try to handle everything with scripts, or Chef/Puppet combinations. This is great if you are already familiar with OpenStack, but if you aren’t there can be many problems on the back end of those provisioning tools that you are either unaware of, or unable to resolve. To really understand how all the pieces of OpenStack fit together, you need to manually install everything, so I set out to find a guide that would let me do just that.

The guide/distro combo that I had the best luck with was RedHat’s OpenStack Folsom Preview, and the accompanying documentation found here. The documentation will get you close to a working finished product, but there were some hiccups that I found and documented along the way. The good news is that RedHat is constantly updating their documentation. Hopefully as they update, less and less of these issues will continue to be a problem.

At this time, I’ve gotten the following setup: a single-node OpenStack environment with VM’s and images. The following components are being used:

• Keystone (Identity, Token, Catalog, and Policy)
• Glance (Image)
• Cinder (Block Storage)
• Nova (Compute)
• Quantum (Network)
• Horizon (Dashboard)

My future plans include adding several compute nodes, configuring Swift object-storage, configuring a dedicated Quantum networking node, and most importantly, integrating EMC storage with Cinder and Swift for block and object storage.

If you have already checked out the RedHat guide, you’re probably wondering why I don’t already have Swift working The answer is that I skipped over Swift for now. You might also be wondering why I want a dedicated network node. Quantum and RHEL 6.4 is hamstrung out of the box. If you follow the guide, you’ll be installing a networking component that has some functionality that does not work. This is due to the fact that RHEL 6.4’s kernel does not have support for namespaces. Without support for namespaces, there can only be one virtual router per node, and you cannot have overlapping IP address ranges. For more information, check out the bug here, which hopefully will be fixed in RHEL 6.5. I plan on using a different distro (either Fedora or Ubuntu) for the Quantum network node.

With that out of the way, let’s get to the “gotchas.” I’ve tried to vet these as much as possible, but some of them may not be best practice. They did work for me, but your mileage may vary. The notes are in chronological order coinciding with the RedHat Getting Started Guide. I would recommend reading this in its entirety before beginning the install, as you’ll need to know some things before you install RHEL.

Before Starting

If you want to have a proper Cinder install, you’ll need to setup your partitions during the RHEL install. To do this, create two physical volumes for LVM during the partitioning phase of the RHEL install. One will be for your OS, the other will be for your Cinder volumes, so size them appropriately for your deployment. You’ll also need to create a boot partition in addition to the two physical volumes for LVM. This boot partition can be small, around 250 MB is fine. For RedHat’s recommended partitioning scheme info, check here. Once you’ve created the two physical volumes for LVM, you’ll need to create a volume-group for each one. Name one of them “cinder-volumes” and name the other whatever you want. The other one is where you’ll create your /home, root, and swap LVM partitions.

For the sake of simplicity, I’d recommend disabling a couple of security measures. First set SELinux to “permissive” mode by editing the /etc/selinux/config file to have the following set:

SELINUX=permissive

Next, disable iptables:

$ chkconfig iptables off
$ service iptables stop

Keystone

Your first issues will likely arise right out of the gate with the method that they use to set the admin_token. I could not get their service token to work, so I simply edited the /etc/keystone/keystone.conf file to manually set the admin token. Perform these steps BEFORE exporting the environment variables as some of them become redundant after doing this.

Open the keystone.conf file and set the “admin_token” to whatever you’d like. Then, set the environment variable to the same thing. For example:

$ sudo vi /etc/keystone/keystone.conf
admin_token = TOKEN_PASSWORD

Save the file and exit, and then set the environment variable to the same:

$ export SERVICE_TOKEN=TOKEN_PASSWORD

Don’t run the commands that are crossed out:

$ export SERVICE_TOKEN=$(openssl rand -hex 10)
$ export SERVICE_ENDPOINT=http://127.0.0.1:35357/v2.0
$ echo $SERVICE_TOKEN > /tmp/ks_admin_token
$ sudo openstack-config --set /etc/keystone/keystone.conf \
DEFAULT admin_token $SERVICE_TOKEN

Glance

In section 9.1, they want you to run through the process of creating your own image to import. This is a lot of work when all you really need is an image. I recommend just downloading a CirrOS image and importing that instead:

$ wget https://launchpad.net/cirros/trunk/0.3.0/+download/cirros-0.3.0-x86_64-uec.tar.gz

Cinder

RedHat’s guide will have you create a temporary file system for Cinder to use. I opted instead to create a “cinder-volumes” volume group using LVM partitioning during the initial RHEL installation. This is the default volume group that Cinder looks for out of the box.

When you get to the part of the Cinder configuration where they want you to setup the volume groups, don’t run the following commands that have been crossed out:

$ sudo grep -q /etc/cinder/volumes /etc/tgt/targets.conf || sudo sed -i '1iinclude /etc/cinder/volumes/*' /etc/tgt/targets.conf
$ sudo service tgtd start
$ sudo chkconfig tgtd on
$ sudo truncate --size 20G ~/cinder-volumes
$ sudo losetup -fv ~/cinder-volumes
$ sudo vgcreate cinder-volumes /dev/loop0
$ sudo vgdisplay cinder-volumes

After this step, here’s some things to check:

In the /etc/cinder/cinder.conf file, set the following options:

volume_name_template = volume-%s
volume_group = cinder-volumes
state_path = /var/lib/cinder/
volumes_dir = /var/lib/cinder/volumes

In /etc/tgt/targets.conf, make sure this command is at the top:

include /etc/cinder/volumes/*

In /etc/tgt/conf.d/cinder.conf uncomment the following line:

include /var/lib/cinder/volumes/*

OpenStack Networking (Quantum)

Unfortunately, I still don’t have Quantum working the way that it should, and I believe that’s due to RHEL 6.4’s lack of namespaces support. Without namespaces, there can only be one virtual router, and duplicate IP addresses aren’t supported. Duplicate IP address support is very important for multi-tenancy.

If you want full Quantum functionality, my recommendation is instead to configure it on its own node with a Linux distro that does support namespaces. Apparently Fedora 17 and Ubuntu 12.04 and later support namespaces. This will change a lot of your config files for Quantum, since they will now be pointing to the controller IP instead of the loopback address. Stay tuned for an update on this once I’ve setup a dedicated network node. For now, just know that you will be slightly limited in what you can and can’t do with Quantum following the instructions in this document and the RedHat guide.

When you run the quantum-server-setup command, and tell it to update the nova configuration files, make sure you double check those files. You will need to update the quantum configuration settings:

quantum_admin_username = openstack_network
quantum_admin_password = password
quantum_admin_auth_url = http://127.0.0.1:35357/v2.0/
quantum_auth_strategy = keystone
quantum_admin_tenant_name = openstack_network
quantum_url = http://localhost:9696/

You also might need to make changes to the quantum config files. Be sure to update your tenant, user, password and region in the quantum plugin.ini, dhcp_agent.ini, l3-agent.ini, api-paste.ini, and quantum.conf files. Case is important, even in the region. Make sure that your auth_region = regionOne and not RegionOne. Also, put the correct SQL credentials in the /etc/quantum/plugins/openvswitch/ovs_quantum_plugin.ini file:

sql_connection = mysql://root:password@fqdn.of.your.server/ovs_quantum

Horizon

Feel free to ignore the “lokkit” commands if you’ve been following this guide as the firewall will have been disabled.

When enabling VNC console access (section 12.2.2), the VNC options will all be appended to the end of the nova.conf file. This is problematic because the VNC options will end up in the [keystone_authtoken] section where they will be completely ignored. Cut and paste all of the VNC related settings and put them before the [keystone_authtoken] section. Also note that the “novncproxy_base_url” should be the outside interface of your controller node that is reachable by end users. When a user clicks on the VNC console for an instance, the base url is what their web browser will try to connect to.

# VNC
novncproxy_host = 0.0.0.0
novncproxy_port = 6080
novncproxy_base_url=http://fqdn.of.your.server:6080/vnc_auto.html
vnc_enabled=true
vncserver_listen=<public IP of your controller>
vncserver_proxyclient_address=<public IP of your controller>
[keystone_authtoken]
admin_tenant_name = %SERVICE_TENANT_NAME%
admin_user = %SERVICE_USER%
admin_password = %SERVICE_PASSWORD%
auth_host = 127.0.0.1
auth_port = 35357
auth_protocol = http

Using OpenStack

If you’re using Quantum for your networking, ignore sections 18 and 19. They are only applicable to Nova networking, which is disabled. Also note that when attaching a Cinder volume to your instances, you may need to specify a mount point other than /dev/vdc.

Troubleshooting

If you’re having problems with something in OpenStack, tail all of the log files for that particular service while you recreate the problem. For example, if you’re having a problem with Quantum:

$ sudo tail -f /var/log/quantum/*.log

It’s also not a bad idea to check all of your .conf and .ini files for any lingering default or bad credentials that weren’t updated. For example, I had an issue where the region for the quantum user had been changed to RegionOne instead of regionOne. To check all of the files for that mistake, I used the following commands:

$ sudo grep RegionOne /etc/quantum/*
$ sudo grep RegionOne /etc/nova/*
$ sudo grep RegionOne /etc/cinder/*

Wrap Up

Please let me know if any of these suggestions helped you out, and if you have anything to add, definitely let me know so I can update the doc, or include the suggestion in a future post. I plan on following up with articles on creating a network node, creating compute nodes, and setting up EMC storage to work with OpenStack.