I’ve been intrigued by OpenStack for some time now, and have tried setting up test beds on numerous occasions. These tests inevitably failed to get off the ground, but I kept coming back to try again. Most of my difficulties have been brought on by poor documentation, and a lack of cohesive software packaging. Many OpenStack guides try to handle everything with scripts, or Chef/Puppet combinations. This is great if you are already familiar with OpenStack, but if you aren’t there can be many problems on the back end of those provisioning tools that you are either unaware of, or unable to resolve. To really understand how all the pieces of OpenStack fit together, you need to manually install everything, so I set out to find a guide that would let me do just that.
The guide/distro combo that I had the best luck with was RedHat’s OpenStack Folsom Preview, and the accompanying documentation found here. The documentation will get you close to a working finished product, but there were some hiccups that I found and documented along the way. The good news is that RedHat is constantly updating their documentation. Hopefully as they update, less and less of these issues will continue to be a problem.
At this time, I’ve gotten the following setup: a single-node OpenStack environment with VM’s and images. The following components are being used:
- Keystone (Identity, Token, Catalog, and Policy)
- Glance (Image)
- Cinder (Block Storage)
- Nova (Compute)
- Quantum (Network)
- Horizon (Dashboard)
My future plans include adding several compute nodes, configuring Swift object-storage, configuring a dedicated Quantum networking node, and most importantly, integrating EMC storage with Cinder and Swift for block and object storage.
If you have already checked out the RedHat guide, you’re probably wondering why I don’t already have Swift working The answer is that I skipped over Swift for now. You might also be wondering why I want a dedicated network node. Quantum and RHEL 6.4 is hamstrung out of the box. If you follow the guide, you’ll be installing a networking component that has some functionality that does not work. This is due to the fact that RHEL 6.4’s kernel does not have support for namespaces. Without support for namespaces, there can only be one virtual router per node, and you cannot have overlapping IP address ranges. For more information, check out the bug here, which hopefully will be fixed in RHEL 6.5. I plan on using a different distro (either Fedora or Ubuntu) for the Quantum network node.
With that out of the way, let’s get to the “gotchas.” I’ve tried to vet these as much as possible, but some of them may not be best practice. They did work for me, but your mileage may vary. The notes are in chronological order coinciding with the RedHat Getting Started Guide. I would recommend reading this in its entirety before beginning the install, as you’ll need to know some things before you install RHEL.
If you want to have a proper Cinder install, you’ll need to setup your partitions during the RHEL install. To do this, create two physical volumes for LVM during the partitioning phase of the RHEL install. One will be for your OS, the other will be for your Cinder volumes, so size them appropriately for your deployment. You’ll also need to create a boot partition in addition to the two physical volumes for LVM. This boot partition can be small, around 250 MB is fine. For RedHat’s recommended partitioning scheme info, check here. Once you’ve created the two physical volumes for LVM, you’ll need to create a volume-group for each one. Name one of them “cinder-volumes” and name the other whatever you want. The other one is where you’ll create your /home, root, and swap LVM partitions.
For the sake of simplicity, I’d recommend disabling a couple of security measures. First set SELinux to “permissive” mode by editing the /etc/selinux/config file to have the following set:
Next, disable iptables:
$ chkconfig iptables off $ service iptables stop
Your first issues will likely arise right out of the gate with the method that they use to set the admin_token. I could not get their service token to work, so I simply edited the /etc/keystone/keystone.conf file to manually set the admin token. Perform these steps BEFORE exporting the environment variables as some of them become redundant after doing this.
Open the keystone.conf file and set the “admin_token” to whatever you’d like. Then, set the environment variable to the same thing. For example:
$ sudo vi /etc/keystone/keystone.conf admin_token = TOKEN_PASSWORD
Save the file and exit, and then set the environment variable to the same:
$ export SERVICE_TOKEN=TOKEN_PASSWORD
Don’t run the commands that are crossed out:
$ export SERVICE_TOKEN=$(openssl rand -hex 10)$ export SERVICE_ENDPOINT=http://127.0.0.1:35357/v2.0 $ echo $SERVICE_TOKEN > /tmp/ks_admin_token $ sudo openstack-config --set /etc/keystone/keystone.conf \ DEFAULT admin_token $SERVICE_TOKEN
In section 9.1, they want you to run through the process of creating your own image to import. This is a lot of work when all you really need is an image. I recommend just downloading a CirrOS image and importing that instead:
$ wget https://launchpad.net/cirros/trunk/0.3.0/+download/cirros-0.3.0-x86_64-uec.tar.gz
RedHat’s guide will have you create a temporary file system for Cinder to use. I opted instead to create a “cinder-volumes” volume group using LVM partitioning during the initial RHEL installation. This is the default volume group that Cinder looks for out of the box.
When you get to the part of the Cinder configuration where they want you to setup the volume groups, don’t run the following commands that have been crossed out:
$ sudo grep -q /etc/cinder/volumes /etc/tgt/targets.conf || sudo sed -i '1iinclude /etc/cinder/volumes/*' /etc/tgt/targets.conf $ sudo service tgtd start $ sudo chkconfig tgtd on
$ sudo truncate --size 20G ~/cinder-volumes $ sudo losetup -fv ~/cinder-volumes $ sudo vgcreate cinder-volumes /dev/loop0 $ sudo vgdisplay cinder-volumes
After this step, here’s some things to check:
In the /etc/cinder/cinder.conf file, set the following options:
volume_name_template = volume-%s volume_group = cinder-volumes state_path = /var/lib/cinder/ volumes_dir = /var/lib/cinder/volumes
In /etc/tgt/targets.conf, make sure this command is at the top:
In /etc/tgt/conf.d/cinder.conf uncomment the following line:
OpenStack Networking (Quantum)
Unfortunately, I still don’t have Quantum working the way that it should, and I believe that’s due to RHEL 6.4’s lack of namespaces support. Without namespaces, there can only be one virtual router, and duplicate IP addresses aren’t supported. Duplicate IP address support is very important for multi-tenancy.
If you want full Quantum functionality, my recommendation is instead to configure it on its own node with a Linux distro that does support namespaces. Apparently Fedora 17 and Ubuntu 12.04 and later support namespaces. This will change a lot of your config files for Quantum, since they will now be pointing to the controller IP instead of the loopback address. Stay tuned for an update on this once I’ve setup a dedicated network node. For now, just know that you will be slightly limited in what you can and can’t do with Quantum following the instructions in this document and the RedHat guide.
When you run the quantum-server-setup command, and tell it to update the nova configuration files, make sure you double check those files. You will need to update the quantum configuration settings:
quantum_admin_username = openstack_network quantum_admin_password = password quantum_admin_auth_url = http://127.0.0.1:35357/v2.0/ quantum_auth_strategy = keystone quantum_admin_tenant_name = openstack_network quantum_url = http://localhost:9696/
You also might need to make changes to the quantum config files. Be sure to update your tenant, user, password and region in the quantum plugin.ini, dhcp_agent.ini, l3-agent.ini, api-paste.ini, and quantum.conf files. Case is important, even in the region. Make sure that your auth_region = regionOne and not RegionOne. Also, put the correct SQL credentials in the /etc/quantum/plugins/openvswitch/ovs_quantum_plugin.ini file:
sql_connection = mysql://root:firstname.lastname@example.org/ovs_quantum
Feel free to ignore the “lokkit” commands if you’ve been following this guide as the firewall will have been disabled.
When enabling VNC console access (section 12.2.2), the VNC options will all be appended to the end of the nova.conf file. This is problematic because the VNC options will end up in the [keystone_authtoken] section where they will be completely ignored. Cut and paste all of the VNC related settings and put them before the [keystone_authtoken] section. Also note that the “novncproxy_base_url” should be the outside interface of your controller node that is reachable by end users. When a user clicks on the VNC console for an instance, the base url is what their web browser will try to connect to.
# VNC novncproxy_host = 0.0.0.0 novncproxy_port = 6080 novncproxy_base_url=http://fqdn.of.your.server:6080/vnc_auto.html vnc_enabled=true vncserver_listen=<public IP of your controller> vncserver_proxyclient_address=<public IP of your controller> [keystone_authtoken] admin_tenant_name = %SERVICE_TENANT_NAME% admin_user = %SERVICE_USER% admin_password = %SERVICE_PASSWORD% auth_host = 127.0.0.1 auth_port = 35357 auth_protocol = http
If you’re using Quantum for your networking, ignore sections 18 and 19. They are only applicable to Nova networking, which is disabled. Also note that when attaching a Cinder volume to your instances, you may need to specify a mount point other than /dev/vdc.
If you’re having problems with something in OpenStack, tail all of the log files for that particular service while you recreate the problem. For example, if you’re having a problem with Quantum:
$ sudo tail -f /var/log/quantum/*.log
It’s also not a bad idea to check all of your .conf and .ini files for any lingering default or bad credentials that weren’t updated. For example, I had an issue where the region for the quantum user had been changed to RegionOne instead of regionOne. To check all of the files for that mistake, I used the following commands:
$ sudo grep RegionOne /etc/quantum/* $ sudo grep RegionOne /etc/nova/* $ sudo grep RegionOne /etc/cinder/*
Please let me know if any of these suggestions helped you out, and if you have anything to add, definitely let me know so I can update the doc, or include the suggestion in a future post. I plan on following up with articles on creating a network node, creating compute nodes, and setting up EMC storage to work with OpenStack.
Cisco just implemented VXLAN capabilities into the newest version of the Nexus 1000v, and I was eager to try them out. VXLAN won’t be functionally integrated with VMware until a patch is released, but in the meantime you can still play around with VXLAN’s to see how they work on the Nexus 1000v.
I’m going to assume that you’ve already got a working VMware environment with two ESX hosts using a 1000v for virtual switching, and a multicast enabled physical switch connecting the two hosts (we’ll be using a Cisco 3750.) You should already have management VMKNICs configured on the ESX hosts. If you need any docs, Cisco’s configuration guides are available here, and VMware’s docs are here.
If you haven’t already, configure ip routing, and multicast routing on your upstream physical switch (you’ll need the ipservices license from Cisco for the multicast routing):
3750(config)# ip multicast-routing distributed 3750(config)# ip routing
Next you’ll need to create two VLAN’s to serve as the VXLAN transport VLAN’s between the ESX hosts and the directly connected physical switch (which will be serving as our router.) I know it seems silly not to use the same VLAN for your VXLAN transport and just trunk it to both ESX’s, but we’re configuring it this way to show that the VXLAN traffic can cross layer 3 boundaries.
Create the VLAN interfaces on your physical switch, and add the new VLAN’s to the trunk ports going to your ESX hosts. In this example the VLAN is being added to a port-channel, but it will work on any Ethernet interface. VXLAN also requires PIM and proxy-ARP on gateway interfaces. Proxy-ARP is enabled by default, so if you’ve previously disabled it, you’ll need to re-enable it with the ip proxy-arp command.
3750(config)# interface vlan 100 3750(config-if)# ip address 10.1.100.1 255.255.255.0 3750(config-if)# ip pim sparse-dense-mode 3750(config-if)# no shut 3750(config)# interface vlan 200 3750(config-if)# ip address 10.1.200.1 255.255.255.0 3750(config-if)# ip pim sparse-dense-mode 3750(config-if)# no shut 3750(config-if)# interface port-channel 1 3750(config-if)# switchport trunk allowed vlan add 100 3750(config-if)# interface port-channel 2 3750(config-if)# switchport trunk allowed vlan add 200
Create the new VLAN’s in your Nexus 1000v:
1000v(config)# vlan 100 1000v(config-vlan)# name vxlan-transport-1 1000v(config-vlan)# vlan 200 1000v(config-vlan)# name vxlan-transport-2
Since the two ESX hosts will be trunked two different VXLAN transport VLAN’s, we’ll configure two upstream port-profiles. For this example, assume that we used VLAN 10 as the management VLAN. You only need the channel-group auto mode on command if you’re applying this port-profile to multiple uplinks using port-channel. We’re also increasing the MTU to 1550 as VXLAN encapsulation adds 50 bytes onto outgoing packets.
1000v(config)# port-profile type ethernet esx-1-uplink 1000v(config-port-prof)# vmware port-group 1000v(config-port-prof)# switchport mode trunk 1000v(config-port-prof)# switchport trunk allowed vlan 10,100 1000v(config-port-prof)# mtu 1550 1000v(config-port-prof)# channel-group auto mode on 1000v(config-port-prof)# no shutdown 1000v(config-port-prof)# system vlan 10,100 1000v(config-port-prof)# state enabled 1000v(config-port-prof)# port-profile type ethernet esx-2-uplink 1000v(config-port-prof)# vmware port-group 1000v(config-port-prof)# switchport mode trunk 1000v(config-port-prof)# switchport trunk allowed vlan 10,200 1000v(config-port-prof)# mtu 1550 1000v(config-port-prof)# channel-group auto mode on 1000v(config-port-prof)# no shutdown 1000v(config-port-prof)# system vlan 10,200 1000v(config-port-prof)# state enabled
MTU side-note: We are using a 3750 as the upstream switch/router which has jumbo frame support of 9198 bytes on gigabit interfaces. However, the routing MTU is set to 1500, and the system MTU is set to 1500. All switches that are to carry VXLAN traffic must have their MTU increased to 1550 to prevent dropped packets, so we’ll do that now. Changing your system MTU requires a reboot on 3750’s.
3750(config)#system mtu routing 9198 3750(config)#system mtu 1550 Changes to the system MTU will not take effect until the next reload is done 3750(config)#exit 3750#show system mtu System MTU size is 1500 bytes On next reload, System MTU will be 1550 bytes System Jumbo MTU size is 9198 bytes System Alternate MTU size is 1500 bytes Routing MTU size is 9198 bytes 3750#write memory Building configuration... [OK] 3750#reload Proceed with reload? [confirm]
After you’ve created these two uplink port-profiles and increased your MTU’s, in vSphere you’ll need to move the physical NIC’s of ESX hosts 1 and 2 to their new uplink port-profiles.
Then you’ll need to create port-profiles for the VMKNIC’s that will be on each ESX host. These VMKNIC’s will exist in the “VXLAN transport VLAN’s” and will be the source/destination interface of all VXLAN traffic that egresses/ingresses the ESX host. For this guide, I’m creating a separate port-profile for each ESX host, but don’t think about it that way. Think about it as creating a port-profile for each VXLAN transport VLAN. In a production environment, you’d likely have multiple VXLAN transport VLAN’s with multiple ESX hosts in those VLAN’s that would use the same port-profile.
1000v(config)# port-profile vxlan-1-vmknic 1000v(config-port-prof)# vmware port-group 1000v(config-port-prof)# switchport mode access 1000v(config-port-prof)# switchport access vlan 100 1000v(config-port-prof)# capability vxlan 1000v(config-port-prof)# no shutdown 1000v(config-port-prof)# state enabled 1000v(config-port-prof)# port-profile vxlan-2-vmknic 1000v(config-port-prof)# vmware port-group 1000v(config-port-prof)# switchport mode access 1000v(config-port-prof)# switchport access vlan 200 1000v(config-port-prof)# capability vxlan 1000v(config-port-prof)# no shutdown 1000v(config-port-prof)# state enabled
Next, go to your vSphere client and create a VMKNIC on your first ESX host using the “vxlan-1-vmknic” port-profile. Make sure that the IP address you use for the VMKNIC is in the VLAN 100 IP space. Repeat for the second ESX host making sure to use “vxlan-2-vmknic” and assigning an IP address in VLAN 200.
After creating the VMKNICS, go back to the Nexus 1000 and enable VXLAN’s:
1000v(config)# feature segmentation
Now we’re going to create a VXLAN for our tenant. In NX-OS, this is called a bridge-domain. Give it a useful name; we’re going to use “tenant-1.” The segment ID is analogous to a VLAN ID. You can’t use a value less than 4096 for the segment ID as that range is reserved for VLAN’s. We also need to specify a multicast address for the bridge-domain to use. Any broadcasts that need to go from a VXLAN on one ESX host to another will be converted in to multicast traffic and will be sent to that group.
1000v(config)# bridge-domain tenant-1 1000v(config-bd)# segment id 5001 1000v(config-bd)# group 18.104.22.168
Now that we have a VXLAN, we need to configure a port-profile to use it. This port profile will be what you attach to your tenant VM’s. The only difference between a VXLAN port-profile and a standard VLAN port-profile is that you’ll be specifying an access bridge-domain instead of an access VLAN.
1000v(config)# port-profile tenant-1-vxlan 1000v(config-port-prof)# vmware port-group 1000v(config-port-prof)# switchport mode access 1000v(config-port-prof)# switchport access bridge-domain tenant-1 1000v(config-port-prof)# no shut 1000v(config-port-prof)# state enabled
Once you’ve created this port-profile, assign it to a VM on each ESX host. Make sure the IP space you’ve configured on your VM’s is in the same network, and you should have connectivity between the two!
The following commands will verify that your VXLAN’s are configured correctly:
1000v# show bridge-domain Bridge-domain tenant-1 (2 ports in all) Segment ID: 5001 (Manual/Active) Group IP: 22.214.171.124 State: UP Mac learning: Enabled Veth24, Veth25
From that command, you can see the Vethernet interfaces that are associated with the bridge-domain, you can verify that they are configured correctly as well:
1000v# show interface vethernet24 Vethernet24 is up Port description is Tenant1 VM 1, Network Adapter 1 Hardware: Virtual, address: 0050.568d.2738 (bia 0050.568d.2738) Owner is VM "Tenant1 VM 1", adapter is Network Adapter 1 Active on module 4 VMware DVS port 320 Port-Profile is tenant-1-vxlan Port mode is access 5 minute input rate 0 bits/second, 0 packets/second 5 minute output rate 0 bits/second, 0 packets/second Rx 64166 Input Packets 2184 Unicast Packets 198 Multicast Packets 62140 Broadcast Packets 3941292 Bytes Tx 2217 Output Packets 2111 Unicast Packets 42 Multicast Packets 64 Broadcast Packets 108 Flood Packets 214312 Bytes 0 Input Packet Drops 0 Output Packet Drops 1000v# show interface vethernet25 Vethernet25 is up Port description is Tenant1 VM 2, Network Adapter 1 Hardware: Virtual, address: 0050.568d.2739 (bia 0050.568d.2739) Owner is VM "Tenant1 VM 2", adapter is Network Adapter 1 Active on module 3 VMware DVS port 321 Port-Profile is tenant-1-vxlan Port mode is access 5 minute input rate 0 bits/second, 0 packets/second 5 minute output rate 0 bits/second, 0 packets/second Rx 8745 Input Packets 2179 Unicast Packets 162 Multicast Packets 6601 Broadcast Packets 614518 Bytes Tx 2194 Output Packets 2113 Unicast Packets 46 Multicast Packets 35 Broadcast Packets 81 Flood Packets 213056 Bytes 0 Input Packet Drops 0 Output Packet Drops
You can also verify that the virtual machines think they are on the same network even though they traversed a layer 3 hop by crosschecking the vNIC MAC addresses and VM ARP tables. You should see the MAC address of the VM in your ARP table, not the MAC address of an upstream router:
admin@tenant1vm1:~$ arp -n Address HWtype HWaddress Flags Mask Iface 10.1.21.22 ether 00:50:56:8d:27:39 C eth0 cie@cie-ubuntu-pcap:~$ ifconfig | grep HWaddr eth0 Link encap:Ethernet HWaddr 00:50:56:8d:27:38
admin@tenant1vm2:~$ arp -n Address HWtype HWaddress Flags Mask Iface 10.1.21.21 ether 00:50:56:8d:27:38 C eth0 cie@cie-ubuntu-pcap:~$ ifconfig | grep HWaddr eth0 Link encap:Ethernet HWaddr 00:50:56:8d:27:39
Invariably during any new implementation, I’ve had to “escalate to Google” to find a solution to whatever networking problem I was grappling with. Answers would come from blogs and forums as frequently as they would from official vendor documentation. With that in mind, I figured I’d try to pay back the networking community by posting some of my own insights. If my posts can help one person get out of the office on-time, I will have considered it a worthy endeavor.