Archive | How-to’s RSS for this section

Getting Started with RedHat’s OpenStack Preview

I’ve been intrigued by OpenStack for some time now, and have tried setting up test beds on numerous occasions. These tests inevitably failed to get off the ground, but I kept coming back to try again. Most of my difficulties have been brought on by poor documentation, and a lack of cohesive software packaging. Many OpenStack guides try to handle everything with scripts, or Chef/Puppet combinations. This is great if you are already familiar with OpenStack, but if you aren’t there can be many problems on the back end of those provisioning tools that you are either unaware of, or unable to resolve. To really understand how all the pieces of OpenStack fit together, you need to manually install everything, so I set out to find a guide that would let me do just that.

The guide/distro combo that I had the best luck with was RedHat’s OpenStack Folsom Preview, and the accompanying documentation found here. The documentation will get you close to a working finished product, but there were some hiccups that I found and documented along the way. The good news is that RedHat is constantly updating their documentation. Hopefully as they update, less and less of these issues will continue to be a problem.

At this time, I’ve gotten the following setup: a single-node OpenStack environment with VM’s and images. The following components are being used:

  • Keystone (Identity, Token, Catalog, and Policy)
  • Glance (Image)
  • Cinder (Block Storage)
  • Nova (Compute)
  • Quantum (Network)
  • Horizon (Dashboard)

My future plans include adding several compute nodes, configuring Swift object-storage, configuring a dedicated Quantum networking node, and most importantly, integrating EMC storage with Cinder and Swift for block and object storage.

If you have already checked out the RedHat guide, you’re probably wondering why I don’t already have Swift working The answer is that I skipped over Swift for now. You might also be wondering why I want a dedicated network node. Quantum and RHEL 6.4 is hamstrung out of the box. If you follow the guide, you’ll be installing a networking component that has some functionality that does not work. This is due to the fact that RHEL 6.4’s kernel does not have support for namespaces. Without support for namespaces, there can only be one virtual router per node, and you cannot have overlapping IP address ranges. For more information, check out the bug here, which hopefully will be fixed in RHEL 6.5. I plan on using a different distro (either Fedora or Ubuntu) for the Quantum network node.

With that out of the way, let’s get to the “gotchas.” I’ve tried to vet these as much as possible, but some of them may not be best practice. They did work for me, but your mileage may vary. The notes are in chronological order coinciding with the RedHat Getting Started Guide. I would recommend reading this in its entirety before beginning the install, as you’ll need to know some things before you install RHEL.

Before Starting

If you want to have a proper Cinder install, you’ll need to setup your partitions during the RHEL install. To do this, create two physical volumes for LVM during the partitioning phase of the RHEL install. One will be for your OS, the other will be for your Cinder volumes, so size them appropriately for your deployment. You’ll also need to create a boot partition in addition to the two physical volumes for LVM. This boot partition can be small, around 250 MB is fine. For RedHat’s recommended partitioning scheme info, check here. Once you’ve created the two physical volumes for LVM, you’ll need to create a volume-group for each one. Name one of them “cinder-volumes” and name the other whatever you want. The other one is where you’ll create your /home, root, and swap LVM partitions.

For the sake of simplicity, I’d recommend disabling a couple of security measures. First set SELinux to “permissive” mode by editing the /etc/selinux/config file to have the following set:

SELINUX=permissive

Next, disable iptables:

$ chkconfig iptables off
$ service iptables stop

Keystone

Your first issues will likely arise right out of the gate with the method that they use to set the admin_token. I could not get their service token to work, so I simply edited the /etc/keystone/keystone.conf file to manually set the admin token. Perform these steps BEFORE exporting the environment variables as some of them become redundant after doing this.

Open the keystone.conf file and set the “admin_token” to whatever you’d like. Then, set the environment variable to the same thing. For example:

$ sudo vi /etc/keystone/keystone.conf
admin_token = TOKEN_PASSWORD

Save the file and exit, and then set the environment variable to the same:

$ export SERVICE_TOKEN=TOKEN_PASSWORD

Don’t run the commands that are crossed out:

$ export SERVICE_TOKEN=$(openssl rand -hex 10)
$ export SERVICE_ENDPOINT=http://127.0.0.1:35357/v2.0
$ echo $SERVICE_TOKEN > /tmp/ks_admin_token
$ sudo openstack-config --set /etc/keystone/keystone.conf \
DEFAULT admin_token $SERVICE_TOKEN

Glance

In section 9.1, they want you to run through the process of creating your own image to import. This is a lot of work when all you really need is an image. I recommend just downloading a CirrOS image and importing that instead:

$ wget https://launchpad.net/cirros/trunk/0.3.0/+download/cirros-0.3.0-x86_64-uec.tar.gz

Cinder

RedHat’s guide will have you create a temporary file system for Cinder to use. I opted instead to create a “cinder-volumes” volume group using LVM partitioning during the initial RHEL installation. This is the default volume group that Cinder looks for out of the box.

When you get to the part of the Cinder configuration where they want you to setup the volume groups, don’t run the following commands that have been crossed out:

$ sudo grep -q /etc/cinder/volumes /etc/tgt/targets.conf || sudo sed -i '1iinclude /etc/cinder/volumes/*' /etc/tgt/targets.conf
$ sudo service tgtd start
$ sudo chkconfig tgtd on
$ sudo truncate --size 20G ~/cinder-volumes
$ sudo losetup -fv ~/cinder-volumes
$ sudo vgcreate cinder-volumes /dev/loop0
$ sudo vgdisplay cinder-volumes

After this step, here’s some things to check:

In the /etc/cinder/cinder.conf file, set the following options:

volume_name_template = volume-%s
volume_group = cinder-volumes
state_path = /var/lib/cinder/
volumes_dir = /var/lib/cinder/volumes

In /etc/tgt/targets.conf, make sure this command is at the top:

include /etc/cinder/volumes/*

In /etc/tgt/conf.d/cinder.conf uncomment the following line:

include /var/lib/cinder/volumes/*

OpenStack Networking (Quantum)

Unfortunately, I still don’t have Quantum working the way that it should, and I believe that’s due to RHEL 6.4’s lack of namespaces support. Without namespaces, there can only be one virtual router, and duplicate IP addresses aren’t supported. Duplicate IP address support is very important for multi-tenancy.

If you want full Quantum functionality, my recommendation is instead to configure it on its own node with a Linux distro that does support namespaces. Apparently Fedora 17 and Ubuntu 12.04 and later support namespaces. This will change a lot of your config files for Quantum, since they will now be pointing to the controller IP instead of the loopback address. Stay tuned for an update on this once I’ve setup a dedicated network node. For now, just know that you will be slightly limited in what you can and can’t do with Quantum following the instructions in this document and the RedHat guide.

When you run the quantum-server-setup command, and tell it to update the nova configuration files, make sure you double check those files. You will need to update the quantum configuration settings:

quantum_admin_username = openstack_network
quantum_admin_password = password
quantum_admin_auth_url = http://127.0.0.1:35357/v2.0/
quantum_auth_strategy = keystone
quantum_admin_tenant_name = openstack_network
quantum_url = http://localhost:9696/

You also might need to make changes to the quantum config files. Be sure to update your tenant, user, password and region in the quantum plugin.ini, dhcp_agent.ini, l3-agent.ini, api-paste.ini, and quantum.conf files. Case is important, even in the region. Make sure that your auth_region = regionOne and not RegionOne. Also, put the correct SQL credentials in the /etc/quantum/plugins/openvswitch/ovs_quantum_plugin.ini file:

sql_connection = mysql://root:password@fqdn.of.your.server/ovs_quantum

Horizon

Feel free to ignore the “lokkit” commands if you’ve been following this guide as the firewall will have been disabled.

When enabling VNC console access (section 12.2.2), the VNC options will all be appended to the end of the nova.conf file. This is problematic because the VNC options will end up in the [keystone_authtoken] section where they will be completely ignored. Cut and paste all of the VNC related settings and put them before the [keystone_authtoken] section. Also note that the “novncproxy_base_url” should be the outside interface of your controller node that is reachable by end users. When a user clicks on the VNC console for an instance, the base url is what their web browser will try to connect to.

# VNC
novncproxy_host = 0.0.0.0
novncproxy_port = 6080
novncproxy_base_url=http://fqdn.of.your.server:6080/vnc_auto.html
vnc_enabled=true
vncserver_listen=<public IP of your controller>
vncserver_proxyclient_address=<public IP of your controller>
[keystone_authtoken]
admin_tenant_name = %SERVICE_TENANT_NAME%
admin_user = %SERVICE_USER%
admin_password = %SERVICE_PASSWORD%
auth_host = 127.0.0.1
auth_port = 35357
auth_protocol = http

Using OpenStack

If you’re using Quantum for your networking, ignore sections 18 and 19. They are only applicable to Nova networking, which is disabled. Also note that when attaching a Cinder volume to your instances, you may need to specify a mount point other than /dev/vdc.

Troubleshooting

If you’re having problems with something in OpenStack, tail all of the log files for that particular service while you recreate the problem. For example, if you’re having a problem with Quantum:

$ sudo tail -f /var/log/quantum/*.log

It’s also not a bad idea to check all of your .conf and .ini files for any lingering default or bad credentials that weren’t updated. For example, I had an issue where the region for the quantum user had been changed to RegionOne instead of regionOne. To check all of the files for that mistake, I used the following commands:

$ sudo grep RegionOne /etc/quantum/*
$ sudo grep RegionOne /etc/nova/*
$ sudo grep RegionOne /etc/cinder/*

Wrap Up

Please let me know if any of these suggestions helped you out, and if you have anything to add, definitely let me know so I can update the doc, or include the suggestion in a future post. I plan on following up with articles on creating a network node, creating compute nodes, and setting up EMC storage to work with OpenStack.

Advertisements

Basic VXLAN Configuration

Cisco just implemented VXLAN capabilities into the newest version of the Nexus 1000v, and I was eager to try them out. VXLAN won’t be functionally integrated with VMware until a patch is released, but in the meantime you can still play around with VXLAN’s to see how they work on the Nexus 1000v.

Starting Point

I’m going to assume that you’ve already got a working VMware environment with two ESX hosts using a 1000v for virtual switching, and a multicast enabled physical switch connecting the two hosts (we’ll be using a Cisco 3750.) You should already have management VMKNICs configured on the ESX hosts. If you need any docs, Cisco’s configuration guides are available here, and VMware’s docs are here.

Configuration

If you haven’t already, configure ip routing, and multicast routing on your upstream physical switch (you’ll need the ipservices license from Cisco for the multicast routing):

3750(config)# ip multicast-routing distributed
3750(config)# ip routing

Next you’ll need to create two VLAN’s to serve as the VXLAN transport VLAN’s between the ESX hosts and the directly connected physical switch (which will be serving as our router.) I know it seems silly not to use the same VLAN for your VXLAN transport and just trunk it to both ESX’s, but we’re configuring it this way to show that the VXLAN traffic can cross layer 3 boundaries.

Create the VLAN interfaces on your physical switch, and add the new VLAN’s to the trunk ports going to your ESX hosts. In this example the VLAN is being added to a port-channel, but it will work on any Ethernet interface. VXLAN also requires PIM and proxy-ARP on gateway interfaces. Proxy-ARP is enabled by default, so if you’ve previously disabled it, you’ll need to re-enable it with the ip proxy-arp command.

3750(config)# interface vlan 100
3750(config-if)# ip address 10.1.100.1 255.255.255.0
3750(config-if)# ip pim sparse-dense-mode
3750(config-if)# no shut
3750(config)# interface vlan 200
3750(config-if)# ip address 10.1.200.1 255.255.255.0
3750(config-if)# ip pim sparse-dense-mode
3750(config-if)# no shut
3750(config-if)# interface port-channel 1
3750(config-if)# switchport trunk allowed vlan add 100
3750(config-if)# interface port-channel 2
3750(config-if)# switchport trunk allowed vlan add 200

Create the new VLAN’s in your Nexus 1000v:

1000v(config)# vlan 100
1000v(config-vlan)# name vxlan-transport-1
1000v(config-vlan)# vlan 200
1000v(config-vlan)# name vxlan-transport-2

Since the two ESX hosts will be trunked two different VXLAN transport VLAN’s, we’ll configure two upstream port-profiles. For this example, assume that we used VLAN 10 as the management VLAN. You only need the channel-group auto mode on command if you’re applying this port-profile to multiple uplinks using port-channel. We’re also increasing the MTU to 1550 as VXLAN encapsulation adds 50 bytes onto outgoing packets.

1000v(config)# port-profile type ethernet esx-1-uplink
1000v(config-port-prof)# vmware port-group
1000v(config-port-prof)# switchport mode trunk
1000v(config-port-prof)# switchport trunk allowed vlan 10,100
1000v(config-port-prof)# mtu 1550
1000v(config-port-prof)# channel-group auto mode on
1000v(config-port-prof)# no shutdown
1000v(config-port-prof)# system vlan 10,100
1000v(config-port-prof)# state enabled
1000v(config-port-prof)# port-profile type ethernet esx-2-uplink
1000v(config-port-prof)# vmware port-group
1000v(config-port-prof)# switchport mode trunk
1000v(config-port-prof)# switchport trunk allowed vlan 10,200
1000v(config-port-prof)# mtu 1550
1000v(config-port-prof)# channel-group auto mode on
1000v(config-port-prof)# no shutdown
1000v(config-port-prof)# system vlan 10,200
1000v(config-port-prof)# state enabled

MTU side-note: We are using a 3750 as the upstream switch/router which has jumbo frame support of 9198 bytes on gigabit interfaces. However, the routing MTU is set to 1500, and the system MTU is set to 1500. All switches that are to carry VXLAN traffic must have their MTU increased to 1550 to prevent dropped packets, so we’ll do that now. Changing your system MTU requires a reboot on 3750’s.

3750(config)#system mtu routing 9198
3750(config)#system mtu 1550
Changes to the system MTU will not take effect until the next reload is done
3750(config)#exit
3750#show system mtu       

System MTU size is 1500 bytes
On next reload, System MTU will be 1550 bytes
System Jumbo MTU size is 9198 bytes
System Alternate MTU size is 1500 bytes
Routing MTU size is 9198 bytes
3750#write memory
Building configuration...
[OK]
3750#reload
Proceed with reload? [confirm]

After you’ve created these two uplink port-profiles and increased your MTU’s, in vSphere you’ll need to move the physical NIC’s of ESX hosts 1 and 2 to their new uplink port-profiles.

  

Then you’ll need to create port-profiles for the VMKNIC’s that will be on each ESX host. These VMKNIC’s will exist in the “VXLAN transport VLAN’s” and will be the source/destination interface of all VXLAN traffic that egresses/ingresses the ESX host. For this guide, I’m creating a separate port-profile for each ESX host, but don’t think about it that way. Think about it as creating a port-profile for each VXLAN transport VLAN. In a production environment, you’d likely have multiple VXLAN transport VLAN’s with multiple ESX hosts in those VLAN’s that would use the same port-profile.

1000v(config)# port-profile vxlan-1-vmknic
1000v(config-port-prof)# vmware port-group
1000v(config-port-prof)# switchport mode access
1000v(config-port-prof)# switchport access vlan 100
1000v(config-port-prof)# capability vxlan
1000v(config-port-prof)# no shutdown
1000v(config-port-prof)# state enabled
1000v(config-port-prof)# port-profile vxlan-2-vmknic
1000v(config-port-prof)# vmware port-group
1000v(config-port-prof)# switchport mode access
1000v(config-port-prof)# switchport access vlan 200
1000v(config-port-prof)# capability vxlan
1000v(config-port-prof)# no shutdown
1000v(config-port-prof)# state enabled

Next, go to your vSphere client and create a VMKNIC on your first ESX host using the “vxlan-1-vmknic” port-profile. Make sure that the IP address you use for the VMKNIC is in the VLAN 100 IP space. Repeat for the second ESX host making sure to use “vxlan-2-vmknic” and assigning an IP address in VLAN 200.

After creating the VMKNICS, go back to the Nexus 1000 and enable VXLAN’s:

1000v(config)# feature segmentation

Now we’re going to create a VXLAN for our tenant. In NX-OS, this is called a bridge-domain. Give it a useful name; we’re going to use “tenant-1.” The segment ID is analogous to a VLAN ID. You can’t use a value less than 4096 for the segment ID as that range is reserved for VLAN’s. We also need to specify a multicast address for the bridge-domain to use. Any broadcasts that need to go from a VXLAN on one ESX host to another will be converted in to multicast traffic and will be sent to that group.

1000v(config)# bridge-domain tenant-1
1000v(config-bd)# segment id 5001
1000v(config-bd)# group 239.1.1.1

Now that we have a VXLAN, we need to configure a port-profile to use it. This port profile will be what you attach to your tenant VM’s. The only difference between a VXLAN port-profile and a standard VLAN port-profile is that you’ll be specifying an access bridge-domain instead of an access VLAN.

1000v(config)# port-profile tenant-1-vxlan
1000v(config-port-prof)# vmware port-group
1000v(config-port-prof)# switchport mode access
1000v(config-port-prof)# switchport access bridge-domain tenant-1
1000v(config-port-prof)# no shut
1000v(config-port-prof)# state enabled

Once you’ve created this port-profile, assign it to a VM on each ESX host. Make sure the IP space you’ve configured on your VM’s is in the same network, and you should have connectivity between the two!

Verification

The following commands will verify that your VXLAN’s are configured correctly:

1000v# show bridge-domain
Bridge-domain tenant-1 (2 ports in all)
Segment ID: 5001 (Manual/Active)
Group IP: 239.1.1.1
State: UP               Mac learning: Enabled
Veth24, Veth25

From that command, you can see the Vethernet interfaces that are associated with the bridge-domain, you can verify that they are configured correctly as well:

1000v# show interface vethernet24
Vethernet24 is up
  Port description is Tenant1 VM 1, Network Adapter 1
  Hardware: Virtual, address: 0050.568d.2738 (bia 0050.568d.2738)
  Owner is VM "Tenant1 VM 1", adapter is Network Adapter 1
  Active on module 4
  VMware DVS port 320
  Port-Profile is tenant-1-vxlan
  Port mode is access
  5 minute input rate 0 bits/second, 0 packets/second
  5 minute output rate 0 bits/second, 0 packets/second
  Rx
    64166 Input Packets 2184 Unicast Packets
    198 Multicast Packets 62140 Broadcast Packets
    3941292 Bytes
  Tx
    2217 Output Packets 2111 Unicast Packets
    42 Multicast Packets 64 Broadcast Packets 108 Flood Packets
    214312 Bytes
    0 Input Packet Drops 0 Output Packet Drops

1000v# show interface vethernet25
Vethernet25 is up
  Port description is Tenant1 VM 2, Network Adapter 1
  Hardware: Virtual, address: 0050.568d.2739 (bia 0050.568d.2739)
  Owner is VM "Tenant1 VM 2", adapter is Network Adapter 1
  Active on module 3
  VMware DVS port 321
  Port-Profile is tenant-1-vxlan
  Port mode is access
  5 minute input rate 0 bits/second, 0 packets/second
  5 minute output rate 0 bits/second, 0 packets/second
  Rx
    8745 Input Packets 2179 Unicast Packets
    162 Multicast Packets 6601 Broadcast Packets
    614518 Bytes
  Tx
    2194 Output Packets 2113 Unicast Packets
    46 Multicast Packets 35 Broadcast Packets 81 Flood Packets
    213056 Bytes
    0 Input Packet Drops 0 Output Packet Drops

You can also verify that the virtual machines think they are on the same network even though they traversed a layer 3 hop by crosschecking the vNIC MAC addresses and VM ARP tables. You should see the MAC address of the VM in your ARP table, not the MAC address of an upstream router:

admin@tenant1vm1:~$ arp -n
Address                  HWtype  HWaddress           Flags Mask            Iface
10.1.21.22               ether   00:50:56:8d:27:39   C                     eth0
cie@cie-ubuntu-pcap:~$ ifconfig | grep HWaddr
eth0      Link encap:Ethernet  HWaddr 00:50:56:8d:27:38
admin@tenant1vm2:~$ arp -n
Address                  HWtype  HWaddress           Flags Mask            Iface
10.1.21.21               ether   00:50:56:8d:27:38   C                     eth0
cie@cie-ubuntu-pcap:~$ ifconfig | grep HWaddr
eth0      Link encap:Ethernet  HWaddr 00:50:56:8d:27:39