Basic VXLAN Configuration

Cisco just implemented VXLAN capabilities into the newest version of the Nexus 1000v, and I was eager to try them out. VXLAN won’t be functionally integrated with VMware until a patch is released, but in the meantime you can still play around with VXLAN’s to see how they work on the Nexus 1000v.

Starting Point

I’m going to assume that you’ve already got a working VMware environment with two ESX hosts using a 1000v for virtual switching, and a multicast enabled physical switch connecting the two hosts (we’ll be using a Cisco 3750.) You should already have management VMKNICs configured on the ESX hosts. If you need any docs, Cisco’s configuration guides are available here, and VMware’s docs are here.

Configuration

If you haven’t already, configure ip routing, and multicast routing on your upstream physical switch (you’ll need the ipservices license from Cisco for the multicast routing):

3750(config)# ip multicast-routing distributed
3750(config)# ip routing

Next you’ll need to create two VLAN’s to serve as the VXLAN transport VLAN’s between the ESX hosts and the directly connected physical switch (which will be serving as our router.) I know it seems silly not to use the same VLAN for your VXLAN transport and just trunk it to both ESX’s, but we’re configuring it this way to show that the VXLAN traffic can cross layer 3 boundaries.

Create the VLAN interfaces on your physical switch, and add the new VLAN’s to the trunk ports going to your ESX hosts. In this example the VLAN is being added to a port-channel, but it will work on any Ethernet interface. VXLAN also requires PIM and proxy-ARP on gateway interfaces. Proxy-ARP is enabled by default, so if you’ve previously disabled it, you’ll need to re-enable it with the ip proxy-arp command.

3750(config)# interface vlan 100
3750(config-if)# ip address 10.1.100.1 255.255.255.0
3750(config-if)# ip pim sparse-dense-mode
3750(config-if)# no shut
3750(config)# interface vlan 200
3750(config-if)# ip address 10.1.200.1 255.255.255.0
3750(config-if)# ip pim sparse-dense-mode
3750(config-if)# no shut
3750(config-if)# interface port-channel 1
3750(config-if)# switchport trunk allowed vlan add 100
3750(config-if)# interface port-channel 2
3750(config-if)# switchport trunk allowed vlan add 200

Create the new VLAN’s in your Nexus 1000v:

1000v(config)# vlan 100
1000v(config-vlan)# name vxlan-transport-1
1000v(config-vlan)# vlan 200
1000v(config-vlan)# name vxlan-transport-2

Since the two ESX hosts will be trunked two different VXLAN transport VLAN’s, we’ll configure two upstream port-profiles. For this example, assume that we used VLAN 10 as the management VLAN. You only need the channel-group auto mode on command if you’re applying this port-profile to multiple uplinks using port-channel. We’re also increasing the MTU to 1550 as VXLAN encapsulation adds 50 bytes onto outgoing packets.

1000v(config)# port-profile type ethernet esx-1-uplink
1000v(config-port-prof)# vmware port-group
1000v(config-port-prof)# switchport mode trunk
1000v(config-port-prof)# switchport trunk allowed vlan 10,100
1000v(config-port-prof)# mtu 1550
1000v(config-port-prof)# channel-group auto mode on
1000v(config-port-prof)# no shutdown
1000v(config-port-prof)# system vlan 10,100
1000v(config-port-prof)# state enabled
1000v(config-port-prof)# port-profile type ethernet esx-2-uplink
1000v(config-port-prof)# vmware port-group
1000v(config-port-prof)# switchport mode trunk
1000v(config-port-prof)# switchport trunk allowed vlan 10,200
1000v(config-port-prof)# mtu 1550
1000v(config-port-prof)# channel-group auto mode on
1000v(config-port-prof)# no shutdown
1000v(config-port-prof)# system vlan 10,200
1000v(config-port-prof)# state enabled

MTU side-note: We are using a 3750 as the upstream switch/router which has jumbo frame support of 9198 bytes on gigabit interfaces. However, the routing MTU is set to 1500, and the system MTU is set to 1500. All switches that are to carry VXLAN traffic must have their MTU increased to 1550 to prevent dropped packets, so we’ll do that now. Changing your system MTU requires a reboot on 3750’s.

3750(config)#system mtu routing 9198
3750(config)#system mtu 1550
Changes to the system MTU will not take effect until the next reload is done
3750(config)#exit
3750#show system mtu       

System MTU size is 1500 bytes
On next reload, System MTU will be 1550 bytes
System Jumbo MTU size is 9198 bytes
System Alternate MTU size is 1500 bytes
Routing MTU size is 9198 bytes
3750#write memory
Building configuration...
[OK]
3750#reload
Proceed with reload? [confirm]

After you’ve created these two uplink port-profiles and increased your MTU’s, in vSphere you’ll need to move the physical NIC’s of ESX hosts 1 and 2 to their new uplink port-profiles.

  

Then you’ll need to create port-profiles for the VMKNIC’s that will be on each ESX host. These VMKNIC’s will exist in the “VXLAN transport VLAN’s” and will be the source/destination interface of all VXLAN traffic that egresses/ingresses the ESX host. For this guide, I’m creating a separate port-profile for each ESX host, but don’t think about it that way. Think about it as creating a port-profile for each VXLAN transport VLAN. In a production environment, you’d likely have multiple VXLAN transport VLAN’s with multiple ESX hosts in those VLAN’s that would use the same port-profile.

1000v(config)# port-profile vxlan-1-vmknic
1000v(config-port-prof)# vmware port-group
1000v(config-port-prof)# switchport mode access
1000v(config-port-prof)# switchport access vlan 100
1000v(config-port-prof)# capability vxlan
1000v(config-port-prof)# no shutdown
1000v(config-port-prof)# state enabled
1000v(config-port-prof)# port-profile vxlan-2-vmknic
1000v(config-port-prof)# vmware port-group
1000v(config-port-prof)# switchport mode access
1000v(config-port-prof)# switchport access vlan 200
1000v(config-port-prof)# capability vxlan
1000v(config-port-prof)# no shutdown
1000v(config-port-prof)# state enabled

Next, go to your vSphere client and create a VMKNIC on your first ESX host using the “vxlan-1-vmknic” port-profile. Make sure that the IP address you use for the VMKNIC is in the VLAN 100 IP space. Repeat for the second ESX host making sure to use “vxlan-2-vmknic” and assigning an IP address in VLAN 200.

After creating the VMKNICS, go back to the Nexus 1000 and enable VXLAN’s:

1000v(config)# feature segmentation

Now we’re going to create a VXLAN for our tenant. In NX-OS, this is called a bridge-domain. Give it a useful name; we’re going to use “tenant-1.” The segment ID is analogous to a VLAN ID. You can’t use a value less than 4096 for the segment ID as that range is reserved for VLAN’s. We also need to specify a multicast address for the bridge-domain to use. Any broadcasts that need to go from a VXLAN on one ESX host to another will be converted in to multicast traffic and will be sent to that group.

1000v(config)# bridge-domain tenant-1
1000v(config-bd)# segment id 5001
1000v(config-bd)# group 239.1.1.1

Now that we have a VXLAN, we need to configure a port-profile to use it. This port profile will be what you attach to your tenant VM’s. The only difference between a VXLAN port-profile and a standard VLAN port-profile is that you’ll be specifying an access bridge-domain instead of an access VLAN.

1000v(config)# port-profile tenant-1-vxlan
1000v(config-port-prof)# vmware port-group
1000v(config-port-prof)# switchport mode access
1000v(config-port-prof)# switchport access bridge-domain tenant-1
1000v(config-port-prof)# no shut
1000v(config-port-prof)# state enabled

Once you’ve created this port-profile, assign it to a VM on each ESX host. Make sure the IP space you’ve configured on your VM’s is in the same network, and you should have connectivity between the two!

Verification

The following commands will verify that your VXLAN’s are configured correctly:

1000v# show bridge-domain
Bridge-domain tenant-1 (2 ports in all)
Segment ID: 5001 (Manual/Active)
Group IP: 239.1.1.1
State: UP               Mac learning: Enabled
Veth24, Veth25

From that command, you can see the Vethernet interfaces that are associated with the bridge-domain, you can verify that they are configured correctly as well:

1000v# show interface vethernet24
Vethernet24 is up
  Port description is Tenant1 VM 1, Network Adapter 1
  Hardware: Virtual, address: 0050.568d.2738 (bia 0050.568d.2738)
  Owner is VM "Tenant1 VM 1", adapter is Network Adapter 1
  Active on module 4
  VMware DVS port 320
  Port-Profile is tenant-1-vxlan
  Port mode is access
  5 minute input rate 0 bits/second, 0 packets/second
  5 minute output rate 0 bits/second, 0 packets/second
  Rx
    64166 Input Packets 2184 Unicast Packets
    198 Multicast Packets 62140 Broadcast Packets
    3941292 Bytes
  Tx
    2217 Output Packets 2111 Unicast Packets
    42 Multicast Packets 64 Broadcast Packets 108 Flood Packets
    214312 Bytes
    0 Input Packet Drops 0 Output Packet Drops

1000v# show interface vethernet25
Vethernet25 is up
  Port description is Tenant1 VM 2, Network Adapter 1
  Hardware: Virtual, address: 0050.568d.2739 (bia 0050.568d.2739)
  Owner is VM "Tenant1 VM 2", adapter is Network Adapter 1
  Active on module 3
  VMware DVS port 321
  Port-Profile is tenant-1-vxlan
  Port mode is access
  5 minute input rate 0 bits/second, 0 packets/second
  5 minute output rate 0 bits/second, 0 packets/second
  Rx
    8745 Input Packets 2179 Unicast Packets
    162 Multicast Packets 6601 Broadcast Packets
    614518 Bytes
  Tx
    2194 Output Packets 2113 Unicast Packets
    46 Multicast Packets 35 Broadcast Packets 81 Flood Packets
    213056 Bytes
    0 Input Packet Drops 0 Output Packet Drops

You can also verify that the virtual machines think they are on the same network even though they traversed a layer 3 hop by crosschecking the vNIC MAC addresses and VM ARP tables. You should see the MAC address of the VM in your ARP table, not the MAC address of an upstream router:

admin@tenant1vm1:~$ arp -n
Address                  HWtype  HWaddress           Flags Mask            Iface
10.1.21.22               ether   00:50:56:8d:27:39   C                     eth0
cie@cie-ubuntu-pcap:~$ ifconfig | grep HWaddr
eth0      Link encap:Ethernet  HWaddr 00:50:56:8d:27:38
admin@tenant1vm2:~$ arp -n
Address                  HWtype  HWaddress           Flags Mask            Iface
10.1.21.21               ether   00:50:56:8d:27:38   C                     eth0
cie@cie-ubuntu-pcap:~$ ifconfig | grep HWaddr
eth0      Link encap:Ethernet  HWaddr 00:50:56:8d:27:39
Advertisements

Tags: , , , , ,

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: