Last week I invited Neeraj Malhotra, a Principal Engineer at Cisco, to present at the Bay Area Network Operators Group (BANOG) on how EVPN can be used to build multi-tenant data center fabrics. Neeraj gave a great presentation and I’m sharing his slides below.
EVPN has many use cases and this presentation focuses on EVPN as a control plane in the data center to support host/VM mobility and active/active multihoming. I’m currently looking at EVPN as alternative to replace the proprietary MLAG technology in my data centers.
The presentation abstract and slides are below. Enjoy.
Abstract:
EVPN-IRB (Integrated Routing and Bridging) is a technology that leverages BGP EVPN as common overlay control plane to enable VPN routing and bridging service over an MPLS or IP underlay fabric. Point to multi-point bridging service enables VLANs to be stretched across data center IP or MPLS fabric, while VPN routing service enables inter-subnet routing across these stretched subnets. It hence allows for flexible workloads with seamless VM mobility across the stretched subnet.
This talk will provide a tutorial of relevant EVPN constructs and procedures used to enable overlay bridged and routed connectivity between tenant workloads in a data center and compare three main design choices with respect to overlay routing architecture: – Centralized EVPN-IRB: with centralized first-hop any-cast GW on the border leafs OR DCI / DC Edge routers – Asymmetric EVPN-IRB: with distributed first-hop any-cast GW on the ToRs – Symmetric EVPN-IRB: with distributed first-hop any-cast GW on the ToRs
It will further focus on a symmetric EVPN-IRB design with distributed any-cast GW, and go thru detailed packet walks to get a good feel for how an EVPN-IRB based DC fabric works to provide any to any L2 and L3 overlay connectivity.
Someone asked me the other day how they could automate the execution of a command on multiple routers without accessing each router manually. Obviously an Ansible playbook can easily do that (or even using Ansible ad-hoc command without a playbook); or you can write a bash script with a for loop that iterates over the devices and connects to each device to run the command. But that question also got me wondering if there was a simpler way to get the job done quickly without installing any configuration management tools or messing with scripting?
It turns out the answer is Yes. You can do that using good old SSH — specifically using parallel SSH on Linux.
Parallel SSH (PSSH) is a great tool to use when you want to run single or multiple commands on more than one host or router at the same time. All what you need is a Linux host with PSSH installed and you are good to go. You can install PSSH on Ubuntu by using the Python package installer pip install pssh (if you don’t have the Python package installed, you can install it by executing apt-get install python-pip).
pssh [OPTIONS] command […]
Let’s look at a quick example. I have two routers, csr and csr3, defined in the hosts.txt file and I want PSSH to save the runnng configuration on each router. The optional -l argument tells PSSH what unsername to use while the -A argument tells it to prompt for a password (alternatively you can use private/public key pair instead of passwords).
➜ ~ pssh -h hosts.txt -l cisco -A “wr mem”
Warning: do not enter your password if anyone else has superuser privileges or access to your account.
Password:
[1] 20:11:59 [SUCCESS] csr3
[2] 20:12:00 [SUCCESS] csr
If you want to gather some data from the routers and write the output to a file, you can do so by adding the -o argument as follows:
Warning: do not enter your password if anyone else has superuser privileges or access to your account.
Password:
[1] 20:14:39 [SUCCESS] csr3
[2] 20:14:39 [SUCCESS] csr
➜ ~
➜ ~ ls /tmp/out/
csr csr3
➜ ~
➜ ~ more /tmp/out/csr
Building configuration…
Current configuration : 2643 bytes
!
! Last configuration change at 14:59:27 UTC Wed Feb 8 2017 by cisco
!
version 15.4
service timestamps debug datetime msec
service timestamps log datetime msec
no platform punt-keepalive disable-kernel-core
platform console virtual
!
hostname CSR
!
➜ ~
The PSSH utility is lightweight, simple, and does the job with minimum overhead. It also runs through routers in parallel which saves time especially when you are executing tasks that take some time to complete.
It’s amazing how much you can do with the mighty SSH. Keep PSSH in your toolbox in case you need it one day.
Did I say there is also PSCP (parallel SCP) utility which you can use to copy an image to multiple devices at the same time when you are upgrading those devices? That is your homework now, Google it and check it out.
The first half of the meetup was an introduction to Ansible for network engineers. The second half was a live demos of Ansible playbooks. You can view and download the slides below
With Cisco IOS, I had to use several modules in my playbook to be able to automate the upgrade process because there was not a single module available that could handle all the tasks.
As I was trying to expand my Ansible knowledge, I began looking at the available Ansible networking modules that run on Juniper devices. It turns out junos_package is a core module (comes installed with Ansible) and can take care of the entire process: copy the package to the device flash, install the package, commit, and reboot.
My Setup:
– Python 2.7.6 and Ansible 2.2 running on Ubuntu 14.4.5 LTS (codename: Trusty)
– Juniper vSRX running version 12.1X47-D10.4
Requirements:
– You need to have Ansible and Junos PyEZ installed. Junos PyEZ is a Python library to manage remotely and automate Junos devices.
– Netconf and SSH enabled on the Junos device.
The following playbook consists of few tasks: Ansible collects first device facts. Then the junos_package module compares the running version on the Junos device with the version defined in the “package” variable and upgrades the device if there is a mismatch. Once the device reboots with the new package, Ansible will wait until that device becomes reachable via Netconf and then attempts to ping a root DNS from the device to check internet connectivity. If the destination is not reachable, Ansible will generate a “ping failed” error.
You can of course expand this playbook to include tasks to verify that routing protocols have come up after the reboot.
Ansible is a nice tool to automate the deployment and configuration of network devices. I wrote the following playbook to automate the upgrade of Cisco IOS devices. This playbook has been tested successfully to upgrade a Cisco CSR1000v router and can be easily tweaked to support Cisco Nexus and Arista switches.
My Setup:
– Python 2.7.6 and Ansible 2.2 running on Ubuntu 14.4.5 LTS (codename: Trusty)
– Cisco CSR1000v running IOS-XE 3.10.S
Requirements:
– Obviously you must have Ansible version 2.2 or higher installed and configured properly to access via SSH the network devices defined in your hosts file.
– You must also have ntc_ansible installed. The ntc_ansible modules were written by Jason Edelman; follow the instructions on Github to install them
– Some of the ntc_ansible modules have dependency on the pyntc library. Pyntc is an open-source multi-vendor library which makes it easier to copy files and upgrade network devices. Follow the steps here to install the library.
Note: the pyntc package install should also install the future library which is required for pyntc to work. If Ansible spits out “No module named builtins” errors when you run the playbook, that means the future library is missing from your system and you can install it by executing sudo pip install future. A quick way to find out if the future library is installed on your system is by doing import pyntc from the Python interpreter. If the import works then both the pyntc and future libraries have been installed successfully.
– In my setup, Ansible is authenticating against the devices using username/password credentials. If you prefer to use Public key authentication instead, here is a quick tutorial on how to enable SSH RSA authentication on a Cisco router.
—
– name: Upgrade a Cisco IOS router
hosts: csr
tasks:
– name: GATHERING FACTS
ios_facts:
gather_subset: hardware
provider: “{{cli}}”
tags: always
– name: COPYING IMAGE TO DEVICE FLASH
ntc_file_copy:
platform: cisco_ios_ssh
local_file: images/{{ new_image }}
host: “{{ inventory_hostname }}”
username: “{{ username }}”
password: “{{ password }}”
when: ansible_net_version != “{{version}}”
tags: copy
– name: SETTING BOOT IMAGE
ios_config:
lines:
– no boot system
– boot system flash bootflash:{{new_image}}
provider: “{{cli}}”
host: “{{ inventory_hostname }}”
when: ansible_net_version != “{{version}}”
tags: install
– name: SAVING CONFIGS
ntc_save_config:
platform: cisco_ios_ssh
host: “{{ inventory_hostname }}”
username: “{{ username }}”
password: “{{ password }}”
local_file: backup/{{ inventory_hostname }}.cfg
when: ansible_net_version != “{{version}}”
tags: backup
– name: RELOADING THE DEVICE
ntc_reboot:
platform: cisco_ios_ssh
confirm: true
timer: 2
host: “{{ inventory_hostname }}”
username: “{{ username }}”
password: “{{ password }}”
when: ansible_net_version != “{{version}}”
tags: reload
– name: VERIFYING CONNECTIVITY
wait_for:
port: 22
host: “{{inventory_hostname}}”
timeout: 300
– ios_command:
commands: ping 8.8.4.4
provider: “{{cli}}”
wait_for:
– result[0] contains “!!!”
register: result
failed_when: “not ‘!!!’ in result.stdout[0]”
tags: verify
The playbook is pretty straightforward and consists of 6 tasks:
1. GATHERING FACTS: the first task uses the ios_facts (core module that ships with Ansible itself) to gather facts about the device and see if it needs an upgrade. If the image running on the device matches the target image, Ansible skips that device. Otherwise it collects the facts and moves on to the next task.
2. COPYING IMAGE TO DEVICE FLASH:This task checks if the image file is available on the device flash and copies the file to flash if it doesn’t exist.
Note: this module uses SCP/Netmiko to copy the file. If you want to use this module with a Cisco Nexus switch, you will need to enable SCP on the switch (not a requirement for Cisco IOS) by doing: ip scp server enable
3. SETTING BOOT IMAGE: this task sets the boot image. Simple.
4. SAVING CONFIGS: this task saves the running-configs as startup configs on the device and also saves a copy of the configs locally on the host in the backup folder
5. RELOADING THE DEVICE: This task reloads the device after a time interval (2 minutes)
6. VERIFYING CONNECTIVITY: the final task waits for the device for 5 minutes to come up and become accessible via SSH before it pings a root DNS to verify internet connectivity. If the ping is not successful, Ansible generates an error.
This is it. Now you can run a single command for all of your devices and let Ansible do its magic.
Next I will be doing some work to automate Junos devices. Stay tuned for that post. You can find the source code including the ansible configuration and variable files in my Github repo.
Here are also some resources that helped me in my Ansible learning journey:
Last month at Networking Field Day (NFD10) Juniper presented their Junos Fusion solution which brings simplicity to the data center by giving you a single pane of glass for managing all the switches in the fabric and allows you to upgrade all the switches from a central interface.
With Junos Fusion, all the access switches (called Satellite devices) are managed from a single or a pair of aggregation devices. The access devices can be either EX 4300 or QFX 5100 series switches and run a Windriver Linux distribution, known as the Linux Forwarding Operating System (LFOS), which is decoupled from the Junos operating system running on the aggregation devices. The aggregation devices are the new QFX 10000 series and run classic Junos. Juniper uses LLDP between the aggregation and access devices for auto discovery/provisioning and 802.1br+ for configuring and monitoring the ports. They also use Netconf between the aggregation devices to sync the configurations.
The feature itself is not new, the Juniper MX edge routers have supported this feature for a while but Juniper just extended the support to their data center switches this year.
From operational perspective, Junos Fusion is very similar to Cisco Fabric Extender (FEX). They both make the fabric look from the outside as a big switch with a single IP address.
So if you are building a data center fabric, should you go with Junos Fusion or Cisco FEX? Well there are few things to consider when comparing the two architectures. Here is a few things to think about:
Port Density: how many server ports do you need? Both Junos Fusion and Cisco FEX architectures support today up to 64 access switches per fabric. The Nexus 2200 (FEX) has only 48 extended (server) ports which gives you a total of 3072 (64 x 48) ports per fabric while the QFX 5100-96S has 96 server ports which gives you a maximum of 6144 (64 x 96) ports per fabric so the Junos Fusion architecture clearly scales better when it comes to port density
Support For Local Switching & Other Features In the Access Layer: The Cisco Nexus 2200 has no brain and therefore has no support for local switching, VLAN tagging, or any other features you typically see in an access switch. It’s an “extender” and doesn’t not have ASICs to switch traffic. The QFX 5100/EX 4300 on the other hand are full blown switches with ASICs & intelligent software and support all the features mentioned above and more. L3 routing is not supported today on the QFX 5100/EX 4300 in Fusion mode, however Juniper stated that this feature is on the roadmap.
The need for local switching is a good debate to have. Some people argue that the Nexus 2200 is not a good fit for the data center because it cannot do local switching, however this is not a fair assessment in my opinion. Traffic patterns in the data center depend heavily on the type of workloads. Some workloads like Hadoop generate heavy east-to-west traffic within the same VLAN and in such case it’s recommended to keep all the server nodes on the same TOR to switch traffic locally and avoid congesting the uplinks. However many of the other workloads (Web applications namely) don’t generate heavy east-west traffic within the same VLAN.
The other thing to keep in mind is that with server virtualization the edge of the network is moving to the hypervisor and much of that intra-VLAN traffic is getting switched in kernel by the hypervisor without leaving the physical host therefore making the need for local switching unnecessary. Even inter-VLAN traffic can now get routed without leaving the physical host if you have a virtual distributed firewall / router.
Cost: This is where the Nexus 2200 really shines. Because it’s an extension and does not have full software/hardware capabilities, it’s very affordable and can reduce your CapEx substantially.
My Take:
Both the Junos Fusion and Cisco FEX architectures simplify managing data center networks. When comparing the two solutions, examine your workload requirements, determine how intelligent your TORs need to be, and from there you can decide which solution best works for you.
Someone asked me the other day how they could automate the execution of a command on multiple routers without accessing each router manually. Obviously an Ansible playbook can easily do that (or even using Ansible ad-hoc command without a playbook); or you can write a bash script with a for loop that iterates over the devices and connects to each device to run the command. But that question also got me wondering if there was a simpler way to get the job done quickly without installing any configuration management tools or messing with scripting?
It turns out the answer is Yes. You can do that using good old SSH — specifically using parallel SSH on Linux.
Parallel SSH (PSSH) is a great tool to use when you want to run single or multiple commands on more than one host or router at the same time. All what you need is a Linux host with PSSH installed and you are good to go. You can install PSSH on Ubuntu by using the Python package installer pip install pssh (if you don’t have the Python package installed, you can install it by executing apt-get install python-pip).
pssh [OPTIONS] command […]
Let’s look at a quick example. I have two routers, csr and csr3, defined in the hosts.txt file and I want PSSH to save the runnng configuration on each router. The optional -l argument tells PSSH what unsername to use while the -A argument tells it to prompt for a password (alternatively you can use private/public key pair instead of passwords).
➜ ~ pssh -h hosts.txt -l cisco -A “wr mem”
Warning: do not enter your password if anyone else has superuser privileges or access to your account.
Password:
[1] 20:11:59 [SUCCESS] csr3
[2] 20:12:00 [SUCCESS] csr
If you want to gather some data from the routers and write the output to a file, you can do so by adding the -o argument as follows:
Warning: do not enter your password if anyone else has superuser privileges or access to your account.
Password:
[1] 20:14:39 [SUCCESS] csr3
[2] 20:14:39 [SUCCESS] csr
➜ ~
➜ ~ ls /tmp/out/
csr csr3
➜ ~
➜ ~ more /tmp/out/csr
Building configuration…
Current configuration : 2643 bytes
!
! Last configuration change at 14:59:27 UTC Wed Feb 8 2017 by cisco
!
version 15.4
service timestamps debug datetime msec
service timestamps log datetime msec
no platform punt-keepalive disable-kernel-core
platform console virtual
!
hostname CSR
!
➜ ~
The PSSH utility is lightweight, simple, and does the job with minimum overhead. It also runs through routers in parallel which saves time especially when you are executing tasks that take some time to complete.
It’s amazing how much you can do with the mighty SSH. Keep PSSH in your toolbox in case you need it one day.
Did I say there is also PSCP (parallel SCP) utility which you can use to copy an image to multiple devices at the same time when you are upgrading those devices? That is your homework now, Google it and check it out.
The first half of the meetup was an introduction to Ansible for network engineers. The second half was a live demos of Ansible playbooks. You can view and download the slides below
With Cisco IOS, I had to use several modules in my playbook to be able to automate the upgrade process because there was not a single module available that could handle all the tasks.
As I was trying to expand my Ansible knowledge, I began looking at the available Ansible networking modules that run on Juniper devices. It turns out junos_package is a core module (comes installed with Ansible) and can take care of the entire process: copy the package to the device flash, install the package, commit, and reboot.
My Setup:
– Python 2.7.6 and Ansible 2.2 running on Ubuntu 14.4.5 LTS (codename: Trusty)
– Juniper vSRX running version 12.1X47-D10.4
Requirements:
– You need to have Ansible and Junos PyEZ installed. Junos PyEZ is a Python library to manage remotely and automate Junos devices.
– Netconf and SSH enabled on the Junos device.
The following playbook consists of few tasks: Ansible collects first device facts. Then the junos_package module compares the running version on the Junos device with the version defined in the “package” variable and upgrades the device if there is a mismatch. Once the device reboots with the new package, Ansible will wait until that device becomes reachable via Netconf and then attempts to ping a root DNS from the device to check internet connectivity. If the destination is not reachable, Ansible will generate a “ping failed” error.
You can of course expand this playbook to include tasks to verify that routing protocols have come up after the reboot.
Ansible is a nice tool to automate the deployment and configuration of network devices. I wrote the following playbook to automate the upgrade of Cisco IOS devices. This playbook has been tested successfully to upgrade a Cisco CSR1000v router and can be easily tweaked to support Cisco Nexus and Arista switches.
My Setup:
– Python 2.7.6 and Ansible 2.2 running on Ubuntu 14.4.5 LTS (codename: Trusty)
– Cisco CSR1000v running IOS-XE 3.10.S
Requirements:
– Obviously you must have Ansible version 2.2 or higher installed and configured properly to access via SSH the network devices defined in your hosts file.
– You must also have ntc_ansible installed. The ntc_ansible modules were written by Jason Edelman; follow the instructions on Github to install them
– Some of the ntc_ansible modules have dependency on the pyntc library. Pyntc is an open-source multi-vendor library which makes it easier to copy files and upgrade network devices. Follow the steps here to install the library.
Note: the pyntc package install should also install the future library which is required for pyntc to work. If Ansible spits out “No module named builtins” errors when you run the playbook, that means the future library is missing from your system and you can install it by executing sudo pip install future. A quick way to find out if the future library is installed on your system is by doing import pyntc from the Python interpreter. If the import works then both the pyntc and future libraries have been installed successfully.
– In my setup, Ansible is authenticating against the devices using username/password credentials. If you prefer to use Public key authentication instead, here is a quick tutorial on how to enable SSH RSA authentication on a Cisco router.
—
– name: Upgrade a Cisco IOS router
hosts: csr
tasks:
– name: GATHERING FACTS
ios_facts:
gather_subset: hardware
provider: “{{cli}}”
tags: always
– name: COPYING IMAGE TO DEVICE FLASH
ntc_file_copy:
platform: cisco_ios_ssh
local_file: images/{{ new_image }}
host: “{{ inventory_hostname }}”
username: “{{ username }}”
password: “{{ password }}”
when: ansible_net_version != “{{version}}”
tags: copy
– name: SETTING BOOT IMAGE
ios_config:
lines:
– no boot system
– boot system flash bootflash:{{new_image}}
provider: “{{cli}}”
host: “{{ inventory_hostname }}”
when: ansible_net_version != “{{version}}”
tags: install
– name: SAVING CONFIGS
ntc_save_config:
platform: cisco_ios_ssh
host: “{{ inventory_hostname }}”
username: “{{ username }}”
password: “{{ password }}”
local_file: backup/{{ inventory_hostname }}.cfg
when: ansible_net_version != “{{version}}”
tags: backup
– name: RELOADING THE DEVICE
ntc_reboot:
platform: cisco_ios_ssh
confirm: true
timer: 2
host: “{{ inventory_hostname }}”
username: “{{ username }}”
password: “{{ password }}”
when: ansible_net_version != “{{version}}”
tags: reload
– name: VERIFYING CONNECTIVITY
wait_for:
port: 22
host: “{{inventory_hostname}}”
timeout: 300
– ios_command:
commands: ping 8.8.4.4
provider: “{{cli}}”
wait_for:
– result[0] contains “!!!”
register: result
failed_when: “not ‘!!!’ in result.stdout[0]”
tags: verify
The playbook is pretty straightforward and consists of 6 tasks:
1. GATHERING FACTS: the first task uses the ios_facts (core module that ships with Ansible itself) to gather facts about the device and see if it needs an upgrade. If the image running on the device matches the target image, Ansible skips that device. Otherwise it collects the facts and moves on to the next task.
2. COPYING IMAGE TO DEVICE FLASH:This task checks if the image file is available on the device flash and copies the file to flash if it doesn’t exist.
Note: this module uses SCP/Netmiko to copy the file. If you want to use this module with a Cisco Nexus switch, you will need to enable SCP on the switch (not a requirement for Cisco IOS) by doing: ip scp server enable
3. SETTING BOOT IMAGE: this task sets the boot image. Simple.
4. SAVING CONFIGS: this task saves the running-configs as startup configs on the device and also saves a copy of the configs locally on the host in the backup folder
5. RELOADING THE DEVICE: This task reloads the device after a time interval (2 minutes)
6. VERIFYING CONNECTIVITY: the final task waits for the device for 5 minutes to come up and become accessible via SSH before it pings a root DNS to verify internet connectivity. If the ping is not successful, Ansible generates an error.
This is it. Now you can run a single command for all of your devices and let Ansible do its magic.
Next I will be doing some work to automate Junos devices. Stay tuned for that post. You can find the source code including the ansible configuration and variable files in my Github repo.
Here are also some resources that helped me in my Ansible learning journey:
Last month at Networking Field Day (NFD10) Juniper presented their Junos Fusion solution which brings simplicity to the data center by giving you a single pane of glass for managing all the switches in the fabric and allows you to upgrade all the switches from a central interface.
With Junos Fusion, all the access switches (called Satellite devices) are managed from a single or a pair of aggregation devices. The access devices can be either EX 4300 or QFX 5100 series switches and run a Windriver Linux distribution, known as the Linux Forwarding Operating System (LFOS), which is decoupled from the Junos operating system running on the aggregation devices. The aggregation devices are the new QFX 10000 series and run classic Junos. Juniper uses LLDP between the aggregation and access devices for auto discovery/provisioning and 802.1br+ for configuring and monitoring the ports. They also use Netconf between the aggregation devices to sync the configurations.
The feature itself is not new, the Juniper MX edge routers have supported this feature for a while but Juniper just extended the support to their data center switches this year.
From operational perspective, Junos Fusion is very similar to Cisco Fabric Extender (FEX). They both make the fabric look from the outside as a big switch with a single IP address.
So if you are building a data center fabric, should you go with Junos Fusion or Cisco FEX? Well there are few things to consider when comparing the two architectures. Here is a few things to think about:
Port Density: how many server ports do you need? Both Junos Fusion and Cisco FEX architectures support today up to 64 access switches per fabric. The Nexus 2200 (FEX) has only 48 extended (server) ports which gives you a total of 3072 (64 x 48) ports per fabric while the QFX 5100-96S has 96 server ports which gives you a maximum of 6144 (64 x 96) ports per fabric so the Junos Fusion architecture clearly scales better when it comes to port density
Support For Local Switching & Other Features In the Access Layer: The Cisco Nexus 2200 has no brain and therefore has no support for local switching, VLAN tagging, or any other features you typically see in an access switch. It’s an “extender” and doesn’t not have ASICs to switch traffic. The QFX 5100/EX 4300 on the other hand are full blown switches with ASICs & intelligent software and support all the features mentioned above and more. L3 routing is not supported today on the QFX 5100/EX 4300 in Fusion mode, however Juniper stated that this feature is on the roadmap.
The need for local switching is a good debate to have. Some people argue that the Nexus 2200 is not a good fit for the data center because it cannot do local switching, however this is not a fair assessment in my opinion. Traffic patterns in the data center depend heavily on the type of workloads. Some workloads like Hadoop generate heavy east-to-west traffic within the same VLAN and in such case it’s recommended to keep all the server nodes on the same TOR to switch traffic locally and avoid congesting the uplinks. However many of the other workloads (Web applications namely) don’t generate heavy east-west traffic within the same VLAN.
The other thing to keep in mind is that with server virtualization the edge of the network is moving to the hypervisor and much of that intra-VLAN traffic is getting switched in kernel by the hypervisor without leaving the physical host therefore making the need for local switching unnecessary. Even inter-VLAN traffic can now get routed without leaving the physical host if you have a virtual distributed firewall / router.
Cost: This is where the Nexus 2200 really shines. Because it’s an extension and does not have full software/hardware capabilities, it’s very affordable and can reduce your CapEx substantially.
My Take:
Both the Junos Fusion and Cisco FEX architectures simplify managing data center networks. When comparing the two solutions, examine your workload requirements, determine how intelligent your TORs need to be, and from there you can decide which solution best works for you.
Here is the Junos Fusion presentation from NFD10:
Your Turn Now
What are you thoughts on this? Have you deployed either solution? I want to hear from you.
Disclaimer: I attended Networking Field Day 10 as a delegate. Vendors sponsoring the event indirectly covered my travel expenses, however I’m not required to write about their products or about the event. If I do write something, it’s because I want to express my opinions.
I had the opportunity to attend Cisco Live 2015 last month in San Diego and meet new people and learn bunch of new stuff. Most of the sessions I had attended were IPv6 related as I wanted to learn more about the protocol and be able to design IPv6 networks.
Now that the on-demand videos and presentation slides have been published, I wanted to share with you some of the sessions that I thought were really great and worth watching.
Here are some of my favorite sessions from Cisco Live 2015.
IPv6 from Intro to Intermediate: This is session by Tim Martin was a great introduction to IPv6 and covered addresses, headers, and link operations. If you are just getting started with IPv6, this would be a good place to start.
Enterprise IPv6 Deployment: another session by Tim Martin which covered general design, host configuration, and translation techniques.
IPv6 Routing Protocols Update: This presentation by Wim Verrydt was a comprehensive overview of the IPv6 routing protocols (OSPFv3, BGP, EIGRP) along with some configuration examples. It also discussed the coexistence of IPv4 and IPv6 routing protocols.
Troubleshooting OSPF: This is a presentation by Faraz Shamim which I could not attend live at the conference but watched the video this week when it became available. It is a great session that covers OSPF LSAs and some of the new commands that make troubleshooting easier. If you work in operations and you deal with OSPF, you definitely want to check out this session.
Do you have favorite sessions? I want to know about them. Share them with us below.
I have been preparing for my VMvware Certified Professional (VCP) exam. Early this year I decided to invest and buy HP ProLiant MicroServer G8 and Synology DS414slim NAS appliance to expand my home lab.
I’m one on those who learn better by doing rather than reading and I wanted to rely on practice labs and hands-on experience instead of books and practice tests to pass the exam.
I bought everything from newegg.com. The HP MicroServer came with 8GB of RAM installed. I then upgraded the RAM to 16GB and downloaded the ESXi 5.5 ISO directly from the HP website which comes with all the drivers required to run ESXi on HP ProLiant servers.
I’m running few VMs on the HP server including the VMware vCenter Server Virtual Appliance (vCSA) which manages my ESXi servers. The HP server is one of two ESXi servers I have running. The other ESXi server runs inside VMware Fusion (nested) on my iMac. Because my HP server and iMac desktop have two different CPU architectures, I had to enable VMware Enhanced vMotion Compatibility (EVC) to provide CPU compatibility and support for vMotion and DRS.
I populated the Synology appliance with two SSDs configured in RAID 1. On it, I have a datastore configured that provides NFS storage to my VMs. I also store there all of my ISOs and OVA files.
I’m happy so far with both devices. The HP MicroServer is relatively quiet compared to other devices I have seen. In terms of noise the HP MicroServer fan generates on average 40 dB-A of noise (equivalent to the noise which a fan of Dell Latitude laptop would put out) according to my iPhone noise meter. The Synology NAS appliance is also pretty quiet. Its fan comes on for few seconds only when the CPU is doing heavy processing. I keep both devices in my home office, which is where I do most of my work.
One thing I wanted to do was to schedule automatic shut down at night to save power. So I searched online for a script to do so but the problem I ran into was that in vSphere 5.5 the host had to be put into maintenance mode before it could shut off gracefully. That meant that the server would come up as a result in maintenance mode when it powered back on and I would need to intervene and take it out of maintenance mode every time.
After experimenting with few ESX CLIs and with some help from the online community I came up with the following Apple script (hack) which basically shuts down the powered on VMs, puts the host in maintenance mode and then issues a shut down command with a delay of 10 seconds. Before the delay timer expires the script executes another command (last command below) and takes the host out of maintenance mode. When the delay timer finally expires the host gracefully shuts down.
In this post I will walk you through some steps you can take to troubleshoot BGP neighbor adjacency. These steps become even more helpful when you have access to only one side on the link (in the case where you are trying to run BGP with a service provider). We will focus in this post on some of the reasons that may prevent two BGP routers from forming a relationship and will demonstrate along the way how the BGP state machine moves from Idle to Established.
I’m using here two Cisco CSR1000v routers as my BGP speakers but the tips below apply in general to any router from any vendor.
In the diagram above I have two routers, R1 and R2, with two parallel physical links between them. The two routers want to peer using the loopback addresses via BGP which is a common way to do load sharing between two routers. However the BGP adjacency is not coming up and stuck in Idle state as you can see from the output below:
R2#sh ip bgp sum
BGP router identifier 2.2.2.2, local AS number 200
BGP table version is 1, main routing table version 1
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
1.1.1.1 4 100 0 0 1 0 0 never Idle
Follow the steps below to verify that your configurations are complete and that there are no connectivity issues between the two routers:
1- First test and verify that R1 is reachable from R2 and vice versa. Issue a ping command from R2 sourcing your ping from the loopback0 interface with R2’s loopback0 interface as the destination as shown below:
R2#ping 1.1.1.1 source loopback 0
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 1.1.1.1, timeout is 2 seconds:
Packet sent with a source address of 2.2.2.2
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms
If your ping fails then you have a connectivity problem and you need to fix that before continuing this process. You want to make sure that the static routes are there on each router and that each router is able to ARP for the other router’s IP address.
2- You also need to verify that there is no firewall between R1 and R2 blocking TCP port 179 which is the port BGP uses to establish the connection. A quick way to ensure whether there is no firewall between R1 and R2 is to use the telnet command with port 179 as the destination port. Perform this test from both routers and don’t forget to source the traffic from the loopback interface:
As you can see from the output above, I got a “Connection refused by remote host” response when I tried to telnet from R2 to R1. This simply means that there is no device in the middle blocking traffic and R1 is rejecting the request obviously because 179 is not a standard port for telnet. If there was a firewall in the middle blocking traffic then you would get a “Unable to connect to remote host” response instead.
3- Now that we have verified that there are no connectivity problems, let’s focus on the BGP configurations. I will turn on “debug ip bgp x.x.x.x” on the router which shows me that the router is failing to establish a TCP connection with its peer. A good show command to use at this point is “show ip bgp neighbor x.x.x.x”
R2#show ip bgp nei 1.1.1.1
BGP neighbor is 1.1.1.1, remote AS 100, external link
BGP version 4, remote router ID 0.0.0.0
Address tracking is enabled, the RIB does have a route to 1.1.1.1
Connections established 0; dropped 0
Last reset never
External BGP neighbor not directly connected.
Transport(tcp) path-mtu-discovery is enabled
Graceful-Restart is disabled
No active TCP connection
The output above tells me two important things. First that the RIB does have a route to reach the peer which confirms that the router has static routes needed to reach its peer’s loopback address.
The second important thing this output shows is this line: “External BGP neighbor not directly connected”. By default only directly connected eBGP peers are allowed to establish relationship.In order to change this default behavior, I have to add the “neighbor disable-connected-check” on both routers.
4- Even after disabling the direct-connected check, BGP relationship is still not coming up so my next step is to enable “debug ip tcp transactions” to see if that tells me why the TCP connection is failing:
R1#deb ip tcp transactions
TCP special event debugging is on
*Mar 13 05:53:52.522: Reserved port 0 in Transport Port Agent for TCP IP type 0
*Mar 13 05:53:52.522: TCP: connection attempt to port 179
*Mar 13 05:53:52.522: TCP: sent RST to 192.168.32.20:27730 from 1.1.1.1:179
The last line in the output above is interesting. It is showing that R1 (1.1.1.1:179) is sending a TCP reset to R2 (192.168.32.20). Which means that it was R2 who initiated the TCP session. What this reveals also is that R2 sourced the connection request from its physical interface which is the default behavior in eBGP. But since I want the routers to peer using the loopback addresses and each router is expecting to receive a connection request from its peer loopback address, then i need to add the “neighbor update-source” to BGP on both ends
5- Now if I look at “debug ip bgp“, I can see that the TCP session is getting established and BGP is transitioning from the Idle -> Connect -> OpenSent -> OpenConfirm as shown below:
*Mar 13 06:49:28.979: BGP: 1.1.1.1 passive open to 2.2.2.2
*Mar 13 06:49:28.979: BGP: Fetched peer 1.1.1.1 from tcb
*Mar 13 06:49:28.979: BGP: 1.1.1.1 passive went from Idle to Connect
*Mar 13 06:49:28.979: BGP: ses global 1.1.1.1 (0x7F028066E270:0) pas Receive OPEN
*Mar 13 06:49:28.979: BGP: ses global 1.1.1.1 (0x7F028066E270:0) pas Send OPEN
*Mar 13 06:49:28.979: BGP: 1.1.1.1 passive went from Connect to OpenSent
*Mar 13 06:49:28.979: BGP: 1.1.1.1 passive went from OpenSent to OpenConfirm
*Mar 13 06:49:28.980: %BGP-3-NOTIFICATION: received from neighbor 1.1.1.1 passive 2/2 (peer in wrong AS) 2 bytes 00C8
*Mar 13 06:49:28.980: BGP: ses global 1.1.1.1 (0x7F028066E270:0) pas Receive NOTIFICATION 2/2 (peer in wrong AS) 2 bytes 00C8
*Mar 13 06:49:28.980: %BGP-5-NBR_RESET: Neighbor 1.1.1.1 *Mar 13 06:49:28.980: BGP: 1.1.1.1 passive went from OpenConfirm to Closing
*Mar 13 06:49:28.980: BGP: 1.1.1.1 passive went from Closing to Idle
When BGP is in the OpenConfirm state it’s one step away from reaching its final state (ESTABLISHED) and while in the OpenConfirm state BGP waits to hear a KEEPALIVE from its peer before it moves to the Established state. As you can see from the output above, after reaching OpenConfirm BGP instead closes the connection and transitions back to Idle because it receives an error (peer in wrong AS).
This is a clear indication that the AS number on R1 is wrong so I will fix that and issue a “clear ip bgp *” command to restart the process.
And now after I corrected AS number in the configs on R1, the BGP state machine transitions to Established as you see below and the two peers can start exchanging routing updates and keepalives.
R2#sh ip bgp neighbors 1.1.1.1
BGP neighbor is 1.1.1.1, remote AS 100, external link
BGP version 4, remote router ID 1.1.1.1
BGP state = Established, up for 00:00:42
Obviously that’s not everything, and there are other reasons that could prevent BGP relationship from being established but I wanted to discuss the most common ones that I have seen in the field. Do you have something to share? Please respond and share below.
Here is the final configs for R1 and R2 for your reference.
Python is every network engineer’s favorite programming language. It’s simple, powerful, and open-source. I started learning Python two months ago as I will be getting into network automation next year. I figured to share the trainings I have been using with those of you who are interested in learning Python.
Below are some of the training resources I have used personally. There is also a lot of other free training available online you can search for if you want to learn Python. All the trainings below except for the Rice University class are self-paced.
Up and Running With Python: This online video tutorial is offered by lynda.com. It is the first Python training I have used and covers advanced topics like working with files, dates & times, and parsing & processing HTML. This class is not free, lynda.com requires a subscription to use their site and it’s usually about $25 a month to get started. You can use either the online Python interpreter or Aptana Studio for this class
Google’s Python Class: This is a free and popular Python class. It combines articles and video lectures. It’s probably the first training people use when learning Python. I’m currently looking into this training. You can the built-in Python interpreter for Mac or download the free Python interpreter for Windows for this class.
Python for Network Engineers: This free ten-week class is offered via email by Kirk Byers and it’s an introduction to Python. I have not taken this training personally but have heard good things about it. Check and find out from the website when is the next available class. Kirk also runs a blog which focuses on network automation.
An Introduction to Interactive Programming in Python: This free online course is offered by Rice University through Coursera. I took this class recently and I can tell you it’s a lot of fun. Be prepared to spend about 2 hours a day studying and writing code if you plan to take this course. You will be required to write few games in this class including the Memory, Pong, and Blackjack games. I did not get a chance to complete all the games but certainly learned a lot in this class. If you are in it for the challenge and have the time, this class is for you.
I attended a panel discussion session at VMUG this week. A guy from the audience asked a question on how to troubleshoot connectivity issues after moving a VM. The guy had a one flat VLAN with one IP subnet and every time he vMotioned a VM to another host, users lost connectivity with that VM.
To answer his question, a guy on the stage advised him to recreate another VLAN/IP subnet, move that VM to the newly created subnet, and then try to do the vMotion again.
Well obviously that wasn’t a good advice and changing the VM IP address in this case would not help. In this post I will explain what happens when a VM moves and steps you can take to troubleshoot network delay and connectivity issues related to vMotion.
First before you do any vMotion, ensure that the VLAN is configured on all switches and allowed on all necessary trunk ports in the network. You can use the commands show vlan and show trunk to verify that that.
When you create a new VM, the host allocates and assigns a MAC address to that VM. The physical switch which the host is connected to eventually learns the VM MAC address (that happens when the VM ARPs for its gateway, starts sending traffic and the switch sees that traffic, or when the Notify Switches in vSphere option is tuned on).
When a VM moves to another host (the new host could be either connected to another port or the same physical switch or connected to a different switch), somebody has to tell the network to change its destination port for that MAC address in order to continue to deliver traffic to that VM. In vSphere that is usually handled by the host when the “Notify Switches” option is turned on. The host in this case notifies the network by sending several RARP messages on behalf of the VM to ensure that the upstream physical network updates its MAC table.
So the first step in troubleshoot this specific issue is to ensure that the “Notify Switches” settings is set to Yes before you vMotion the VM. In vSphere 5.5 you would go to the vSwitch/vDS settings and then to Teaming and Failover to verify the setting.
If that does not help, try to do the followings:
On the physical switch (the switch the destination host is connected to), issue a command to look up MAC table and find out which destination port the switch is using to reach the VM. On a Cisco Catalyst switch, you can use the “show mac address-table” command for example. If the switch is still using the old MAC-to-port mapping, then it’s either that the switch is not reaching the RARP notification from the host or the host itself is not sending it.
If you don’t do anything to correct the problem, the aging timer for that MAC address on the switch (default is set to 5 mins on Cisco switches) will eventually expire and the switch will learn the MAC address via the new port but to speed things up you can alway flush out the VM MAC address from the switch MAC table (clear mac address-table on Cisco catalyst).
Obviously having to manually intervene and troubleshoot every time you move a VM defeats the whole purpose of vMotion. vMotion is supposed to do live migration of a VM and preserves all active network connections during the process. But hopefully the above steps will give you some pointers on where to look and start to find the root cause.
Additionally VMware recommends disabling the followings on the physical ports connected to the host to minimize networking delay:
– Port Aggregation Protocol (PAgP) and Link Aggregation Control Protocol (LACP).
– Dynamic Trunking Protocol (DTP) or trunk negotiation.
I’m starting a new post series to walk you through building a private cloud. This post is the first one in the series and will be just an introduction to private clouds.
In the next posts I will discuss collecting the requirements, building the network and network services, building the storage, designing the server infrastructure, and putting the orchestration layer on top of that.
Why Private Cloud?
There are number of public cloud providers out there that have good and affordable offerings. However customers sometimes prefer private cloud over public cloud for specific reasons or requirements they need to meet. Here are some the reasons that come to mind:
Regulatory Compliance: Some companies have have to comply with certain regulations and therefore have to keep their customer data or their own data on dedicated/isolated infrastructure behind their firewall. In this case the multi-tenant public cloud model won’t be a good fit for them.
Control: Some companies require more control over the cloud infrastructure than what public cloud providers usually offer or they might have a specific need to integrate a solution/product from certain vendor. Most public cloud providers offer standard services that may not be customized or tweaked easily.
Avoid Lock-in: I’m sure you heard this before. People don’t like to put all of their eggs in one basket nor like to heavily rely on a specific vendor/provider. By building their own cloud in house companies can have the freedom of choosing multiple vendors and can also leverage open source technology if they wish.
Cost: This might be a bit of surprise to you but depending on how much the monthly bill coming from the public cloud provider is, operating a private cloud might become a cheaper option. This is especially true if you are using the cloud to run production workloads that need to be up 24/7 and cannot be turned off while not in use.
Barriers to Private Cloud Deployments:
Lack of Technical Resources: Designing, implementing, and supporting a private cloud require having the right IT resources in house. Some companies just don’t have that.
Time to Market: Getting a private cloud up and running is at least 6-12 month long process. Some companies have to respond to market pressure quickly and simply don’t have the luxury of spending a year to build a cloud in house.
Cost: Building and operating a private cloud is expensive and it makes more sense in some cases to leverage a public cloud offering than spending that CapEx up front.
You have the capital and technical talent and want to build a private cloud? Great. In the next post I will walk you through the process of collecting the requirements and making high-level design decisions before we get into building the infrastructure itself. Stay tuned.
Make sure that you either subscribe to my blog or follow me on Twitter to get notified when I add a new content.
This post assumes a working knowledge of Locator/ ID Separation Protocol (LISP), you may first want to review Part 1 before reading through this post.
In Part 1 we created two LISP sites and enabled connectivity between these two sites. We also enabled VM mobility and showed how you can migrate a VM to the cloud without changing its IP address, mask, or default gateway.
In this post I will expand on Part 1 and enable connectivity between non-LISP and LISP sites. As shown in the diagram below I have a branch office (non-LISP site) which needs to reach the VM (IP address 10.122.139.91) in the cloud.
To do so we have three design options…
Option 1 is to enable LISP on the branch edge router. In this design traffic leaving the branch will be redirected by the branch edge router to the cloud however this option is costly and requires deploying a router with LISP support on each branch router.
Option 2 is to deploy a LISP proxy router in the service provider WAN to intercept traffic and re-direct it to cloud. This option is less expensive than option 1 but requires routing all traffic thru the proxy router which could cause this router to become a choke point.
Option 3 is to configure the already LISP-enabled router in the data center as proxy which will redirect traffic to the cloud. This is the least expensive option but obviously does not provide optimal routing as you need to backhaul traffic to the data center first, however this might be acceptable design when local internet access is not available at the branch and internet-bound traffic needs to traverse the data center anyway or when the customer wants the traffic from the branch to traverse the data center firewall first for policy enforcement.
In this post I will show you what you need to do to enable option 3.
To add the proxy functionality to the design, first we need to configure CSR1 as Proxy Tunnel Router (PxTR) which tells CSR1 to intercept the traffic coming from any non-LISP site and redirect it to the cloud:
CSR_1#sh run | i ipv4
no ipv4 itr
ipv4 proxy-etr
ipv4 proxy-itr 1.1.1.1
Now if I run a show command I can see that CSR1 is configured as PxTR:
CSR_1#sh ip lisp
Instance ID: 0
Router-lisp ID: 0
Locator table: default
EID table: default
Ingress Tunnel Router (ITR): disabled
Egress Tunnel Router (ETR): enabled
Proxy-ITR Router (PITR): enabled RLOCs: 1.1.1.1
Proxy-ETR Router (PETR): enabled
The second step is to configure CSR3 to use CSR1 as its default proxy ETR (PETR). This is necessary as CSR3 is not going to know how to get back to the source 2.2.2.2 since this prefix is not a LISP entry and does not exist in its cache table.
CSR_3# show run | b lisp
router lisp
ipv4 use-petr 1.1.1.1
Now let’s run a ping test from R2 (2.2.2.2) to see if we can reach the VM:
R2#ping 10.122.139.92 source 2.2.2.2
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.122.139.92, timeout is 2 seconds:
Packet sent with a source address of 2.2.2.2
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 48/51/63 ms
Comment below if you have any questions. Also please help me spread out the word if you like this post and share with your network on Twitter/Facebook/Linkedin.
If you have been around Cisco devices for a while you probably know how to enable them for SSH access and log in using a username/password. Yesterday however I ran into a situation while deploying Ansible where i needed to enable logging in to the router using an RSA key instead of a password and had to try few things to get it to work.
Why would you want to use RSA based user authentication for SSH instead of a password based authentication?
1- RSA keys are much more secure than passwords. Passwords (even when they are stronger than your dog’s name) are susceptible to brute-force attacks and can be compromised
2- Using RSA key is easier as you don’t have to enter or remember your password every time
3- You might need to use RSA authentication if you are using management or automation tools (such as Ansible) to manage the devices via SSH.
Here is what you need to do to enable SSH RSA authentication on a Cisco router:
Step 1: Enable the router for SSH server by entering the following commands:
ip domain name example.com
!
!generate the RSA key for SSH
crypto key generate rsa
!
username bob password 0 smith
!
line vty 0 98
login local
At this point you should be able to SSH to the router using the username/password defined in the configs above. Fix any issues you may have before you move on to the next step. A good debug command to use for troubleshooting is: debug ip ssh
Step 2: Enable Public/RSA Key Authentication
First make sure that you generate a public/private key pair on the machine you are trying to SSH from if you don’t already have one. SecureCRT and Putty for Windows have a built-in program to generate the key pair. If you are on a Mac or a Linux/Unix machine, you can use the command ssh-keygen to generate the key pair.
Next enter the following commands on the router:
R_Ent(config)#ip ssh pubkey-chain
R_Ent(conf-ssh-pubkey)#username bob
R_Ent(conf-ssh-pubkey-user)#key-string
R_Ent(conf-ssh-pubkey-data)#!ENTER YOUR PUBLIC KEY HERE
R_Ent(conf-ssh-pubkey-data)#exit
R_Ent(conf-ssh-pubkey-user)#end
At this point you should be able to SSH to the router without entering a password:
There are times when you want to leverage public cloud but want to keep some workloads in your data center. You can do that with Hybrid Cloud.
According to Gartner the most important use case for Hybrid Cloud is cloud bursting. That’s running the workloads internally in the data center and bursting to a public cloud during peak times. The other common use cases for hybrid cloud is disaster recovery where you migrate the workloads to a public cloud during disasters or maintenance windows.
Migrating a workload to the cloud is time consuming especially if you have to re-IP all the VMs.
In this post I will show how to migrate an application to the cloud without changing its IP address, mask, default gateway, or security policies. This makes the migration easier especially if you are migrating the workload temporarily and do not want to change the IP addresses or DNS settings.
Why I Like This Solution
You can certainly enable workload mobility between two data centers using a Layer 2 extension technology such as VPLS or Cisco OTV. However I prefer LISP over any L2 extension simply because it’s based on Layer 3 and therefore does not exntend the broadcast/failure domain like L2 extensions and you get all the benefits and scale of L3 routing.
I also think that using LISP is better than relying on any cloud extension technology like Cisco Nexus 1000v InterCloud or Verizon CloudSwitch because those technologies add a shim layer to allow the VM to run in a cloud provider environment and that degrades the VM performance. With this LISP design the VM runs natively on the hypervisor and behaves like any VM local to the cloud provider environment.
Prerequisites
– To get this solution to work to you need two Cisco routers that support Locator/ID Separation Protocol (LISP). The data center router can be either virtual or physical while the cloud provider router has to be virtual. I will be using here two Cisco CSR 1000v routers running IOS XE 3.12S for this solution.
– I will briefly explain how LISP works along the way and will be focusing only on the required components for this design. You can find more information on LISP on Cisco.com
– I will be using Verizon Terremark as my cloud provider. You may use other providers like Microsoft Azure or AWS if you wish.
Before The Migration
The diagram above shows a VM running in a data center/private cloud with IP address 10.122.139.92. The default gateway for this VM is currently CSR-1 and from the outside DNS resolves to the VM public IP address which is part of the data center addressing scheme. Now because CSR-1 is a LISP enabled router and because it’s also acting as a LISP map-server/map-resolver, it will cache the VM IP address in its database so it can inform other LISP routers about the VM location when asked.
In the cloud provider, CSR-3 is also a LISP enabled router and configured to query CSR-1 for any hosts trying to reach. So if CSR-3 receives any traffic destined for the VM it queries CSR-1 for the location of that VM. CSR-1 replies back to CSR-3 and tells it to encapsulate and send all traffic designed for the VM to him because the VM is located on its LAN.
The output below shows that the VM is located on CSR-1 local subnet at the moment:
CSR_1#sh ip lisp forwarding eid local
Prefix
10.122.139.92/32
The next output below shows that CSR-1 (1.1.1.1) is currently the locator for the VM:
On CSR-3 if I issue “where is the VM” command I see that CSR-1 (1.1.1.1) is currently the locator:
CSR_3#lig 10.122.139.92
Mapping information for EID 10.122.139.92 from 10.0.20.1 with RTT 49 msecs
10.122.139.92/32, uptime: 00:02:28, expires: 23:59:59, via map-reply, complete
Locator Uptime State Pri/Wgt
1.1.1.1 00:02:28 up 1/100
That’s good. Both routers are in sync and know that the VM is located in the data center. Let’s now migrate this bad boy to the public cloud.
After the Migration
Now the VM has migrated (cold migration) to public cloud as shown in the diagram above. When the VM boots up of the first time it will ARP for its default gateway (which is CSR-3 in this case). When CSR-3 detects that the VM is now on its LAN it registers it with the map-server (CSR-1) and declares itself as the new locator for that VM. Now the DNS mapping for the VM has not changed and the outside world still thinks that the VM resides in the enterprise data center. So when CSR-1 receives traffic destined for that VM it does a lookup in its database, encapsulates the traffic, and sends it to the new locator (CSR-3) which in turn decapsualtes the traffic and sends it off to the VM.
Now back to the router CLI. If I issue the “where is the VM now” command again, I see that CSR-3 (3.3.3.3) is now the locator:
CSR_1#lig 10.122.139.92
Mapping information for EID 10.122.139.92 from 10.0.20.3 with RTT 48 msecs
10.122.139.92/32, uptime: 00:02:27, expires: 23:59:59, via map-reply, complete
Locator Uptime State Pri/Wgt
3.3.3.3 00:02:27 up 1/100
If I ping the VM from CSR-1 sourcing my packets from the LAN interface, the ping is successful:
CSR_1#ping 10.122.139.92 source loopback 10
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.122.139.92, timeout is 2 seconds:
Packet sent with a source address of 10.10.10.1
.!!!!
Success rate is 80 percent (4/5), round-trip min/avg/max = 47/47/48 ms
Conclusion
LISP allows migrating workloads to the cloud without relying on L2 extensions. Use this design if you are migrating your workload temporarily. If you are migrating the workload permanently you might want to consider adding a LISP router at the WAN edge or at the branch office to redirect user traffic to the public cloud provider before its enters the enterprise data center.
Ping me if you want the full configurations.
Part 2 of this post is coming soon, make sure to subscribe to the blog or follow me on Twitter to get notified when new content is added.
I’m in the process of putting together a new course called Introduction to Cloud Computing. It will be delivered via email so you can work at your own pace and above all it’s free. If you are interested go ahead and subscribe below to receive the weekly emails from me once the course starts. The course starts in October and runs for 6 weeks. You may unsubscribe at any time.
Here is what you will learn in this course:
Week 1 – The Basics of Cloud Computing
Week 2 – Is Cloud Right For Your Business?
Week 3 – Cloud Models: Private, Public, and Hybrid Cloud
I have big goals set for this year and one of the things I need to archive these goals is a home lab to train on OpenStack, network virtualization, hypervisor networking, and just virtualization in general. In this post I will be sharing some details on my new home lab which I’m in the process of building.
For me the one thing I was trying to avoid was investing in physical hardware as I don’t have much space at the moment to place the servers. So far I have managed to run everything in Virtual Machine (VM) form factor. However as the lab grows I will probably need to invest in some physical servers.
Few Upgrades First
Before deploying any VMs I had to do some upgrades:
– Upgraded from VMware Fusion 4.0 to 6.0 Professional on my iMac which is 30% faster than 4.0 and supports creating additional virtual networks (similar to Network Editor in VMware Workstation)
– Upgraded my iMac memory from 12GB to 16GB to make room for additional VMs
– 2 x VMware ESXi 5.5 hosts running on Fusion for VM mobility, HA, etc..
– VMware vCener 5.5 server running on Windows 2008 server to mange the two ESXi hosts (I will be also trying the vCenter virtual appliance shortly)
– Windows 7 VM to run the vSphere Client and run other monitoring/debugging tools.
I will be sharing over the next few weeks some interesting use cases I have been working on. Make sure to subscribe to the blog to get notified when I add new content.
I have been using the Cisco CSR 1000v as a default gateway in my home lab and I run an IPSec tunnel & LISP between it and my cloud provider (more on LISP in a separate post). The CSR 1000v runs on VMware ESXi, Microsoft Hyper-V, and Amazon Xen hypervisors but it can also run on a laptop/desktop hypervisor like VirtualBox or VMWare Fusion for testing or training purposes.
In this post I will show how to spin up a CSR 1000v instance in VMware Fusion for Mac.
Requirements:
1) I’m running VMWare Fusion 6.0 Professional but you can also run this virtual router on Fusion 4 or 5. I highly recommend using Fusion 5 Professional or 6 Professional if you want to create additional networks and assign the CSR 1000v interfaces to those networks. The feature to add additional networks (similar to Network Editor in VMWare Workstation) was added by VMWare in Fusion 5 Professional and it’s not available in the standard edition of Fusion. Alternatively if you don’t want to pay the extra bucks for the Professional edition, you can use this free tool from Nick Weaver to create the networks if you are running the standard edition. Nick’s tool isn’t the best but it does the job.
2) Go to Cisco and download the latest and greatest software version of the CSR 1000v. The CSR runs Cisco IOS XE and you need to download the OVA package for the deployment.
Installation:
The installation process is simple and quick:
1- Launch Fusion and go to File -> New from the top bar menu
2- Once the Installation wizard starts click on More options. Select Install an existing virtual machine and click Continue.
3- On the next screen click Choose File. Navigate to the folder containing the OVA package, select the file, and click Open and then Continue.
4- The next window will ask you to save the VM name, choose a name and click Save.
5- The final step of the wizard is to either customize or click fire up the VM. The CSR 1000v by default comes with three interfaces. If you need to add more interfaces click on Customize, otherwise click Finish.
6- Once you click Finish, the CSR 1000v will boot a couple of times and then you will be in traditional Cisco router User Mode.
Interfaces Management:
Enter Exec mode and enter “show ip int bri” to see the three interfaces.
By default interfaces will become part of the Ethernet or WiFi network (that’s done by Fusion) depending on which adapter is active during the installation and you can assign from here IP interfaces and default gateway.
If you wish to put the interfaces on separate networks (VLANs), select the CSR VM and go to Virtual Machine -> Setting and choose the desired Network Adapter. You may also create custom networks by going to VMWare Fusion -> Preferences -> Network. From there add the + sign to add additional network and choose wether to enable DHCP or not.
Activating The License
In order to enable the full features of the CSR 1000v, you need to purchase a license from Cisco. If you want to try the full features before purchasing, Cisco offers 60 day free trial license. To activate the free trial license, go into the router configuration mode and enter: license boot level premium. You will be asked to boot the router after you enter the command.
One of the common use cases for the CSR 1000v is a secure VPN gateway in the cloud to terminate VPN tunnels. So if you run some applications in the cloud and you want to allow your branch offices to access those application over a secure network, you can run IPsec tunnels between those branch offices and the CSR 1000v in the cloud. From performance perspective this would work much better than back hauling the traffic to the data center (HQ) and then to the cloud.
In this post I will show you how to set up a VPN tunnel between Cisco CSR 1000v and a branch office router using Cisco Easy VPN. You may also use standard IPSec VPN but in my configurations I use Easy VPN because it’s, well, easier 🙂
The following configurations are part of a demo I gave at Cisco Live 2013 in Orlando where the CSR 1000v was the VPN server and the branch server (Cisco 2900 in this case) was the VPN client.
You will also need to open UDP port 500 and IP protocol ID 50 (ESP) on all firewalls sitting between the CSR 1000v and the branch router for the IPSec tunnel to be established successfully. Additionally depending on your design you may need to configure NAT.
In my setup, I’m using:
Cisco CSR 1000v with IOS XE 3.9x running in Verizon Terremark Enterprise Cloud
Cisco 2900 running IOS 153.2.T (branch office)
Here are the configs of the CSR 1000v (VPN Server). Only relevant configs are shown:
CSR-1#sh run hostname CSR-1 ! aaa new-model ! ! aaa authentication login hw-client-groupname local aaa authorization network hw-client-groupname local ! ! aaa session-id common ! ! username myusername password 0 mypassword ! ! crypto isakmp policy 1 encr 3des authentication pre-share group 2 crypto isakmp client configuration address-pool local dynpool ! crypto isakmp client configuration group hw-client-groupname key hw-client-password dns 8.8.8.8 domain domain.com pool dynpool save-password ! ! crypto ipsec transform-set transform-1 esp-3des esp-sha-hmac mode tunnel ! ! ! crypto dynamic-map dynmap 1 set transform-set transform-1 reverse-route ! ! crypto map dynmap client authentication list hw-client-groupname crypto map dynmap isakmp authorization list hw-client-groupname crypto map dynmap client configuration address respond crypto map dynmap 1 ipsec-isakmp dynamic dynmap ! ! ! interface GigabitEthernet2 description WAN ip address 10.22.36.71 255.255.255.224 negotiation auto crypto map dynmap ! interface GigabitEthernet3 description LAN ip address 10.22.39.91 255.255.255.240 negotiation auto ! ip local pool dynpool 192.168.1.1 ! ! ip access-list extended split_t permit ip 10.22.39.0 0.0.0.255 any ! end
Here are the configs for the Cisco 2900 router (VPN client).
no aaa new-model ! ip cef ! ! username vpntest password 0 vpntest ! ! crypto ipsec client ezvpn hw-client connect auto group hw-client-groupname key hw-client-password mode client peer 10.22.36.71 username myusername password mypassword xauth userid mode local ! ! interface GigabitEthernet0/0 description WAN ip address 10.35.120.104 255.255.255.0 duplex auto speed auto crypto ipsec client ezvpn hw-client ! interface GigabitEthernet0/1 description LAN ip address 172.16.4.102 255.255.255.0 duplex auto speed auto crypto ipsec client ezvpn hw-client inside ! ip route 0.0.0.0 0.0.0.0 10.35.120.1 ! end
I have been looking for a way (other than VPN) to secure my internet connection when I’m working outside the office at Starbucks or attending a conference and using unsecure/open WiFi hotspot.
I stumbled upon Sidestep for Mac the other day and decided to give it a try. It has worked well so far and i really like it because it’s simple, lightweight, free, and best of all connects automatically if it detects I’m on unsecure wireless connection.
Sidestep uses SSL tunneling to connect to a proxy server (SSH server) and encrypts your data so that other people connected to the same network cannot intercept your unencrypted traffic.
I first used a server I had at home as my proxy gateway but the speed was not so great especially when I was streaming videos and decided to set up a proxy server on Amazon Web Services (AWS) instead. That obviously solved my problem.
In this post I’m going to show how easy it’s to set up a proxy server on AWS and secure your open wireless connection:
Head over to AWS and sign up for an account if you don’t already have one. If you are a new customer you can use their Free Usage Tier which gives you the services you need for this setup for a year. If you are existing customer and no longer eligible for their free tier, you would have to pay for the service but it is actually pretty cheap and you can shut down the instance when you are not using it to even save more.
Use the EC2 Launch Instance wizard and create an m3.medium Ubuntu instance. Make sure to allow SSH (TCP port 22) from Anywhere in your security group and download the public SSH key to your machine.
Once the instance is ready, click on it and copy the public DNS name from the bottom pane.
On your Mac, launch a terminal window, go to the folder where you stored the SSH key (cd <directory>)and execute the following command to make the SSH key readable only by you: chmod 400 <ssh_key.pem>
Now it’s time to test the connection and connect to the server. Launch Sidestep on your Mac and go to Preferences. From the General tab make sure there that Reroute automatically when insecure and Run Sidestep on login are both checked.
Click on the Proxy Server tab and enter your username (the default should be “ubuntu”) and the hostname (which is the DNS name of the instance)
Click on the Advanced tab and in the Additional SSH Arguments field enter the following argument, this will tell Sidestep to use the SSH key and where to find it: -i <path to your SSH key>/<SSH key>
Go back to the Proxy Server tab and click on Test Connection to Server. At this point your should see a “Connection succeeded!” message.
Close the Preferences window now and click on Connect to get connected. If you open your browser now and search for “what is my IP”, your IP address will be the same as the EC2 instance IP address. From here Sidestep will automatically terminate the connection if you switch to a secure connection.
Note: if you decide to shut down your EC2 instance when you are not using it, please be aware that its public IP address and DNS name will be different after you turn it back on. That means you will need to update the hostname field in Sidestep every time you stop/start your instance. One way to work around that is to assign an Elastic IP (static) to your instance or install Linux dynamic update client on your Ubuntu.
I will be attending Cisco Live 2014 next month in San Francisco. I’m not presenting this year so hopefully will have more time to talk to vendors/solution providers and hang out more. If you are a fellow networking professional and attending drop me a line if you would like to meet up and chat over coffee or lunch.
The Cisco Cloud Services Router (CSR) 1000v is Cisco’s first virtual router and it runs as a VM on x86 virtualized server. The CSR 1000v runs Cisco IOS XE and brings many benefits to a cloud environment where it can operate as a secure VPN gateway to terminate site-to-site IPSEC tunnels. Other use cases of the CSR 1000v include MPLS WAN termination and control & traffic redirection. You can find more information n this virtual router on the Cisco CSR 1000v page.
In this post I’m going to show you how to install the CSR 1000v on the Verizon Terremark eCloud. If you are not familiar with the Verizon Terremark cloud console, you can go here for some video tutorials.
Download A Free Trial Image
Go to the Cisco Download Center and log in using your Cisco credential. Select the latest software release from the left pane and then download the .iso package from the right pane to your desktop. Don’t download the OVA or BIN packages. This free trial should be good up to 6 months.
You can purchase a license from Cisco and upgrade to a full version after you install the CSR.
Create the VM
Go over to the Verizon Terremark Enterprise Cloud (eClould) page and log in using your credential.
The first step here is to create a blank server. A blank server is just a container inside the VMware environment that will hold the Cisco IOS later. So go to Devices -> Create Server -> Blank Server to launch the wizard.
On the next screen select Linux for the OS Family, Other 2.6x Linux (64-bit) for the Version. Give the server a name and a description (optional). Then choose an INT network andchoose in which Row and Group you want to place the router.
On the next screen choose 4 or more for the Processors and 4096 MB or more for Memory.
On the next screen enter 8 for the Disk size or choose a Detached Disk that’s at least 8GB. Click Next.
From the next window you can optionally assign tags or just click Next if you don’t want any tag.
On the next window review your configurations, check the Agreement box, and click Deploy. Give it about 2 minutes and the server should be ready. If you go to the row and group where you placed the router you should see a greyed out icon (powered off) there.
If you recall we attached only one network to the server earlier, however the CSR requires a minimum of 3 interfaces to run so next we are going to create two additional networks and attach them to the blank server. Right click on the server and go to Configure to launch the wizard.
On the next screen choose Configure the Server and click Next.
On the next screen click on Network Settings from the top. At this point you should see only one INT network under the Current Connection. Click Add a Connection from the top right to add a new DMZ Network.
Go click again on Add a Connection and this time choose an INT network (INT_XXX). Choose a unique INT network different from the one you have used when you created the blank server. Click Save.
Now the VM will take a minute or so to reconfigure itself.
Once the wheel stops spinning and VM is ready, right click on it and go to Power On. Powering on this VM should take a minute or so and while the machine is powering on an icon with gear spinning is presented. The greyed out icon should turn into an icon with “blue light” once the server is completely powered on.
Console to the VM
In order to console to the blank server you must VPN first and use the VMware remote console plugin in Firefox or Internet Explorer to connect to the server for the first time. So for the following steps make sure you are using either Firefox or IE. Once the CSR is up and running you can switch back to your favorite browser or use a Telnet/SSH client to connect it directly instead of using the console.
Click on the VPN Connect link from the top of the page. This will install the Cisco AnyConnect Mobility Client on your machine if you don’t already have it and connect you via secure SSL VPN to eClould. If the browser prompts you to install any Java or ActiveX plugin during this step make sure to accept.
Now click on the router icon and then click on Connect from the bottom pane. This will start installing the VMware remote console plugin if it’s not already installed on your machine and will connect you to the server console to install the CSR.
By now you should see “Operating system not found” message on the console screen. Go to Devices-> CD/DVD drive -> Connect to Disk Image File ISO. This will open a new window for you to locate the ISO package.
Select the ISO package from your local machine and click OK. The system will start uncompressing/unzipping the package and installing software. Depending on the file size and your network speed this process may take up to 2 or 3 hours. Come back in 3 hours to check on the install process. If the console session times out, reconnect again as you did previously.
Once the CSR is successfully installed, you should see the traditional Cisco router prompt and you can then log in and start configuring it.
Assigning IP Addresses
In this section we will assign IP addresses to the CSR 1000v interfaces. When we created the VM we added 3 interfaces for the CSR 1000v to use (INT_XXX, DMZ_XXX, and INT_XXX). Each CSR interface will map to a logical vNIC assigned by the VMware hypervisor. The vNIC in turn is mapped to a physical MAC address.
The following steps are very important and you should not begin configuring any features on the CSR 1000v before you execute all of them.
First we need to find the CSR interface to vNIC mappings so go ahead and create the following table on a piece of paper. You will be populating its fields along the way:
NIC ID
Network
MAC
CSR Interface
IP/Mask
Default Gateway
1 (eth0)
INT
2 (eth1)
DMZ
3 (eth2)
INT
Log in to the CSR console and issue the following command:
show platform software vnic-if interface-mapping
You should see three interfaces with their vNIC and MAC addresses mapping. If you are running IOS XE 3.10S or earlier the first interface from the top should be GigabitEthernet0 and it’s the management interface. On the other hand if you are running 3.11S or higher the first interface is usually GigabitEthernet1. Start populating the table you have created n the previous step with the information from the command you just executed (IP information will come in the next steps) .
Go back to the eClould portal, click on the router and go then to Administrative Tasks -> Manage PIs
From the Networks section, click on the drop-down menu and select your DMZ network. Then from the left pane Available IPs select one IP address and click on the rightgreen arrow to assign it to the server. Click Save.
Repeat the previous step to assign IP addresses for the INT (internal) interfaces.
Go back to the table and fill out the IP addresses, mask, gateway sections before you move on.
Now that you have the IP addresses, masks, and interface mappings start configuring the DMZ and INT interfaces of the CSR by issuing the following commands for each interface:
config t int gX ip add <ip> <mask> description <Outside or LAN interface>
Configure A Default Route
The final step here is to assign a default route so that the CSR can reach the internet. The CSR must use the DMZ default gateway to reach the internet because only the DMZ network can reach the internet because the INT networks are not visible from the outside. So copy from the table you have created the default gateway IP address for the DMZ network.
Go back to the CSR console and issue the following commands. Make sure that you replace the text in italic with the IP address of the default gateway
config t
ip route 0.0.0.0 0.0.0.0 <DMZ_Default_gateway>
end
write mem
Try now pining the default gateway, you should get a response:
ping <DMZ_default_gateway>
At this point, the CSR should be able to reach the internet. Issue the following ping to make sure:
ping 8.8.8.8
If you don’t get a response then something is wrong and you should revisit the previous steps.
Private VLAN (PVLAN) is a security feature which has been available for quite some time on most modern switches. It adds a layer of security and enables network admins to restrict communication between servers in the same network segment (VLAN). So for example let’s say you have an Email and Web servers in the DMZ in the same VLAN and you don’t want them to communicate with each other but still want each server to communicate with the outside world. Obviously one way to prevent the servers from talking directly to each other is to place each server in a separate VLAN and apply ACLs on the firewall/router preventing communication between the two VLANs. This solution though requires using multiple VLANs and IP subnets. It also requires you to re-IP the servers in an existing environment. But what if you are running out of VLANs or IP subnets and/or re-IPing is too disruptive? Well then you can use PVLAN instead.
With PVLAN you can provide network isolation between servers or VMs which are in the same VLAN without introducing any additional VLANs or having to use MAC access control on the switch itself.
While you can configure PVLAN on any modern physical switch, this post will focus on deploying PVLAN on a virtual distributed switch in a VMware vSphere environment.
But first let me explain briefly how PVLAN works. The basic concept behind Private VLAN (PVLAN) is to divide up the existing VLAN (now referred to as Primary PVLAN) into multiple segments , called secondary PVLANs. Each Secondary PVLAN type then can be one of the following:
Promiscuous: VMs in this PVLAN can talk to any other VMs in the same Promiscuous PVLAN or any other secondary PVLANs. On the diagram above, VM E can communicate with A, B, C, and D.
Community: VMs in this secondary PVLAN can communicate with any VM in the same Community PVLAN and it can communicate with the Promiscuous PVLAN as explained above. However VMs in this PVLAN cannot talk to the Isolated PVLAN. So on the diagram, VM C and D can communicate with each other and communicate also with E.
Isolated: A VM in this secondary PVLAN cannot communicate with any VM in the same Isolated PVLAN nor with any VM in the Community PVLAN. It can only communicate with the Promiscuous PVLAN. So looking at the diagram again, VM A and B cannot communicate with each other nor with C or D but can communicate with E.
There are few things you need to be aware of when deploying PVLAN in a VMware vSphere environment,:
PVLAN is supported only on distributed virtual switches with Enterprise Plus license. PVLAN is not supported on a standard vSwitch.
PVLAN is supported on vDS in vSphere 4.0 or later; or on Cisco Nexus 1000v version 1.2 or later.
All traffic between VMs in the same PVLAN on different ESXi hosts need to traverse the upstream physical switch. So the upstream physical switch must be PVLAN-aware and configured accordingly. Note that this required only if you are deploying PVLAN on s vSphere vDS since VMware applies PVLAN enforcement at the destination while Cisco Nexus 1000v applies enforcement at the source and therefore allowing PVLAN support without upstream switch awareness.
Next I’m going to demonstrate how to configure PVLAN on VMware vDS and Cisco Nexus 1000v. Stay tuned for that. In the meantime feel free to leave some comments below.
Q. Is the FAB1 module supported with SUP2 or SUP2E?
A. Yes, supported with both supervisors.
Q. What minimum software release do I need to support SUP2 or SUP2E?
A. NX-OS 6.1
Q. Can I still run NX-OS version 6.0 on SUP1?
A. Yes.
Q. Can I upgrade SUP2 to SUP2E?
A. Yes. You would need to upgrade both the CPU and memory on board.
[8/14/2014] update: after further investigation I found that the answer is no (upgrade is not possible).
Q. I need to enable high-availability (HA), can use one SUP1 with one SUP2 in the same chassis?
A. No, for high-availability the two supervisors must be of the same type so you would need to use either SUP1/SUP1 or SUP2/SUP2.
Q. How many I/O modules can I have in a 7004?
A. Maximum of 2. The other 2 slots are reserved for the supervisors and you cannot use them for I/O modules.
Q. FAB1 or FAB2 on 7004?
A. The Nexus 7004 chassis does not actually use any FAB’s. The I/O modules are connected back to back.
Q. How many FEX’s can the Nexus 7000 support?
A. 32 FEX’s with SUP1 or SUP2; and 48 FEX’s with SUP2E. [8/14/2014] update: 64 FEXs with SUP2E or SUP2
Q. How many VDC’s can the Nexus 7000 support?
A. 4 VDC’s (including 1 VDC for management) with SUP1 or SUP2; and 8 + 1 (management) VDC’s with SUP2E.
Q. Which modules support FabricPath, FCoE, and FEX connectivity?
A. FabricPath is supported on all F1 and F2 modules. FCoE is supported on all F1 modules and F2 modules except on the 48 x 10GE F2 (Copper) module. FEX is supported on all F2 modules. Use this link from Cisco as a reference.
[8/14/2014] update: The F2e module supports FCoE, FEX, and FabricPath. The F3 module (12 port 40GE) supports FEX, FabricPath, FCoE, OTV, MPLS and LISP.
Q. Which modules support LISP, MPLS, and OTV?
A. All M1 and M2 modules support MPLS and OTV. LISP is supported only on the 32 x 10GE M1 module.
Q. Does the Nexus 7004 support SUP1?
A. No, the Nexus 7004 supports only SUP2 and SUP2E.
Q. Can I place an F2 module in the same VDC with F1 or M module?
A. No, the F2 module must be placed in a separate VDC so if you plan to mix F2 with F1, and M modules in the same chassis you would need a VDC license.
[8/14/2014] update: The F2e and F3 (12 port 40GE) modules can interoperate with the M-series in the same VDC.
Q. How can I upgrade from FAB1 to FAB2 modules during operation without any service disruption?
A. Yes, if you replace each module within a couple of minutes. Just make sure to replace all FAB1 with FAB2 modules within few hours. If you mix FAB1 with FAB2 modules in the same chassis for a long time, the FAB2 modules will operate in backward compatible mode and downgrade their speed to match the FAB1 modules peed. You can follow this link for step-by-step procedure for upgrading the FAB modules on the Nexus 7000.
Q. Can I use FAB1 modules in a 7009 chassis?
A. No, the Nexus 7009 uses only FAB2 modules.
Q. Does the Nexus 7000 support native Fibre Channel (FC) ports?
A. No, FC ports are not supported on the Nexus 7000. You would need either the Nexus 5500 or the MDS 9000 to get FC support.
Here is a video tour in HD of Cisco’s state-of-the-art data center in Allen, TX. It’s very impressive data center design featuring unique fresh air cooling system. It works together with the Richardson data center to provide active-active failover. Watch Robb Boyd and Jimmy Ray Purser from Cisco’s TechWise TV give the tour:
Below is also a cool interactive tour of Cisco’s data centers in Richardson, TX (Building 9) and Northern California where you can explore approaches to design, implementation, and management of Cisco’s global data centers and get insights from their architect and facilities manager.
When deploying blade servers it’s always recommended to use blade switches in the chassis for cabling reduction, improved performance and lower latency. However blade switches increase the complexity of the server access layer and introduce extra layer between the servers and the network. In this post I will go though few options for connecting the popular HP C-class BladeSystem to a Cisco unified fabric.
HP Virtual Connect Flex-10 Module
The Virtual Connect Flex-10 Module is a blade switch for the HP BladeSystem c7000 and c3000 enclosures. It reduces cabling uplinks to the fabric and offers oversubscription ratio of 2:1.
It has (16) 10GE internal connectors for downlinks and (8) 10Gb SFP+ for uplinks. This module however does not support FCoE so if you are planning on supporting Fibre Channel (FC) down the road you would need to add separate module to the chassis for storage. It also does not support QoS so that means you will need to carve up manually the bandwidth on the Flex-NICs exposed to the vSphere ESX kernel for vMotion, VM data, console, etc. This could be inefficient way of bandwidth assignment as the Flex-NIC would get only what’s assigned to it even if the 10G link is idle.
This module adds additional management point to the network as it has to be managed separately from the fabric (usually by the server team). The HP Virtual Connect Flex-10 module is around $12,000 list price.
HP 10GE Pass-Thru Module
The HP 10GE Pass-Thru module for the BladeSystem c7000 and c3000 enclosures acts like a hub and offers 1:1 oversubscription ratio. It has 16 connectors for downlinks and (16) 1/10GE uplink ports. It supports FCoE and the uplink ports accept SFP or SFP+ optics.
As shown the picture above this module can be connected to a Nexus Fabric Extender (FEX) such as the Nexus 2232PP which offers (32) 10GE ports for server connectivity or you can connect the module to another FEX with support for only 1GE downlinks if your server do not need the extra bandwidth. This solution is more attractive than the first option of using Virtual Connect Flex-10 module because it’s pass-thru and supports FCoE so you would not need another module for storage. And because it’s a pass-through it wouldn’t act like a “man in the middle” between the fabric and the blade servers.
Finally with this solution you have the option of using VM-FEX technology on the Nexus 5500 since both the HP pass-thru module and the Nexus 2200 FEX are transparent to the Nexus 5500. This module is around $5000 list price.
Cisco Fabric Extender (FEX) For HP BladeSystem
The Cisco B22 FEX was designed specifically to support the HP BladeSystem c7000 and c3000 enclosures. Similar to the Cisco Nexus 2200 it works like a remote card and is managed from the parent Nexus 5500 eliminating multiple provisioning and testing points. This FEX has 16 internal connectors for downlinks and (8) 10GE uplink ports. It does support FCoE and the supported features on it are on par with the Nexus 2200.
By far this is the most attractive solution for connecting HP BladeSystem to a Cisco fabric. With this solution you need to manage only the Nexus 5500 switches and you have support for FCoE, VM-FEX, and other NX-OS features. This B22 FEX is sold by HP (not Cisco) and its priced around $10,000 list price. The Nexus 5500 supports up to 24 fabric extenders in Layer 2 mode and up to 16 fabric extenders in Layer 3 mode.
When building data center networks using Cisco Nexus switches you can choose to attach the Nexus 2000 Fabric Extender (FEX) to a Nexus 5000 or 7000 depending on your design requirements and budget. In a previous post I briefly described the benefits of Virtual PortChannel (vPC) and discussed design options for the Nexus 2000/5500/5000. In this post I will go over the vPC design options for the Nexus 2000/7000 and important things to consider while creating the design.
Without vPC
The picture above shows how you can connect a Nexus 2000 to its parent switch Nexus 7000 without using vPC. Topology A on the left shows a single attached Nexus 2000 to a 7000 and a server connected to a server port on the Nexus 2000. There is no redundancy in this topology and failure of the Nexus 7000 or 2000 would cause the server to lose connectivity to the fabric. In this design you can have up to 32 FEX’s per Nexus 7000 with Sup1/2 or 48 FEX’s with Sup2E.
Topology B on the right has also no vPC and NIC teaming in this case is used for failover. The solid blue link is the primary connection and the dotted link is the backup. It’s up to the OS on the server to detect any failure upstream and fail over to the backup link. Similar to A in this design you can have up to 32 FEX’s per Nexus 7000 with Sup1/2 or 48 FEX’s with Sup2E.
With vPC
The picture above hows the supported vPC topology for the Nexus 7000. Topology C is called straight-through vPC in which each Nexus 2000 (FEX) is connected to one parent Nexus 7000 while the server is dual attached to a pair of Nexus 2000. In this case NIC on server must support LACP so that the two FEX’s appear as a single switch. Most modern Intel and HP NIC’s support LACP today. This topology supports up to 64 FEX’s (32 per Nexus 7000) with Sup1/2 or 96 FEX’s (48 per Nexus 7000) with Sup 2E.
Maximum Supported Nexus FEX As of Today:
Nexus 7000
Without vPC
32 with Sup1/2; 48 with Sup2E
Straight-through
64 with Sup1/2 (32 per Nexus 7000); 96 with Sup2E (48 per Nexus 7000)
Notes:
The Nexus 7000 modules that support FEX are: N7K-M132XP-12L (32 x 10GbE SFP+), N7K-F248XP-25 (48 x 10GbE SFP/SPF+), and all M2 modules. The F1, F2 copper, and 1GbE M1 modules don’t support FEX
All FEX uplinks must be placed in the same VDC on the Nexus 7000
Dual attaching the FEX to pair of Nexus 7000 is not supported as of today on the Nexus 7000 which means in the event of I/O module failure all FEX’s hanging off of that module will be knocked out. For this reason it’s recommended to have at least two IO modules in the chassis that support FEX and distribute the uplinks across those two modules for redundancy
If the FEX is going to be within 100 meters from the Nexus 7000, you can use Cisco Fabric Extended Transceiver (FET) on the uplinks which offers cost-effective way to connect the FEX to its parent switch. The FET is much cheaper than the 10G SFP+ optic
When building scalable Leaf-Spine fabrics, network architects look for low-latency, high-density switches to use at the leaf layer. There are many fixed switches that can be used for Top-0f-rack (ToR) at the leaf layer to provide connectivity upstream to the spine layer. What I’m about to compare are 3 ultra-low-latency ToR switches based on merchant silicon available in the market today for that purpose.
Cisco Nexus 3064
The 3064 is 1 RU heigh and has a low latency and low power consumption per port. It has (48) 1/10GbE ports and (4) 40 GbE uplinks which can be each used in native 40 GbE or split into four 10GbE ports. It runs the same Nx-OS as the Nexus 7000 and 5000 series.
The Nexus 3064 is Cisco’s first switch in the Nexus family to use merchant silicon (Broadcom Trident+ chip). I’m curious to see whether Cisco will continue to use merchant silicon in future products or stick to their propreitery Nuova ASIC of the 7000 and 5000 series.
Arista 7050S-64
Arista 7050S-64 is very similar to the Cisco Nexus 3064 in terms of latency, interface types, and switching capacity. Its power consumption is less than the Nexus 3064 though. Arista’s fixed switches are known for their low power consumption and the 7050S-64 is no exception. Its power consumption is under 2W per port. You really cannot beat that!
Dell Force10 S4810
The Dell Force10 S4810 is another great ToR switch that can be used to build leaf-spine fabrics. It offers the same interface types as the Nexus 3064 and Arista 7050s-64; and similar form factor. It does however have slightly higher power consumption per port.
When connecting the UCS chassis to the Fabric interconnect, it’s important to follow the design rules or otherwise you may end up with unexpected behavior. UCS supports up to two fabric extenders (2100/2200 series) per chassis and two fabric interconnects (6100/6200 series) per cluster. To have a fully redundant system you will need two fabric extenders and two fabric interconnects connected as shown below the first picture (topology in the top right and top left).
Here are some UCS design rules to keep in mind:
Direct one-to-one relationship between the FEX and fabric interconnect. Meaning each Fabric Extender (FEX) can be connected only to a single fabric interconnect. You cannot dual-home a FEX. Similarly a fabric interconnect cannot connect to more than one FEX.
If you choose to have only one FEX in the chassis you must place that FEX into the left bay (as viewed from the rear of the enclosure)
When using two fabric interconnects for redundancy you must establish a cluster link between them by connecting the L1/L2 ports on the first fabric interconnect to the L1/L2 pots on the second one.
Correct IOM connectivity options:
The picture below shows the supported IOM connectivity options for UCS
Incorrect IOM connectivity options:
The picture below shows some unsupported IOM connectivity options for UCS
The first topology (upper left) is not supported because the links between the fabric interconnects are missing
The second topology (lower left) is not supported because both FEX’s are uplinked to the same fabric interconnect.
The third topology (upper right) is not supported because the FEX is dual-homed to the fabric interconnects.
The fourth topology (lower right) is not supported because there is only one FEX in the chassis so that FEX should be placed into the left bay.
Virtual PortChannel (vPC) allows two links that are connected to two different physical Cisco Nexus 5000 or 7000 switches to appear to the downstream device as a single PortChannel link. That downstream device could be a server, Nexus 2000, or any Classical Ethernet switch.
vPC is useful to prevent spanning tree from blocking redundant links in the topology. After all you have spent fortune and bought those expensive 10G ports and the last thing you want is for spanning tree to block them.
Having said that they are several ways to connect the Cisco Nexus Fabric Extender (FEX) to its parent the Nexus 5000 or 7000 switch. In this post I’m going to discuss supported vPC topologies for the Nexus series. I’m going to start with the Nexus 2000/5000 now and will add a separate post for the Nexus 2000/7000 options later.
Without vPC
The picture above shows the supported non-vPC topologies. Topology A on the left shows a straight forward connectivity between Nexus 2000 and 5000 with a server connected to a server port on the Nexus 2000. There is no redundancy in this topology and failure of the Nexus 5000 or 2000 would cause the server to lose connectivity to the fabric. In this design you can have up to 24 FEX’s per Nexus 5500 in L2 mode and 16 FEX’s in L3.
Topology B on the right has also no vPC and NIC teaming in this case is used for failover. The solid blue link is the primary connection and the dotted link is the backup. It’s up to the OS on the server to detect any failure upstream and fail over to the backup link. Similar to A in this design you can have up to 24 FEX’s per Nexus 5500 in L2 mode and 16 FEX’s in L3.
With vPC
The picture above hows the supported vPC topologies for the Nexus 5000. Topology C is called straight-through vPC in which each Nexus 2000 (FEX) is connected to one parent Nexus 5000 while server is dual-homed. In this case NIC on server must support LACP so that the two FEX’s appear as a single switch. Most modern Intel and HP NIC’s support LACP today. This topology supports up to 48 FEX’s (24 per Nexus 5500) in L2 mode and 32 FEX’s (16 per Nexus 5500) in L3 mode.
In topology D on the other hand each FEX is dual-homed and so is the server. So the NIC on the server must support LACP as in C. In this topology you can have up to 24 FEX’s in L2 mode and 16 FEX’s in L3.
Topology E is similar to D where each FEX is dual-homed but the server is single-homed. In this topology you can have up to 24 FEX’s in L2 mode and 16 FEX’s in L3.
I know this speech is all over the internet but I’m transferring some posts from my old blog to this new and it’s worth sharing this speech again it’s really classic.
This is really a must-see speech by Steve Jobs who shares 3 interesting stories from his life: college drop-out, work at Apple, and his battle with cancer. The speech is from his Commencement address at Stanford in 2005.
Some pictures from my wonderful trip to Ecuador and the Galapagos Islands. I visited San Cristobal, Santa Maria, Santa Cruz, and Espanola. And got to see all kinds of birds and animals. Here are some of the pics I have taken while exploring the wild and marine life including some pictures of sexy boobies
This interview with Steve Jobs surfaced a day after his death. I listened to it for the first time this week and could not help but notice how accurate Steve’s vision of the future was.
In the interview he predicted:
The use of Dynamic Web Pages would continue to grow
People would start searching the web for information and not just products
Graphical user interface is the way to go; users would want to see buttons and nice pictures on their screen
Everyone would eventually be using a computer and not just 5% of the population
All of his predictions from 1996 have come true today..That’s really the genius of Steve Jobs; he was a visionary who saw into the future – something many leaders are incapable of doing today.