Year: 2012

How To Connect HP BladeSystem c7000/c3000 To Cisco Unified Fabric

August 8, 2012 / / 1 Comment

When deploying blade servers it’s always recommended to use blade switches in the chassis for cabling reduction, improved performance and lower latency. However blade switches increase the complexity of the server access layer and introduce extra layer between the servers and the network. In this post I will go though few options for connecting the popular HP C-class BladeSystem to a Cisco unified fabric.

HP Virtual Connect Flex-10 Module

The Virtual Connect Flex-10 Module is a blade switch for the HP BladeSystem c7000 and c3000 enclosures. It reduces cabling uplinks to the fabric and offers oversubscription ratio of 2:1.

It has (16) 10GE internal connectors for downlinks and (8) 10Gb SFP+ for uplinks. This module however does not support FCoE so if you are planning on supporting Fibre Channel (FC) down the road you would need to add separate module to the chassis for storage. It also does not support QoS so that means you will need to carve up manually the bandwidth on the Flex-NICs exposed to the vSphere ESX kernel for vMotion, VM data, console, etc. This could be inefficient way of bandwidth assignment as the Flex-NIC would get only what’s assigned to it even if the 10G link is idle.

This module adds additional management point to the network as it has to be managed separately from the fabric (usually by the server team). The HP Virtual Connect Flex-10 module is around $12,000 list price.

HP 10GE Pass-Thru Module

The HP 10GE Pass-Thru module for the BladeSystem c7000 and c3000 enclosures acts like a hub and offers 1:1 oversubscription ratio. It has 16 connectors for downlinks and (16) 1/10GE uplink ports. It supports FCoE and the uplink ports accept SFP or SFP+ optics.

As shown the picture above this module can be connected to a Nexus Fabric Extender (FEX) such as the Nexus 2232PP which offers (32) 10GE ports for server connectivity or you can connect the module to another FEX with support for only 1GE downlinks if your server do not need the extra bandwidth. This solution is more attractive than the first option of using Virtual Connect Flex-10 module because it’s pass-thru and supports FCoE so you would not need another module for storage. And because it’s a pass-through it wouldn’t act like a “man in the middle” between the fabric and the blade servers.

Finally with this solution you have the option of using VM-FEX technology on the Nexus 5500 since both the HP pass-thru module and the Nexus 2200 FEX are transparent to the Nexus 5500. This module is around $5000 list price.

Cisco Fabric Extender (FEX) For HP BladeSystem

The Cisco B22 FEX was designed specifically to support the HP BladeSystem c7000 and c3000 enclosures. Similar to the Cisco Nexus 2200 it works like a remote card and is managed from the parent Nexus 5500 eliminating multiple provisioning and testing points. This FEX has 16 internal connectors for downlinks and (8) 10GE uplink ports. It does support FCoE and the supported features on it are on par with the Nexus 2200.

By far this is the most attractive solution for connecting HP BladeSystem to a Cisco fabric. With this solution you need to manage only the Nexus 5500 switches and you have support for FCoE, VM-FEX, and other NX-OS features. This B22 FEX is sold by HP (not Cisco) and its priced around $10,000 list price. The Nexus 5500 supports up to 24 fabric extenders in Layer 2 mode and up to 16 fabric extenders in Layer 3 mode.

Share This:

Cisco Nexus 2000/7000 vPC Design Options

August 3, 2012 / / 1 Comment

When building data center networks using Cisco Nexus switches you can choose to attach the Nexus 2000 Fabric Extender (FEX) to a Nexus 5000 or 7000 depending on your design requirements and budget. In a previous post I briefly described the benefits of Virtual PortChannel (vPC) and discussed design options for the Nexus 2000/5500/5000. In this post I will go over the vPC design options for the Nexus 2000/7000 and important things to consider while creating the design.

Without vPC

The picture above shows how you can connect a Nexus 2000 to its parent switch Nexus 7000 without using vPC. Topology A on the left shows a single attached Nexus 2000 to a 7000 and a server connected to a server port on the Nexus 2000. There is no redundancy in this topology and failure of the Nexus 7000 or 2000 would cause the server to lose connectivity to the fabric. In this design you can have up to 32 FEX’s per Nexus 7000 with Sup1/2 or 48 FEX’s with Sup2E.

Topology B on the right has also no vPC and NIC teaming in this case is used for failover. The solid blue link is the primary connection and the dotted link is the backup. It’s up to the OS on the server to detect any failure upstream and fail over to the backup link. Similar to A in this design you can have up to 32 FEX’s per Nexus 7000 with Sup1/2 or 48 FEX’s with Sup2E.

With vPC

The picture above hows the supported vPC topology for the Nexus 7000. Topology C is called straight-through vPC in which each Nexus 2000 (FEX) is connected to one parent Nexus 7000 while the server is dual attached to a pair of Nexus 2000. In this case NIC on server must support LACP so that the two FEX’s appear as a single switch. Most modern Intel and HP NIC’s support LACP today. This topology supports up to 64 FEX’s (32 per Nexus 7000) with Sup1/2 or 96 FEX’s (48 per Nexus 7000) with Sup 2E.

Maximum Supported Nexus FEX As of Today:

	Nexus 7000
Without vPC	32 with Sup1/2; 48 with Sup2E
Straight-through	64 with Sup1/2 (32 per Nexus 7000); 96 with Sup2E (48 per Nexus 7000)

Notes:

The Nexus 7000 modules that support FEX are: N7K-M132XP-12L (32 x 10GbE SFP+), N7K-F248XP-25 (48 x 10GbE SFP/SPF+), and all M2 modules. The F1, F2 copper, and 1GbE M1 modules don’t support FEX
All FEX uplinks must be placed in the same VDC on the Nexus 7000
Dual attaching the FEX to pair of Nexus 7000 is not supported as of today on the Nexus 7000 which means in the event of I/O module failure all FEX’s hanging off of that module will be knocked out. For this reason it’s recommended to have at least two IO modules in the chassis that support FEX and distribute the uplinks across those two modules for redundancy
If the FEX is going to be within 100 meters from the Nexus 7000, you can use Cisco Fabric Extended Transceiver (FET) on the uplinks which offers cost-effective way to connect the FEX to its parent switch. The FET is much cheaper than the 10G SFP+ optic

Share This:

Ultra-low-latency ToR Switches For Building Scalable Leaf-Spine Fabrics

July 26, 2012 / / 0 Comments

When building scalable Leaf-Spine fabrics, network architects look for low-latency, high-density switches to use at the leaf layer. There are many fixed switches that can be used for Top-0f-rack (ToR) at the leaf layer to provide connectivity upstream to the spine layer. What I’m about to compare are 3 ultra-low-latency ToR switches based on merchant silicon available in the market today for that purpose.

Cisco Nexus 3064

The 3064 is 1 RU heigh and has a low latency and low power consumption per port. It has (48) 1/10GbE ports and (4) 40 GbE uplinks which can be each used in native 40 GbE or split into four 10GbE ports. It runs the same Nx-OS as the Nexus 7000 and 5000 series.

The Nexus 3064 is Cisco’s first switch in the Nexus family to use merchant silicon (Broadcom Trident+ chip). I’m curious to see whether Cisco will continue to use merchant silicon in future products or stick to their propreitery Nuova ASIC of the 7000 and 5000 series.

Arista 7050S-64

Arista 7050S-64 is very similar to the Cisco Nexus 3064 in terms of latency, interface types, and switching capacity. Its power consumption is less than the Nexus 3064 though. Arista’s fixed switches are known for their low power consumption and the 7050S-64 is no exception. Its power consumption is under 2W per port. You really cannot beat that!

Dell Force10 S4810

The Dell Force10 S4810 is another great ToR switch that can be used to build leaf-spine fabrics. It offers the same interface types as the Nexus 3064 and Arista 7050s-64; and similar form factor. It does however have slightly higher power consumption per port.

Ultra-low-latency 10/40 GbE Top-of-Rack Switches

	Cisco Nexus 3064	Arista 7050S-64	Dell Force10 S4810
Ports	48 x 1/10GbE SPF+ and 4 x 40GbE QSFP+	48 x 1/10GbE SPF+ and 4 x 40GbE QSFP+	48 x 1/10GbE SPF+ and 4 x 40GbE QSFP+
Packet Latency (64 bytes)	824ns	800ns	700ns
OS	Nx-OS	Arista EOS	FTOS
Form Factor	1 RU	1 RU	1 RU
Switching Capacity	1.28 Tbps	1.28 Tbps	1.28 Tbps
Power Supply	2 Redundant & Hot swappable power supplies	2 Redundant & Hot swappable power supplies	2 Redundant & Hot swappable power supplies
Typical Operating Power	177W	103W	220W
Full Data Sheet	Data Sheet	Data Sheet	Data Sheet

Share This:

Cisco UCS Supported IOM Connectivity Options

July 22, 2012 / / 2 Comments

When connecting the UCS chassis to the Fabric interconnect, it’s important to follow the design rules or otherwise you may end up with unexpected behavior. UCS supports up to two fabric extenders (2100/2200 series) per chassis and two fabric interconnects (6100/6200 series) per cluster. To have a fully redundant system you will need two fabric extenders and two fabric interconnects connected as shown below the first picture (topology in the top right and top left).

Here are some UCS design rules to keep in mind:

Direct one-to-one relationship between the FEX and fabric interconnect. Meaning each Fabric Extender (FEX) can be connected only to a single fabric interconnect. You cannot dual-home a FEX. Similarly a fabric interconnect cannot connect to more than one FEX.
If you choose to have only one FEX in the chassis you must place that FEX into the left bay (as viewed from the rear of the enclosure)
When using two fabric interconnects for redundancy you must establish a cluster link between them by connecting the L1/L2 ports on the first fabric interconnect to the L1/L2 pots on the second one.

Correct IOM connectivity options:

The picture below shows the supported IOM connectivity options for UCS

Incorrect IOM connectivity options:

The picture below shows some unsupported IOM connectivity options for UCS

The first topology (upper left) is not supported because the links between the fabric interconnects are missing

The second topology (lower left) is not supported because both FEX’s are uplinked to the same fabric interconnect.

The third topology (upper right) is not supported because the FEX is dual-homed to the fabric interconnects.

The fourth topology (lower right) is not supported because there is only one FEX in the chassis so that FEX should be placed into the left bay.

Share This:

Cisco Nexus 2000/5000 vPC Design Options

June 29, 2012 / / 5 Comments

Virtual PortChannel (vPC) allows two links that are connected to two different physical Cisco Nexus 5000 or 7000 switches to appear to the downstream device as a single PortChannel link. That downstream device could be a server, Nexus 2000, or any Classical Ethernet switch.

vPC is useful to prevent spanning tree from blocking redundant links in the topology. After all you have spent fortune and bought those expensive 10G ports and the last thing you want is for spanning tree to block them.

Having said that they are several ways to connect the Cisco Nexus Fabric Extender (FEX) to its parent the Nexus 5000 or 7000 switch. In this post I’m going to discuss supported vPC topologies for the Nexus series. I’m going to start with the Nexus 2000/5000 now and will add a separate post for the Nexus 2000/7000 options later.

Without vPC

The picture above shows the supported non-vPC topologies. Topology A on the left shows a straight forward connectivity between Nexus 2000 and 5000 with a server connected to a server port on the Nexus 2000. There is no redundancy in this topology and failure of the Nexus 5000 or 2000 would cause the server to lose connectivity to the fabric. In this design you can have up to 24 FEX’s per Nexus 5500 in L2 mode and 16 FEX’s in L3.

Topology B on the right has also no vPC and NIC teaming in this case is used for failover. The solid blue link is the primary connection and the dotted link is the backup. It’s up to the OS on the server to detect any failure upstream and fail over to the backup link. Similar to A in this design you can have up to 24 FEX’s per Nexus 5500 in L2 mode and 16 FEX’s in L3.

With vPC

The picture above hows the supported vPC topologies for the Nexus 5000. Topology C is called straight-through vPC in which each Nexus 2000 (FEX) is connected to one parent Nexus 5000 while server is dual-homed. In this case NIC on server must support LACP so that the two FEX’s appear as a single switch. Most modern Intel and HP NIC’s support LACP today. This topology supports up to 48 FEX’s (24 per Nexus 5500) in L2 mode and 32 FEX’s (16 per Nexus 5500) in L3 mode.

In topology D on the other hand each FEX is dual-homed and so is the server. So the NIC on the server must support LACP as in C. In this topology you can have up to 24 FEX’s in L2 mode and 16 FEX’s in L3.

Topology E is similar to D where each FEX is dual-homed but the server is single-homed. In this topology you can have up to 24 FEX’s in L2 mode and 16 FEX’s in L3.

Maximum Supported Cisco FEX As of Today:

	Nexus 5000	Nexus 5500
Without vPC (L2 Mode)	12	24
Without vPC (L3 Mode)	X	16
Straight-through (L2 Mode)	24 (12 per Nexus 5000)	48 (24 per Nexus 5500)
Straight-through (L3 Mode)	X	32 (16 per Nexus 5500)
Dual-homed FEX (L2 Mode)	12	24
Dual-homed FEX (L3 Mode)	x	16

Share This:

Steve Jobs: How To Live Before You Die

January 7, 2012 / anas / 0 Comments

I know this speech is all over the internet but I’m transferring some posts from my old blog to this new and it’s worth sharing this speech again it’s really classic.

This is really a must-see speech by Steve Jobs who shares 3 interesting stories from his life: college drop-out, work at Apple, and his battle with cancer. The speech is from his Commencement address at Stanford in 2005.

Share This:

Pictures From My Trip To the Galapagos Islands & Ecuador

January 7, 2012 / anas / 1 Comment

Some pictures from my wonderful trip to Ecuador and the Galapagos Islands. I visited San Cristobal, Santa Maria, Santa Cruz, and Espanola. And got to see all kinds of birds and animals. Here are some of the pics I have taken while exploring the wild and marine life including some pictures of sexy boobies