Month: September 2014

Troubleshooting vMotion Connectivity Issues

I attended a panel discussion session at VMUG this week. A guy from the audience asked a question on how to troubleshoot connectivity issues after moving a VM. The guy had a one flat VLAN with one IP subnet and every time he vMotioned a VM to another host, users lost connectivity with that VM.

To answer his question, a guy on the stage advised him to recreate another VLAN/IP subnet, move that VM to the newly created subnet, and then try to do the vMotion again.

Well obviously that wasn’t a good advice and changing the VM IP address in this case would not help. In this post I will explain what happens when a VM moves and steps you can take to troubleshoot network delay and connectivity issues related to vMotion.

First before you do any vMotion, ensure that the VLAN is configured on all switches and allowed on all necessary trunk ports in the network. You can use the commands show vlan and show trunk to verify that that.

When you create a new VM, the host allocates and assigns a MAC address to that VM. The physical switch which the host is connected to eventually learns the VM MAC address (that happens when the VM ARPs for its gateway, starts sending traffic and the switch sees that traffic, or when the Notify Switches in vSphere option is tuned on).

When a VM moves to another host (the new host could be either connected to another port or the same physical switch or connected to a different switch), somebody has to tell the network to change its destination port for that MAC address in order to continue to deliver traffic to that VM. In vSphere that is usually handled by the host when the “Notify Switches” option is turned on. The host in this case notifies the network by sending several RARP messages on behalf of the VM to ensure that the upstream physical network updates its MAC table.

So the first step in troubleshoot this specific issue is to ensure that the “Notify Switches” settings is set to Yes before you vMotion the VM. In vSphere 5.5 you would go to the vSwitch/vDS settings and then to Teaming and Failover to verify the setting.  

Notify switches


If that does not help, try to do the followings:

On the physical switch (the switch the destination host is connected to), issue a command to look up MAC table and find out which destination port the switch is using to reach the VM. On a Cisco Catalyst switch, you can use the “show mac address-table” command for example. If the switch is still using the old MAC-to-port mapping, then it’s either that the switch is not reaching the RARP notification from the host or the host itself is not sending it.

If you don’t do anything to correct the problem, the aging timer for that MAC address on the switch (default is set to 5 mins on Cisco switches) will eventually expire and the switch will learn the MAC address via the new port but to speed things up you can alway flush out the VM MAC address from the switch MAC table (clear mac address-table on Cisco catalyst).

Obviously having to manually intervene and troubleshoot every time you move a VM defeats the whole purpose of vMotion. vMotion is supposed to do live migration of a VM and preserves all active network connections during the process. But hopefully the above steps will give you some pointers on where to look and start to find the root cause.

Additionally VMware recommends disabling the followings on the physical ports connected to the host to minimize networking delay:

– Port Aggregation Protocol (PAgP) and Link Aggregation Control Protocol (LACP). 

– Dynamic Trunking Protocol (DTP) or trunk negotiation.

– Spanning Tree Protocol (STP).


Share This:

Building A Private Cloud – Introduction


I’m starting a new post series to walk you through building a private cloud. This post is the first one in the series and will be just an introduction to private clouds. 


In the next posts I will discuss collecting the requirements, building the network and network services, building the storage, designing the server infrastructure, and putting the orchestration layer on top of that.   



Why Private Cloud?


There are number of public cloud providers out there that have good and affordable offerings. However customers sometimes prefer private cloud over public cloud for specific reasons or requirements they need to meet. Here are some the reasons that come to mind:


  • Regulatory Compliance: Some companies have have to comply with certain regulations and therefore have to keep their customer data or their own data on dedicated/isolated infrastructure behind their firewall. In this case the multi-tenant public cloud model won’t be a good fit for them.
  • Control: Some companies require more control over the cloud infrastructure than what public cloud providers usually offer or they might have a specific need to integrate a solution/product from certain vendor. Most public cloud providers offer standard services that may not be customized or tweaked easily. 
  • Avoid Lock-in: I’m sure you heard this before. People don’t like to put all of their eggs in one basket nor like to heavily rely on a specific vendor/provider. By building their own cloud in house companies can have the freedom of choosing multiple vendors and can also leverage open source technology if they wish.
  • Cost: This might be a bit of surprise to you but depending on how much the monthly bill coming from the public cloud provider is, operating a private cloud might become a cheaper option. This is especially true if you are using the cloud to run production workloads that need to be up 24/7 and cannot be turned off while not in use.  


Barriers to Private Cloud Deployments:


  • Lack of Technical Resources: Designing, implementing, and supporting a private cloud require having the right IT resources in house. Some companies just don’t have that.
  • Time to Market: Getting a private cloud up and running is at least 6-12 month long process. Some companies have to respond to market pressure quickly and simply don’t have the luxury of spending a year to build a cloud in house.
  • Cost: Building and operating a private cloud is expensive and it makes more sense in some cases to  leverage a public cloud offering than spending that CapEx up front.


You have the capital and technical talent and want to build a private cloud? Great. In the next post I will walk you through the process of collecting the requirements and making high-level design decisions before we get into building the infrastructure itself. Stay tuned.


Make sure that you either subscribe to my blog or follow me on Twitter to get notified when I add a new content.





Share This:

© 2018

Theme by Anders NorenUp ↑