Fun lesson on VRRP on Nexus

Date April 8, 2014

I'm in the middle of migrating our upstream links from the old 6500 core to the new Nexus switches, and I discovered something fun today that I thought I'd share.

Before, since I only had a single switch, each of my subnets had a VLAN interface which had the IP address of the gateway applied, such as this:

interface Vlan100
 description VLAN 100 -- Foo and Bar Stuff
 ip address 192.168.100.1 255.255.255.0
 no ip redirects
 ip dhcp relay information trusted
 ip route-cache flow
end

Pretty simple. But in the new regime, there are two switches, and theoretically, each switch should be fully capable of acting as the gateway. This is clearly a case for VRRP.

One Core01, the configuration looks like this:

interface Vlan100
  description Foo and Bar Stuff
  no shutdown
  ip address 192.168.100.2/24
  management
  vrrp 100
    authentication text MyVrrpPassword
    track 1 decrement 50
    address 192.168.100.1
    no shutdown

and on Core02, it looks like this:

interface Vlan100
  description Foo and Bar Stuff
  no shutdown
  ip address 192.168.100.3/24
  management
  vrrp 100
    authentication text MyVrrpPassword
    track 1 decrement 50
    address 192.168.100.1
    no shutdown

The only difference is the IP address assigned to the interface.

(Incidentally, I'm tracking the upstream interface. If the link goes dead, it decrements 50 points to make sure that VRRP changes over the Virtual IP to the other interfaces).

When both switches were configured this way, "show vrrp" reported that both switches were set to Master:

core01(config-if)# sho vrrp
      Interface  VR IpVersion Pri   Time Pre State   VR IP addr
---------------------------------------------------------------
        Vlan100 100   IPV4     100    1 s  Y  Master 192.168.100.1

core02(config-if)# sho vrrp
      Interface  VR IpVersion Pri   Time Pre State   VR IP addr
---------------------------------------------------------------
        Vlan100 100   IPV4     100    1 s  Y  Master 192.168.100.1

That's clearly not good. I verified that both switches were actually sending traffic:

2014 Apr 8 10:58:20.968603 vrrp-eng: Vlan100[Grp 100]: Sent packet for VR 100, intf 0x9010073

When digging in, what I found was that the switches were getting traffic from duplicate IPs:

Apr 8 11:06:33 core01 %ARP-3-DUP_VADDR_SRC_IP: arp [3573] Source address of packet received from 0000.5e00.0173 on Vlan100(port-channel1) is duplicate of local virtual ip, 192.168.100.1

On a whim, the other admin I was working with at the upstream provider had me disable the "management" flag. And of course, that made things start working immediately.

Apparently, setting the management flag (which ostensibly allows you to manage the switch in-band by using the address assigned to the interface) ALSO aggressively uses the VIP as its source address. I don't know why. It seems like a bug to me, but I'm going to get in touch with the TAC today and see if it's a known thing or not.

I thought you might be interested in knowing about this, anyway. Thanks for reading! (and if you have more information as to why this happens, please comment!)

  • Howard Wilkerson

    What exatcly is the management flag and how did you remove it? Thanks