Cisco Switch-Profile Issues

Date December 27, 2013

So, whenever you've got a Fabric Extender attached to more than one switch, you need to configure the shared ports in both places. To make life "easier", Cisco has the concept of "switch profiles", where the two switches will have what amounts to a template, that gets applied. That way you don't have to worry about doing the same thing in each place. Theoretically, it's an awesome idea. In practice, though, I'm not even running the things in production and it's causing me all kinds of problems.

Here's an example:


core01# config sync
Enter configuration commands, one per line. End with CNTL/Z.
core01(config-sync)# switch-profile core-shared
Switch-Profile started, Profile ID is 1
core01(config-sync-sp)# int po1
core01(config-sync-sp-if)# switchport trunk allowed vlan 1-314,1050-1065
Error: This PC has already been configured outside switch-profile. Please configure further commands outside switch-profile or import the port-channel within switch-profile

So, lets check on it.

core01# sh run int po1

!Command: show running-config interface port-channel1
!Time: Fri Dec 27 19:55:43 2013

version 6.0(2)N1(2)

interface port-channel1
switchport mode trunk
switchport trunk allowed vlan 1050-1065
spanning-tree port type network
vpc peer-link

Alright. And in the switch-profile?


core01# sh run switch-profile | section .*interface.port-channel1$
interface port-channel1
switchport mode trunk
switchport trunk allowed vlan 1050-1065
vpc peer-link

Alright, yes, As you can see in the running-config, the 'spanning-tree port type network' line exists, but not in the switch-profile. So, now we have to get it out of there. The obvious answer seems to be to go into 'configure terminal' mode (now just 'configure' in NX-OS) and take it out. Lets do that:


core01(config-sync-sp-if)# buffer-delete all
core01(config-sync-sp)# config t
core01(config)# int po1
core01(config-if)# switchport trunk allowed vlan 1-314,1050-1065
Error: Command is not mutually exclusive
core01(config-if)#

So we can't update the switch-profile because it's got configuration outside of the profile, and we can't update it outside of the profile, because it's managed by the profile? Ooooookay. Maybe we can import it.


core01# config sync
Enter configuration commands, one per line. End with CNTL/Z.
core01(config-sync)# switch-profile core-shared
Switch-Profile started, Profile ID is 1
core01(config-sync-sp)# import interface po1
core01(config-sync-sp-import)# verify
Failed: Verify Failed

Well, huh.


core01(config-sync-sp-import)# sh switch-profile status

switch-profile : core-shared
----------------------------------------------------------

Start-time: 840920 usecs after Fri Dec 27 20:06:57 2013
End-time: 59573 usecs after Fri Dec 27 20:06:59 2013

Profile-Revision: 86
Session-type: Import-Verify
Session-subtype: -
Peer-triggered: No
Profile-status: Verify Failed

Local information:
----------------
Status: Verify Success
Error(s):

Peer information:
----------------
IP-address: 129.10.108.61
Sync-status: In sync
Status: Verify Failure
Error(s):
Following commands failed mutual-exclusion checks:
interface port-channel1
spanning-tree port type network

Alright, so the local switch successfully verified, but the remote switch failed. Lets check over there.


core02# sh run int po1

!Command: show running-config interface port-channel1
!Time: Fri Dec 27 20:09:10 2013

version 6.0(2)N1(2)

interface port-channel1
switchport mode trunk
switchport trunk allowed vlan 1050-1065
spanning-tree port type network
vpc peer-link

core02# sh run switch-profile | section .*interface.port-channel1$
interface port-channel1
switchport mode trunk
switchport trunk allowed vlan 1050-1065
vpc peer-link

Yeah, that's pretty much exactly what core01 said, too.

Things like this are why I want to drink.

sigh. I'm going to be calling the TAC. Again, for the second time this week.

UPDATE

Alright, after talking with Cisco's Technical Assistance Center (TAC), here's what I've found out.

  • The answer to almost all of the inconsistency problems I've seen regarding switch profiles is to break the peer-sync (using no sync-peers destination <destination>), make each of the changes locally so that each switch is locally coherent between its local config and the switch-profile config, make sure that the two switches' switch-profiles are 100% completely and absolutely identical, and then set up the peer-sync again just as it was before.
  • Although the guy on the phone said that there's no document stating this, they have found that most of the problems related to switch-profile syncing is on non-FEX interfaces. That is, interfaces that are 'local' to the Switch itself, or a port channel. He recommended that I only use switch-profiles for those interfaces that are actually shared between the switches because they live on a dual-homed FEX.

There's no way to remove an interface from a switch-profile, either. The only solution is to blow away the profile and re-create. As much...uhh...fun, as that sounds, it's not something that I'd like to be doing a lot of. Anyway, hopefully this can help someone else out. Good luck, and let me know if you come across anything weird with switch-profiles, too. I'm interested to hear how many people are using them versus manually configuring each Nexus for each change you make. I'm SUPER interested if you use another tool to automate shared port configs on multiple switches. Comment below!

  • http://www.mostlynetworks.com Scott McDermott

    Yeah, I'm dealing with the same basic problem. I'm retiring an old pair of 5010s and we can't config any of the ports on one of the FEXen because no matter what, it's not consistent on the one of the switches. Very frustrating.

    We've started moving to a new pair of 5596s and we aren't using config-sync with those. We're just making the config in both places and put a note in the motd to remind us that you have to make the changes on both. We're more comfortable with this and after a lot of issues with config-sync over several years we are happy to be getting rid of it.