fryguypa

Archive for August, 2010|Monthly archive page

Nexus Install – after thoughts – coming soon…

In Uncategorized on August 24, 2010 at 13:54

Well, the Nexus 7010 pair have been installed and are working like champs now.  It was a nightmare of an install, nothing like shoehorning a new technology switch to replace an old technology switch.  Even with planning, planning, and more planning we got bit by bugs, old design flaws, and just some ancillary problems that might have not been related to what we did – but are being blamed on it because they coincided.  The norm as we all know it.

I have received a copy of the TAC case notes and will digest the information over the next few days and post some of the mistakes that we made so that you do not make them in the future.

The first deployment is always the hardest and where you learn the most.  Stuff like this makes you either love what you do or hate it, but usually both at times 🙂  Still, I love what I do – the problems that I run into are a wonderful opportunity to learn.

Advertisements

Smart Call Home Alert Groups

In Nexus on August 11, 2010 at 13:56

As promised, here is a list of the Smart Call Home groups and the commands that they execute.

All commands start with SHOW, I have left that out due to spacing constraints.

Group         Description           Commands
------------  --------------------- -------------------------------
Cisco TAC     All critical alerts   Executed commands based on the
              from the other groups alert group that originates the
                                    alert

Configuration Periodic events       module
              related to configs    running-configuration vdc-all all
                                    start-configuration vdc-all
                                    vdc current
                                    vdc membership
                                    version

Diagnostic    Events generated by   diagnostic result module all detail
              diagnostics           diagnostic result module number detail
                                    hardware
                                    logging last 200
                                    module
                                    sprom all
                                    tech-support gold
                                    tech-support platform
                                    tech-support sysmgr
                                    vdc current
                                    vdc membership
                                    version

EEM           Events generated by   diagnostic result module all detail
              by Embedded Event     diagnostic result module number detail
              Manager               module
                                    tech-support gold
                                    tech-support platform
                                    tech-support sysmgr
                                    vdc current
                                    vdc membership

Environmental Events related to     environment
              power, fan, etc       logging last 200
                                    module
                                    vdc current
                                    vdc membership
                                    version

Inventory     Inventory status      inventory
              that is provided      license usage
              when a unit is is     module
              cold booted or FRU    system uptime
              inserted or removed   sprom all
                                    vdc current
                                    vdc membership
                                    version

License       Events related to     license usage vdc all
              to licensing          logging last 200
                                    vdc current
                                    vdc membership

Linemodule    Events related to     diagnostic result module detail
 hardware     standard or smart     diagnostic result module number detail
              switching mods.       hardware
                                    logging last 200
                                    module
                                    sprom all
                                    tech-support ethpm
                                    tech-support gold
                                    tech-support platform
                                    tech-support sysmgr
                                    vdc current
                                    vdc membership
                                    version

Supervisor    Events related to     diagnostic result module all detail
 hardware     the supervisor        hardware  
                                    logging last 200
                                    module
                                    sprom all
                                    tech-support ethpm
                                    tech-support gold
                                    tech-support platform
                                    tech-support sysmgr
                                    vdc current
                                    vdc membership
                                    version

Syslog port   Events generated      license usage
 group        syslog PORT facility  logging last 200
                                    vdc current
                                    vdc membership

System        Events generated      diagnostic result module all detail
              by a failure of a     hardware
              system that is        logging last 200
              critical to the       module
              units operation       sprom all
                                    tech-support ethpm
                                    tech-support gold
                                    tech-support platform
                                    tech-support sysmgr
                                    vdc current
                                    vdc membership

Test          This group is for     module
              user generated test   vdc current
              messages              vdc membership
                                    version

Nexus 7000 Smart CallHome Configuration

In Nexus on August 11, 2010 at 10:16

CallHome feature is not anything new, it has been around for quite some time now.  I remember hearing about it when the Sup720 years ago.  Believe it or not, I never really bothered to configure it or really tinker with it as our corporate support contract is through a VAR and not direct with TAC.

Well, with our Nexus switches I figured it was time to take a good look at this feature and configure it for internal use as well as for connectivity to our VAR.  Since I am configuring this feature, figured it was a good time to blog on how to!

This excert is taking directly from Cisco’s website on what Smart CallHome is:
Cisco® Smart Call Home is an award-winning, embedded support feature available on a broad range of Cisco products. Smart Call Home enabled devices continuously perform proactive diagnostics on their own components to provide real-time alerts and remediation advice when an issue is detected.

So, without future ado, here is how to configure Smart CallHome on the Nexus 7000:

1. Enter configuration mode
     N7K2# conf t
     Enter configuration commands, one per line.  End with CNTL/Z.
    
2. Once in Configuration mode, enter the callhome configuraiton section
     N7K2(config)# callhome
    
3. The first thing that you should do is define a the system contact information.  This is the group or person(s) who are resposible for the system.  My suggestion is to first designate a small group of key individuals, fine-tune the features, and then deploy to a larger group, or preferrably, your Support Center (NOC)
     N7K2(config-callhome)# email-contact NexusSupportTeam@YourCompany.Com
     N7K2(config-callhome)# phone-contact +1-610-555-0542
     N7K2(config-callhome)# streetaddress 124 Colo Blvd, Colo City, Co 12345
     N7K2(config-callhome)# contract-id 12345
     N7K2(config-callhome)# site-id 2468
     N7K2(config-callhome)#  
  
4. Now, lets check the config via the SHOW CALLHOME
     N7K2(config-callhome)# show callhome
     callhome disabled
     Callhome Information:
     contact person name(sysContact):who@where
     contact person’s email:NexusSupportTeam@YourCompany.Com
     contact person’s phone number:+1-610-555-0542
     street addr:124 Colo Blvd, Colo City, Co 12345
     site id:2468
     customer id:
     contract id:12345
     switch priority:7
     duplicate message throttling : enabled
     periodic inventory : enabled
     periodic inventory time-period : 7 days
     periodic inventory timeofday : 08:00 (HH:MM)
     Distribution : Disabled
     N7K2-CoreSwitch2(config-callhome)#

5. Now we need to define a destination configuration. This is who the notifications will be sent to.  Again, this should be either a key group of people or your Support NOC.
     N7K2(config-callhome)# destination-profile Nexus-Support-NOC
     N7K2(config-callhome)# destination-profile Nexus-Support-NOC email-addr
NexusSupport@YourCompany.Com
     N7K2(config-callhome)# destination-profile Nexus-Support-NOC format full-txt
     N7K2(config-callhome)# destination-profile Nexus-Support-NOC message-level 5
     N7K2(config-callhome)# destination-profile Nexus-Support-NOC alert-group all
     N7K2(config-callhome)#

6. To check to see what is now configured, execute the command show callhome destination-profile profile Nexus-Support-NOC
     N7K2(config-callhome)# show callhome destination-profile profile Nexus-Support-NOC
     Nexus-Support-NOC destination profile information
     maximum message size:2500000
     message format:full-txt
     message-level:5
     transport-method:email
     email addresses configured:
    
NexusSupport@YourCompany.Com

     url addresses configured:
    
     alert groups configured:
     all

     N7K2(config-callhome)#

7. Now you will need to configure your SMTP server information
     N7K2(config-callhome)# transport email smtp-server 10.100.100.100 port 25 use-vrf default
     N7K2(config-callhome)# transport email from
N7K2@YourCompany.Com
     N7K2(config-callhome)# transport email reply-to NexusSupportTeam@YourCompany.Com
    
8.  To check the configuration, use the show callhome transport-email command
     N7K2(config-callhome)# show callhome transport-email
     from email addr:N7K2@YourCompany.Com
     reply to email addr:NexusSupportTeam@YourCompany.Com
     smtp server:10.100.100.100
     smtp server port:25
    
9. Now all that is left to do is enable CallHome
     N7K2(config-callhome)# enable
    
10.To check the config, issue the show callhome command
     N7K2(config-callhome)# show callhome
     callhome enabled
     Callhome Information:
     contact person name(sysContact):who@where
     contact person’s email:NexusSupportTeam@YourCompany.Com
     contact person’s phone number:+1-610-555-0542
     street addr:124 Colo Blvd, Colo City, Co 12345
     site id:2468
     customer id:
     contract id:12345
     switch priority:7
     duplicate message throttling : enabled
     periodic inventory : enabled
     periodic inventory time-period : 7 days
     periodic inventory timeofday : 08:00 (HH:MM)
     Distribution : Disabled
     N7K2-CoreSwitch2(config-callhome)#

 I will post a seperate post on the Alert Groups.

Nexus 7000 NXOS Upgrade via ISSU

In Uncategorized on August 10, 2010 at 12:40

Before we release our Nexus 7000’s to the productions packets, we are upgrading to a newer version of code.  Since we are doing this, I figured it would be a good opportunity to  document the ISSU upgrade process.

The Nexus 7000 has two images in order to run.  The first is a Kickstart image and the second is the actual System image. The  Kickstart image contains the Linux kernel, basic drivers,   and initial file system. The System Image contains the system  software and infrastructure code.

I prefer to use FTP to transfer large files; you can also copy  them via TFTP or USB if you wish.  For this post, I will  demonstrate FTP.

1.  Start your local FTP server (or place files on common FTP  server). In my case the IP of the FTP server is 10.1.3.11

2.  On the Nexus 7000, execute the following command to copy the kickstart image using the management VRF in this example:
N7K1# copy ftp://nexus@10.1.3.11/n7000-s1-kickstart.5.0.3.bin bootflash://sup-local/
Enter vrf (If no input, current vrf ‘default’ is considered): management
Password: nexus
[##                       ]         4.15MB

This is the progress indicator bar that shows you transfer status

***** Transfer of file Completed Successfully *****

3. Now you can copy the NXOS code to the bootflash:
N7K1# copy ftp://nexus@10.1.3.11/n7000-s1-dk9.5.0.3.bin bootflash://sup-local/
Enter vrf (If no input, current vrf ‘default’ is considered): management
Password: nexus
[##                       ]         4.15MB
***** Transfer of file Completed Successfully *****

4.  Now that you have the files on the Active supervisor, copy them to the standby supervisor:
N7K1# copy bootflash:/n7000-s1-dk9.5.0.3.bin bootflash://sup-2/
Copy progress 100% 107430KB !This is the progress indicator bar that shows you transfer status
N7K1# copy bootflash:/n7000-s1-kickstart.5.0.3.bin bootflash://sup-2/
Copy progress 100% 24522KB

5.  To validate that you have successfully copied to the standby supervisor, you should also attach to it and check the directory.
N7K1# attach module 6
Attaching to module 6 …
To exit type ‘exit’, to abort type ‘$.’
Cisco Nexus Operating System (NX-OS) Software
TAC support: http://www.cisco.com/tac
Copyright (c) 2002-2010, Cisco Systems, Inc. All rights reserved.
The copyrights to certain works contained in this software are
owned by other third parties and used and distributed under
license. Certain components of this software are licensed under
the GNU General Public License (GPL) version 2.0 or the GNU
Lesser General Public License (LGPL) Version 2.1. A copy of each
such license is available at (SNIP)
N7K1(standby)# dir
3610    Jul 14 20:37:16 2010  aaa_cnv.log
16384    Jan 13 12:40:30 2008  lost+found/
100118021    Jan 14 11:11:23 2008  n7000-s1-dk9.4.2.4.bin
107369112    Jul 14 19:58:04 2010  n7000-s1-dk9.5.0.2a.bin
107430217    Aug 10 14:14:19 2010  n7000-s1-dk9.5.0.3.bin

24727552    Jan 14 11:11:37 2008  n7000-s1-kickstart.4.2.4.bin
23613440    Jul 14 19:56:49 2010  n7000-s1-kickstart.5.0.2a.bin
24522752    Aug 10 14:21:11 2010  n7000-s1-kickstart.5.0.3.bin

4096    Jan 13 14:29:21 2008  vdc_2/
4096    Jan 13 14:29:21 2008  vdc_3/
4096    Jan 13 14:29:21 2008  vdc_4/

Usage for bootflash://
513769472 bytes used
1296130048 bytes free
1809899520 bytes total
N7K1(standby)#

6.  Now you can return to the active supervisor by typing exit
N7K1(standby)# exit
rlogin: connection closed.
N7K1#

7.  Before continuing, make sure that all your configs are saved and backed up to an external device.  I suggest copying them to a TFTP server for disaster recovery.  A rollback will automatically be created when you perform the upgrade, but an added insurance policy is suggested.
N7K1# checkpoint file bootflash:081010_MainVDC.cfg
Done
N7K1# switchto vdc coreSwitch1
Cisco Nexus Operating System (NX-OS) Software
TAC support: http://www.cisco.com/tac
Copyright (c) 2002-2010, Cisco Systems, Inc. All rights reserved.
The copyrights to certain works contained in this software are
owned by other third parties and used and distributed under
license. Certain components of this software are licensed under
the GNU General Public License (GPL) version 2.0 or the GNU
Lesser General Public License (LGPL) Version 2.1. A copy of each
such license is available at
(SNIP)
N7K1-CoreSwitch1# checkpoint file bootflash:081010_VDC1.cfg
Done
N7K1-CoreSwitch1# switchback

N7K1#

8.  Prior to the upgrade of the code, I highly recommend checking the impact of the upgrade.  This is a good command to run
when you are preparing your Chance Controls as you can validate the file as well as the impact.
You can do this by using the following command:
N7K1# show install all impact kickstart bootflash:n7000-s1-kickstart.5.0.3.bin system bootflash:n7000-s1-dk9.5.0.3.bin

Verifying image bootflash:/n7000-s1-kickstart.5.0.3.bin for boot variable “kickstart”.
[####################] 100% — SUCCESS

Verifying image bootflash:/n7000-s1-dk9.5.0.3.bin for boot variable “system”.
[####################] 100% — SUCCESS

Verifying image type.
[####################] 100% — SUCCESS

Extracting “lc1n7k” version from image bootflash:/n7000-s1-dk9.5.0.3.bin.
[####################] 100% — SUCCESS

Extracting “bios” version from image bootflash:/n7000-s1-dk9.5.0.3.bin.
[####################] 100% — SUCCESS

Extracting “lc1n7k” version from image bootflash:/n7000-s1-dk9.5.0.3.bin.
[####################] 100% — SUCCESS

Extracting “lc1n7k” version from image bootflash:/n7000-s1-dk9.5.0.3.bin.
[####################] 100% — SUCCESS

Extracting “system” version from image bootflash:/n7000-s1-dk9.5.0.3.bin.
[####################] 100% — SUCCESS

Extracting “kickstart” version from image bootflash:/n7000-s1-kickstart.5.0.3.bin.
[####################] 100% — SUCCESS

Extracting “lc1n7k” version from image bootflash:/n7000-s1-dk9.5.0.3.bin.
[####################] 100% — SUCCESS

Extracting “lc1n7k” version from image bootflash:/n7000-s1-dk9.5.0.3.bin.
[####################] 100% — SUCCESS

Extracting “lc1n7k” version from image bootflash:/n7000-s1-dk9.5.0.3.bin.
[####################] 100% — SUCCESS

Extracting “lc1n7k” version from image bootflash:/n7000-s1-dk9.5.0.3.bin.
[####################] 100% — SUCCESS

Extracting “cmp” version from image bootflash:/n7000-s1-dk9.5.0.3.bin.
[####################] 100% — SUCCESS

Extracting “cmp-bios” version from image bootflash:/n7000-s1-dk9.5.0.3.bin.
[####################] 100% — SUCCESS

Performing module support checks.
[####################] 100% — SUCCESS

Notifying services about system upgrade.
[####################] 100% — SUCCESS

Compatibility check is done:
Module  bootable          Impact  Install-type  Reason
——  ——–  ————–  ————  ——
2       yes  non-disruptive       rolling
3       yes  non-disruptive       rolling
4       yes  non-disruptive       rolling
5       yes  non-disruptive         reset
6       yes  non-disruptive         reset
7       yes  non-disruptive       rolling
8       yes  non-disruptive       rolling
9       yes  non-disruptive       rolling
10       yes  non-disruptive       rolling

[==== Output Omitted ====]
N7K1#

Note: The key piece of information here is the Impact column.  As you can see here,
this upgrade is non-disruptive (call it minimal) to traffic.  The only two modules
that need to be reset are the Supervisors (Mod 5 and 6)


9.  Once you have validated the images, you can now proceed with the upgrade.  To upgrade the images, use the install command.
N7K1#  install all kickstart bootflash:n7000-s1-kickstart.5.0.3.bin system bootflash:n7000-s1-dk9.5.0.3.bin

Verifying image bootflash:/n7000-s1-kickstart.5.0.3.bin for boot variable “kickstart”.
[####################] 100% — SUCCESS

Verifying image bootflash:/n7000-s1-dk9.5.0.3.bin for boot variable “system”.
[####################] 100% — SUCCESS

Verifying image type.
[####################] 100% — SUCCESS

Extracting “lc1n7k” version from image bootflash:/n7000-s1-dk9.5.0.3.bin.
[####################] 100% — SUCCESS

Extracting “bios” version from image bootflash:/n7000-s1-dk9.5.0.3.bin.
[####################] 100% — SUCCESS

Extracting “lc1n7k” version from image bootflash:/n7000-s1-dk9.5.0.3.bin.
[####################] 100% — SUCCESS

Extracting “lc1n7k” version from image bootflash:/n7000-s1-dk9.5.0.3.bin.
[####################] 100% — SUCCESS

Extracting “system” version from image bootflash:/n7000-s1-dk9.5.0.3.bin.
[####################] 100% — SUCCESS

Extracting “kickstart” version from image bootflash:/n7000-s1-kickstart.5.0.3.bin.
[####################] 100% — SUCCESS

Extracting “lc1n7k” version from image bootflash:/n7000-s1-dk9.5.0.3.bin.
[####################] 100% — SUCCESS

Extracting “lc1n7k” version from image bootflash:/n7000-s1-dk9.5.0.3.bin.
[####################] 100% — SUCCESS

Extracting “lc1n7k” version from image bootflash:/n7000-s1-dk9.5.0.3.bin.
[####################] 100% — SUCCESS

Extracting “lc1n7k” version from image bootflash:/n7000-s1-dk9.5.0.3.bin.
[####################] 100% — SUCCESS

Extracting “cmp” version from image bootflash:/n7000-s1-dk9.5.0.3.bin.
[####################] 100% — SUCCESS

Extracting “cmp-bios” version from image bootflash:/n7000-s1-dk9.5.0.3.bin.
[####################] 100% — SUCCESS

Performing module support checks.
[####################] 100% — SUCCESS

Notifying services about system upgrade.
[####################] 100% — SUCCESS

Compatibility check is done:
Module  bootable          Impact  Install-type  Reason
——  ——–  ————–  ————  ——
2       yes  non-disruptive       rolling
3       yes  non-disruptive       rolling
4       yes  non-disruptive       rolling
5       yes  non-disruptive         reset
6       yes  non-disruptive         reset
7       yes  non-disruptive       rolling
8       yes  non-disruptive       rolling
9       yes  non-disruptive       rolling
10       yes  non-disruptive       rolling

Images will be upgraded according to following table:
Module       Image                  Running-ersion(pri:alt)           New-Version  Upg-Required
[===== Output Omitted ====]


Note: As you can see, it does the impact analysis for you anyways when you do the install.
You will now need to confirm if you do want to do the upgrade by entering Y

Do you want to continue with the installation (y/n)?  [n] y

Install is in progress, please wait.

Performing runtime checks.
[####################] 100% — SUCCESS

Syncing image bootflash:/n7000-s1-kickstart.5.0.3.bin to standby.
[####################] 100% — SUCCESS

Syncing image bootflash:/n7000-s1-dk9.5.0.3.bin to standby.
[####################] 100% — SUCCESS

Setting boot variables.
[####################] 100% — SUCCESS

Performing configuration copy.
[####################] 100% — SUCCESS

Module 2: Refreshing compact flash and upgrading bios/loader/bootrom.
Warning: please do not remove or power off the module at this time.
[####################] 100% — SUCCESS

Module 3: Refreshing compact flash and upgrading bios/loader/bootrom.
Warning: please do not remove or power off the module at this time.
[####################] 100% — SUCCESS

Module 4: Refreshing compact flash and upgrading bios/loader/bootrom.
Warning: please do not remove or power off the module at this time.
[####################] 100% — SUCCESS

Module 5: Refreshing compact flash and upgrading bios/loader/bootrom.
Warning: please do not remove or power off the module at this time.
[####################] 100% — SUCCESS

Module 6: Refreshing compact flash and upgrading bios/loader/bootrom.
Warning: please do not remove or power off the module at this time.
[####################] 100% — SUCCESS

Module 7: Refreshing compact flash and upgrading bios/loader/bootrom.
Warning: please do not remove or power off the module at this time.
[####################] 100% — SUCCESS

Module 8: Refreshing compact flash and upgrading bios/loader/bootrom.
Warning: please do not remove or power off the module at this time.
[####################] 100% — SUCCESS

Module 9: Refreshing compact flash and upgrading bios/loader/bootrom.
Warning: please do not remove or power off the module at this time.
[####################] 100% — SUCCESS

Module 10: Refreshing compact flash and upgrading bios/loader/bootrom.
Warning: please do not remove or power off the module at this time.
[####################] 100% — SUCCESS
2010 Aug 10 15:22:34 N7K1 %PLATFORM-2-MOD_REMOVE: Module 6 removed (Serial number JAF14XXXXXX)

2010 Aug 10 15:26:43 N7K1 %CMPPROXY-STANDBY-2-LOG_CMP_UP: Connectivity Management processor(on module 6) is now UP

Module 6: Waiting for module online.
— SUCCESS

Notifying services about the switchover.
[####################] 100% — SUCCESS

“Switching over onto standby”.
2010 Aug 10 15:33:55 N7K1 %$ VDC-1 %$ %CMPPROXY-STANDBY-2-LOG_CMP_UP: Connectivity Management processor(on module 5) is now UP

Module 2: Non-disruptive upgrading.
[####################] 100% — SUCCESS

Module 3: Non-disruptive upgrading.
[####################] 100% — SUCCESS

Module 4: Non-disruptive upgrading.
[####################] 100% — SUCCESS

Module 7: Non-disruptive upgrading.
[####################] 100% — SUCCESS

Module 8: Non-disruptive upgrading.
[####################] 100% — SUCCESS

Module 9: Non-disruptive upgrading.
[####################] 100% — SUCCESS

Module 10: Non-disruptive upgrading.
[####################] 100% — SUCCESS

Install has been successful.

User Access Verification
N7K1 login:


10. And just like that, both supervisors have been upgraded and are running the new code:

N7K1# sh ver
Cisco Nexus Operating System (NX-OS) Software
TAC support: http://www.cisco.com/tac
Copyright (c) 2002-2010, Cisco Systems, Inc. All rights reserved.
The copyrights to certain works contained in this software are
owned by other third parties and used and distributed under
license. Certain components of this software are licensed under
the GNU General Public License (GPL) version 2.0 or the GNU
Lesser General Public License (LGPL) Version 2.1. A copy of each
such license is available at
(SNIP)

Software
BIOS:      version 3.22.0
loader:    version N/A
kickstart: version 5.0(3)
system:    version 5.0(3)

BIOS compile time:       02/20/10
kickstart image file is: bootflash:/n7000-s1-kickstart.5.0.3.bin
kickstart compile time:  7/12/2010 18:00:00 [07/24/2010 11:47:30]
system image file is:    bootflash:/n7000-s1-dk9.5.0.3.bin
system compile time:     7/12/2010 18:00:00 [07/24/2010 13:21:35]

Philosophical security post for Nexus 7000

In Uncategorized on August 5, 2010 at 09:18

History:
I was configuring our production Nexus 7000 today and was also moving some modules around in order to accommodate better cabling options.  The one problem that I had with our Nexus switches initially was insufficient 110 power leads in our lab to power them fully.  Today I was able to resolve that by acquiring C19 pigtail cables to power al the power supplies on just 8 outlets (needed 12 to run all the power supplies). 

Philosophical:
What I have also done was move a few modules around in the chassis that where originally powered-down in order to run the chassis.  What I noticed on the Nexus 7000 now is that you are able to poweroff a module even though a module in not installed:
     N7K2# sh run | inc power
     poweroff module 1
     power redundancy-mode combined force
     N7K2# sh module 1
     N7K2#

As you can see there is no card installed in Module 1 but yet the power is disabled to that module. 

I am trying to figure out if it is a safer option to keep the module powered-off for security and sanity or is it better to leave power to that slot.

Current Direction:
For now, I am going to leave the power-off to that slot in order to
     1) Prevent anyone from adding a module in the future
     2) Also prevent a sudden power draw that could have a negative impact in the future (exceed power requirements)
     3) Keep people honest – ensure a change will be submitted if more capacity is ever needed

%d bloggers like this: