Cisco DCNM server unresponsive

Cisco MDS

You’re running a DCNM virtual appliance and when you’re trying to log on, it looks like the appliance’s webserver is no longer active.

Connect to the Linux prompt of the appliance and do the following:

Stop DCNM services:

/usr/local/cisco/dcm/dcnm/bin/stopLANSANServer.sh

Open the psql prompt:

/usr/local/cisco/dcm/db/bin/psql dcmdb

log on with root password (user root is assumed)

Type the command:

VACUUM FULL ANALYZE VERBOSE;

This will produce tons of output, but just let it run and eventually it all ends and you just need to start the DCNM services again.

When done, quit the psql prompt:

dcmdb=> \q

Start DCNM services:

/usr/local/cisco/dcm/dcnm/bin/startLANSANServer.sh

Source: cisco.com

Reclaim thin provisioned space in VMware datastore

We were using a portable storage array as temporary storage array to store data from an older array that had to be reconfigured. Capacity was quite an issue and everything would barely fit (with dedupe and compression doing its best on the Dell EMC XtremIO we were using for this), but we discovered that provisioned space wasn’t immediately reclaimed, so we got a bit into trouble when space wasn’t immediately reclaimed when data was deleted from a VMware version 5 datastore.

We were facing two options:

  • Datastore VMFS version 6: Unmap automatically every 12 hours (not 100% sure about this interval)
  • Datastore VMFS version 5: Unmap manually from the ESXi command line:

esxcli storage vmfs unmap -l XIO-05-VMFS-013

Before:

After running the command “esxcli storage vmfs unmap -l MGG-XIO-05-VMFS-013” and a few hours of patience and waiting:

Source: : https://kb.vmware.com

Brocade SAN Switch commands that prove to be very useful

Brocade commands are usually very easy to find out. Simply type “help” and look for the command you think you need, but what exactly does each command do? Here’s a list:

Info

uptime                          – Same as unix uptime
date                              – Same as unix date
version                          – Gives versions of firmwares & OS

Hardware State

faultshow                     – Show switch faults
fanshow                       – Show switch FAN faults
psshow                        – Show switch POWER SUPPLY faults
tempshow                    – Show switch TEMPERATURE values
switchstatusshow          – Overall status of switch

Config

agtcfgshow                  – Show SNMP config
configshow                  – Show switch config
gbicshow                     – Show GBIC slots and serial numbers
licenseshow                 – Show license data
supportshow                – Like Sun’s explorer – gets many configs at once
switchshow                  – Show switch ports and connections
zoneshow                    – Show zone and switch aliases

IP

bcastshow                   – Show broadcast routing
ifmodeshow                 – show interface mode (duplex)
ifshow                         – Like unix netstat -s
ipaddrshow                  – Like unix netstat -i
interfaceshow               – Like unix ndd

Performance

ifshow                          – Like unix netstat -s
portperfshow                – Show interface mode (duplex)
portshow #                  – Show stats on a port
portrouteshow #           – Show routes on a port
portstatsshow #            – Show stats (netstat -s) on a port

Misc Show

diagshow                     – Show diagnostics – paged output
errdump                      – Show diagnostics – no paged output
fabricshow                   – Show fabric
fspfshow                      – Show FSPF protocol info
mqshow                       – Show queues
nbrstateshow                – Show FSPF neighbor states
nsshow                        – Show name servers
nsallshow                     – Show all name servers
porterrshow                  – Like mpstat – shows port info
switchstatuspolicyshow  – Show config at when errors are flagged

 

Brocade Data Collection

There are 2 types of Data Collection associated with Brocade switches, “supportshow” and “supportsave”.

If the problem is related to basic “port issues”, a “supportshow” is sufficient. However, a more complex problem that involves Zoning, Performance, Reboots, CP Failovers, Fabric wide problems etc, requires a “SupportSave” type of collection. If the problem is multiswitch related (ISL connections, long distance configurations etc.) we need the supportsave from all relevant switches in the fabric.

The amount and type of information displayed in a supportshow output is depending on the type of switch as well as the collection options enabled in the switch configuration. This can be configured with the “supportshowcfgenable [option]” and “supportshowcfgdisable [option]”.

The output from the “supportshowcfgshow” command shows which subscripts will be enabled. The default groups are always enabled. Only on special occasions you will be asked to enable additional groups if necessary.

 

Performance and Intermittent Error Related Issues

If there is no sign of any obvious physical issue there might be link related problems which can identify performance problems and/or protocol related errors. Brocade counters are cumulative and keep doing so until a certain counter wraps, a switch reboots or the statistics are manually cleared.

Storage vendors for example require in these circumstances that a new baseline is created, a certain run-time has been achieved and separate commands are submitted against the suspected switch or switches.

To create a new baseline with cleared counters do the following:

  1. Log in to the switch via Telnet or SSH
  2. Submit the “statsclear” command
  3. Submit the “slotstatsclear” command

After the agreed (mostly around one hour) run-time capture a new supportsave and upload this to the vendor’s service request.

SupportSave Data Collection (CLI Method)

When the problem is more sophisticated a supportsave from the switch is required. The supportsave command is available as of Fabric OS version 4.4 however, Fabric OS versions (> 6.2.x) provide a significant better collection of logs which represent the status overview of the switch and fabric. If you have a director class switch with two CP’s and/or core plus function blades it will also collect information from all the blades.

The supportsave will upload between 25 and 80 files depending on platform, Fabric OS level and enabled features to an FTP or SCP server. These will not be tarred or zipped into one file so it is important you create such an archive with a meaningful name. (ie. switchname-domainid-fabricid.zip)

Example

Fabosv4.4switch:admin> supportsave -u anonymous -p password -h xxx.xxx.xxx.xxx -d /directory -l ftp

This command collects RASLOG, TRACE, supportShow, core file, FFDC data and then transfer them to a FTP/SCP server or a USB device. You could also simply type the “supportsave” command without any parameters and you will be asked for the extra information before the command is actually executed. The operation will usually take several minutes to complete.
NOTE: supportSave will transfer existing trace dump file first, then automatically generate and transfer latest one. There will be two trace dump files transferred after this command.

OK to proceed? (yes, y, no, n): [no] y
Saving support information for switch:BR4100_IP127, module:RAS…
Saving support information for switch:BR4100_IP127, module:CTRACE_OLD…
Saving support information for switch:BR4100_IP127, module:CTRACE_NEW…
etc……

To upload the files you can specify the FTP parameters inline (as modeled above) or through the supportftp command (see Fabric OS V5.1 command Reference guide) .

  1. Host IP: XXX.XXX.X.X (example 192.168.1.1)
  2. User Name: admin
  3. Password:
  4. Remote Directory: tmp (example: tmp)
  5. Saving support information

SupportShow Data Collection

This is a non disruptive procedure and can be performed by the CE or the customer.

You can use your favorite terminal emulation utility. Refer to the respective documentation how to turn on capturing output to a file.

Telnet or SSH

  1. Telnet or SSH into Brocade switch
  2. Enter username and password
  3. Start logging to file on the Telnet session.
    NOTE: For Windows standard telnet, this is under the terminal pulldown menu.
  4. Enter command: supportShow
  5. Upload the telnet log to TUF.

What is the Principal switch in a Brocade SAN

Principal switches maintain unique domain ID across the fabric. Principal switch ensures that each switch in a SAN have different domain ID. Any ISL of a switch that takes to the Principal switch is a upstream. Any ISL of a switch that goes away from principle switch is a downstream. All non principal switches are called subordinate switch. Zoning updated in a principal switch or a non principal switch will update zoning across the fabric. “date” command will be a read-only if a switch is configured with a Time server – NTP. Principal switch will update the time in all the non principal switches. tsclockserver is the command used to associate a switch to a NTP time server.

To manually set a subordinate switch to a principal switch use “fabricprincipal” command.

To elect a new principal switch:

>fabricprincipal -1         # This command will see the appropriate principal switch and make it as a new principal switch. Also, this will elect new upstream and downstream ISLs

To force a switch to be a principal switch:

> fabricprincipal -f         # This command will force the switch to be the principal switch of the fabric. This will also elect new upstream and downstream ISLs

To see the current settings:

> fabricprincipal -q      # This command will query the current settings and displays

How to check the Principle switch:

  • Fabicshow shows the Princple switchSwitch ID   Worldwide Name           Enet IP Addr    FC IP Addr      Name
  • ————————————————————————-
  • 121: fffc79 10:00:00:05:1e:36:0b:42 10.2.59.52      0.0.0.0        >”IBM_2109_M48_21″
  • 130: fffc82 10:00:00:05:33:cd:a4:c2 10.2.59.8       0.0.0.0         “DS_6505B”
  • The Fabric has 2 switches
  1. Switchshow shows the principle

This Fabric has 2 switches

IBM_2109_M48_21:admin> switchshow

switchName:     IBM_2109_M48_21
switchType:     42.2
switchState:    Online
switchMode:     Native
switchRole:     Principal
switchDomain:   121
switchId:       fffc79
switchWwn:      10:00:00:05:1e:36:0b:42
zoning:         ON (Cfg_080123)
switchBeacon:   ON
blade1 Beacon:  OFF
blade2 Beacon:  OFF
blade3 Beacon:  OFF
blade4 Beacon:  OFF
blade7 Beacon:  OFF

DS_6505B:admin> switchshow
switchName:     DS_6505B
switchType:     118.1
switchState:    Online
switchMode:     Native
switchRole:     Subordinate
switchDomain:   130
switchId:       fffc82
switchWwn:      10:00:00:05:33:cd:a4:c2
zoning:         ON (Cfg_080123)
switchBeacon:   OFF
FC Router:      OFF
FC Router BB Fabric ID: 1
Address Mode:   0

How to Replace the Switch:

  • Load the Licenses to the Switch as of the other switches
  • Assign the Network IP address to the switch and do not connect to the SAN Fabric
  • Save the configuration by issuing the command configupload
  • Upgrade the code to the same Firmware version
  • Update the credentials to the standard credential as per your org standards
  • Disable your switch with ‘switchdisable’
  • Change the domain id of the switch next available
  • Change the principleswitch role to subordinate  by issuing the command principle –f 0
  • Connect the Switch to the SAN  and this will download the config from the core switch or principle switch

Free Dell EMC trainings

For those that want some extra training on Unity or the VMAX All Flash for example, there’s free training available from Dell EMC Education at this moment:

This list is a summary only. For a complete list, take a look at DECN (Dell EMC Community Network) for the complete list: https://community.emc.com/docs/DOC-34286

How to create a snapview snapshot on an existing LUN

I apologize in advance for this (6 years or so too late) post, since it’s for creating a snapview snapshot on a LUN on a VNX. It’s simply meant as a reminder for the command line syntax:

Examples for creating snapview snapshots (it only defines it, no COFW is happening at this point):
naviseccli -h 172.16.20.96 snapview -createsnapshot 17 -snapshotname VMFS-001-SNAP
naviseccli -h 172.16.20.96 snapview -createsnapshot 18 -snapshotname VMFS-003-SNAP
naviseccli -h 172.16.20.166 snapview -createsnapshot 27 -snapshotname VMFS-002-SNAP
naviseccli -h 172.16.20.116 snapview -createsnapshot 5 -snapshotname VMFS-004-SNAP

To start an actual point in time session (and the start of COFWs):
naviseccli -h [ip address] snapview -startsession [session name] -snapshotname VMFS-001-SNAP

To stop a session:
cnaviseccli -h [ip address] snapview -stopsession [session name]

To activate a snapview session (make the data visible):
naviseccli -h [ip address] snapview -activatesnapshot [session name] -snapshotname VMFS-001-SNAP

To deactivate a snapview session (stop presenting the data to the hosts):
naviseccli -h [ip address] snapview -deactivatesnapshot [session name] -snapshotname VMFS-001-SNAP

Dell EMC Elect 2017: the list

Dell EMC Elect

It’s been a while since my last post and there’s no good reason why I didn’t post this earlier, except that life’s been very very busy for me lately. But then again, so it is for all of us, right?

Anyway, last week, on March 29 to be precise, the judging was over and we presented the new list of Elect members. After the merger of Dell Rockstar and EMC Elect, we had to come up with a new program that would combine both groups and form a representative list of professionals who share their passion for the Dell EMC enterprise environment. Being a part of the founders, again, me and the other founders discussed the name of the new group and how we would judge the nominees. I wrote a post on that a little while ago actually. This new advocacy program will be know as the

Dell EMC Elect

Read more »

The king is dead, long live the king: Dell EMC Elect is born!

Dell EMC Elect

Now that Dell and EMC have merged, both advocacy programs have ended as well. No more Dell Rockstar and EMC Elect, but a brand “new” program has emerged!!

As an EMC Elect founder I loved being part of the EMC Elect from the very beginning in Q4 of 2012, but times are changing and a new era has started: I guess that’s how life changes as well and this time it was a good change! The advocacy program we’ve all been proud of, whether that was the Dell Rockstar or the EMC Elect, was booming! I can only speak of the Elect and being able to talk to people about it, as a real advocate should be doing, was great.

Michael Dell finally meets the Elect

Read more »

How to change a (domain) password on a Windows 2012 server

Security

When logged on to a Windows 2012 server through an RDP through another RDP connection on an Apple laptop, changing the password can be a challenge, especially if the account on this 2012 machine is not the same as on the machine you used to connect to the 2012 machine (try saying that sentence 5 times in a row).

So you’re logged on to a Windows 2012 server and you need to change the password of the actual account you’re using at that moment.

Click on the windows flag in the lower left of the screen and type this Powershell command:

Powershell -noprofile -nologo -noninteractive -command “(new-object -ComObject shell.application).WindowsSecurity()”

It doesn’t look pretty, but since I cannot find any alternative, it’s the best I can do.

It works!

Using putty instead of the default ssh from Cisco device manager

DCNM

How do I configure Cisco DCNM so Putty starts when I select to go to the command line of a switch?

right click on switch

If you right click on a switch in the overview section in DCNM, you can go to the command line of that switch, but how do you change the default CLI SSH into Putty (or another telnet / SSH capable tool)?

Read more »

Dell EMC World 2016 Austin, TX, USA

Dell EMC World 2016

You’ve all heard the news last week (at least, I think you did): On September 7 the huge merger between Dell and EMC took place, making the Dell Technologies transaction the biggest technology company integration in history! So the digital transformation is really here, and Dell Technologies / Dell EMC is leading the way. From October 18 to 20, Dell EMC will host business and IT leaders from around the world for the biggest enterprise technology event of the year, Dell EMC World 2016 in Austin, Texas. And I received an invite to join the EMC Elect and Dell Rock Stars to report on the news that Michael Dell and others will bring us. Many thanks go to Mark Browne, who made sure EMC Elect people were invited to be present at this event. We’ll be in super secret influencer meetings, sneak peek preview breakfasts and we’ll be sitting in a reserved seat section to watch Michael Dell perform on stage! Yeah, I’m pretty excited! After being invited to the VNX2 launch in Milan in 2013, I’m once again present at a major event as part of the EMC Elect.

Read more »

How to add new members to an existing Cisco smart zone

Cisco MDS

DCNM is down. I love the tool, but the downside is that you forget how to use the CLI. I faced downtime of our DCNM appliance and was forced to use the CLI instead. No big deal actually, but I still want to post the commands to use when you need to add new hosts to existing zones.

First you might want to create a new device alias for the new hosts:

device-alias database
device-alias name server1 pwwn 20:11:00:15:b9:00:00:00
device-alias name server2 pwwn 20:11:00:15:b9:00:00:01
device-alias commit

And then you want to add the new aliases to the existing (smart) zone:

config
zone name NameOfZone vsan 123
member device-alias server1 init
member device-alias server2 init
zone commit vsan 123

You don’t need to commit the whole zoneset again, since that one didn’t change. Check it by running:

zoneset activate name ZoneSetName vsan 123

That’s it! there’s not much to it, and can save you a lot of time as well!

Useful EMC VMAX CLI commands

VMAX

Over the years I collected a number of useful CLI commands to control the DMX / VMAX machines I worked with. Even though nowadays Unisphere for VMAX is a useful tool, nothing really beats the command line!
Read more »

How to list Host LUN ids in VMAX Masking Views

VMAX

Creating tdevs and masking them to hosts (storage groups in a masking view) is relatively easy in the Unisphere for VMAX interface, but what if you add a few tdevs of exactly the same size and you want to make sure that the VMware administrator uses the right LUN for each VMFS he’s going to create? One way to make sure he knows which LUN corresponds with what tdev is the Host LUN id. To list the host LUN ids

symaccess -sid 1234 show view mv_some-maskingview-name

or (a bit more verbose)

symaccess -sid 1234 list view -name mv_some-maskingview-name -detail

The second command shows each initiator group nested within other initiator groups as well.

Both generate a table with the following headers:

Sym                                                            Host
Dev     Dir:Port    Physical    Device    Name    Lun    Attr    Cap(MB)

The column under “Host Lun” shows the Host LUN ids.

How to enable SSH on a VMware 5.5 ESXi host using the vSphere client

virtual machine

Recently I needed to make some edits on vmdk files in order to get rid of a reference to the change tracking files, as mentioned in Gabrie’s post: Cannot open the disk and could not open change tracking file. An SSH connection to a VMware ESXi host was required. SHH was disabled and I needed to enable it temporarily.

How to enable SSH on an ESXi host using the vSphere client

In vSphere, select the host you want to enable SSH on.

Select the “Configuration” TAB, then “Security Profile”. In the upper right select “Properties.

A new window opens, now select SSH (you may need to scroll down a bit) and in the lower right select “Options”.

Again a new window opens and here you can either “Start” or “Stop” the SSH daemon.

enable SSH

Free EMC trial software / virtual appliances

virtual machine

A quick heads-up this time about building your own lab environment

Sometimes you just want to run a VNX, Avamar, PowerPath, Data Domain or Isilon as a virtual machine to see how things work, or to write work instructions. And EMC offers a lot of these virtual appliances for free!

Take a look at these:

 

EMC Unity: simplicity redefined

EMC Unity

It’s been a while since the VNX2 was born: September 2013, I remember it very well. Being a part of the EMC Elect, I was invited to be at the actual launch in Milan (Italy) and what a ride it was! The whole launch was wrapped around Formula 1 technology and it sure was “speed 2 lead“. That “old” VNX2, which I’m still perfectly happy with by the way, was a revolution in my humble opinion: multi-core everything, in short MCx. And yes, it was like everything just went faster, smoother and better.

New technologies

But with new technologies popping up every so many months now, it was time for a new mid-range storage array. Flash storage isn’t a novelty anymore, it’s a must! And the “old” hybrid arrays were fine, but needed some fine-tuning. With flash devices growing bigger every quarter or half a year and faster as well, the whole back-end needed an upgrade. The old 6 Gb back-end (x4) needed an upgrade.

Read more »

How to gather performance statistics from an EMC XtremIO

XtremIO

The XtremIO GUI looks very slick and it has this chronometer with IOps and MBps and everything, but when you need to have MiTrend analyze the statistics, the questions comes to mind: what statistics, what files?

How do I get my hands on these files?

First log on to the XtremIO GUI. Then click on the administration button

Administration

Now click on the “CLI Terminal TAB” and type:

create-debug-info

Read more »

A list of (social) activities during EMC World in Las Vegas 2016

Las Vegas

It’s a list of activities. No big story or my thoughts on it, it’s just a list and yes, there’s a lot of overlap. I’ll be editing this list when more events come to my attention, so keep an eye on this blog post at all times!

I’ll list as much activities as are known to me, but feel free to add comments to this post and I’ll edit any unlisted events to the list asap!

Read more »

Create and retrieve spcollect files from the command line (VNX)

Hard Drive

Java, it’s a curse. And now you suddenly need to upload the spcollect files to EMC and Java isn’t installed or incompatible and Unisphere won’t start.

Now what?

Make sure you have NAVISECCLI installed and just do it from the CLI!

Read more »

Cisco MDS “fabric merge” and “switch add” prerequisites

Cisco MDS

Prerequisites when adding a new SAN switch to an existing SAN fabric

When using enhanced device aliases, make sure you enable this on the new switch as well:

device-alias mode enhanced
device-alias commit

You can verify if this is enabled on an existing switch by this command:

sh run | grep “device-alias mode”

Read more »

%d bloggers like this: