Stretching and unstretching ActiveCluster PODs on Pure arrays

Scenario

Suppose you have two (or more) datacenters and you’re running a true active / active setup, meaning that clusters are spread over both sites and each host has access to both the local volume as well as the identical writable copy on the second site. You’ve accomplished this by setting up ActiveCluster and some PODs with volumes in them and everything is working fine and the PODs are in sync, so volumes can be written to on both locations.

The OTA team is testing a new application on one of the sites without the cluster being spread over both sites. When the testing is done it’s time to go to production and move the application to a production cluster, so formerly local volumes need to be added to a POD, because they need to be written to on the second location as well (or simply need to be replicated to the DR site to have the data on two locations).

Moving one or more volumes to a POD – steps

In order to move one or more volumes into PODS, you can either create a new unstretched POD or – and this can be tricky – you need to unstretch an existing POD first. If a POD contains volumes that are actually being accessed on both sites, you need the hosts to stop using the volumes on the array that’s going to be deleted from the POD for a short while. When hosts are configured correctly (having access to each volume on two arrays), you don’t need to take any action as the I/O will automatically be redirected to the only surviving Pure array. Of course this only applies to the hosts that are actively acessing volumes in the remote POD that you want to unstretch; hosts on the “surviving” array will continue to access that array and will not be able to access the volumes on the second site for a short while. Stopping I/O is necessary because when unstretching PODs, one side loses all access to the volumes in that particular POD.

If hosts are not configured to access volumes on both sites, you will need to failover the resources to the surviving site. When no I/O is going to the array that doesn’t contain a copy of the OTA volumes, the POD that will contain these formerly OTA volumes will need to be unstretched.

If this POD contains a large number of volumes, there’s something interesting to be seen later on! Pay attention!

When you’re sure all I/O to volumes in the POD on the second array is stopped or multi-site access is configured correctly, you can start unstretching the POD: select the POD, delete the remote array as seen from the volume(s) that you need to add to that POD and as soon as the POD contains only 1 array, you can add the volumes that need to be present on both sites.

Resyncing

When all OTA volumes in our example are added to the unstretched POD, you can add the remote Pure array and the resyncing will start.

Depending on the activity on the volumes in that POD, this can be a short transition or somewhat longer (sometimes even hours if it’s a really active POD), but there’s one interesting thing that I noticed during the numerous resyncing activities that I’ve whitnessed so far: when the resync starts, it looks like it’s taking ages to get somewhere and it might even show a disappointing progress, until the resync reaches 20%.

All of a sudden the resync will look like it’s speeding up and as soon the the counter is over 20% it will reach 100% in a matter of seconds (or minutes). I cannot find any proof that this progress has a different meaning other than what it looks like: 20% meaning only 20% is actually resynced, but my assumption is that 20% means it reached 100% and above 20% means it’s catching up on the last few “dirty” blocks on the sides that are being written to.

The good thing is that as soon as you realize that 20 is the new 100%, so waiting for the resync isn’t half as bad as it looks at first!

Happy resyncing, everybody!

How to test the alerting in a Pure Storage FlashArray

When configuring SMTP or syslog for alerting the easiest way to configure is in the GUI, simply because everything you need to configure is right there in plain sight.

For this example, we will assume the syslog servers and SMTP relay host and sender domain have been specified. If not, these can be set from the GUI under Settings > System > Syslog servers and Settings > System > Alert Routing.

But to test if it all works is a different story: there’s no test button!

So we need to log on to the CLI. Use your favorite SSH client and log on the the array.

Syslog

First you can view if the syslog was configured according to your liking by entering the command:

purelog list

You should now see the configured syslog servers and the ports that are used.

To test if syslogging works enter the command:

purelog test

SMTP

For emails the command to view the settings is:

purealert watcher list

You should now see the email addresses that will be used whenever the array needs to send an email. To test this you need to use the following command:

purealert watcher test [email protected]

If all goes well you will then receive an email similar to:

Hello,

This is a test message from your Pure Storage Array.

Controller Serial: PCTFL1953173B
-Pure Storage Array PUREARRAY-027-ct0

Troubleshooting error-disabled issues between a Cisco UCS fabric interconnect and a Cisco MDS

Cisco MDS

When connecting a high speed cisco UCS Fabric Interconnect (like the 6454) to Cisco MDS Fibre Channel switches, you might encounter error-disabled ports causing the port-channel to go down. Specifically for 8 Gbps speeds you need to enter 1 additional command per port that’s participating in the port-channel:

int fc1/17
switchport fill-pattern IDLE speed 8000

For each fc port on the MDS you need to enter this command, followed by a “shut / no shut” and the error disabled problems are most likely gone.

Check to see the port config:

show run int fc1/17

Or whatever port you configured to use the fill-pattern.

Source: Cisco

How to upgrade a Cisco MDS SAN switch

Cisco MDS

After a few years of not having had to perform switch upgrades, I had to perform a few firmware upgrades on MDS switches. I decided to write down the steps I had to take:

  • First make sure you have a reachable ftp server somewhere and write down its ip address
  • Log on to the Cisco MDS SAN switch you need to upgrade
  • Make sure the bootflash: has enough available space to hold the 2 new files

# dir bootflash:

4096 Feb 23 14:58:04 2020 lost+found/
26049024 Feb 10 12:57:55 2020 m9100-s5ek9-kickstart-mz.8.3.2.bin
121121564 Feb 10 12:56:47 2020 m9100-s5ek9-mz.8.3.2.bin

Usage for bootflash://sup-local
463962112 bytes used
2894815232 bytes free
3358777344 bytes total

Read more »

I’m in Tech Field Day 22!

From Wednesday December 9 to Friday December 11 I’ll be a delegate in Tech Field Day 22. I attended once before, in San Jose in 2014, for Storage Field Day 5 and now, 6 years later I’m in #TFD22! Time sure flies! This week we’ll be talking to some interesting companies once again.

These are the presenting companies and the delegates that will be attending:

Wednesday Thursday Friday
Early Presenting Companies Commvault
Quantum
Veeam
Red Hat
VMware
Delegate Panel Arjan Timmerman, Bart Heungens, Bruno Wollmann, Jasper Bongertz, Kori Younger, Lino Telera, Max Mortillaro, Michael Davis, Ned Bellavance, Phillip Sellers, Rob Koper, Steven Cortez
Late Presenting Companies Cisco
MemVerge
Riverbed Illumio
Delegate Panel Al Rasheed, Ben Mason, Chris Hildebrandt, Chris Maki, David Chapa, Erik Ableson, Jody Lemoine, Justin WarrenMatt Callaway, Matthew LeibNico Stein, Robert Novak, Scott Bollinger

 

It’s going to be an interesting few days and I’m very much looking forward to a renewed participation in this TFD event!

 

For more detailed information, please visit the TechField Day website.