Networks... Woot!

24 January 2018

Cisco ACI APIC and Spine/Leaf Upgrade Process

I'm just getting started with ACI in general. Here's the general process to upgrade the APICs/spine/leaf.

First check out the Cisco document here. In case the link moves/etc the document is "Cisco APIC Management, Installation, Upgrade and Downgrade Guide". This is essentially an abbreviated version of that document.

Of particular importance in that link is the section around "Supported Upgrade Paths for APIC Controller and Switch Software" and associated downgrade section. Make sure you can jump from where you are to where you want to go. If it ain't listed, it ain't supported... and prepare for headaches.

The basic process is:

Get files from Cisco onto a HTTP/SCP server and then uploaded to APIC
Get APICs upgraded
Wait for things to stabilise.
Get Leaf/Spines upgraded
Wait for things to stabilise.

This whole process took a few hours to complete... but I was gifted with having a fast internet connection to download files/etc with. I do the above using the (wimpy) GUI methods but the linked document lists ways to do the same using REST/CLI/Console/etc.

Getting the files from Cisco, to an intermediate HTTP/SCP box and onto the APICs

I couldn't believe it when I downloaded them but the files are gigantic. There is basically two main bits of software to get; APIC and ACI switch software. Thankfully Cisco put the matching APIC and ACI versions in the same sub-heading/version "Application Policy Infrastructure Controller (APIC)" on their download site. Basically if you click on 3.0 it has the APIC version (3.0.1 in my case) and the leaf/spine version (13.0.1) in the same section. In my case (going from 2.2 to 3.0) the files were:

aci-apic-dk9.3.0.1k.iso - For APICs
aci-n9000-dk9.13.0.1k.bin - For ACI Leaf/Spine

Grab them from Cisco per normal process. Upload them to a HTTP/SCP server.

In APIC, create a "Download Task" (Admin > Firmware > Download Tasks) point to each file individually. Once the task is created the file will be downloaded to APIC. You can see the status under the "Operational" tab of this page.

It looks like you can upload files directly onto the APIC from the GUI as well now (I didn't try that here though). This looks to be done through "Firmware Repository" under Admin > Firmware > Firmware Repository and clicking the "Upload Firmware to APIC" action.

Upgrading the APICs

In Admin > Firmware > Controller Firmware you'll have an action to "Upgrade Controller". Select the version/scheduled/etc and off you go. The screen will update the Upgrade Progress status bar for each APIC. The system will do one APIC at a time automatically/etc so just sit back and let it do its thing.... which brings me to...

Waiting for things to Stabilise
Just note that during this waiting time APICs will reload. This is non-disruptive as APICs aren't involved in production traffic but are only used to push policy to nodes/etc. This was a good 10-30min process for me. Had to reload the APIC browser session after they rebooted as well.

The APICs all appeared in the "Controller Firmware" screen as being "Upgraded Successfully"

Upgrading the Leaf/Spines
Similar to the APICs, except that you are going to be potentially impacting production traffic if things go bad. Basically under "Firmware Groups" in Admin > Firmware > Fabric Node Firmware > Firmware Groups create a group of AllNodes and select the ACI version you want to go to.

Before worrying about doing all at the same time... just keep in mind the next bit is to create Maintenance Groups whereby you dictate which switches to upgrade at the same time. under "Maintenance Groups" in Admin > Firmware > Fabric Node Firmware > Maintenance Groups. Make a primary and a secondary maintenance group of nodes.

You kick off the upgrade by clicking "Upgrade Now" action of the primary Maintenance Group... then you wait patiently for things to come back and do the same for the secondary group. \

Based on the link (I've not tested though):

Up to 20 nodes are upgraded at the same time
Only one member of a VPC peer is ever upgraded at the same time (nice!)

Waiting for things to Stabilise
Up to 12 minutes is the estimate on how long it will take in the guide... be patient. The nodes will come back and things will be good (hopefully). During the upgrade process nodes will reboot and production traffic will experience some minor disruption provided everything is dual-mode connected.

Obviously take this all with a grain of salt... I am not an ACI expert but wanted to write some notes to summarise the wordy Cisco process. Some of my colleagues have screwed this up in the past and managed to get things going again (albeit onsite) using some of the other methods (i.e. CLI/etc).

Good luck! Hope this helps...

12 January 2017

VPN over Satellite - Why it doesn't always work at full bandwidth

I came across an interesting situation recently whereby a customer had to move datacentres and had to reconnect their Satellite connected remote sites to connect to a different datacentre using different VPN technologies. Through the move we discovered and interesting situation regarding Satellite and VPN that I thought warranted a quick post.

Some background about Satellite communications...
Satellite communications have inherit limitations in regards to round trip time (RTT) due to the distance packets travel in order to get from source to destination (i.e. propagation delay). RTT of Satellite links are typically above the 500ms mark. Consider what this delay does to network communications. UDP will suffer the RTT delay associated with opening the connection (i.e. handshake) with a back-n-forth of 500ms required for each part of the handshake but after that the performance across these high-RTT links may not be so bad. Once the stream is opened data will quite happily flow until its finished as UDP doesn't care if bits are lost along the way it just sends one after another until its sent everything (i.e. connectionless).

On the other hand consider TCP. It also has to establish a handshake and will require the 500ms delay for each back-n-forth of the setup. However, once the handshake is completed the data stream, compared to UDP, behaves nothing like the UDP stream! It chugs!

TCP is a connection orientated protocol and it will only send the next chunk of data once the last chunk it has sent has been acknowledged and received successfully. Over short-RTT links this isn't such a big deal, we lose a packet and we simply kick it off again and wait for the other end to acknowledge. In standard networks, because the RTT is low, the penalty for doing this is negligible... and life goes on. Over satellite links the delay associated with a lost chunk of data in a TCP stream is very detrimental due to the 500ms per-direction back-n-forth required to restart the chunk that was lost.

To get around the inherit limitations of TCP over Satellite (and make it workable), network boffins came up with some fancy optimization. Satellite providers these days typically provide some sort of TCP/UDP header optimization for their services that buffers packets and tricks the PCs at either end of the Satellite into thinking the network is more responsive than it is or that data can be sent quicker than what the typical RTT back-n-forth would dictate under normal circumstances. This works great provided the optimization technology supports the protocols you're using.... (*ominous fore-shadowing*).

Now what about VPNs? Typical IPSEC VPNs over Satellite behave pretty similar to the vanilla TCP situation described above (i.e poorly) except (you guessed it) the Satellite modems can't optimize the TCP/UDP packet headers tunnels across the VPN. This is because the TCP packet headers are encapsulated and encrypted inside ESP packets (i.e. the modem can't see it's TCP or optimize it). Similar is true for IP Tunnels (i.e. using GRE)!

Overall this results in a lot of frustration and stock-standard VPNs connecting without issue but not forwarding traffic at speeds that non-VPN traffic operates at.

How to configuring VPNs over Satellite links in a way that can be optimized...
So in the wall of text above, we noted that UDP is connectionless, that satellite modems do some form of spoofing to trick TCP/UDP into working fast despite the poor RTT and that VPNs hide the underling TCP/UDP from the modems preventing this optimization! Phew!

A standard VPN uses ESP packets for the tunnel... what if we made it use UDP as the transport instead? On a Cisco router this would be configured using the following (note this is default in newer software)

crypto isakmp nat-traversal 20

The above command points to the fact that NAT traversal suffers from the similar issues.

Hope it helps someone!

ASA - Which serial number to lodge PAKs against

ASAs sometimes show one serial number in the "show version" output and a completely different one in the "show inventory" output. If you are tasked with applying a license to your ASA you will probably ask "which serial number do I use?" The simple answer is; use the "show inventory" serial number.

But what if I've already lodged my license against the wrong serial number? Simply email/contact the licensing team (licensing@cisco.com) and they'll sort it out for you.

Solution for nuisance phone calls from 0198339100 to my mobile

So I've been very annoyed lately. I've been getting phone calls from 0198339100 randomly on my mobile... 24/7. The problem was the service would hang up after one or two rings or would hang up on me even if I answered it. Sometimes I'd answer it in time for it to wait 5 seconds before reading me a text message. This was only happening from a single person. As you can imagine, this is really annoying and I'll be honest... a few swear words were said said over the last few days.

Googling the number reveals it's related to Telstra's Talking Text service. Basically if the phone the SMS was sent to is not SMS capable Telstra convert it into a talking text instead where a robot reads it back to you. The problem is I was receiving these calls on my mobile... which handles SMS perfectly well.... so I was stumped as to why Telstra was sending these through to me. The phone calls and annoyance continued for days.

It took me a while but I figured out what the problem was for me and my situation. I have no doubts someone else will have the same issue so I thought I'd make a post.

My desk phone at work is call-forwarded to mobile... So what was happening was a specific person was innocently sending SMSs to me from his mobile but his mobile had my desk phone listed as my primary contact number. Hence I was getting talking text calls on my desk phone... which forwarded to my mobile 4-5 rings afterwards. By the time they connect through to my mobile they're at the end of the default "wait for answer" period and hang up.

How to properly fix it:

Get the friend/enemy sending SMSs to your desk number to fix it on his end (i.e. change primary contact number to be your mobile rather than desk phone for your contact on his phone).

How to prevent Talking Text from reaching you on mobile:

Block the 0198339100 number on your mobile. For my Samsung S6 it was a case of clicking [...] next to number and saying "block". It won't stop the friend/enemy from sending SMSs to the wrong spot but will stop your mobile phone from ringing. NOTE: Your desk phone will still ring!

How to disable Talking Text service:

Disable the message bank service on your desk phone

How to prevent the calls from being forwarded to your mobile:

Disable call-forwarding on your desk phone.

How to make it work (i.e. keep talking texts from desk phone forwarding to mobile without hanging up):

Increase the "time to wait for answer" by calling 0198339999 from your desk phone and follow the prompts.

There is a bunch of technical information here if you have other requirements/etc.

Hope this helps someone. Good luck!

19 March 2015

Cisco IronPort (WSA/WSAV) - Licensing/Feature Key Enablement

The process to enable feature keys provided by Cisco is vague. As of right now (19/03/2015) I've noted the following as a general guide to the experience:

You need to go through the standard Cisco licensing process and get your license, associated with your device.

If you are migrating from one device to another you can "try" re-host the license yourself on the licensing website or you can email licensing@cisco.com and let them do it for you.

The Cisco Licensing site/team will send you a zip file (or you can download it from the Cisco licensing page). In that zip file a ".bin" file is located as well as a generic "instructions" HTML file. The instructions file should be similar to the below (I've been provided variations to the below throughout my experience):

To apply the key(s), please follow the instructions below:

    In the Web Interface, use the System Administration > Feature Keys page.
    In the Command Line Interface, issue the command 'featurekey'.

Virtual Appliance License Installation Instructions:
1. From the console, note the IP address of the appliance (e.g. enter interfaceconfig)
2. From SSH or telnet, login to the virtual appliance with admin/ironport

3. Enter loadlicense, then
    a. Use option <1> to input the license file by pasting its contents and pressing Ctrl-D, or
    b. Use option <2> to load the license file that has been uploaded to the virtual appliance via FTP:
               i. Using FTP to transfer license file to appliance: ftp to appliance with admin/Ironport, cd into directory configuration, put license.xml, exit
               ii. Using SCP to copy license file to appliance: scp license.xml admin@:configuration

4. Read and agree to the End User License Agreement
5. Enter showlicense to view the license details
6. Log on to the web UI (http://:8080) and run the system setup wizard

The first section is the part you are concerned with if you just have a physical appliance.
The "Virtual Appliance License Installation Instructions" are aimed at WSAV devices and has to do with defining a shared "serial number" across all virtual appliances.

Open the ".bin" file in a text editor. You'll note that a single/series of features, dates and keys are listed similar to the below:

Type: McAfee Anti-Malware, apply by: 2015/04/17, expires: 2015/10/09(12 months, 0 days)