Cisco 887VAG - Unable to get the 3G working

Today I was tasked with configuring a Cisco 887VAG's 3G interface... and had some issues. Here in Australia I've done plenty of 3G on Cisco router configurations so this one kind of frustrated me a little
The basics of the issue are that the Cisco 887VAG doesn't seem to support "encapsulation ppp" even though the router happily accepts the command.
The configuration for most 3G varieties with Cisco looks something like the following:

TEST#

cellular 0 gsm profile create 1 <APN> chap <username@domain.com.au> <password>

TEST(config)#
chat-script gsm "" "ATDT*98*1#" TIMEOUT 60 "CONNECT"

!

interface Cellular0

description NextG Modem Dialer Interface

no ip address

encapsulation ppp

dialer in-band

dialer pool-member 1

async mode interactive

!

interface Dialer1

ip address negotiated

ip virtual-reassembly

encapsulation ppp

dialer pool 1

dialer idle-timeout 0

dialer string gsm

dialer persistent

dialer-group 1

no fair-queue

no cdp enable

ppp chap hostname <username@domain.com.au>

ppp chap password <password>

ppp ipcp dns request

!

!

ip route 0.0.0.0 0.0.0.0 Dialer1 250

!

!

access-list 1 permit any

dialer-list 1 protocol ip list 1

!

!

line 3

script dialer gsm

no exec

transport input all

transport output telnet

In the above, you have a 3G profile, a cellular interface and dialer interface all joined together. On the 887VAG the encapsulation ppp command spat out an issue as follows:

TEST(config)#int cell 0

TEST(config-if)#encapsulation ppp

Cellular0: Only SLIP encapsulation supported

Without PPP we can't authenticate. Without authentication... it won't allow me to connect. Regardless, I couldn't find much information out there on this specific issue so I thought a blog entry would be handy. I've got a TAC case lodged with Cisco now... will update as more information comes to hand.
The Cisco 887VAG is running:

TEST#show ver

Cisco IOS Software, C880 Software (C880DATA-UNIVERSALK9-M), Version 15.1(4)M3, RELEASE SOFTWARE (fc1)

Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2011 by Cisco Systems, Inc.
Compiled Tue 06-Dec-11 22:52 by prod_rel_team
ROM: System Bootstrap, Version 15.1(2r)T2, RELEASE SOFTWARE (fc1)

TEST uptime is 1 hour, 11 minutes
System returned to ROM by power-on
System restarted at 00:56:10 UTC Mon Apr 2 2012
System image file is "flash:c880data-universalk9-mz.151-4.M3.bin"
Last reload type: Normal ReloadTEST(config)#int cell 0

=======================================

Update 1:
As John Marshall indicated in the comments below, there is a configuration guide available from Cisco here. I was directed to that guide by TAC just over the last week or so. I've had some delays with getting through to TAC for me due to SmartNet coverage and so forth.

For me, the guide indicated didn't work.... I have several devices all fresh from factory. I tried setup a second 887VAG with the guide configuration from scratch like for like to what John did and still no go.

I'm making some progress with TAC now... we will see how we go. Oh, and the SIM definitely works (it was confirmed on a 887G on my desk using the old style configuration security).

This isn't all to say I haven't been making "some" progress though. The cellular interface on the new routers "HAS" to be slip encapsulation. It is just the way the new modem interfaces with the chassis. Refer to the configuration guide from Cisco for more information on that.

I'll probably write another post with configuration/troubleshooting steps once I get it sorted completely... for the moment I'm still stuck but am working with TAC to get this resolved.

For the meantime here are some useful commands:

• debug cellular 0 messages all - If you want to see if the cellular interface is doing "something" chuck this command in to have a look at the back/forth between the cellular interface and the cell tower.
• show cell 0 radio history all - This shows you RSSI history on the modem much like "show proc cpu history" shows you CPU loading history. Cool.
• debug ppp negotiation / debug ppp authentication - In theory (at least on the old routers) these commands were useful because they confirmed either you were at the PPP level of the connection. I'm not sure if these work on the newer models though... yet. I plan to find out.

Useful information:

• The cellular profile referenced by the "AT" command seems to still be configured using the "cell 0 gsm profile create" command according to the guide
• The chat-script line referenced by the old (AT98*1)/new (AT!SCACT=1,1) routers differs but achieves the same thing. These are the AT commands the chassis runs to initiate a connection on the onboard Sierra Wireless card to connect using a profile. The differences are mainly cosmetic and you shouldn't need to tinker with them beyond defining "1" as the profile.

What I am thinking:

• At the moment if I telnet into the modem itself (you shouldn't need to do this) and run the AT command specified by Cisco's guide I get an error:
  
  Router#telnet 1.1.1.1 2003
  Trying 1.1.1.1, 2003 ... Open
  AT!SCACT=1,1
  +CME ERROR: no network service
  Now I also tried creating a profile on the modem itself using AT commands that specified PAP authentication... and when I ran the AT command described above "OK". Which means "successful". Frustrating. (I've been careful to only play with AT commands on one device in case it breaks it)

I'll continue to work through with Cisco...

=======================================

Update 2: Resolution
In the end the Cisco provided configuration in the link above (or in comments) is correct. Use that configuration. My issue was related to the cellular profile not being the correct type. Telstra need IPv4 type profiles for internet. I was configuring PPP type profiles. I will write up a more detailed guide at some point.

Outlook 2010 and the curse of the invisible mail rules...

I just had a fun morning troubleshooting outlook. My mailbox had certain rules stopping noisy/mis-configured monitoring servers from popping up in my inbox all day. The monitoring server was repaired today and I wanted to remove the rule that pushed the "noise" mail to a particular folder and marked it as read... Easy enough, navigated to manage rules in Outlook 2010 and... huh? It looks like there are no rules pushing the false-positive alerts to the folder. Mails are still definitely being redirected though. This must be fixed!

Just to be sure, I disable all rules in Outlook and wait for a few monitoring alerts to come through and of course they are still being caught by an invisible rule somewhere, and being moved to the specific folder and marked as read. So what's going on here?

A few minutes Google-teching the issue I came across a few possible ways to resolve the issue.

1. Quick-rebuild the Outlook profile - Remove profile in Windows Mail control panel item and pull it down again from Exchange server. This didn't work for me...
2. Hard-rebuild of Outlook profile - Remove the profile completely (navigate to and delete AppData files/etc) and pull it down from the server again. Again, this didn't work for me...
3. Break the rule - Removed the Outlook folder that contains the moved monitoring mails and also rebooted Outlook with a "/clearrules" argument - This worked...

I know that removing folders that rules rely upon to breaks the Outlook rule logic thus the rule stops... so this may have resolved it. I am thinking the "/clearrules" flag also helped as well.

Which one fixed it? Not exactly sure... but it works now. Regardless, hopefully this helps someone else...

What happens when multiple ADSL circuits with same ISP authenticate with same credentials?

A customer had a fun situation that chewed up plenty of time to get to the bottom of. The basics are that he has multiple ADSL circuits onsite (some considered "spare"), and he wanted to reuse one of these circuits for a new project he was working on. He wanted a new router to be configured remotely and shipped to site to do the job. Sounds easy enough...


First thing I ask for is the user/pass he wants us to use on the new ADSL circuit. He obliges my request instantly. We then configure the router, ship it to site and think nothing more of it. This week he installed the router and informed me that "its not working". Skeptical at first, I try to SSH into the Cisco router on its public IP and no go. Immediate thought is "something's not configured on the router or its an ISP issue".


We work towards canceling out the configuration first. Configuration file is shared amongst the team... no issues flagged. IOS in use was good. Nothing weird or out of place... At this point it was time to bite the bullet and call the ISP.... Telstra. They do their testing and note the following: "I see the ADSL authenticating and disconnecting every 60 seconds or so on the phone line you've quoted. The site isn't losing ADSL Sync/Carrier detect and the line looks great". 


Now we're in firefighting mode as everyone else is adamant it's installed correctly and the line is authenticating. We send  our guys onsite to do some troubleshooting. We get logs similar to below (from http://www.cisco.com/en/US/tech/tk175/tk15/technologies_configuration_example09186a008071a7c2.shtml#l1b):

Router#debug ppp negotiation


PPP protocol negotiation debugging is on


Router#
2w3d: Vi1 PPP: No remote authentication for call-out
2w3d: Vi1 PPP: Phase is ESTABLISHING
2w3d: Vi1 LCP: O CONFREQ [Open] id 146 len 10
2w3d: Vi1 LCP: MagicNumber 0x8CCF0E1E (0x05068CCF0E1E)
2w3d: Vi1 LCP: O CONFACK [Open] id 102 Len 15
2w3d: Vi1 LCP: AuthProto CHAP (0x0305C22305)
2w3d: Vi1 LCP: MagicNumber 0xD945AD0A (0x0506D945AD0A)
2w3d: Di1 IPCP: Remove route to 20.20.2.1
2w3d: Vi1 LCP: I CONFACK [ACKsent] id 146 Len 10
2w3d: Vi1 LCP: MagicNumber 0x8CCF0E1E (0x05068CCF0E1E)
2w3d: Vi1 LCP: State is Open
2w3d: Vi1 PPP: Phase is AUTHENTICATING, by the peer
2w3d: Vi1 CHAP: I CHALLENGE id 79 Len 33 from "6400-2-NRP-2"
2w3d: Vi1 CHAP: O RESPONSE id 79 Len 28 from "John"
2w3d: Vi1 CHAP: I SUCCESS id 79 Len 4
2w3d: Vi1 PPP: Phase is UP
2w3d: Vi1 IPCP: O CONFREQ [Closed] id 7 Len 10
2w3d: Vi1 IPCP: Address 0.0.0.0 (0x030600000000)
2w3d: Vi1 IPCP: I CONFREQ [REQsent] id 4 Len 10
2w3d: Vi1 IPCP: Address 20.20.2.1 (0x030614140201)
2w3d: Vi1 IPCP: O CONFACK [REQsent] id 4 Len 10
2w3d: Vi1 IPCP: Address 20.20.2.1 (0x030614140201)
2w3d: Vi1 IPCP: I CONFNAK [ACKsent] id 7 Len 10
2w3d: Vi1 IPCP: Address 40.1.1.2 (0x030628010102)
2w3d: Vi1 IPCP: O CONFREQ [ACKsent] id 8 Len 10
2w3d: Vi1 IPCP: Address 40.1.1.2 (0x030628010102)
2w3d: Vi1 IPCP: I CONFACK [ACKsent] id 8 Len 10
2w3d: Vi1 IPCP: Address 40.1.1.2 (0x030628010102)
2w3d: Vi1 IPCP: State is Open
2w3d: Di1 IPCP: Install negotiated IP interface address 40.1.1.2
2w3d: Di1 IPCP: Install route to 20.20.2.1

The above is a completely normal connection on the Cisco router. We're not getting anything weird from the router. It gets its public IP correctly from the ISP... everything looks good. I ask the onsite tech to do a "show users" and get the PPP next-hop IP... that's ok. "show routes" that's ok as well everything is pointing in the right direction. 


"Let's try ping the next-hop from the site router"... works ok. "Let's try ping something I know responds to pings on the internet"... this fails. Ok. Check the routes again (still good). Do a traceroute from the router to a public IP... it goes to next-hop IP then dies. "Has to be ISP routing problem then," I decide.

Back to the ISP "Everything looks normal. I can see it's been connected for weeks now". The Telstra tech  sounded absolutely confident... but he does go through and rebuild the circuit from scratch regardless (very nice thing to do to cancel out sticky configurations on their DSLAM gear and kudos to the guy for the effort).

It is at this point where things started to click. "The ADSL was only connected to the router a few days ago...how could it be active for weeks?" I forgot the one golden rule to never forget... don't trust information from customers.

I ask the ISP to do some more snooping around.... and "Bam!" There it was... the customer had already allocated the user/pass we were authenticating with on the new on another ADSL circuit onsite. Telstra finally found the second circuit using the same credentials. The customer had not kept track of the ADSL authentication details used throughout his network and had effectively re-allocated the same service details (ADSL user/pass) twice.

Things to take away from this:

• If an ADSL circuit is already in use by another account you will:
• Be able to authenticate (depends on the ISP and their radius setup)
• Be able to get a public IP address you would normally get (static public IP in this case)
• Be unable to inject a static-framed route into the ISP network to advertise the public IP availability via the L2 ATM link. The ISP network will already have an active route with the same IP/MASK.
• The above will result in "normal" looking router connection but no route-abiltiy from/to the public world
• If a customer provides you information... you should double-check it. :)