What happens when multiple ADSL circuits with same ISP authenticate with same credentials?

A customer had a fun situation that chewed up plenty of time to get to the bottom of. The basics are that he has multiple ADSL circuits onsite (some considered "spare"), and he wanted to reuse one of these circuits for a new project he was working on. He wanted a new router to be configured remotely and shipped to site to do the job. Sounds easy enough...


First thing I ask for is the user/pass he wants us to use on the new ADSL circuit. He obliges my request instantly. We then configure the router, ship it to site and think nothing more of it. This week he installed the router and informed me that "its not working". Skeptical at first, I try to SSH into the Cisco router on its public IP and no go. Immediate thought is "something's not configured on the router or its an ISP issue".


We work towards canceling out the configuration first. Configuration file is shared amongst the team... no issues flagged. IOS in use was good. Nothing weird or out of place... At this point it was time to bite the bullet and call the ISP.... Telstra. They do their testing and note the following: "I see the ADSL authenticating and disconnecting every 60 seconds or so on the phone line you've quoted. The site isn't losing ADSL Sync/Carrier detect and the line looks great". 


Now we're in firefighting mode as everyone else is adamant it's installed correctly and the line is authenticating. We send  our guys onsite to do some troubleshooting. We get logs similar to below (from http://www.cisco.com/en/US/tech/tk175/tk15/technologies_configuration_example09186a008071a7c2.shtml#l1b):

Router#debug ppp negotiation


PPP protocol negotiation debugging is on


Router#
2w3d: Vi1 PPP: No remote authentication for call-out
2w3d: Vi1 PPP: Phase is ESTABLISHING
2w3d: Vi1 LCP: O CONFREQ [Open] id 146 len 10
2w3d: Vi1 LCP: MagicNumber 0x8CCF0E1E (0x05068CCF0E1E)
2w3d: Vi1 LCP: O CONFACK [Open] id 102 Len 15
2w3d: Vi1 LCP: AuthProto CHAP (0x0305C22305)
2w3d: Vi1 LCP: MagicNumber 0xD945AD0A (0x0506D945AD0A)
2w3d: Di1 IPCP: Remove route to 20.20.2.1
2w3d: Vi1 LCP: I CONFACK [ACKsent] id 146 Len 10
2w3d: Vi1 LCP: MagicNumber 0x8CCF0E1E (0x05068CCF0E1E)
2w3d: Vi1 LCP: State is Open
2w3d: Vi1 PPP: Phase is AUTHENTICATING, by the peer
2w3d: Vi1 CHAP: I CHALLENGE id 79 Len 33 from "6400-2-NRP-2"
2w3d: Vi1 CHAP: O RESPONSE id 79 Len 28 from "John"
2w3d: Vi1 CHAP: I SUCCESS id 79 Len 4
2w3d: Vi1 PPP: Phase is UP
2w3d: Vi1 IPCP: O CONFREQ [Closed] id 7 Len 10
2w3d: Vi1 IPCP: Address 0.0.0.0 (0x030600000000)
2w3d: Vi1 IPCP: I CONFREQ [REQsent] id 4 Len 10
2w3d: Vi1 IPCP: Address 20.20.2.1 (0x030614140201)
2w3d: Vi1 IPCP: O CONFACK [REQsent] id 4 Len 10
2w3d: Vi1 IPCP: Address 20.20.2.1 (0x030614140201)
2w3d: Vi1 IPCP: I CONFNAK [ACKsent] id 7 Len 10
2w3d: Vi1 IPCP: Address 40.1.1.2 (0x030628010102)
2w3d: Vi1 IPCP: O CONFREQ [ACKsent] id 8 Len 10
2w3d: Vi1 IPCP: Address 40.1.1.2 (0x030628010102)
2w3d: Vi1 IPCP: I CONFACK [ACKsent] id 8 Len 10
2w3d: Vi1 IPCP: Address 40.1.1.2 (0x030628010102)
2w3d: Vi1 IPCP: State is Open
2w3d: Di1 IPCP: Install negotiated IP interface address 40.1.1.2
2w3d: Di1 IPCP: Install route to 20.20.2.1

The above is a completely normal connection on the Cisco router. We're not getting anything weird from the router. It gets its public IP correctly from the ISP... everything looks good. I ask the onsite tech to do a "show users" and get the PPP next-hop IP... that's ok. "show routes" that's ok as well everything is pointing in the right direction. 


"Let's try ping the next-hop from the site router"... works ok. "Let's try ping something I know responds to pings on the internet"... this fails. Ok. Check the routes again (still good). Do a traceroute from the router to a public IP... it goes to next-hop IP then dies. "Has to be ISP routing problem then," I decide.

Back to the ISP "Everything looks normal. I can see it's been connected for weeks now". The Telstra tech  sounded absolutely confident... but he does go through and rebuild the circuit from scratch regardless (very nice thing to do to cancel out sticky configurations on their DSLAM gear and kudos to the guy for the effort).

It is at this point where things started to click. "The ADSL was only connected to the router a few days ago...how could it be active for weeks?" I forgot the one golden rule to never forget... don't trust information from customers.

I ask the ISP to do some more snooping around.... and "Bam!" There it was... the customer had already allocated the user/pass we were authenticating with on the new on another ADSL circuit onsite. Telstra finally found the second circuit using the same credentials. The customer had not kept track of the ADSL authentication details used throughout his network and had effectively re-allocated the same service details (ADSL user/pass) twice.

Things to take away from this:

• If an ADSL circuit is already in use by another account you will:
• Be able to authenticate (depends on the ISP and their radius setup)
• Be able to get a public IP address you would normally get (static public IP in this case)
• Be unable to inject a static-framed route into the ISP network to advertise the public IP availability via the L2 ATM link. The ISP network will already have an active route with the same IP/MASK.
• The above will result in "normal" looking router connection but no route-abiltiy from/to the public world
• If a customer provides you information... you should double-check it. :)

Cisco Catalyst Express 500 - Is there a CLI interface available?

Overall the Cisco CatalystExpress 500 is not designed to be used as a command-line configured switch. It is ideally designed for small-business/home office style solutions and runs a whole bunch of features automatically that are typically manually configured on the more enterprise level Catalyst switches. This is generally "OK" provided your customer doesn't want to get too fancy and doesn't have a mixed Catalyst and CatalystExpress switching environment. Overall, I'm not a big fan.

One thing that has come to be a requirement for myself is compare the configuration of a CE500 against a typical Catalyst switch (say a Cisco 2960) this is impossible using the CE500's GUI.... until you know about the hidden URL to get a CLI-like interface with the router. :)

To get to the CLI GUI:

"http://x.x.x.x/level/15/exec/"

I would strongly advise against trying to configure the switch "fully" through this interface as it is not designed to be used for this purpose (you will break the GUI and/or future GUI use will break your config). This CLI part of the GUI is best used to get an understanding for the underlying IOS commands configured on this not-quite-a-Catalyst switch. Happy tinkering...

How to bridge ADSL to Ethernet on a Cisco 800 series router?

Short and sweet. To convert a fully-functioned Cisco ADSL router into a ATM/Ethernet converter (bridge) do the following:

interface ATM0
 no ip address
 no atm ilmi-keepalive
 pvc 8/35
  encapsulation aal5snap
 !
 dsl operating-mode auto
 bridge-group 1
!
!
interface Vlan1
 no ip address
 bridge-group 1
!
bridge 1 protocol ieee

Make sure the ATM interface does not have any sub-interfaces/Dialer interfaces associated and remove any firewalls on the router. Having no IP addresses on interfaces is important as is retaining the PVC settings for the internet provider on the ATM interface.

This is useful for when you need to have a non-ATM device behind the Cisco router do the ISP authentication or hold the IP address that the ISP provides (without doing double-nat).

By doing this you are taking your shiny Cisco router and making it do a job I'd prefer to see a D-Link/Netgear do. Such a waste of a good router. Hope it helps...

How to troubleshoot ADSL on Cisco routers?

The absolute basics of ADSL is that it is a technology that runs over a normal phone line (PSTN). Like Ethernet, the ADSL connection should be looked at using a OSI Layered approach (i.e. work through the layers!):


Layer Breakdown

Layer 1 - Physical

• Noise
- The PSTN can transfer both voice and data at the same time. The basics of this are that there are voice frequencies and a data frequencies. The router reserves the audible-frequencies for voice and then basically allocates whatever is left (and viable) for data. ADSL lives in the data frequencies. "Noise" is basically a measure of how much interference the data frequencies are getting. This can be from devices sharing the PSTN (i.e. fax/phone) or from poor ADSL filters (replace them) or from poor patching/cabling in premises or off-premises. Cisco routers  track and show this value on "connected "ADSL services. 

• Attenuation
- Is a measure of how much a signal degrades over distance. You may have heard that ISPs sometimes say "you're too far from an exchange". This value measures that distance. A signal enters a wire and is "loud". As the signals travels over the wire the signal loses strength. Attenuation is a measure of this loss over distance. When there is too much attenuation, the signal is not strong enough when it is reached by the other end. Cisco routers can track and show this value on connected ADSL connections. 

• Filtering
- If there are other devices sharing the phone line (i.e fax or phone) and there is no filtering in place. The non-filtered non-ADSL device will potentially inject signals into the voice frequencies. The ADSL router will typically respond to this with varying noise readings/ADSL that drops while faxes are received. Filtering at all points where the phone is shared will prevent this. Filtering works by isolating non-ADSL devices to "voice-only" frequencies (i.e. audible frequencies). The ADSL plugs on filters do the inverse (i.e. prevent ADSL from using "voice-only" frequencies). From a router, the only way to spot this is varying noise margins (i.e. it is good, good, good, bad, good, good/etc). 

• Sync/Exchange Configuration
- This is a little harder to explain, you get a PSTN from a provider. The provider then enables specific codes on the DSLAM/Exchange to enable ADSL codes for that PSTN (sometimes ISPs say the DSLAM isn't ADSL ready, which basically means the equipment at the local DSLAM is really old). Routers, before doing any authentication need to build "sync" with the DSLAM (basically a circuit).  If the sync light on your router is flashing it usually means the router isn't able to "build" that circuit to the DSLAM over the PSTN. On Cisco routers this is the CD (Carrier Detect). Ring your ISP and/or make sure you have the right phone line.

Layer 2 - Link-layer

• ATM PVC
- There is a PVC ("circuit") between the router and the Exchange. On a Cisco router you can actually do some ATM pings to see if you can get to your local DSLAM ("segment") or to the exchange ("end").

• PPP/Authentication -
The ADSL protocol in Australia is typically authenticated against a radius server using the PPP protocol. There are two varieties of this protocol PAP/CHAP. Telstra do both but some ISPs only do one or the other. Another thing to be aware of is there is PPPoA or PPPoE (these really just mean PPP over ATM or PPP over Ethernet). The configuration for each PPPoA or PPPoE is different on a router... basically the same thing. Slightly different packet structures/overheads on each. Settings on your router need to be correct though.

Layer 3 - Network

• IP Address -
Once the router authenticates successfully, the radius server should supply an IP address to the end-router (sometimes not depending on ISP). If no address is supplied this may mean that the ISP has not configured the radius server correctly. Are you getting a dynamic IP where you paid for a static? Ring the ISP.

• Routing
- If incorrect default route information is passed onto the router or no default route is configured on the router then the router will not be able to talk to the external world. No routes = no directions for where to go for internet/WAN.

Cisco Commands

Commands on Cisco Routers (837/877/887)

• "show user"
- show connected users VTY/dialer interfaces

• "show dsl int atm 0"
- shows ADSL information of connected interface ATM 0. Includes noise, attenuation, speed in both up/down varieties.

• "debug ppp authentication"
/ "debug ppp negotiation" - shows the ADSL authentication packets going back and forth between Telstra and the router. It is possible to see just authentication requests going out and no response from Telstra.

• "clear int ATM 0"
- Drops and brings back an ADSL connection. Good if you are remote and want to do this.

• "ping atm ..."
 - If you have sync but nothing else, this allows you see whether you can reach the exchange or not

Hope it helps someone...

How to migrate Putty settings from one PC to another?

On old machine:

1. Start > Run > "regedit" > Ok
2. Ctrl + F > Find "SimonTatham"
3. A folder will be found. Right click the folder "SimonTatham" and click Export.
4. Save this file somewhere you can find it again.
5. Copy the file to USB and insert into new machine.

On new machine:

1. Get the USB and locate the ".reg" file from the old machine you saved above
2. Double-click the ".reg" file

Restart Putty. Old settings are now transferred!

Welcome to the blog

This is just tech blog designed to provide day-to-day advice on things I find useful/interesting. Basically this is just somewhere to dump my thoughts of my normal work role and activities as a Network Engineer.