Spoofed packets are a big problem with on the Internet, they are commonly used in DNS amplification attacks, and TCP SYN floods. Unfortunately there is no simple way to totally fix all spoofed packets on the Internet but if service providers implement ingress filtering on their network, it effectively stops such attacks with spoofed source addresses coming from their patch.

The process is actually standardised Best Practice in BCP 38 “Network Ingress Filtering” which all service providers should implement if they have Internet facing services for good karma.

There are a number of ways of implementing ingress filtering, one of the technically simplest is to create ACLs of your customers global address ranges and only allow packets sourced from those ranges to leave your network. Configuration wise Unicast Reverse Path Forwarding (uRPF) is in my opinion the simplest way of managing this and it has a couple of extra features.

uRPF checks incoming unicast packets and validates that a return path exists, there is not much point in forwarding a packet if it doesnt know how to return it right?

There are 2 methods of implementation of uRPF strict and loose. Strict mode is where the source of the packet is reachable via the interface that it came from, this is nice for extra security on the edge of your network but not so good if you have multiple edges towards the Internet eg you peer at multiple IXPs where you might expect asymmetric routing. In such cases loose mode is used which checks that a return route exists in the routing table.

Configuration
The configuration is super simple, after CEF has been enabled just go to the interface you wish to check inbound traffic and use the following command, with the “rx” option for strict mode or “any” for loose mode.

Router(config-if)#ip verify unicast source reachable-via ?
  any  Source is reachable via any interface
  rx   Source is reachable via interface on which packet was received

Verification
Obviously you can check the running config to see if its configured but if your a fan of using other show commands its visible under the sh cef interface and sh ip interface as shown below;

Router#sh cef  interface fastEthernet 0/0 | i RPF
  IP unicast RPF check is enabled

Router# sh ip int fa0/0 | i verify
  IP verify source reachable-via RX

Netflow is a great tool developed by Cisco which is commonly used for bandwidth monitoring & traffic analysis. Its used quite heavily where I work for the detecting and dealing with security related incidents (I talk about it at the last UKNOF meeting here). Although originally developed by Cisco other vendors have support for it under their own product names. And there are standardised versions of it under the name IPFIX.

Like with anything I thoroughly recommend you test this out before rolling it out to your production systems as on high traffic networks it can cause CPU problems in such cases most people change the amount of sampling. If you are worried about this Cisco has produced a white paper on working out the resource utilisation of Netflow.

Configuration

First of all to configure it choose the interfaces you want to monitor and decide if you want to monitor the ingress, egress or both;

Router(config-if)#ip flow ?
egress Enable outbound NetFlow
ingress Enable inbound NetFlow

There are plenty of extra global options available, below is an example of the configuration to capture the packet length, TTL and the MAC addresses.

ip flow-capture packet-length
ip flow-capture ttl
ip flow-capture mac-addresses

Most implementations of Netflow export the data to a remote server for analysis. There are plenty of Netflow analysis software choices. There are 3 versions supported on Cisco routers at present 1, 5 & 9. Version 1 was designed for classful networks and is almost obsolete, Version 5 is designed for IPv4 Unicast flows, and Version 9 is the newest built and can carry BGP Next Hop information, IPv6, Multicast, and MPLS.

Netflow wont export each packets headers as a single packet but it will collect the packet headers together and report them in a single UDP packet via export. In my example at the bottom of this 715 flows have been exported in 48 UDP packets.

ip flow-export version 9
ip flow-export destination 10.10.10.10 555

Another handy option is configuring logging of the top talkers, this is handy if you have a problem with a remote site where they are complain of problems with this enabled you can see the biggest bandwidth hogs in the period of the cache timeout.

ip flow-top-talkers
        top 100
        sort-by bytes
        cache-timeout 3600000
        match protocol tcp

Verification

Most importantly verification its actually working, show ip flow interface below confirms which interfaces are sampling for netflow

Router#sh ip flow interface
Dot11Radio0
  ip flow ingress
  ip flow egress
BVI1
  ip flow ingress
  ip flow egress

If you configured top talkers then show ip flow top-talkers will show you the bandwidth hogs.

Router#sh ip flow top-talkers

SrcIf         SrcIPaddress    DstIf         DstIPaddress    Pr SrcP DstP Bytes
Di0           74.125.165.145  BV1*          10.10.10.100    06 0050 D89F    92K
BV1           10.10.10.100    Local         10.10.10.1      06 D7DC 0017    19K
BV1           10.10.10.100    Di0           74.125.165.145  06 D89F 0050  7381
BV1           10.10.10.100    Di0           209.85.229.138  06 D89E 0050  1466
Di0           209.85.229.138  BV1*          10.10.10.100    06 0050 D89E  1379
5 of 100 top talkers shown. 5 of 9 flows matched.

Finally if you configured netflow export verify it with the show ip flow export command.

Router#sh ip flow export
Flow export v9 is enabled for main cache
  Export source and destination details :
  VRF ID : Default
    Destination(1)  10.10.10.10 (555)
  Version 9 flow records
  715 flows exported in 48 udp datagrams
  0 flows failed due to lack of export packet
  48 export packets were sent up to process level
  0 export packets were dropped due to no fib
  0 export packets were dropped due to adjacency issues
  0 export packets were dropped due to fragmentation failures
  0 export packets were dropped due to encapsulation fixup failures

Just received version 1.0.0.0 of the IPExpert Video on Demand (VoD) the updated for the v4 R&S CCIE track. I have gone through the first day over the weekend and thought it would be helpful to post my initial thoughts.

For your money you get 45 videos, which are 116kbps MP4 files, consisting of 84kbps of video (only slides so this is fine), and 32kbps of audio. The menu splits the video up over 4 days but from the amount of content it could easily be over 5 days,.

The product gets shipped on a DVD and the files are 4GB, I have copied the files straight off the DVD onto my hard drive and they have worked fine. I have tested it with Windows 7 x64 and on my Mac with Snow Leopard and it worked really fine, just click on the .exe or .app file and you get presented the menu below.

Interface
Below is an example of the video interface which is the same on Windows or Mac. You might notice that there is no full screen option as there was in the v3 Video, I hope they put that in a future version as the videos could be a little larger especially if you have a large screen with high resolution.

One of my main issues with the old VoD was that the slides did not appear for long enough and the video was split between the slides and Scot Morris. It was sometimes a welcome break to pause it and try to get Scott Morris to pull strange faces mid talk. With the update there is no video of the instructor just the slides, command line and audio which is best but no funny faces of Joe like we could get with Scott.

Bypassing Obfuscation
One of the downsides to the product is they dont give you easy access to the MP4 files, personally I would like to have the media on my phone, ipod etc. Now from the looks of they havn’t encrypted the video files just obfuscated them. If you look in the /com/ipexpert/data.dat/ folder you will see 45 files with the .ipx extension. Changing the extension to MP4 allows you to play them in any player you like. You just need to match up the file names with the actual video content, it would be pretty simply for somebody to write a batch/shell script to do this

Quality of Content
The videos are presented by Joe Astorino who trained with the v3 BLS as he explained in the videos. From what I have seen so far (after day 1) Joe presents a lecture on the theory which lasts around and hour long and then a few 30 to 60 mins hands on labs on the topics talked about. The level of detail is really good and he doesn’t spend to long on the basics. There are 45 video files which are split over 4 official days.

Joe is really good at explaining every little detail, but is especially on the configuring sections. Sometimes there is a lot of detail on the slides, but if you want to refer back to the slides they are available to download from the members area of the IPExpert web site, which is a nice touch. There are a couple of really minor problems which should get fixed eg after one of the video Joe says words to the affect of “thats the end of day 1″ when there is another video, but that should get fixed pretty quickly and does not affect the quality of the training.

The quality of the audio is also excellent, almost too good as you can hear every creak of Joe’s chair and gulp of water!

Pros
Excellent quality instruction by Joe
Really high quality video
Good value for money, much cheaper than a boot camp

Cons
No full screen button
No easy access to the MP4 files

Overall
Its a damn good set of videos updated for the new track, I am very happy with it but could only be a little more if it had a full screen button or easier access to the MP4 files. Joe has done a brilliant job and I am defiantly feeling better about the configuration section regarding the topics he has covered. Its part of the BLS so contact IPExpert if you have the old version to get them to send you the updated one.

Quick bit of a refresh on OSPF network types, these are specified on a per link basis and it changes the way OSPF behaves on that link specifying the wrong type can cause bad things to happen such as if you mix DR & Non DR types adjacencies wont form correctly.

There are 5 network types,

1. Point to Point – Should be used as the name implies on P2P links which support broadcast, nearly all OSPF traffic is sent to the ALLSPFRouters address which is 224.0.0.5, although retransmitted LSAs are sent to the unicast address as such these links should support broadcasting.
2. Broadcast – This is the default network type for Ethernet networks, as this network type is capable of having adjacencies with multiple routers. To prevent unnecessary meshing DR & BDR elections take place. Communications from the DR & BDR take place on the ALLDRRouters address 224.0.0.6 whilst all other communications on 224.0.0.5. As the name implies the layer 2 network must support broadcasting.
3. Non Broadcast Multiple Access (NBMA) – This is the default configuration on a serial interface, and is classified as a link which is able to send to multiple other routers via the single link but with no broadcasting, therefore neighbour statements must be manually set. A DR & BDR will still be elected but all communication is unicast, the hello and dead timers change from the defaults to 30 and 120 seconds.
4. Point to Multipoint networks – All communication is unicast and no DR or BDR election takes place an ideal placement for this would be in the hub in a hub and spoke topology.
5. Virtual Links – Packets sent over Virtual Links are unicast and perceived as unnumbered point to point connections

Its pretty simple to change the network type its all done under the interface like so.

Router(config-if)#ip ospf network ?
  broadcast            Specify OSPF broadcast multi-access network
  non-broadcast        Specify OSPF NBMA network
  point-to-multipoint  Specify OSPF point-to-multipoint network
  point-to-point       Specify OSPF point-to-point network

In a nutshell On-Demand Routing (ODR) is simply freaking awesome for hub and spoke environments.

In hub and spoke enviornemnts before ODR there was generally two choices to the deployment

1. Have default routes on the spokes pointing to the hub and static routes back. But frankly this was a pain as there was an administrative burden as you have to manually add/remove each spoke as needed.
2. Run a routing protocol on the spokes which communicates reachability information with the hub but this is unnecessary as the spokes are dead end streets they don’t need more topology information than a default route and generally the spoke routers aren’t the best in the network so it an unneeded overhead.

But this is were On-Demand Routing (ODR) comes to the rescue . Now ODR is not a routing protocol, the spoke routers send details of attached networks via CDP to neighbouring routers and when ODR is enabled the network information is simply installed into the IP routing table. Each of the spokes only needs to have a default route pointing to the hub and have CDP enabled which is on by default and thats it. The routes it collects have an AD of 160 with a metric of 1 (only 1 hop away), the routes that ODR collected can then be redistributed into your IGP of choice.

If you don’t believe me that its that simple check out the configuration below

Firstly we just enable it with the

router odr

command, this is a single command which only needs to be configured on the hub, the spokes dont need it.

R0(config)#router odr

There are not many options in the protocol itself the most common you might wish to change is the timers

R0(config-router)#?
Router configuration commands:
  default            Set a command to its defaults
  default-metric     Set metric of redistributed routes
  distance           Define an administrative distance
  distribute-list    Filter networks in routing updates
  exit               Exit from routing protocol configuration mode
  help               Description of the interactive help system
  maximum-paths      Forward packets over multiple paths
  neighbor           Specify a neighbor router
  network            Enable routing on an IP network
  no                 Negate a command or set its defaults
  passive-interface  Suppress routing updates on an interface
  redistribute       Redistribute information from another routing protocol
  timers             Adjust routing timers
  traffic-share      How to compute traffic share over alternate paths

We can validate that its running and see the routes it has collected.

R0#sh ip protocols
Routing Protocol is "odr"
  Sending updates every 60 seconds, next due in 47 seconds
  Invalid after 180 seconds, hold down 0, flushed after 240
  Outgoing update filter list for all interfaces is not set
  Incoming update filter list for all interfaces is not set
  Maximum path: 4
  Routing Information Sources:
    Gateway         Distance      Last Update
    10.1.2.1             160      00:00:30
    10.1.1.1             160      00:00:47
  Distance: (default is 160)

And on the hub se should be able to see the routes installed into the IP routing table with the AD of 160 and metric of 1

R0#sh ip route
Codes: C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route

Gateway of last resort is not set

     10.0.0.0/24 is subnetted, 2 subnets
o       10.1.2.0 [160/1] via 10.1.2.1, 00:01:45, Serial0/1
                 [160/1] via 10.1.2.1, 00:00:45
o       10.1.1.0 [160/1] via 192.168.1.2, 00:00:02, Serial0/0
                 [160/1] via 10.1.1.1, 00:02:02, Serial0/0
                 [160/1] via 10.1.1.1, 00:01:02
     192.168.1.0/24 is variably subnetted, 2 subnets, 2 masks
C       192.168.1.0/24 is directly connected, Serial0/0
C       192.168.1.2/32 is directly connected, Serial0/0
     192.168.2.0/24 is variably subnetted, 2 subnets, 2 masks
C       192.168.2.2/32 is directly connected, Serial0/1
C       192.168.2.0/24 is directly connected, Serial0/1

Now to test it the view from R2 just shows the default route up to the hub

R2#sh ip route
Codes: C - connected, S - static, R - RIP, M - mobile, B - BGP
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2
i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
ia - IS-IS inter area, * - candidate default, U - per-user static route
o - ODR, P - periodic downloaded static route

Gateway of last resort is 0.0.0.0 to network 0.0.0.0

10.0.0.0/24 is subnetted, 1 subnets
C       10.1.2.0 is directly connected, Loopback0
192.168.2.0/24 is variably subnetted, 2 subnets, 2 masks
C       192.168.2.0/24 is directly connected, Serial0/0
C       192.168.2.1/32 is directly connected, Serial0/0
S*   0.0.0.0/0 is directly connected, Serial0/0

Now to test if we can ping the other spoke router

R2#ping 10.1.1.1

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.1.1.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 12/39/60 ms

I am almost speechless at how simple it was to get spoke to spoke connectivity! Beautiful!

Sometimes you need to know the MTU of a path across a network, the MTU commonly changes for things like Q in Q or GRE tunnelling so its important to accurately know it to prevent undue fragmentation.

The middle bit in the 3 bit fragmentation flags field in the IP header is the Don’t Fragment (DF) flag, if it set to 1 the packet will not be fragmented, therefore if the packet exceeds the MTU of a link it will generate a ICMP type 3 code 4 error message and be dropped.

Using the extended commands in the ping we can manually determine the specific MTU across a path, there are a couple of pointers on doing this;

1. Change the repeat count to 1 or it will sweep the entire range a default of 5 times which is usually unnecessary
2. Don’t sweep the default range, it defaults to an MTU of between 36 to 18024 bytes, considering it sends 1 packer per byte of the MTU that’s 17,988 packets. That a hell of a lot of default 2 second time outs to wait for… so try a smaller range like 10 or so bytes
3. Remember to set the DF bit in the IP header or the ICMP packets will fragment and you actually be testing the MTU
4. Use the verbose option to see the results of each packet
5. Change the sweep interval to 1, again there is little point in sweeping the range multiple times

Router#ping
Protocol [ip]:
Target IP address: 201.14.1.1
Repeat count [5]: 1
Datagram size [100]:
Timeout in seconds [2]:
Extended commands [n]: y
Source address or interface:
Type of service [0]:
Set DF bit in IP header? [no]: yes
Validate reply data? [no]:
Data pattern [0xABCD]:
Loose, Strict, Record, Timestamp, Verbose[none]: Verbose
Loose, Strict, Record, Timestamp, Verbose[V]:
Sweep range of sizes [n]: y
Sweep min size [36]: 1495
Sweep max size [18024]: 1505
Sweep interval [1]:
Type escape sequence to abort.
Sending 11, [1495..1505]-byte ICMP Echos to 201.14.1.1, timeout is 2 seconds:
Packet sent with the DF bit set
Reply to request 0 (4 ms) (size 1495)
Reply to request 1 (4 ms) (size 1496)
Reply to request 2 (4 ms) (size 1497)
Reply to request 3 (4 ms) (size 1498)
Reply to request 4 (4 ms) (size 1499)
Reply to request 5 (8 ms) (size 1500)
Request 6 timed out (size 1501)
Request 7 timed out (size 1502)
Request 8 timed out (size 1503)
Request 9 timed out (size 1504)
Request 10 timed out (size 1505)
Success rate is 54 percent (6/11), round-trip min/avg/max = 4/4/8 ms
Router#

As shown in the example above it indicates that the maximum MTU along the path is 1500 bytes.

Quite often when you are trying to verify that tasks are working  as expected you use the #debug ip packet command which works great… except if there is any other traffic on the router and you can quickly become swamped with the debug output. Luckily the #debug ip packet command can also take an ACL so if you wanted to verify traffic was going to the host 10.10.10.1 and only to that host you can do this by;

• - Creating an ACL for 10.10.10.1

  Router#Access-list 101 permit ip any host 10.10.10.1

• - Applying that ACL to the debug command

Router#Debug ip packet detail 101

• - Bob’s your uncle

Because the #debug ip packet detail takes an ACL you can change the ACL to whatever you like, so if you just wanted to see ICMP its simple just change it to the following and reapply your Debug ip packet detail 101.

Router#Access-list 101 permit ip any any icmp

Another handy way is that if you accidently perform a debug command which causes a backlog on your console connection that you could not perform an un all due to so much output on the console, you should be able to telnet into the router and issue the usual undebug all immediately

The debugging will instantly disappear from the Console session as well, this is as debugging wont be sent to vty lines without the following command issuing the command terminal monitor, so you can stop debugging without having to wait to get your command across the Console if you have filled it up with debugging.

This can save valuable minutes or even a reload just remember to setup remote access before getting into a pickle

Just found out that at CCIE Flyer is giving away Narbik’s Soup to Nuts, the link is on his post at http://ccieflyer.com/2009-October-toc.php. I purchased the book a while ago and it is a great resource for getting to grips with the key technologies but there are a few errors. It is aimed at v3 of the R&S track but the topics in v3 and v4 are very similar and its free so take a look!

I also highly recommend Packet Life’s “cheat sheets”, they are brilliantly made quick reference guides, I print a couple of them off and keep them in my desk, so that the next time that I am unsure if that annoying fibre is really an ST, SC, or LC I can rest easy http://packetlife.net/library/cheat-sheets/

Also its a couple of old links but both IP Expert and Internetwork Expert have some free lectures to tempt you into their products at;
https://www.ipexpert.com/index.cfm/product/sku/ccie_lab_vlecture_series
http://www.internetworkexpert.com/free_ccie_vseminar.htm

Frame Relay Local Management Interface (LMI) is a set of enchantments to frame relay, originally agreed upon in 1990 by a consortium consisting of Cisco, Digital Equipment Corporation (DEC), Northern Telecom, and StrataCom. Eventually ANSI and CCITT came along created standard versions but most vendors support both standard version alongside the one defined by the consortium.

There are very few differences between them, mainly Cisco LMI type use the DLCI of 1023 and also DLCI number between 16 and 1007 for usable DLCI number and the ANSI & ITU(q933a) standard use a DLCI of 0 and usable DLCI numbers between 16 and 976. The LMI type can be changed on the frame relay enabled interface with the command;

Router(config-if)#frame-relay lmi-type ?
  cisco
  ansi
  q933a

The LMI messages messages sent between Routers and Frame Relay switches provide the the following additional features;
- Inverse ARP – to find out the layer 3 address of device on the other end of the DLCI so you dont need to static map it or similar
- Signalling of Virtual Circuit status messages, therefore if a PVC becomes unreachable all nodes along its path can be aware of this failure so data can be prevented from being sent to indirect failures.
- Multicasting – This extension allows multicast groups to be used over frame relay networks, the higher DLCI numbers reserved by the LMI type are used for this
- Globally signficiant DLCIs – Brilliant!
- Retro flow control with XON/XOFF if the applications using the Frame Relay network know understand FECNs and BECNs

By default LMI messages are sent every 10 seconds, and every sixth message a full status message will be sent which contains more detailed information about each VC, the interface will fail if the interfaces does not receive an LMI message 3 times the hello time, so 30 seconds. You cant actually turn off LMI but you can disable the keepalives with the highly ambiguous command;

Router(config-if)#frame-relay lmi-n391dte ?
  <1-255>  event

Finally you can check the status of LMI with the command;

Router#sh frame lmi

LMI Statistics for interface Serial1/0 (Frame Relay DTE) LMI TYPE = CISCO
  Invalid Unnumbered info 0             Invalid Prot Disc 0
  Invalid dummy Call Ref 0              Invalid Msg Type 0
  Invalid Status Message 0              Invalid Lock Shift 0
  Invalid Information ID 0              Invalid Report IE Len 0
  Invalid Report Request 0              Invalid Keep IE Len 0
  Num Status Enq. Sent 18               Num Status msgs Rcvd 0
  Num Update Status Rcvd 0              Num Status Timeouts 18
  Last Full Status Req 00:00:03         Last Full Status Rcvd never

Finally pretty much every command on the interface beings with frame-relay;

Router(config-if)#frame-relay ?
  accounting             Special accounting instruction
  address-reg            ELMI address registration
  broadcast-queue        Define a broadcast queue and transmit rate
  class                  Define a map class on the interface
  congestion-management  Enable Frame Relay congestion management
  de-group               Associate a DE group with a DLCI
  fragment               Enable end-to-end fragmentation for all PVCs
  fragmentation          Adaptive fragmentation

UPDATE: On some versions of IOS it forces you to set set a LMI hello of between 1 and 255 thereby the above method of using frame relay encapsulation on a P2P link by turning off LMI does not work, in these cases your going to have to use a different encapsulation type or a frame relay switch.

R5(config-if)#frame-relay lmi-n391dte
% Incomplete command.

Recently I have been playing around with RSA SecurID which is commonly used to authenticate VPN users using 2 part authentication, which is great. Although I am way more concerned about using 2 part authentication on admins logging into network devices, luckily its really easy to setup.

The RSA authentication is managed by an RSA application called the RSA Authentication Manager which is currently on version 7.1, this handles replication between servers in a primary to replica’s fashion. Each Authentication Manager can have the RSA “Steel Belted RADIUS” server installed which allows RADIUS clients to authenticate against it. If you are using 2 part authentication you really need to have build in redundancy as if the devices are at a remote site from the RADIUS server they will not be able to perform 2 part authentication in the case of a network or server failure, but we all know that single points of failure are a bad idea?

Authentication Manager Configuration

I wont go to much into how to install and manage the RSA Authentication Manager as there is loads of documentation from RSA on that matter, but there is not much on getting it to authenticate console or vty line users, but luckily its really simple. For each Cisco device that you wish to authenticate users on add it as a RADIUS client in the Authentication Manager, with the model set to “IOS11.1 or later” and then add an associated RSA agent. After this you will need to remember to force replicate the information to the replica RADIUS servers as it does not do this automatically (authentication on the replicas will fail if you don’t do this!)

IOS Configuration

As like most AAA things make sure the aaa new-model is on

aaa new-model

Next we should have sure we have a local user name and password on the system, if the RADIUS servers does not send back a response then we can login using this account. If the response from the RADIUS server comes back as failed from the RADIUS server it will NOT check the local username and password, so this will only be used if the system receive an invalid or no response form the RADIUS server.

username cisco privilege 15 password 0 cisco

Now to enable the RADIUS servers group to be used for the authenticating users on login.

aaa authentication login default group radius line enable local

Now to configure the RADIUS servers, add a separate line for each additional server. It defaults to the standard ports 1645 for authorisation and 1646 for accounting. If there are multiple RADIUS servers it will do a full set of timeouts and retransmits to each one before moving onto the next server.

radius-server host 10.0.0.10 auth-port 1645 acct-port 1646 key password

As I mentioned if anything goes wrong with the RADIUS servers it will fail back to using the locally set username and password, using the default time outs, as waiting for the default timemouts this feels like an eternity so we should change them.

radius-server retransmit 3
radius-server timeout 10

Now give it a test, it even happily support next tokencode mode (see the image), this is where the Authentication Manager can ask you to enter your next token if it thinks your token is starting to drift time abit or it just wants to make sure you actually have the token.

Troubleshooting
#show radius statistics is a good show command to check the response times and make sure everything is singing along well

DistSwitch#show radius statistics
       Maximum inQ length: 1
     Maximum waitQ length: 1
     Maximum doneQ length: 1
     Total responses seen: 53
   Packets with responses: 53
Packets without responses: 17
   Average response delay: 3336 ms
   Maximum response delay: 35048 ms
Number of Radius timeouts: 73
     Duplicate ID detects: 0

  Elapsed time since counters last cleared: 1w2d6h22m

Troubleshooting is pretty simple with #debug radius

User Access Verification

Username: exampleuser
Password:

1w1d: RADIUS: ustruct sharecount=1
1w1d: RADIUS: Initial Transmit tty0 id 45 10.10.0.10:1645, Access-Request, len 66
1w1d:         Attribute 4 6 0A0A34A0
1w1d:         Attribute 5 6 00000000
1w1d:         Attribute 61 6 00000000
1w1d:         Attribute 1 10 62722164
1w1d:         Attribute 2 18 DC6BDD95
DistSwitch>
1w1d: RADIUS: Received from id 45 10.10.20.158:1645, Access-Accept, len 80
1w1d:         Attribute 25 60 53425232
1w1d: RADIUS: saved authorization data for user 80E96540 at 80E95D30
DistSwitch>
DistSwitch>ena
DistSwitch#

If that’s not enough information and your getting alot of timeouts try using Wireshark with a port filter on UDP 1645 will allow you to see the RADIUS rejects, accepts, and challenges.

If the server appears to be rejecting some requests using the real time authentication activity monitor on the RSA Authentication Manager will provide much more detailed information of the reason for the reject or accept.