New posts to come soon, though, featuring lots of maths and glib commentary. Honest!

EVENT #
345641
EVENT LOG
Application
EVENT TYPE
Error
OPCODE
Info
SOURCE
MSExchangeIS
CATEGORY
General
EVENT ID
9646
COMPUTERNAME
MAILBOXES
DATE / TIME
08/07/2010 8:54:38 PM

MESSAGE
Mapi session “/o=Org/ou=Exchange Administrative Group/cn=Recipients/cn=User ” exceeded the maximum of 32 objects of type “session”.

BINARY DATA: ……

In our case, these were caused by Riverbed Steelhead appliances running RiOS 6.0.2 (though downrevs will likely experience similar issues) with MAPI Transparent Pre-population enabled.  Users were accessing a VPN in one site, while their mailboxes were in another.  Intersite traffic passes through Riverbed Steelhead appliances in each site, subjecting the user’s Outlook client to Riverbed optimization.

When the client disconnected, the Riverbed appliance maintained the MAPI sessions opened by the Outlook client in order to prefetch mail/calendering/etc.  As the client reconnected/disconnected over the course of a day, this would cause build-up of MAPI session state in the mailbox server, eventually hitting the default limit of 32 sessions.  Subsequent attempts to load Outlook by the user would result in their being prompted for their domain username/password; extremely annoying for the user.

While this 32-session limit could be increased, or the stale connections could be pruned, we preferred a more permanent fix.  After the pre-pop feature was disabled, this problem went away.  This comes at the cost of disconnected clients having to download their new mail from a remote mailserver the next time they connect (as opposed to the Riverbed pre-fetching it for them and handing mail to them locally) — but if mail is a relatively low-priority app in your environment, this is probably more acceptable than users not being able to access it at all (or being prompted for domain credentials all the time).

I should note that in all likelihood, a compromise could have been reached by tweaking the pre-pop timers to time out sessions after a few hours instead of the default 96 hours. YMMV.

I’ll just leave this here in the event that Event ID 9646 gets googled or something.  Outlook clients are generally well-behaved; you shouldn’t see an error like this unless a) the Exchange mailbox server is holding connections open for no reason –not generally a steady-state condition– or b) something in the path is holding the connection to your Exchange mailbox server(s) open on behalf of the client.  Look for anything in the path that “touches” the application-layer and you’ve found a likely culprit.

For Riverbed customers with a valid support contract, this issue can be found as KB article #1053.

Cue Kelly: “took you long enough”

I’m in the middle of moving apartments and pushing out a large-scale replacement of my employer’s remote-access VPN solution.  Plenty to write about, in no particular order:

-QoS on Cisco Catalyst switch hardware*

-Toying with Juniper SRX firewalls

-Forays into virtualization for the SMB

and most excitingly, finally getting my feet wet with IPv6.

I’ll just leave those there as subtle reminders to write more often.

R3#sho controllers serial 0/0

Interface Serial0/0
Hardware is PowerQUICC MPC860
DTE V.35 TX and RX clocks detected.
idb at 0×81122904, driver data structure at 0x8112A398
…

The truly fun part came in figuring out how to do the routing.  My newly acquired remote sites are all set up as EIGRP stub routers, since I don’t need them transiting traffic for other sites.  Sites that are NOT set up as stub routers (i.e. the default when bringing up EIGRP), when brought into a dual-hub-and-spoke topology such as ours (where all the remote sites have at least two VPN tunnels: one to  each “hub” site), will advertise routes learned from Hub1 back to Hub2 and vice versa.  That makes for a messy and complicated EIGRP topology table, and makes routes from Hub1 appear (from Hub2′s perspective) to be available via Branch1 (and vice versa), meaning if the links between Hub1 and Hub2, Branch1 could be used as a transit site for Hub1-Hub2 traffic.  EIGRP stubs stop this behaviour; they will accept routes from their peers, but EIGRP stub routers by default will only advertise connected and summary routes back to their peers.

This is fine when the site has just one router; it’s more complicated when there are routers downstream from your stub router, as is the case with Branch1.  A router downstream of the EIGRP Stub router carries traffic for the 172.29.18.0/24 LAN in a different building at the facility.  Setting up EIGRP between this router and the Stub router will cause 172.29.18.0/24 to be advertised to the Stub router, but the Stub router will not advertise the 172.29.18.0/24 route to Hub1 or Hub2, making the network unreachable from our hub sites.  You could just point default on the downstream router, and redistribute a static route on the Stub router, but where’s the fun in that…

In my case, the Branch1 office LAN and the adjacent building’s LAN are bitwise-adjacent: 172.29.19.0/24 and 172.29.18.0/24, respectively.  This neatly summarizes to 172.29.18.0/23, so all I had to do was configure the summary route to be advertised on the tunnel interfaces of R2, and point default on R3 –  since stub routers will by default advertise summaries, no more work had to be done.  There are other options to do this, for the curious (mostly covered here: http://www.nil.com/ipcorner/EigrpStub/).

Config on the stub is as follows (for simplicity, the link to only one hub site is shown).  No need to configure a static route to 172.29.18.0/24 on R2, because it will learn the route from its neighbour R3 — but since R2 is a stub, it won’t advertise the 172.29.18.0/24 upstream to R1.  I had a static route in anyway from earlier testing, and I’ll get to that in a second.

interface Tunnel1011105
description “To R1 Hub Site”
bandwidth 3072
ip summary-address eigrp 102 172.29.18.0 255.255.254.0 5
!
i
!
router eigrp 102
eigrp stub connected summary

As mentioned a second ago, for a while I was using a static route on the Stub router, pointing to the downstream router in the other building, and redistributing static into EIGRP.  When I actually brought up EIGRP between the two, got a chance to see something neat.  Picture attached for ease of reference.

Doing a “show ip route” showed the proper next hop as per the configured static route:

R2#sho ip route 172.29.18.0
Routing entry for 172.29.18.0/24
Known via “static”, distance 1, metric 0
Routing Descriptor Blocks:

• 172.29.254.62

Route metric is 0, traffic share count is 1

Admin distance of 1 beats everything but a better-match (i.e. longer prefix) or a directly connected route.

But with EIGRP now running between the two, it also showed up in the EIGRP topology table, but with a max metric (note that the composite metric is actually correct; it’s a function of min bandwidth and total delay along with path, and the configured K values i.e. “eigrp metric …” and we use weird K-values in our environment, so YMMV):

R2#sho ip eigrp topology 172.29.18.0 255.255.255.0
IP-EIGRP (AS 102): Topology entry for 172.29.18.0/24
State is Passive, Query origin flag is 1, 0 Successor(s), FD is 4294967295
Routing Descriptor Blocks:
172.29.254.62 (Serial0/0/1), from 172.29.254.62, Send flag is 0×0
Composite metric is (4230656/38400), Route is Internal
Vector metric:
Minimum bandwidth is 1544 Kbit
Total delay is 20100 microseconds
Reliability is 255/255
Load is 1/255
Minimum MTU is 1500
Hop count is 1

I then removed the static route, and ran the same commands.  EIGRP took over, and nary a packet was lost.  The metric given by FD also correctly matches that given by the “composite metric”:

R2#sho ip route 172.29.18.0
Routing entry for 172.29.18.0/24
Known via “eigrp 102”, distance 90, metric 4230656, type internal
Redistributing via eigrp 102
Last update from 172.29.254.62 on Serial0/0/1, 00:00:51 ago
Routing Descriptor Blocks:

• 172.29.254.62, from 172.29.254.62, 00:00:51 ago, via Serial0/0/1

Route metric is 4230656, traffic share count is 1
Total delay is 20100 microseconds, minimum bandwidth is 1544 Kbit
Reliability 255/255, minimum MTU 1500 bytes
Loading 1/255, Hops 1

R2#sho ip eigrp topology 172.29.18.0 255.255.255.0
IP-EIGRP (AS 102): Topology entry for 172.29.18.0/24
State is Passive, Query origin flag is 1, 1 Successor(s), FD is 4230656
Routing Descriptor Blocks:
172.29.254.62 (Serial0/0/1), from 172.29.254.62, Send flag is 0×0
Composite metric is (4230656/38400), Route is Internal
Vector metric:
Minimum bandwidth is 1544 Kbit
Total delay is 20100 microseconds
Reliability is 255/255
Load is 1/255
Minimum MTU is 1500
Hop count is 1

Of course, the rest of the network doesn’t see any of this complexity at all.  It just sees the summary route.  From R1:

R1#sho ip route 172.29.18.0
Routing entry for 172.29.18.0/23
Known via “eigrp 102”, distance 90, metric 7246080, type internal
Redistributing via eigrp 102
Last update from 10.1.1.106 on Tunnel1011105, 01:29:47 ago
Routing Descriptor Blocks:

• 10.1.1.106, from 10.1.1.106, 01:29:47 ago, via Tunnel1011105

Route metric is 7246080, traffic share count is 1
Total delay is 50100 microseconds, minimum bandwidth is 3072 Kbit
Reliability 255/255, minimum MTU 1440 bytes
Loading 1/255, Hops 1

Also note the change in minimum bandwidth; locally in Branch1,  the minimum bandwidth to 172.29.18.0/24 is that of the T1 interconnecting the two EIGRP peers: 1544 kbps.  When this route gets hidden behind the summary, it inherits the minimum bandwidth as seen on the path between remote sites and the originator of the summary route.  Since the bandwidth of the tunnel between R1 and the summary originator (a tunnel on vpn-01.mtj with configured bandwidth of 3072), that explains the change in min bandwidth.

The delay value comes from the sum of delays along the path: the delay of the lowest-delay component route on the summary-originating router (which is the fa0/1 interface – 172.29.19.0/24 — on vpn-01.mtj, with a delay of 100 usec) + the delay of the next-hop tunnel interface (default 50,000 usec, seen with “sho int tunnel…” command) == 50,100 usec.

So if you ever see something like that max metric in the EIGRP topology table, that’s where it came from: there was already a route with a better administrative distance on the router, but EIGRP keeps its route installed in its topology table “just in case.”  Contrast with RIP, which only keeps a route in its database if it’s the best route; all others will not show up in the output of “show ip rip database.”

http://www.cisco.com/en/US/docs/solutions/Enterprise/Campus/campover.html

Among the highlights: “proper” use of the 3750 series Catalyst switches.  Apart from what may be ever-so-slightly higher backplane capacity that will never be used in a my existing $dayJob environment anyway, there is one architectural difference between the 3750 and the 3650, and one architectural difference only: the ability to do a proper switch stack via the StackWise feature.

If one does not plan to stack the switches, there is no reason to shell out the extra money for a 3750 when the 3650 line will do the job more than adequately.  If the money’s already been spent, however, you can do some great things to simplify and optimize your network.  These include reduced sevice requirements (no need to run STP* or HSRP), simplified configuration and manageability (done only once on the master switch in the stack as opposed to two physical switches) and increased availability and load-balancing.  Of course, this precludes physical separation of the switches by distances greater than the longest available StackWise cable; but if its physical separation one wanted, it’s cheaper to do so with the 3560.

By running etherchannels (which can span multiple switches in the stack) to your up/downstream switches, you can make use of multiple links to your “core” stack instead of having the redundant links blocked by STP.  By definition, loops cannot form, reducing complexity associated with STP (especially in multi-vendor environments, with HP defaulting to MSTP, Cisco defaulting to proprietary PVST+, Dell defaulting to god knows what…), and more easily facilitating the spanning of VLANs across access-layer switches – generally not desired for your user-access, but sometimes preferred for server-access where layer-2 adjacencies are preferred for convenience/clustering’s sake.  This is best illustrated on page 18 of the document, with the next page illustrating the benefit in terms of number of active, traffic-passing links vs. an STP design.

Not running HSRP elimates the risk of split-braining, which occurs with physically separated layer-3 switches when they lose connectivity to one another; both switches go active for the HSRP virtual IP address, but return traffic only goes towards one of the two switches, blackholing traffic destined for one of the switches.

And it’s certainly easy enough to do; assuming your existing separate 3xxx series each have links to the same upstream and downstream devices, just take one ‘of ‘em offline, configure the other with a nice high priority (“switch priority 15”), connect the “secondary” switch via the StackWise cables and power it up.  Recommend administratively disabling all the ports until the appropriate EtherChannels have been set up on all switches.

In summation, a quick general-usage guide for the current models of Catalyst switches, all of which come in PoE and non-PoE, and 12/24/48-port flavours of various speeds (10/100/1000); FYI, their QoS architectures and configuration are identical, differing only in buffer pool size and policer granularity (1Mbps policer intervals on the 2960, 8kbps intervals on the 3560/3750):

2960: Dumb layer-2 wiring closet switch.  Replaces the 2950.  No layer-3, no routing protocols, no VRF stuff, nothing.  Can be managed via IPv4/v6.

3560: Smarter layer-3 wiring-closet/distribution/core switch; successor to the 3550.  Can route IPv4 and v6 over all protocols.

3750: Smartest layer-3 1RU switch.  Exactly the same as the 3560, but with the StackWise feature to enable virtual switching — presenting multiple physical switches as a single switch.

3650-E and 3750-E: Same as their counterparts, but more backplane capacity (StackWise backplane doubles from 32 Gbps to 64 Gbps), dual AC power supplies, and support for 10GigE.

Save for the E-series, all of the above feature a single AC power supply, and a proprietary DC power input.  It can be fed redundant power to this DC input via the RPS 675 (EoS) and RPS 2300.  Each RPS can feed up to six different devices; however, the RPS 675 can only actively back up one device.  The 2300 can provide redundant power for up to two.  If you load up 6 switches on a 2300 and all of their internal supplies fail, four of the switches are out of luck.

As far as dual AC supplies go (again, save for the E-series), one has to take a large step up to the 4500 and 6500-series chassis.  Each features the potential for in-box power and supervisor redundancy.  4500 is relatively (of the two) low-cost choice for port-density and minimal features; 6500 fully loaded can hold most if not all of the internet routing table, all manner of line-cards for all manner of port types, and a variety of service modules (load-balancing and SSL-terminating ACE modules, firewall modules, network analysis modules, etc), and a ton of features.  Not likely to see deployment in $dayJob any time soon, but the datacenter aggregation switch of choice in large environments.

*Yah, yah, you should still “run” STP so as to prevent accidental loops from forming, I know, I know: http://www.cisco.com/web/strategy/docs/gov/turniton_stpt.pdf

As far as personal, home-use computing goes, I find the talk that goes on about the relative insecurity of my Windows Vista and XP boxes to be little more than histrionics.  If someone wants the pictures from my digital camera, it’s easier just to add themselves to the Ottawa, ON network on Facebook, and they can probably get a look at them that way.  There are lots of easier ways to steal information about a person than breaking into their PC, when we so readily make information about ourselves available publicly.

(And even once they’ve done that, what can happen, exactly?  Steal my banking information?  That’s not exactly “risky.” A phone-call to my bank, and the problem goes away.  At worst, I show up to a branch in person to verify my identity with all manner of official documentation.  Inconvenient, but hardly life-endangering.)

Want to get someone’s phone number?  Look them up on Facebook.  Check out their friends list.  Randomly send messages to people on the list, say that you’re an old friend trying to contact them for a high-school reunion, but they aren’t responding to your notes and you’re not sure if they check Facebook that often.  Use your imagination.

From there, you can start to draw the conclusion that security isn’t just technical;  it’s social.  It’s about who and what you trust.

But since not everyone can root through my trash for banking statements and hydro bills, remote compromise of my machine may be their only convenient option.  The goal isn’t complete and total security; as the saying goes, the only way to completely secure your PC is to turn it off.  The goal is to make accessing your PC as inconvenient as possible.  For many users, sticking their PC behind a broadband router provides a cheap form of firewalling that’s more than enough to protect them from outside threats.  Personally, I just turn on Windows Firewall and connect directly to my cable modem.  Never was one for trying to hide my PC behind a router.

Next up, the absolute minimum required to easily get around the internets is a browser.  I’m primarily a Firefox man; Google Chrome is incredibly fast and lightweight, but seems a bit lacking in the feature department.  I’ll confess to not giving it a true college try.

But here’s what I use:

Firefox: http://www.mozilla.com/en-US/firefox/personal.html

With the following addons:

AdBlock Plus: http://adblockplus.org/en/

NoScript:  http://noscript.net/getit

Install the add-ons, configure ABP to subscribe to the easylist filter, and that’s it.  NoScript is a bit annoying to work with at first, but you’ll soon get a feel for it.  Under Firefox’s menu, go Tools -> Options -> Advanced -> Update, and make sure everything is checked off to automatically check for and install updates to the browser and add-ons.

After that, configure Windows Update to check for updates frequently, and to download and install frequently.

I don’t run real-time signature-based AV e.g. Norton Anti-Virus, at all.  Vista takes up 800MB of RAM on its own; I don’t need a few million signatures adding another 250MB.  Every month, Microsoft releases an updated Malicious Software Removal Tool (MSRT website: http://www.microsoft.com/security/malwareremove/default.aspx) which runs in the background and is very quiet (as in, you don’t even know it’s running unless it finds something).  You can force it to run manually so that you can actually watch what it’s doing by going Start -> Run -> mrt.exe.

As a backup to the MSRT, I’ll occassionally run MalwareBytes’ Anti-Malware tool: http://www.malwarebytes.org/mbam.php

I run Threatfire (http://www.threatfire.com/) on its most sensitive settings.  It’s another one of those things that appears obnoxious at first, but you get use to it.

For generic cleanup, I run ccleaner (http://www.ccleaner.com/).  I check off just about everything possible, with the exception of “Wipe Free Space,” because that takes FOREVER.  I clear out all histories and saved-form information.  If you’re the kind of person who checks off “Have browser remember passwords,” you may find using this annoying.  But that’s the kind of person who makes themselves most vulnerable when sharing their PC/laptop with someone else.  If you’re incredibly serious about wanting to secure your information, clear out that crap and start getting better at memorizing complicated passwords.  You can configure the program to add itself to your Recycle Bin, so you can just right-click the bin and open it up.  Close all your browsers (so it can access and delete browser caches) and run it every few days.  Clean out everything, and do the “Clean Registry” step, too.

As a bonus, the program also includes a feature to uninstall programs and disable programs that start on bootup.  Check to see if there are programs listed that you don’t recognize, or those which you know don’t need to actually start on boot.  Disable them until needed.

Complicated passwords are another obvious thing;  the more valuable the information, the harder it should be to access.  If you protect it with a username and password of admin/admin, it obviously isn’t that valuable to you.

If you need to fill out a registration form or something, make up a gmail account that you’ll never actually use (I sign up to things using qos.recyclebin@gmail.com, for example, and if you need a gmail invite, let me know), and fill in all the registration forms with fake information unless absolutely necessary.

That way, you hide as many things as you can from being harvested, and you have a convenient place to find your account info when you use it to sign up for something.  If you’re trying to get vendor whitepapers from a place like http://techrepublic.com.com/, for example, do they REALLY need your work email and phone number?  Hell no.  Make some up.  The only reason you need a “real” fake email address that you never check is so that you can check it occassionally in the event that they require you to verify your email address.  You can publish this email address ANYWHERE and never worry because you know for a fact that there’s nothing useful in it anyway.

If you approach every form on the internet with the attitude that it might some day be used against you, you protect yourself against all manner of information harvesting.

Last and most importantly, I don’t install anything I don’t need, and I don’t open weird shit that I’m not expecting.

The more you add to anything, the more you have to protect/trust.  The larger your friends list, the higher probability of someone telling someone else something you don’t want them to know.  Do you really need 4-5 different cellphone related applications on your PC?  No, get them off there.  Limit your exposure to applications to only those which you can easily update, and update often.

And don’t bother giving someone’s chain-letter email due courtesy.  It doesn’t deserve it.  Especially if there’s some sort of weird attachment or link.  Most web-based email clients can disable the display of images or running of scripts, and have good anti-spam and malware practices established.   Don’t turn on the display of images or run weird attachments.  Chances are, you didn’t request it, ergo you’re unlikely to need it.

As mentioned at the beginning, I am hardly a security professional, and am quite amenable to comments and adjustments.  Happy internetting!

Will resume over the coming days with more than one could ever care to learn about QoS architecture of the Catalyst 2960/2970/3560/3750 series switches!

NetFlow is supported on any and all recent IOS routers (read as: 1800/2800/3800 ISR series, 7200/7600, etc).  Alas, no support on Catalyst dumb Layer-2 and multilayer switches outside of the 4500/6500 line, and even then it requires special hardware in the form of proper line cards/Supervisor Engine(s).

However, you can do a poor-man’s NetFlow by building a “probe” that accepts mirrored traffic from a SPAN port on a switch, and crafts its own NetFlow data from the observed traffic (see also: nTop).  Your mileage may vary depending on your IOS version; this note’s test router uses 12.4(15)T7.

You don’t need a collector to get some use out of the feature, though; it maintains a local cache and that’s what this note’s going to be about.  Quite easy to turn on:

interface FastEthernet0/1
ip address x.x.x.x y.y.y.y
ip flow ingress

Verification:

TEST-VPN-Hub-01#sho ip flow interface
FastEthernet0/1
ip flow ingress

Then turn on the top-talkers feature:

TEST-VPN-Hub-01#conf t
TEST-VPN-Hub-01(config)#ip flow-top-talkers
TEST-VPN-Hub-01(config-flow-top-talkers)#top 100

Then we get the option of viewing un-aggregated cache data, or aggregated cache data:

TEST-VPN-Hub-01#sho ip flow top-talkers ?

Display aggregated top talkers:
<1-100>  Number of aggregated top talkers to show

Display unaggregated top flows:
verbose  Display extra information about unaggregated top flows
|        Output modifiers

Un-aggregated provides a very granular view of flows stored in cache; one flow per source/dest IP/port and IP Protocol number (with protocol number and src/dst ports reported in very obnoxious hex), and by default sorted by bytes ingress to the interface:

TEST-VPN-Hub-01#sho ip flow top-talkers

SrcIf         SrcIPaddress    DstIf         DstIPaddress    Pr SrcP DstP Bytes
Tu110232      10.0.30.63      Fa0/1         192.168.141.81  06 170C 88ED  2074K
Tu110232      10.0.30.40      Fa0/1         192.168.141.71  06 0FD9 F727  1519K
Tu110232      10.0.30.140     Fa0/1         192.168.141.144 06 0BFE DB22  1275K
Tu110232      10.0.30.140     Fa0/1         10.1.250.81     06 0BFE B70E  1243K
Tu110232      10.0.30.140     Fa0/1         10.1.250.81     06 0BFE B70F  1242K
Tu110232      10.0.30.62      Fa0/1         192.168.141.80  06 170C 83ED   532K
Tu110232      10.0.30.140     Fa0/1         10.1.250.81     06 0BFE A204   340K
Tu110232      10.0.30.140     Fa0/1         192.168.141.144 06 0BFE D3D8   251K
Fa0/1         192.168.141.81  Tu110232      10.0.30.63      06 88ED 170C    69K
Fa0/1         192.168.141.80  Tu110232      10.0.30.62      06 83ED 170C    60K
Fa0/1         192.168.141.144 Tu110232      10.0.30.140     06 D3D8 0BFE    38K

Useful if you have a source that’s just pounding away; you can easily see where it’s coming from (and the interface through which it enters) and where it’s going (and the interface through which it leaves).

Aggregated view allows you to aggregate the NetFlow data a whole bunch of different ways (I’ve cut a bunch of ways out for sake of brevity):

TEST-VPN-Hub-01#sho ip flow top-talkers 100 aggregate ?
bytes                  number of bytes
destination-address    Destination address
destination-interface  Destination interface
destination-port       Destination port
icmp                   ICMP type and code
ip-nexthop-address     IP nexthop address
max-packet-length      Maximum packet length
min-packet-length      Minimum packet length
packets                number of packets
source-address         Source address
source-interface       Source interface
source-port            Source port
tcp-flags              TCP flags

What follows are ways to find the hot destination ports from your router’s point of view:

TEST-VPN-Hub-01#sho ip flow top-talkers 100 aggregate destination-port sorted-by packets

There are 20 top talkers:

TRNS DST PORT       bytes        pkts       flows
=============  ==========  ==========  ==========
35053     1638362        8922           1
54232     1462512        4017           1
33773      861529        3757           1
63271     1161960        2904           1
56098      950000        2609           1
46862      916876        2518           1
46863      916472        2516           1
5900      110858        2226           2
0      688278        1030          13
2048      658800         549           1
3070       12480         312           5
4056        3492          70           1
4057        2680          67           1
57556        6804          67           1
41476       15288          42           1
3092        2860          35           3
161        2556          35           3

Note that “Port 0” shows up in the above; I believe this may be related to packet fragmentation.  Non-initial fragments will not contain a transport-layer header; rather, they’ll simply have more transport-layer payload.  NetFlow can relate such a packet to a particular transport-layer protocol on account of the IP Protocol field of the IP packet (6 = UDP, 17 = TCP), but that’s as good as it can do without reassembling the entire packet.

Mind you, the traffic could also be IPSEC, which uses IP Protocol 50 or 51 for AH or ESP, respectively, and does not have port numbers for NetFlow to count.  This test bed was also running EIGRP and GRE tunnels; this traffic may have also been counted as “Port 0” traffic.

And to see some equally hot source hosts:

TEST-VPN-Hub-01#sho ip flow top-talkers 100 aggregate source-add sorted-by packets

There are 25 top talkers:

IPV4 SRC ADDR         bytes        pkts       flows
===============  ==========  ==========  ==========
10.0.30.63          1758749        9609           1
10.0.30.140         3161180        8681           5
10.0.30.62           996875        4319           1
10.0.30.40          1266040        3226           5
192.168.141.80       121738        2444           1
10.1.250.81           35960         899           3
192.168.139.66       990000         825           1
192.168.139.129      988800         824           1
192.168.141.144       24640         616           2
192.168.141.81        22520         451           2
192.168.141.71        12372         309           2
192.168.191.234       19008         288           1
192.168.191.242        9900         150           1
192.168.141.70         3944          81           2
192.168.141.66         3360          56           1
192.168.141.65         3300          55           1
192.168.191.238        2508          38           1
192.168.141.70         1680          28           1
192.168.141.76         1680          28           1
192.168.141.75         1680          28           1
192.168.141.71         1620          27           1
192.168.141.72         1620          27           1
192.168.191.230        1650          25           1
10.1.40.169              72           1           1

The command “show ip cache flow” also produces interesting results, including timers associated with the flow cache.

TEST-VPN-Hub-01#sho ip cache flow

IP packet size distribution (26090 total packets):
1-32   64   96  128  160  192  224  256  288  320  352  384  416  448  480
.001 .500 .155 .007 .005 .005 .006 .007 .007 .007 .006 .204 .045 .004 .004

512  544  576 1024 1536 2048 2560 3072 3584 4096 4608
.003 .002 .002 .012 .008 .000 .000 .000 .000 .000 .000

IP Flow Switching Cache, 278544 bytes
60 active, 4036 inactive, 675 added
29520 ager polls, 0 flow alloc failures
Active flows timeout in 30 minutes
Inactive flows timeout in 15 seconds
IP Sub Flow Cache, 25800 bytes
0 active, 1024 inactive, 0 added, 0 added to flow
0 alloc failures, 0 force free
1 chunk, 0 chunks added
last clearing of statistics 00:07:23

Protocol         Total    Flows   Packets Bytes  Packets Active(Sec) Idle(Sec)
——–         Flows     /Sec     /Flow  /Pkt     /Sec     /Flow     /Flow
TCP-other          338      0.7        29   155     22.6      18.6       9.6
UDP-NTP             37      0.0         1    76      0.0       0.0      15.4
UDP-other          184      0.4        13    75      5.8       6.2      15.5
ICMP               100      0.2         2   757      0.5       0.5      15.6
Total:             659      1.4        19   151     29.1      11.4      12.5

From the above output, you can see that flows will age out of the cache 15 seconds after data associated with the flow stops flowing.  You can test this by pinging something through the router (in my tests, locally-originated ICMP traffic was not counted by NetFlow, but there’s a chance I may have just been doing it wrong), and filtering the output of “show ip flow top-talkers” or “show ip cache flow”, until there’s been enough transferred data associated with the flow for it to work its way into the cache.

Then stop the ping.  15 seconds later, the flow won’t be there anymore; so by definition, flows that have accumulated a lot of traffic have been active for a very, very long time.  This technique is incredibly handy for tracking DoS activity; if you’re able to log into a terminal, you can work backwards to find the source address and input interface of potential DoS’ers, misbehaving hosts, etc.  Taken to its logical conclusion – assuming cooperation with a supportive and clueful ISP — you can even trace a spoofed IP address back to its real source. How this would be accomplished is left as an exercise for the reader.

There’s also a packet-size histogram; from the above, you can deduce that 50% of the packets transiting the router are between 32-64 bytes; 15.5% are between 64-96 bytes; and 20% are between 352-384 bytes.

Over at $dayJob, I use http://www.plixer.com/products/free-netflow.php to keep track of a day’s worth of NetFlow data; for a free tool, it’s incredible for providing point-in-time analysis of application use on my network.  As they say,  in network analysis, there is no substitute for knowing your network.  While longer-term analysis would be ideal, I don’t have long-term enterprise NetFlow collection in my budget, nor the time to build out my own; though after you’ve kept a watchful eye on links for a few weeks, you start to see patterns, and deviations from that pattern should be either easily explained or quickly investigated.

line vty 0 15
transport input ssh

I’m going to digress already and say that it’s a good idea to restrict access to certain networks:

line vty 0 15
access-class 101 in
transport input ssh

And that some go a step further and protect the last VTY line as a last resort in the event that the other 14 or so are occupied by someone with less-than-benevolent purposes; that way, the host(s) specified in ACL 102 can still manage the router:

line vty 0 14
access-class 101 in
transport input ssh

line vty 15
access-class 102 in
transport input ssh

But back to the point.  In the early days of IOS 12.3, they introduced the “login” command-set for login security enhancement (http://www.cisco.com/en/US/docs/ios/12_3t/12_3t4/feature/guide/gt_login.html).

login block-for 60 attempts 3 within 60
login delay 3
login on-failure log
login on-success log

This gives you three chances to pass the test before the router blocks all logins for 60 seconds (this period is called “quiet mode”).  There’s also a 3-second delay between attempts.  This mitigates someone throwing the kitchen sink at your device; it takes them 9 seconds just to try three times, and they can only do so once a minute without changing IP addresses.  A “quiet mode” list can be configured to allow certain hosts to get around these restrictions; this is a good idea, because someone spamming login attempts can lock you out, and it’s a race to log in when quiet time ends.  Luckily, the “on-failure log” will tell you which IP address is responsible for the attack. Info on configuring quiet-mode bypass is in the documentation linked at the end of this note.

Of course, the problem with this is that by default, IOS will let you attempt four SSH logins before terminating the session.  You can fix that, too. I use this:

ip ssh authentication-retries 2
ip ssh logging events
ip ssh version 2

“authentication-retries” is, literally, retries.  It lets you make two additional attempts after the first failed attempt; hence three in total, which matches up with the three attempts before you’re locked out for a minute, configured above in the “login” section. “Version 2” forces the use of SSHv2 by the client side; SSHv1/v1.5 considered insecure and deprecated for well over a decade.

Finally, the built-in config-change archiver/logger:

archive
log config
logging enable
logging size 200
notify syslog
hidekeys

This will take any change made in config mode, save a small local copy of said changes to a local buffer, and spit them out to syslog. “hidekeys” keeps sensitive info obscured (syslog packets being unencrypted and all).  How many times have you asked yourself “well, what’s changed?”  This lets you know in real-time.

All this and more over at the IOS Security Configuration Guide and Command Reference, which can be found here for IOS 12.4: http://www.cisco.com/en/US/docs/ios/security/configuration/guide/12_4/sec_12_4_book.html

Whole bunch of examples below the cut!

=======================================================================

Ex 1: failure logged by “ip ssh logging events”

SSH-5-SSH2_USERAUTH    Notice    11443: 19963486: Jun  9 2009 15:05:31.145 UTC: %SSH-5-SSH2_USERAUTH: User ‘rawn’ authentication for SSH2 Session from x.x.x.x (tty = 1) using crypto cipher ‘aes256-cbc’, hmac ‘hmac-sha1′ Failed

=======================================================================

Ex 2: success logged by “ip ssh logging events”

SEC_LOGIN-5-LOGIN_SUCCESS    Notice    9494: 252827: Jun  9 16:01:32.087: %SEC_LOGIN-5-LOGIN_SUCCESS: Login Success [user: rawn] [Source: 172.16.6.166] [localport: 22] at 16:01:32 UTC Tue Jun 9 2009

=======================================================================

Ex 3: Verify “login” config

vpn-01.xxx#sho login
A login delay of 3 seconds is applied.
No Quiet-Mode access list has been configured.
All successful login is logged.
All failed login is logged.

Router enabled to watch for login Attacks.
If more than 3 login failures occur in 60 seconds or less,
logins will be disabled for 60 seconds.

Router presently in Normal-Mode.
Current Watch Window
Time remaining: 2 seconds.
Login failures for current window: 0.
Total login failures: 15.

=======================================================================

Ex 4: Monitor failed logins (held in a local buffer, cleared after a reboot):

vpn-01.xxx#sho login failures
Total failed logins: 9
Detailed information about last 50 failures

Username        SourceIPAddr    lPort Count TimeStamp
fwwed           x.x.x.x    22    3     09:53:48 EDT Tue Jun 9 2009
fw3552          x.x.x.x    22    3     10:32:23 EDT Tue Jun 9 2009
rawn            x.x.x.x    22    3     16:21:16 UTC Tue Jun 9 2009

=======================================================================

Ex 5: syslog produced by failure and subsequent blocking of all login attempts configured with the “login” command set (reverse chronological order, ending with the router “unlocking” itself and permitting logins again)

6/9/2009 12:22:18 PM        x.x.x.x    SEC_LOGIN-5-QUIET_MODE_OFF    Notice    6618: 20268049: Jun  9 2009 16:22:16.775: %SEC_LOGIN-5-QUIET_MODE_OFF: Quiet Mode is OFF, because block period timed out at 16:22:16 UTC Tue Jun 9 2009

6/9/2009 12:21:17 PM        x.x.x.x    SEC_LOGIN-1-QUIET_MODE_ON    Alert    6615: 20267993: Jun  9 2009 16:21:16.775: %SEC_LOGIN-1-QUIET_MODE_ON: Still timeleft for watching failures is 6 secs, [user: rawn] [Source: x.x.x.x] [localport: 22] [Reason: Login Authentication Failed] [ACL: sl_def_acl] at 16:21:16 UTC Tue Jun 9 2009

6/9/2009 12:21:17 PM        x.x.x.x    SEC_LOGIN-4-LOGIN_FAILED    Warning    6614: 20267992: Jun  9 2009 16:21:16.775: %SEC_LOGIN-4-LOGIN_FAILED: Login failed [user: rawn] [Source: x.x.x.x] [localport: 22] [Reason: Login Authentication Failed] at 16:21:16 UTC Tue Jun 9 2009

6/9/2009 12:21:14 PM        x.x.x.x    SEC_LOGIN-4-LOGIN_FAILED    Warning    6613: 20267988: Jun  9 2009 16:21:13.331: %SEC_LOGIN-4-LOGIN_FAILED: Login failed [user: rawn] [Source: x.x.x.x] [localport: 22] [Reason: Login Authentication Failed] at 16:21:13 UTC Tue Jun 9 2009

6/9/2009 12:21:11 PM        x.x.x.x    SEC_LOGIN-4-LOGIN_FAILED    Warning    6612: 20267984: Jun  9 2009 16:21:09.918: %SEC_LOGIN-4-LOGIN_FAILED: Login failed [user: rawn] [Source: x.x.x.x] [localport: 22] [Reason: Login Authentication Failed] at 16:21:09 UTC Tue Jun 9 2009

=======================================================================

Ex 6: syslog of config change produced by “Archive” command set:

6/9/2009 11:55:09 AM        x.x.x.x    PARSER-5-CFGLOG_LOGGEDCMD    Notice    605: 359657: Jun  9 2009 15:55:08.167: %PARSER-5-CFGLOG_LOGGEDCMD: User:lsdb  logged command:service timestamps log datetime msec year

6/9/2009 11:55:07 AM        x.x.x.x    PARSER-5-CFGLOG_LOGGEDCMD    Notice    604: 359655: %PARSER-5-CFGLOG_LOGGEDCMD: User:lsdb  logged command:service timestamps debug datetime msec year

6/9/2009 11:55:06 AM        x.x.x.x    PARSER-5-CFGLOG_LOGGEDCMD    Notice    603: 359654: %PARSER-5-CFGLOG_LOGGEDCMD: User:lsdb  logged command:no service timestamps debug

6/9/2009 11:55:04 AM        x.x.x.x    PARSER-5-CFGLOG_LOGGEDCMD    Notice    602: 359653: %PARSER-5-CFGLOG_LOGGEDCMD: User:lsdb  logged command:no service timestamps log

=======================================================================

Ex 7: locally buffered config changes produced by “archive” commands:

vpn-01.xxx#sho archive log config all | inc service
295    29   rawn@vty0     |service nagle
296    29   rawn@vty0     |service tcp-keepalives-in
297    29   rawn@vty0     |service tcp-keepalives-out
298    29   rawn@vty0     |service timestamps debug datetime msec show-timezone year
299    29   rawn@vty0     |service timestamps log datetime msec show-timezone year
300    29   rawn@vty0     |service password-encryption
301    29   rawn@vty0     |service sequence-numbers
302    30   rawn@vty0     |no service timestamps debug datetime msec show-timezone year
303    30   rawn@vty0     |no service timestamps log datetime msec show-timezone year
304    30   rawn@vty0     |service timestamps debug datetime msec year
305    30   rawn@vty0     |service timestamps log datetime msec year

=======================================================================

Appendix: All commands in convenient copy/paste format.

login block-for 60 attempts 3 within 60
login delay 3
login on-failure log
login on-success log

ip ssh authentication-retries 2
ip ssh logging events
ip ssh version 2

archive
log config
logging enable
logging size 200
notify syslog
hidekeys

=======================================================================