IPv6

Y’know, I really wanted to throw myself into IPv6 this year; then I went and got distracted by a large-scale VMware deployment (hence the lack of posting over the last…three months).  We’re a three-person shop at $dayJob these days, supporting 700+ users across 20+ different countries, and that’s regrettably meant that the things that aren’t required RIGHT NOW get pushed off to the side.

Now that I’m finally finishing that up and comfortable enough with my giant NetApp storage array that I can go without looking at it for a few days, I’m starting to look back into IPv6 again.

I’ve some familiarity with the way the header looks and some basic deployment scenarios — but mostly just those acquired from my CCNP studies of old. Having gone through months of NANOG archives and found disagreement all over the ISP community with respect to the best way(s) to deploy IPv6, I’m even more intimidated.

(That said, I’ve done a paint-by-numbers deployment of IPv6 over MPLS VPN with some Cisco 3800-series routers we snagged from a decommissioned branch to bring some of my BGP/MPLS studies together; that was a ton of fun)

I’ve been prepping for it for a while, though, in terms of all my new hardware acquisitions. Anyone pushing something that wasn’t v6-aware right NOW has been shown the door since 2007, so I’m just about ready to go dual-stack across the enterprise (though few if any of my ISP’s are ready to support this deployment). Going to be one of those things where I’ll just have to take the documentation and start pushing it out and breaking it to see what works and what doesn’t.

But the most frightening thing of all is the sheer size of the address space. Jesus Christ, it’s big. Like, really big. Big enough that I completely forgot how subnetting worked in the first place. 32-bit dotted-decimal was easy to wrap one’s head around; hard to find anyone who’s been doing this for a while who doesn’t have a few hundred critical infrastructure/server addresses committed to memory — safe to say those days are gone.

Think of all the pages wasted on teaching those new to networking how to properly subnet in order to efficiently provision what was once a scarce resource, and how those practices are still being taught without a really big caveat: “Oh by the way, you don’t really have to know this anymore; the value of these pages is going to plummet in the next five years, and here’s why…”

For a lot of people, it’s going to be the first large technical revolution they’ve had to face.  IP hasn’t changed in over three decades; new features were merely layered on top of a fully functional protocol on demand.  But now everything that uses that fundamental protocol has to change; the magnitude of this project is enormous and IT departments who haven’t yet begun planning are years behind the curve (and this is a lot of IT departments, by my anecdotal measure).

I look around at the people who’ve been doing this stuff for years; they’d probably hoped to not have to face this before retirement, but that’s not going to be the case. How does one best go about convincing them that not only is a an IPv6 /64 a completely valid way to address a point-to-point link^[1], but a way that’s encouraged over the old practice of allocating an IPv4 /30 (or in the case of IPv6, a /127)?

There’s going to be a lot of money to be had in the IPv6-migration consulting business.

[1]: http://tools.ietf.org/html/draft-palet-v6ops-point2point-01

I’m going to get back to this, I swear; just took some time off to start working on the CCIP. 

Pleased to report that I passed the BGP exam yesterday, having recently completed the Implementing BGP on Cisco Routers course as delivered by Elan Beer — who, at #1837, is one of the first 1000 CCIE’s. 

For a full week, we had the opportunity to pick the brain of someone who has acted as a technical reviewer for Cisco Press products; impossible to come out of that and not know a little something about the stuff. : )

Anyway, happy new year, etcetcetc.  Lots of nerdly goodness to come, honest!

Non-routing router: bug or feature?

Trying to connect to some routers in Rio, couldn’t do it.  Had to access from a local device, instead.  No problem, just needs a default gateway, right? 

Log in, add default gateway.  Try again.  No luck.  ACL?  No.  “show ip route” completely blank.  What the.  Bug?  Reload.  Still nothing. 

“show run”?

“no ip routing”

 /groan

Quis custodiet ipsos custodes?

I’m playing around with a lot of bleeding edge IOS releases lately, with my newest trick based on Embedded Event Manager documentation from Cisco’s mgmt configuration guide: http://www.cisco.com/en/US/docs/ios/netmgmt/configuration/guide/nm_erm_resource_ps6441_TSD_Products_Configuration_Guide_Chapter.html

 

Ever want to immediately pinpoint a transient CPU spike?  Consistently high utilization is one thing, but quick spikes are less obvious and don’t tend to show up on graphs averaged over 5-minute intervalic measurements.  The following is employed on an 1841, using IOS 12.4(15)T7:

 

resource policy

  policy HighCPU type iosprocess

   system

    cpu process

     critical rising 40 falling 25

     major rising 20 falling 10

    !

   !

  !

  user group Hogger type iosprocess

   instance “IP Input”

   policy HighCPU

  !

snmp-server enable traps resource-policy

 

The (utilization percentage-based) numbers are arbitrary, but I’ll use those in production based on the fact that our resources are heavily oversized for the kind of work they have to do.  To test it out, I hammered the router’s control plane with 100 Mbps worth of ICMP to the router’s interface.  Here’s what immediately happened:

 

000247: *Sep  2 2008 16:38:28.991 UTC: %SYS-4-CPURESRISING: Resource group Hogger is seeing local cpu util 25% at process level more than the configured major limit 20 %

 

Then immediately after I stopped:

 

000278: *Sep  2 2008 16:39:33.971 UTC: %SYS-6-CPURESFALLING: Resource group Hogger is no longer seeing local high cpu at process level for the configured major limit 10%, current value 0%

 

EEM is phenomenal; it basically lets your router monitor itself.  The above will generate a warning and informational-level syslog message for alert triggering and reset, respectively.

 

Next on the list is Control-Plane Policing, or how to mitigate the effect of someone trying to blast your router’s interfaces just like I did for the purposes of this test (which isn’t to say that a sufficiently motivated user couldn’t simply point the firehose at your link to fill it full of DoS, but that’s a seperate issue best dealt with controls elsewhere).  =P

 

Have you s…god dammit

<Rawn> I hate when you know for a fact that every person you work with has seen Office Space
<Rawn> and you can’t find your fucking stapler.
<Knehi> HAHAHA
<Rawn> Fuck you.
<Rawn> I have to go on a scavenger hunt for my stapler.  I can’t let anyone know it’s missing.
<Xir> stapler?
<Knehi> just walk around threatening to burn down the building, it will show up
<Rawn> <Xir> stapler?
<Rawn> Have you seen my…god dammit, I’d never hear the fucking end of it.
<Rawn> fuck it, i’ll use a hole punch and an elastic.
<Trolan> just swipe someone else’s
<Thayne> Just send out an email  Subject:  Stapler  Body:  Have you seen mine?
<Ayrahvon> You’ll never hear the end of it on IRC either, bad move. =P
<Rawn> I had to tell SOMEONE
<Rawn> I didn’t need this following me around irl
<Knehi> http://tinyurl.com/6j7e5h
* Trolan makes a note: Ask Ron about his stapler at Blizzcon.
[S+Z] Alicia (ottertothe@c-24-10-235-250.hsd1.ut.comcast.net) has joined channel #OnTheBounty
<Thayne> Hay Ali…Have you seen Ron’s stapler?
<Alicia> Is it red?
<Alicia> Because that would be awesome.
<Thayne> It is in my mind….
<Rawn> -_-
<Cal> I take Ron’s glare to mean that it is, in fact, red.
<Knehi> I take Ron’s glare to mean he still has a love of asian boys, but I take everything Ron does to mean that
<Trolan> so when he says red stapler, he means he wants something reddish, to bend over like a stapler is bent, and get banged hard.
<Trolan> got it.
<Alicia> He is going to poison all of your drinks.
<Rawn> This says a lot more about you people than it does about me.
<Rawn> And I would never waste a drink like that.
<Rawn> I’d just steal theirs.
<Alicia> haha
<Thayne> Shouldn’t you be looking for your stapler?
<Rawn> shouldn’t you be dying in a fire?
<Knehi> http://www.landoverbaptist.org/2008/august/olympicvolleyball.html
<Thayne> YOur vitriol will not help you find your stapler any quicker Sir.

These days, I just hard-code for job security.

Ultimately, it’s a lot easier to keep one’s job if you can convince your manager that it takes highly skilled, motivated, and incredibly rare individuals to log into a switch and change a port default or two.  I suspect this is also the reason why I read about cable runs taking nine-and-a-half weeks.  The whole stack of cards comes tumbling down the minute one of us admits that it should really only take 2-3 minutes, given the proper cable length and a path with minimal obstruction between distribution frames.

But I digress: http://etherealmind.com/2008/07/15/ethernet-autonegotiation-works-why-how-standard-should-be-set/

Some interesting notes there. I’ve actually been a fan of set-and-forget over the years, but never could explain why. It just seemed to fix a lot of errors, and ultimately that was what was important. Having had the chance to work with quite a few makes and models over the last two years or so, I’m getting more comfortable with the idea of auto-negotiation.

After all, auto-neg IS a standard, and the Gigabit Ethernet specification is pretty clear on the matter of its importance. If it doesn’t work between two devices, it’s ultimately not a design problem (as I preferred to think of it), but rather a driver incompatibility problem (one of your vendors is making garbage NIC’s). These days, my thinking is more along the lines of “if you have to disable a standard to make it work, you’ve got bigger problems than a few collisions.”

You can indeed force Gigabit Ethernet on a Cisco switch to “speed 1000,” but you will be unable to set the duplex; it will default to full and stay there. You’d think this was a good thing until you see the GigE spec excerpted in the article, where it states that GigE uses auto-neg to detect which end of the link will provide clocking. : )

Ex:

lab-sw-3(config)#int gi0/24

lab-sw-3(config-if)#speed 1000

lab-sw-3(config-if)#duplex full ! accepts this command without complaint; it defaulted to “full” in the first place

lab-sw-3(config-if)#duplex half

Gigabit port is restricted to full duplex ! gets upset if you try to set it to “half”

lab-sw-3(config-if)#duplex auto

Gigabit port is restricted to full duplex ! equally upset if you try to set it to “auto”

lab-sw-3(config-if)#

Little food for thought!

Getting to the root of the problem.

From the “Learn something new every day” department…this one could actually be relevent to my studies.*  It was certainly relevant to my job.

In building a three-switch lab to simulate one site’s access-layer and test out some link bundling, I was seeing that my access switch’s uplink had a cost to the root of 16.  This did not make sense.  The path went through a Gigabit uplink to the secondary root, which is connected to the primary root by way of a two-link EtherChannel (two 100Mbps links).  I was expecting a cost related to the sum of costs for one 1Gb and one 100Mb link (4 + 19 = 23).

In looking up the path costs to try and work backwards, I found the cost for 200 Mb is 12.  This combined with the cost of the Gig uplink would give me my 16 value, but where was the 200 Mb coming from?

The answer was that the EtherChannel’s virtual interface gets its own bandwidth, equal to the product of the individual link bandwidth and the number of links in the channel.  Doing a “show interface port-channel” would display a bandwidth of 200 Mbps, and STP uses this interface’s characteristics to derive its path costing:

Port-channel21 is up, line protocol is up (connected)
Hardware is EtherChannel, address is 0019.5669.5382 (bia 0019.5669.5382)
Description: *** HQD_CORE_01 ***
MTU 1500 bytes, BW 200000 Kbit, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 2/255
Encapsulation ARPA, loopback not set
Full-duplex, 1000Mb/s, link type is auto, media type is unknown

This led me to wonder if this could be changed with the interface-level “bandwidth” command.  So for kicks, I changed the bandwidth of a random Gig link to 100 Mbps.  Here’s the before and after:

SW-1>sho span int gi2/0/9

Vlan             Role Sts Cost Prio.Nbr Type
—————- —- — ——— ——– ——————————–
VLAN0102         Desg FWD 4 128.61   P2p

Then the change…

SW-1(config)#int gi2/0/9
SW-1(config-if)#bandwidth 100000

Then the result:

SW-1>sho span int gi2/0/9

Vlan             Role Sts Cost Prio.Nbr Type
—————- —- — ——— ——– ——————————–
VLAN0102         Desg FWD 19 128.61   P2p

The cost to reach the root switch will be the primary determining factor in root-port selection, for any STP topology that involves uplinks to non-root switches. ** This can be influenced by the bandwidth of your links, which I knew already, but what I didn’t know was that the path-cost may inadvertently change with link-bundling or manual interface bandwidth configuration.

*Picture the following scenario: Given a topology consisting of three switches, make Fa0/1 on Switch-C the alternate port…but change nothing on its upstream switches; do not disable STP or employ FlexLinks; and do not manually change the port cost with any “spanning-tree” commands.

**If you’re merely running multiple uplinks from one switch to the root, given interfaces of equal bandwidth, your root bridge ID, path cost, and sender bridge ID will be equal.  The tie is broken based on port ID (2-tuple value consisting of Priority:Interface-Number).  Of course, if you’re doing that, you should really consider EtherChannel; additional path-redundancy and bandwidth, what’s not to love!

Bombs over Grandma

Came across some old and interesting news while researching older and equally interesting news related to the Cogent/Telia de-peering dispute* (resolved well over a year ago; as a quaint addendum, Cogent as of one month ago has become a transit-free AS). From NetworkWorld last year:

If the United States found itself under a major cyberattack aimed at undermining the nation’s critical information infrastructure, the Department of Defense is prepared, based on the authority of the president, to launch a cyber counterattack or an actual bombing of an attack source.

Anyone else get visions of a phalanx of oblivious grandmothers — with zombie-bots simultaneously attempting to exploit Kaminski’s recent DNS vulnerability — suddenly finding themselves on the receiving end of Apache gunfire?

I am so turned on right now.

*The long-term goal here is to be able to solidly and confidently converse in the language of large-scale backbone providers, so that I might not make an ass out of myself immediately upon joining their ranks.

Whole lot of awesome going on here.

Not only the most excellent song from Dr Horrible’s Sing-Along Blog, but quite clearly the best song of this year.

Air Travel is Stupid

Hay guyz, wuts goin’ on hear.