Thursday, October 4, 2012

Regional leadership in IPv6

Network World claims "The US is overtaking the world on IPv6" , but I'm not entirely convinced.

Seems like when the numbers can support the claim such as overall traffic, North America vs. all of Asia numbers are shown.  When it's a bit closer, such as in number of addresses or vendor support the numbers are broken down by country to show the US is slightly ahead of China but not all of Asia.

It's nice to see that North America is no longer ignoring the protocol and the future of the Internet.  Now let's see how we are doing in percentage of bandwidth and percentage of total users!

Thursday, September 20, 2012

SSL and TLS - What do browsers use to encrypt?

I'll spare you all the gory details you can read on wikipedia, but these are protocols used to encrypt data exchanged by browsers and web servers for keeping information private and unchanged.  Secure Sockets Layer (SSL) has been through versions 1, 2, and 3.  All by 1996 !   Transport Layer Security (TLS) was released as version 1.0 in 1999.  (Also a !).

You may often see references to SSL3/TLS1.0 because "TLS 1.0 does include a means by which a TLS implementation can downgrade the connection to SSL 3.0, thus weakening security."

TLS1.1 was defined in 2006.   TLS1.2 was defined in 2008.  As of 2012, IE9 appears to be the only major browser to support it.

In his great BlackHat 2010 presentation, Ivan Ristic presents great statistics on server support for these protocols discovered during surveying:

Monday, August 13, 2012

nping, a ping for more protocols

Nping is really handy when you suspect that ping or traceroute aren't giving you clear results.  I really like it for a TCP Connect to a website:

$ nping --tcp-connect -c 3 --delay 2s

Starting Nping 0.6.01 ( ) at 2012-08-13 12:09 MDT
SENT (0.0067s) Starting TCP Handshake > (
RECV (0.0209s) Handshake with ( completed
SENT (2.0086s) Starting TCP Handshake > (
RECV (2.0250s) Handshake with ( completed
SENT (4.0113s) Starting TCP Handshake > (
RECV (4.0244s) Handshake with ( completed

Max rtt: 16.390ms | Min rtt: 13.108ms | Avg rtt: 14.507ms
TCP connection attempts: 3 | Successful connections: 3 | Failed: 0 (0.00%)
Tx time: 4.00581s | Tx bytes/s: 59.91 | Tx pkts/s: 0.75
Rx time: 4.01892s | Rx bytes/s: 29.86 | Rx pkts/s: 0.75
Nping done: 1 IP address pinged in 4.02 seconds

You can also scan MAC addresses on your local subnet:

 $ sudo nping --arp-type ARP --count 1

SENT (4.8530s) ARP who has Tell
RCVD (4.8546s) ARP reply is at 1C:DF:DF:53:91:91
SENT (5.8538s) ARP who has Tell
RCVD (5.8548s) ARP reply is at 00:26:9F:37:33:33
SENT (6.8556s) ARP who has Tell
RCVD (6.9364s) ARP reply is at A4:EF:57:E3:EA:EA
SENT (7.8575s) ARP who has Tell
SENT (8.8582s) ARP who has Tell
SENT (9.8587s) ARP who has Tell
SENT (10.8600s) ARP who has Tell
SENT (11.8612s) ARP who has Tell
SENT (12.8621s) ARP who has Tell
RCVD (12.8656s) ARP reply is at 00:21:CF:BD:1F:1F
SENT (13.8634s) ARP who has Tell

Raw packets sent: 10 (420B) | Rcvd: 4 (184B) | Lost: 6 (60.00%)
Tx time: 9.01162s | Tx bytes/s: 46.61 | Tx pkts/s: 1.11
Rx time: 10.01221s | Rx bytes/s: 18.38 | Rx pkts/s: 0.40
Nping done: 10 IP addresses pinged in 14.86 seconds

Friday, July 13, 2012

Integrating routing with CDN

A content delivery network (CDN) is a large distributed system of servers deployed in multiple data centers in the Internet.[citation needed] The goal of a CDN is to serve content to end-users with high availability and high performance.

Uniform Resource Identifier (URI) is a string of characters used to identify a name or a resource on the Internet. -

Border Gateway Protocol (BGP) is the protocol which is the core routing decisions on the Internet. It maintains a table of IP networks or 'prefixes' which designate network reach-ability -

The multiprotocol extensions of BGP has been used to carry many types of information.  I stumbled across a great progression on the idea from this NANOG presentation from the guys at Comcast.

Here's their packet capture screenshot.  Notice the Address Family -  URI.

Friday, June 22, 2012

IPv1 , v2, and v3

I always hear "What happened to IP version 5?" But we've seen the answer many times.  "Some experimental testing version".

While reading Charles Kozierok's excellent TCP/IP Guide, I found the history of why IPv4 was actually the first version of IP as in a Layer 3 on it's own.  The addressing layer was part of the original Network Control Program and then Transmission Control Program combined into a single Layer3 and 4 functionality.   TCP became Transmission Control Protocol by its 'fourth iteration'.

Tuesday, June 19, 2012

Why hasn't networking followed Moore's Law ?

Andy Bechtolsheim's keynote at NANOG55 in Vancouver looks into what's been going on with networking versus Moore's Law:

Follow the link to his slides in PDF.  There are some fantastic graphs on:

  1. Server 10/40/100G Adoption Cycle
  2. Total Datacenter Switch Revenue by Protocol & Speed
Today's custom switch silicon support 64 10G ports on a single chip but forecast to scale by 4X in the next 3 years.  He wraps up the presentation looking into ways to alleviate the current high cost of optics slowing down 40G/100G adoption.

Wednesday, June 6, 2012

IPv6 Day

Early today, major web sites enabled their DNS 'quad A' or AAAA entries.

These include,,,, and

I especially like Facebook's IPv6 address:


It looks like traffic to Google over one backbone immediately doubled... Albeit up to almost 2% of total traffic.

I'm told that unlike last year, these updates are planned to be permanent. Sure is a step in the right direction!

Thursday, May 31, 2012

A Brief History of SDN

Software Defined Networking is a new and exciting development in networking.  However, it's been a long time coming:

1985    IGRP  planned to take into count metrics beyond hop count to include reliability and load

1991    OSPF  specified calculating paths to deliver better throughput, lower latency, etc.  John Moy's excellent book remains one of my very favorites.

1996    Cisco Policy Based Routing was introduced with IOS 11.0 and "provides a flexible mechanism for network administrators to customize the operation of the routing table and the flow of traffic within their networks."

2006    Cisco OER  became PfR sometime around 2010.  It allows the traffic forwarding decision to be based on delay, packet loss, and link loading.  Arguably the closest precursor to SDN (at least from Cisco) in that it has a 'separation of control plane', 'a centralized controller' and possibly 'programmability by external applications.

Please send me other examples!

Thursday, May 24, 2012

Slow TCP but no packet loss

Many times I've heard the claim "out-of-order packets" wreak havoc with your application.  Many interpretations conclude that must be because of poor application coding.  Blaming others can flow both ways, it seems.  My favorite comment in all threads about Duplicate ACKs and Fast Retransmit concludes with " how the network performance is improved by a server reboot. "

Back to our main concern here.  When a receiver gets TCP datagrams 1,2,3 and 5 it will immediately send an ACK expecting 4 as it already did to confirm receipt of datagram 3.  Ding.  First 'duplicate ACK'.

Now, listen closely as I don't plan to get in this habit.  The RFC for TCP Congestion Control is a moderately pain-free read and gives the authoritative fundamentals on how TCP should behave.  Give it a look!

The key to dig out is somewhere around page 6.  Duplicate ACKs cause sender to reduce window size.  The whole mechanism and scenarios for SLOWING down transmission rates of the TCP flow are detailed.

MORAL:  nothing has to be lost and TCP can still determine congestion has been encountered and slow down.  Sometimes significantly.  Also came across these great course slides.

Thursday, May 17, 2012

Two missing but vital Cisco Nexus vPC commands

So you come across a pair of Nexus 5548 switches and dive right into setting up a Virtual PortChannel following the configuration guides.  You'll wind up with something like this:

Things go swimmingly!  That is until you want to upgrade or have to power cycle one of the switches.  Your server can't get anywhere nor be reached for quite a few seconds.  Odd, you think.  Surely frames can happily flow along the remaining active link.  Isn't that what vPC is for?  As one switch comes back online, you then get ANOTHER period of lost connection to your server.

Tuesday, May 8, 2012

TCP Throughput across the Internet

After reading Brad Hedlund's excellent overview and digging into the great tools at SWITCH, I ran some quick numbers for what transfers might see across the Internet.

First, taking low packet loss of 0.01% and checking across various delays from 5 to 60ms, I get:

Then, using 60 ms as a rather normal round-trip time (RTT) across the Internet, I tried values for packet loss increasing to a very normal 0.1% :

Looks like it's pretty hard to fill that 25Mbps DSL.  In the future, I will look at tuning options for improving that.  Comments welcome!

Friday, May 4, 2012

Response Time Composition

Waaaay back in 2006, I used a network monitoring appliance from Network Physics whom I'd lost touch with.  It delivered this graph that quickly became my best friend:

One glance at the Response Time Composition and I could see what had changed from a baseline.  I could also see where to dig in further as to the cause of reported 'slowness'. Of course it was never the network and quite often the server or sheer massiveness of application chattiness.

I'm pleased to say my old friend lives on and has been seriously enhanced by the folks at OPNET in their AppResponse Xpert tool.  Even more super-cool is the SaaS edition that fires up right in your #cloud -running application via a sweet bit of JavaScript.

Friday, April 27, 2012

10Gig options within the rack

This is a roundup of my preliminary look into the current state of 10 Gigabit Ethernet cabling within a rack.  While 10GBASE-SR was the common compatibility choice, it's also one of the more expensive.  I've seen it commonly supported in devices ranging from firewalls to storage.

While a bit dated now, Lisa Huff gives a great analysis of the copper lay of the land in 10GBASE-T vs 10GBASE-CR :
        "Both SFP+ DAC and 10GBASE-T products will be needed in the long term – 10GBASE-T for inexpensive connections         and 10GBASE-CR (SFP+ DAC) for lower latency and lower power consumption."

Howard Marks gives a great update on what's happening with the previous power-hog champion, originally weighing in at 8W PER PORT :
All I Want For Christmas Is 10Gbase-T - Network Computing

..which is partially in response to the original (and convincing) attack on 10GBASE-T by the illustrious Mr. Ferro

Monday, April 23, 2012

DIY Router or Switch Management port

Say you've become very fond of that dedicated management port on your Cisco Nexus switch. Now how to connect your Catalyst 6500 into that out-of-band management network?

Local policy to the rescue!  This works like a champ and is not to be confused with Policy-based Routing. (Although PBR does have a great ring to it!).

A local policy is triggered by packets involving the device itself.  Not traffic being routed through.  Here's a sample:

ip local policy route-map local-mgmt
ip access-list extended mgmt
 permit icmp host
route-map local-mgmt permit 10
 match ip address mgmt
 set ip next-hop

Where is a loopback or other management-related interface on your switch and is the next hop router that gets you to your management networks.  Of course you could just alternately make your OOB management a single, large, flat subnet.  Connected routes for the win!

For more information, Petr Lapukhov has several other examples, his fourth coinciding with mine above.

Wednesday, April 18, 2012

Brief Netflix analysis

Firing up Netflix  in a browser, I decided to see what could be found.  Wireshark was my main cohort in this exercise.  Selecting 'Statistics...Conversations' and then sorting by Bytes gave me these:
Back in the main capture, I looked at what order these remote IP's were referred and then tried a reverse DNS lookup:





So it seems the main 'logic' of the website such as account login, pulling up my favorites, and searching for movies is hosted on Amazon.  My guess is that the movie poster images are hosted on Akamai.  The HTTP content of that field confirms:

GET /en_us/boxshots/166/60020865.jpg HTTP/1.1\r\n
Keep-Alive: 115\r\n
Connection: keep-alive\r\n

Finally, I pulled out the TCP stream for the top flow based on Bytes.  

  A 'whois' on the IP reveals it is from Level3:
 NetRange: -
NetType: Direct Allocation
OrgName: Level 3 Communications, Inc.

  The Wireshark decode shows an HTTP/1.1 stream (Content-Type: application/octet-stream\r\n , Server: Level-3 Origin Storage/1.5\r\n)

 A little bit of packet capture tells us they use Amazon Web Services for the front-end and account logic, Akamai for static content, and Level3 Storage for the media streams.  Some to-do items: check browser data for the application type that plays the media and look for TCP header data to learn more about flow control and media streaming.

Monday, April 16, 2012

North American IPv6 Summit

Last week, Denver hosted the 2012 North American IPv6 Summit.

The keynote by John Curran of ARIN kicked things off nicely.  My favorite points were:

  1.  A few years ago, NAT was sounding the death knell for IPv6 adoption.
  2. Today, media providers are hindered by NAT. (Think tracking and targeting...)  Follow the money people - IPv6 is gaining support.

Other interesting tidbits overheard:

  1. Enable a client for IPv6 and see content come in that way as much as 10% of the time.
  2. Enable a web server for IPv6 and only get <1% traffic over it.
  3. Over 30% of DNS domains are IPv6-enabled largely due to efforts of Godaddy and other leading registrars.

I learned from Oracle's Paul Zawacky that 'dual stack' is becoming the transition method of choice.  Owen Delong from Hurricane Electric advised to stick with a /64 per VLAN and stay away from "IPv4 thinking" when considering alternate subnet masks.  Finally, NAT64 is a very interesting method for moving all hosts to a pure-IPv6 environment and maintaining v4 Internet access.

@scotthogg organized and summarized the event quite nicely.  Thanks!