Regional leadership in IPv6

Network World claims "The US is overtaking the world on IPv6" , but I'm not entirely convinced.

Seems like when the numbers can support the claim such as overall traffic, North America vs. all of Asia numbers are shown.  When it's a bit closer, such as in number of addresses or vendor support the numbers are broken down by country to show the US is slightly ahead of China but not all of Asia.

It's nice to see that North America is no longer ignoring the protocol and the future of the Internet.  Now let's see how we are doing in percentage of bandwidth and percentage of total users!

SSL and TLS - What do browsers use to encrypt?

I'll spare you all the gory details you can read on wikipedia, but these are protocols used to encrypt data exchanged by browsers and web servers for keeping information private and unchanged.  Secure Sockets Layer (SSL) has been through versions 1, 2, and 3.  All by 1996 !   Transport Layer Security (TLS) was released as version 1.0 in 1999.  (Also a !).

You may often see references to SSL3/TLS1.0 because "TLS 1.0 does include a means by which a TLS implementation can downgrade the connection to SSL 3.0, thus weakening security."

TLS1.1 was defined in 2006.   TLS1.2 was defined in 2008.  As of 2012, IE9 appears to be the only major browser to support it.

In his great BlackHat 2010 presentation, Ivan Ristic presents great statistics on server support for these protocols discovered during surveying:

nping, a ping for more protocols

Nping is really handy when you suspect that ping or traceroute aren't giving you clear results.  I really like it for a TCP Connect to a website:


$ nping --tcp-connect -c 3 --delay 2s www.google.com

Starting Nping 0.6.01 ( http://nmap.org/nping ) at 2012-08-13 12:09 MDT
SENT (0.0067s) Starting TCP Handshake > www.google.com:80 (74.125.225.176:80)
RECV (0.0209s) Handshake with www.google.com:80 (74.125.225.176:80) completed
SENT (2.0086s) Starting TCP Handshake > www.google.com:80 (74.125.225.176:80)
RECV (2.0250s) Handshake with www.google.com:80 (74.125.225.176:80) completed
SENT (4.0113s) Starting TCP Handshake > www.google.com:80 (74.125.225.176:80)
RECV (4.0244s) Handshake with www.google.com:80 (74.125.225.176:80) completed

Max rtt: 16.390ms | Min rtt: 13.108ms | Avg rtt: 14.507ms
TCP connection attempts: 3 | Successful connections: 3 | Failed: 0 (0.00%)
Tx time: 4.00581s | Tx bytes/s: 59.91 | Tx pkts/s: 0.75
Rx time: 4.01892s | Rx bytes/s: 29.86 | Rx pkts/s: 0.75
Nping done: 1 IP address pinged in 4.02 seconds

You can also scan MAC addresses on your local subnet:


 $ sudo nping --arp-type ARP 172.16.16.1-10 --count 1

SENT (4.8530s) ARP who has 172.16.16.1? Tell 172.16.16.11

RCVD (4.8546s) ARP reply 172.16.16.1 is at 1C:DF:DF:53:91:91

SENT (5.8538s) ARP who has 172.16.16.2? Tell 172.16.16.11

RCVD (5.8548s) ARP reply 172.16.16.2 is at 00:26:9F:37:33:33

SENT (6.8556s) ARP who has 172.16.16.3? Tell 172.16.16.11

RCVD (6.9364s) ARP reply 172.16.16.3 is at A4:EF:57:E3:EA:EA

SENT (7.8575s) ARP who has 172.16.16.4? Tell 172.16.16.11

SENT (8.8582s) ARP who has 172.16.16.5? Tell 172.16.16.11

SENT (9.8587s) ARP who has 172.16.16.6? Tell 172.16.16.11

SENT (10.8600s) ARP who has 172.16.16.7? Tell 172.16.16.11

SENT (11.8612s) ARP who has 172.16.16.8? Tell 172.16.16.11

SENT (12.8621s) ARP who has 172.16.16.9? Tell 172.16.16.11

RCVD (12.8656s) ARP reply 172.16.16.9 is at 00:21:CF:BD:1F:1F

SENT (13.8634s) ARP who has 172.16.16.10? Tell 172.16.16.11

Raw packets sent: 10 (420B) | Rcvd: 4 (184B) | Lost: 6 (60.00%)
Tx time: 9.01162s | Tx bytes/s: 46.61 | Tx pkts/s: 1.11
Rx time: 10.01221s | Rx bytes/s: 18.38 | Rx pkts/s: 0.40
Nping done: 10 IP addresses pinged in 14.86 seconds

Integrating routing with CDN

A content delivery network (CDN) is a large distributed system of servers deployed in multiple data centers in the Internet.[citation needed] The goal of a CDN is to serve content to end-users with high availability and high performance.


Uniform Resource Identifier (URI) is a string of characters used to identify a name or a resource on the Internet. - en.wikipedia.org/wiki/URI


Border Gateway Protocol (BGP) is the protocol which is the core routing decisions on the Internet. It maintains a table of IP networks or 'prefixes' which designate network reach-ability - en.wikipedia.org/wiki/BGP

The multiprotocol extensions of BGP has been used to carry many types of information.  I stumbled across a great progression on the idea from this NANOG presentation from the guys at Comcast.


Here's their packet capture screenshot.  Notice the Address Family -  URI.

IPv1 , v2, and v3

I always hear "What happened to IP version 5?" But we've seen the answer many times.  "Some experimental testing version".

While reading Charles Kozierok's excellent TCP/IP Guide, I found the history of why IPv4 was actually the first version of IP as in a Layer 3 on it's own.  The addressing layer was part of the original Network Control Program and then Transmission Control Program combined into a single Layer3 and 4 functionality.   TCP became Transmission Control Protocol by its 'fourth iteration'.

Why hasn't networking followed Moore's Law ?

Andy Bechtolsheim's keynote at NANOG55 in Vancouver looks into what's been going on with networking versus Moore's Law:

Follow the link to his slides in PDF.  There are some fantastic graphs on:

1. Server 10/40/100G Adoption Cycle
2. Total Datacenter Switch Revenue by Protocol & Speed

Today's custom switch silicon support 64 10G ports on a single chip but forecast to scale by 4X in the next 3 years.  He wraps up the presentation looking into ways to alleviate the current high cost of optics slowing down 40G/100G adoption.

IPv6 Day

Early today, major web sites enabled their DNS 'quad A' or AAAA entries.

These include google.com, bing.com, facebook.com, yahoo.com, youtube.com and aol.com.

I especially like Facebook's IPv6 address:

2A03:2880:2110:3F03:FACE:B00C::

It looks like traffic to Google over one backbone immediately doubled... Albeit up to almost 2% of total traffic.

I'm told that unlike last year, these updates are planned to be permanent. Sure is a step in the right direction!

A Brief History of SDN

Software Defined Networking is a new and exciting development in networking.  However, it's been a long time coming:

1985    IGRP  planned to take into count metrics beyond hop count to include reliability and load

1991    OSPF  specified calculating paths to deliver better throughput, lower latency, etc.  John Moy's excellent book remains one of my very favorites.

1996    Cisco Policy Based Routing was introduced with IOS 11.0 and "provides a flexible mechanism for network administrators to customize the operation of the routing table and the flow of traffic within their networks."

2006    Cisco OER  became PfR sometime around 2010.  It allows the traffic forwarding decision to be based on delay, packet loss, and link loading.  Arguably the closest precursor to SDN (at least from Cisco) in that it has a 'separation of control plane', 'a centralized controller' and possibly 'programmability by external applications.

Please send me other examples!

Slow TCP but no packet loss

Many times I've heard the claim "out-of-order packets" wreak havoc with your application.  Many interpretations conclude that must be because of poor application coding.  Blaming others can flow both ways, it seems.  My favorite comment in all threads about Duplicate ACKs and Fast Retransmit concludes with " how the network performance is improved by a server reboot. "

Back to our main concern here.  When a receiver gets TCP datagrams 1,2,3 and 5 it will immediately send an ACK expecting 4 as it already did to confirm receipt of datagram 3.  Ding.  First 'duplicate ACK'.

Now, listen closely as I don't plan to get in this habit.  The RFC for TCP Congestion Control is a moderately pain-free read and gives the authoritative fundamentals on how TCP should behave.  Give it a look!

The key to dig out is somewhere around page 6.  Duplicate ACKs cause sender to reduce window size.  The whole mechanism and scenarios for SLOWING down transmission rates of the TCP flow are detailed.

MORAL:  nothing has to be lost and TCP can still determine congestion has been encountered and slow down.  Sometimes significantly.  Also came across these great course slides.

Two missing but vital Cisco Nexus vPC commands

So you come across a pair of Nexus 5548 switches and dive right into setting up a Virtual PortChannel following the configuration guides.  You'll wind up with something like this:

Things go swimmingly!  That is until you want to upgrade or have to power cycle one of the switches.  Your server can't get anywhere nor be reached for quite a few seconds.  Odd, you think.  Surely frames can happily flow along the remaining active link.  Isn't that what vPC is for?  As one switch comes back online, you then get ANOTHER period of lost connection to your server.

TCP Throughput across the Internet

After reading Brad Hedlund's excellent overview and digging into the great tools at SWITCH, I ran some quick numbers for what transfers might see across the Internet.

First, taking low packet loss of 0.01% and checking across various delays from 5 to 60ms, I get:

Then, using 60 ms as a rather normal round-trip time (RTT) across the Internet, I tried values for packet loss increasing to a very normal 0.1% :

Looks like it's pretty hard to fill that 25Mbps DSL.  In the future, I will look at tuning options for improving that.  Comments welcome!

Response Time Composition

Waaaay back in 2006, I used a network monitoring appliance from Network Physics whom I'd lost touch with.  It delivered this graph that quickly became my best friend:

One glance at the Response Time Composition and I could see what had changed from a baseline.  I could also see where to dig in further as to the cause of reported 'slowness'. Of course it was never the network and quite often the server or sheer massiveness of application chattiness.

I'm pleased to say my old friend lives on and has been seriously enhanced by the folks at OPNET in their AppResponse Xpert tool.  Even more super-cool is the SaaS edition that fires up right in your #cloud -running application via a sweet bit of JavaScript.

10Gig options within the rack

This is a roundup of my preliminary look into the current state of 10 Gigabit Ethernet cabling within a rack.  While 10GBASE-SR was the common compatibility choice, it's also one of the more expensive.  I've seen it commonly supported in devices ranging from firewalls to storage.

While a bit dated now, Lisa Huff gives a great analysis of the copper lay of the land in 10GBASE-T vs 10GBASE-CR :

        "Both SFP+ DAC and 10GBASE-T products will be needed in the long term – 10GBASE-T for inexpensive connections         and 10GBASE-CR (SFP+ DAC) for lower latency and lower power consumption."

Howard Marks gives a great update on what's happening with the previous power-hog champion, originally weighing in at 8W PER PORT :
All I Want For Christmas Is 10Gbase-T - Network Computing

..which is partially in response to the original (and convincing) attack on 10GBASE-T by the illustrious Mr. Ferro

DIY Router or Switch Management port

Say you've become very fond of that dedicated management port on your Cisco Nexus switch. Now how to connect your Catalyst 6500 into that out-of-band management network?

Local policy to the rescue!  This works like a champ and is not to be confused with Policy-based Routing. (Although PBR does have a great ring to it!).

A local policy is triggered by packets involving the device itself.  Not traffic being routed through.  Here's a sample:


ip local policy route-map local-mgmt
!
ip access-list extended mgmt
 permit icmp host 10.2.3.4 10.161.161.0 0.0.0.255
!
route-map local-mgmt permit 10
 match ip address mgmt
 set ip next-hop 10.2.161.1
!
end

Where 10.2.3.4 is a loopback or other management-related interface on your switch and 10.2.161.1 is the next hop router that gets you to your management networks.  Of course you could just alternately make your OOB management a single, large, flat subnet.  Connected routes for the win!


For more information, Petr Lapukhov has several other examples, his fourth coinciding with mine above.

Brief Netflix analysis

Firing up Netflix  in a browser, I decided to see what could be found.  Wireshark was my main cohort in this exercise.  Selecting 'Statistics...Conversations' and then sorting by Bytes gave me these:

Back in the main capture, I looked at what order these remote IP's were referred and then tried a reverse DNS lookup:

>nslookup 50.19.81.73
Name:    ec2-50-19-81-73.compute-1.amazonaws.com
Address:  50.19.81.73

>nslookup 65.126.84.9
Name:    a65-126-84-9.deploy.akamaitechnologies.com
Address:  65.126.84.9

>nslookup 65.126.84.11
Name:    a65-126-84-11.deploy.akamaitechnologies.com
Address:  65.126.84.11

>nslookup 65.126.84.18
Name:    a65-126-84-18.deploy.akamaitechnologies.com
Address:  65.126.84.18

So it seems the main 'logic' of the website such as account login, pulling up my favorites, and searching for movies is hosted on Amazon.  My guess is that the movie poster images are hosted on Akamai.  The HTTP content of that field confirms:

GET /en_us/boxshots/166/60020865.jpg HTTP/1.1\r\n

Host: cdn-5.nflximg.com\r\n

Keep-Alive: 115\r\n

Connection: keep-alive\r\n

Referer: http://movies.netflix.com/WiHome\r\n

Finally, I pulled out the TCP stream for the top flow based on Bytes.  

  A 'whois' on the IP reveals it is from Level3:

 NetRange: 8.0.0.0 - 8.255.255.255

CIDR: 8.0.0.0/8
NetType: Direct Allocation
OrgName: Level 3 Communications, Inc.


  The Wireshark decode shows an HTTP/1.1 stream (Content-Type: application/octet-stream\r\n , Server: Level-3 Origin Storage/1.5\r\n)

 A little bit of packet capture tells us they use Amazon Web Services for the front-end and account logic, Akamai for static content, and Level3 Storage for the media streams.  Some to-do items: check browser data for the application type that plays the media and look for TCP header data to learn more about flow control and media streaming.

North American IPv6 Summit

Last week, Denver hosted the 2012 North American IPv6 Summit.

The keynote by John Curran of ARIN kicked things off nicely.  My favorite points were:

1.  A few years ago, NAT was sounding the death knell for IPv6 adoption.
2. Today, media providers are hindered by NAT. (Think tracking and targeting...)  Follow the money people - IPv6 is gaining support.

Other interesting tidbits overheard:

1. Enable a client for IPv6 and see content come in that way as much as 10% of the time.
2. Enable a web server for IPv6 and only get <1% traffic over it.
3. Over 30% of DNS domains are IPv6-enabled largely due to efforts of Godaddy and other leading registrars.

I learned from Oracle's Paul Zawacky that 'dual stack' is becoming the transition method of choice.  Owen Delong from Hurricane Electric advised to stick with a /64 per VLAN and stay away from "IPv4 thinking" when considering alternate subnet masks.  Finally, NAT64 is a very interesting method for moving all hosts to a pure-IPv6 environment and maintaining v4 Internet access.

@scotthogg organized and summarized the event quite nicely.  Thanks!