I examined paths of my network traffic to three sites: nytimes.com, amazon.com, and this one (rashidakamal.com). I looked at the traceroute results from a few locations I frequent: NYU Washington Square, NYU Brooklyn and my home; but also, a new spot: the Billy Bishop Airport in Toronto. I’ve seperated out the traceroute results by website below, with analysis of patterns and a disccusion of terms to follow.
Oddities in the Maps
One of the more (initially) intriguing aspects of the mapping results was that many results wound its way to the Cheney Reservoir in Kansas. Of course, as the blogs of former ITPers will reveal, this is no secret networking hotspot, but rather an artefact of maps and the IP geolocating service we used: the reservoir is the center of the United States and the location of some IP lookups defaulted to the centroid.
This meant some of the maps are actually quite boring: many look as though the route suddenly appears in Kansas and sometimes, stays there. Other maps seem to obscure or time out of hops. See below:
Amazon.com from Toronto
1 [AS0] 10.14.0.1 (10.14.0.1) 2.988 ms 4.952 ms 6.114 ms
2 [AS577] 207.164.85.241 (207.164.85.241) 4.185 ms 3.893 ms 4.297 ms
3 [AS0] 10.55.55.109 (10.55.55.109) 6.666 ms 4.191 ms 5.189 ms
4 [AS577] bx3-torontoxn_hundredgige0-2-0-0.net.bell.ca (64.230.97.147) 7.531 ms
[AS577] bx3-torontoxn_hundredgige0-1-0-0.net.bell.ca (64.230.97.145) 7.742 ms
[AS577] bx3-torontoxn_hundredgige0-2-0-0.net.bell.ca (64.230.97.147) 5.268 ms
5 [AS0] 99.82.178.222 (99.82.178.222) 6.447 ms 7.792 ms 7.810 ms
6 * * *
7 * * *
8 [AS16509] 54.239.44.173 (54.239.44.173) 30.954 ms
[AS16509] 54.239.44.171 (54.239.44.171) 35.208 ms
[AS16509] 52.93.3.159 (52.93.3.159) 32.770 ms
9 [AS16509] 54.239.44.187 (54.239.44.187) 36.668 ms
[AS16509] 52.93.129.122 (52.93.129.122) 42.031 ms
[AS16509] 54.239.44.173 (54.239.44.173) 33.610 ms
10 [AS16509] 54.239.44.77 (54.239.44.77) 43.102 ms
[AS16509] 52.93.128.103 (52.93.128.103) 39.937 ms
[AS16509] 54.240.229.143 (54.240.229.143) 32.033 ms
11 * [AS16509] 52.93.131.182 (52.93.131.182) 47.817 ms *
12 * * *
13 * * *
14 * * *
15 * * *
16 * * *
17 * * *
18 * * *
19 * * *
20 [AS0] 52.93.29.18 (52.93.29.18) 40.513 ms
Interesting Hops
[AS577] 207.164.85.241 (207.164.85.241): AS577 Bell Canada - this is an ISP.[AS0] 99.82.178.222 (99.82.178.222): this is Virginia Beach, VA[AS16509] 52.93.131.182 (52.93.131.182): AS16509 Amazon.com, Inc. - this lands us in Ashburn, VA.
Amazon.com from Washington SQ
1 [AS0] 10.17.0.2 (10.17.0.2) 31.251 ms 93.620 ms 38.694 ms
2 [AS0] coregwb-te5-8-vl901-wlangwa-wwh.net.nyu.edu (10.254.4.32) 1.585 ms 1.827 ms 3.148 ms
3 [AS19905] 128.122.1.36 (128.122.1.36) 4.842 ms 1.934 ms 1.784 ms
4 [AS0] ngfw-palo-vl1500.net.nyu.edu (192.168.184.228) 2.588 ms 2.480 ms 2.265 ms
5 [AS19905] nyugwa-outside-ngfw-vl3080.net.nyu.edu (128.122.254.114) 2.355 ms 2.314 ms 1.944 ms
6 [AS0] nyunata-vl1000.net.nyu.edu (192.168.184.221) 2.258 ms 3.900 ms 6.475 ms
7 [AS19905] nyugwa-vl1001.net.nyu.edu (192.76.177.202) 2.685 ms 2.670 ms 2.535 ms
8 [AS19905] dmzgwb-ptp-nyugwa-vl3082.net.nyu.edu (128.122.254.111) 3.183 ms 3.690 ms 3.334 ms
9 [AS19905] 128.122.254.66 (128.122.254.66) 3.155 ms 2.719 ms 3.173 ms
10 [AS0] 199.109.5.5 (199.109.5.5) 3.077 ms 3.713 ms 3.449 ms
11 [AS0] i2-newy-nyc-9208.nysernet.net (199.109.5.2) 3.411 ms 3.106 ms 3.040 ms
12 [AS11537] ae-3.4079.rtsw.wash.net.internet2.edu (162.252.70.138) 8.548 ms 11.111 ms 8.502 ms
13 [AS11537] ae-0.4079.rtsw2.ashb.net.internet2.edu (162.252.70.137) 9.559 ms 8.690 ms 9.201 ms
14 [AS11537] 64.57.30.39 (64.57.30.39) 8.659 ms 8.793 ms 8.907 ms
15 [AS16509] 54.239.108.144 (54.239.108.144) 22.740 ms
[AS16509] 54.239.108.76 (54.239.108.76) 25.750 ms
[AS16509] 54.239.109.24 (54.239.109.24) 31.214 ms
16 [AS16509] 52.93.114.45 (52.93.114.45) 21.628 ms
[AS16509] 52.93.114.119 (52.93.114.119) 8.946 ms
[AS16509] 52.93.114.15 (52.93.114.15) 10.122 ms
17 * * *
18 * * *
19 [AS16509] 52.93.28.82 (52.93.28.82) 10.899 ms
[AS16509] 52.93.28.80 (52.93.28.80) 9.275 ms
[AS16509] 52.93.28.78 (52.93.28.78) 8.992 ms
20 * * *
21 * * *
22 * * *
23 * * *
24 * * *
25 * * *
26 * * *
27 * * *
28 * * *
29 [AS0] 205.251.248.45 (205.251.248.45) 14.240 ms 10.951 ms
Interesting Hops
[AS19905] 128.122.1.36 (128.122.1.36): AS19905 NeuStar, Inc. (hosting) Interestingly, the IP address lookup notes the associated organization as AS12 New York University. I didn’t realize there could be multiple AS assocaited with a node, and I’m curious what it means.[AS0] i2-newy-nyc-9208.nysernet.net (199.109.5.2): NYSERNet - registered in Syracuse, NY.[AS11537] ae-3.4079.rtsw.wash.net.internet2.edu (162.252.70.138): Internet2 - registered in Ann Arbor, MI.[AS16509] 52.93.28.78 (52.93.28.78): AS16509 Amazon.com, Inc. - this lands us in Ashburn, VA.
Amazon.com from ITP
1 [AS0] 10.18.0.2 (10.18.0.2) 2.512 ms 1.422 ms 1.383 ms
2 [AS0] coregwc-te7-8-vl901-wlangwc-7e12.net.nyu.edu (10.254.6.44) 1.502 ms 1.599 ms 1.482 ms
3 [AS19905] 128.122.1.4 (128.122.1.4) 1.511 ms 1.604 ms 1.514 ms
4 [AS0] ngfw-palo-vl1500.net.nyu.edu (192.168.184.228) 2.405 ms 2.948 ms 2.481 ms
5 [AS19905] nyugwa-outside-ngfw-vl3080.net.nyu.edu (128.122.254.114) 3.262 ms 2.585 ms 2.365 ms
6 [AS0] nyunata-vl1000.net.nyu.edu (192.168.184.221) 2.630 ms 2.460 ms 2.453 ms
7 [AS19905] nyugwa-vl1001.net.nyu.edu (192.76.177.202) 2.355 ms 2.529 ms 2.279 ms
8 [AS19905] dmzgwb-ptp-nyugwa-vl3082.net.nyu.edu (128.122.254.111) 3.132 ms 3.262 ms 3.177 ms
9 [AS19905] 128.122.254.66 (128.122.254.66) 2.474 ms 2.607 ms 2.550 ms
10 [AS0] 199.109.5.5 (199.109.5.5) 2.847 ms 2.938 ms 2.900 ms
11 [AS0] i2-newy-nyc-9208.nysernet.net (199.109.5.2) 3.296 ms 3.396 ms 8.341 ms
12 [AS11537] ae-3.4079.rtsw.wash.net.internet2.edu (162.252.70.138) 9.391 ms 8.536 ms 8.839 ms
13 [AS11537] ae-0.4079.rtsw2.ashb.net.internet2.edu (162.252.70.137) 9.773 ms 8.663 ms 10.082 ms
14 [AS11537] 64.57.30.39 (64.57.30.39) 8.861 ms 8.742 ms 9.264 ms
15 [AS16509] 54.239.108.76 (54.239.108.76) 22.125 ms
[AS16509] 54.239.108.178 (54.239.108.178) 32.156 ms
[AS16509] 54.239.109.24 (54.239.109.24) 34.250 ms
16 [AS16509] 52.93.114.115 (52.93.114.115) 15.361 ms
[AS16509] 52.93.114.37 (52.93.114.37) 8.921 ms
[AS16509] 52.93.114.121 (52.93.114.121) 10.217 ms
17 * * *
18 * * *
19 [AS16509] 52.93.29.14 (52.93.29.14) 11.307 ms
[AS16509] 52.93.29.8 (52.93.29.8) 9.899 ms
[AS16509] 52.93.29.14 (52.93.29.14) 10.502 ms
...
64 * * *
Interesting Hops
- Interestingly, the routers in the path from Washington SQ and Brooklyn campuses look remarkable similar.
- By now, we may recognize the NYU, Internet2, and Amazon AS numbers.
[AS16509] 52.93.29.14 (52.93.29.14): AS16509 Amazon.com, Inc. - this lands us in Virginia Beach, VA.
Amazon.com from home
1 [AS198949] openrg (192.168.1.1) 1.957 ms 1.020 ms 0.926 ms
2 [AS0] 10.240.161.169 (10.240.161.169) 10.212 ms 9.006 ms 9.996 ms
3 [AS0] 67.59.229.26 (67.59.229.26) 11.643 ms 12.979 ms 9.660 ms
4 [AS6128] ool-4353dd1a.dyn.optonline.net (67.83.221.26) 12.614 ms
[AS6128] ool-4353dd18.dyn.optonline.net (67.83.221.24) 14.783 ms
[AS6128] ool-4353dd1a.dyn.optonline.net (67.83.221.26) 18.600 ms
5 [AS0] 65.19.100.2 (65.19.100.2) 17.038 ms
[AS0] 64.15.4.108 (64.15.4.108) 13.086 ms 14.709 ms
6 [AS0] 64.15.2.94 (64.15.2.94) 13.688 ms
[AS0] 64.15.2.46 (64.15.2.46) 15.473 ms
[AS0] 64.15.3.116 (64.15.3.116) 15.338 ms
7 [AS0] 99.82.176.182 (99.82.176.182) 31.835 ms
[AS0] 99.82.176.184 (99.82.176.184) 16.275 ms 18.797 ms
8 * * *
9 * * *
10 * * *
11 * [AS16509] 54.240.229.155 (54.240.229.155) 22.785 ms
[AS16509] 54.240.229.141 (54.240.229.141) 21.470 ms
...
64 * * *
Interesting Hops
[AS198949] openrg (192.168.1.1): AS198949 SECURITYDAM LTD - a network operator out of Israel?[AS6128] ool-4353dd1a.dyn.optonline.net (67.83.221.26): AS6128 Cablevision Systems Corp. - Rhinebeck, NY.[AS16509] 54.240.229.141 (54.240.229.141): AS16509 Amazon.com, Inc. - this lands us in Virginia Beach, VA.
NYTimes.com from Toronto
1 [AS0] 10.14.0.1 (10.14.0.1) 3.574 ms 4.100 ms 2.294 ms
2 [AS577] 207.164.85.241 (207.164.85.241) 7.237 ms 6.495 ms 4.529 ms
3 [AS0] 10.55.55.109 (10.55.55.109) 4.447 ms 4.786 ms 33.960 ms
4 [AS577] bx6-chicagodt_0-6-0-0.net.bell.ca (64.230.79.85) 16.091 ms
[AS577] bx6-chicagodt_0-7-0-0.net.bell.ca (64.230.79.87) 17.604 ms
[AS577] bx6-chicagodt_0-6-0-0.net.bell.ca (64.230.79.85) 15.054 ms
5 * * *
...
64 * * *
Interesting Hops
[AS577] 207.164.85.241 (207.164.85.241): AS577 Bell Canada
NYTimes.com from Washington SQ
1 [AS0] 10.17.0.2 (10.17.0.2) 13.495 ms 1.726 ms 1.325 ms
2 [AS0] coregwb-te5-8-vl901-wlangwa-wwh.net.nyu.edu (10.254.4.32) 1.552 ms 1.728 ms 1.337 ms
3 [AS19905] 128.122.1.36 (128.122.1.36) 2.337 ms 40.272 ms 1.669 ms
4 [AS0] ngfw-palo-vl1500.net.nyu.edu (192.168.184.228) 2.535 ms 2.243 ms 2.208 ms
5 [AS19905] nyugwa-outside-ngfw-vl3080.net.nyu.edu (128.122.254.114) 2.437 ms 2.265 ms 3.323 ms
6 [AS0] nyunata-vl1000.net.nyu.edu (192.168.184.221) 2.406 ms 2.494 ms 4.174 ms
7 [AS19905] nyugwa-vl1001.net.nyu.edu (192.76.177.202) 2.572 ms 2.477 ms 2.503 ms
8 [AS19905] dmzgwb-ptp-nyugwa-vl3082.net.nyu.edu (128.122.254.111) 2.886 ms 3.231 ms 2.931 ms
9 [AS19905] 128.122.254.74 (128.122.254.74) 2.773 ms 2.797 ms 2.804 ms
10 [AS3356] 6-1-30.ear3.newyork1.level3.net (4.28.130.117) 3.211 ms 3.009 ms 3.177 ms
...
64 * * *
Interesting Hops
AS3356] 6-1-30.ear3.newyork1.level3.net (4.28.130.117): Level 3 Parent, LLC - this is an ISP.
NYTimes.com from ITP
1 [AS0] 10.18.0.2 (10.18.0.2) 134.872 ms 22.293 ms 17.798 ms
2 [AS0] coregwc-te7-8-vl901-wlangwc-7e12.net.nyu.edu (10.254.6.44) 2.418 ms 1.353 ms 1.428 ms
3 [AS19905] 128.122.1.4 (128.122.1.4) 1.679 ms 1.621 ms 1.719 ms
4 [AS0] ngfw-palo-vl1500.net.nyu.edu (192.168.184.228) 2.522 ms 3.108 ms 2.439 ms
5 [AS19905] nyugwa-outside-ngfw-vl3080.net.nyu.edu (128.122.254.114) 2.428 ms 2.379 ms 2.193 ms
6 [AS0] nyunata-vl1000.net.nyu.edu (192.168.184.221) 2.699 ms 2.585 ms 2.833 ms
7 [AS19905] nyugwa-vl1001.net.nyu.edu (192.76.177.202) 2.533 ms 2.736 ms 2.694 ms
8 [AS19905] dmzgwb-ptp-nyugwa-vl3082.net.nyu.edu (128.122.254.111) 3.339 ms 3.843 ms 3.287 ms
9 [AS19905] 128.122.254.74 (128.122.254.74) 2.898 ms 3.207 ms 3.038 ms
10 [AS3356] 6-1-30.ear3.newyork1.level3.net (4.28.130.117) 3.052 ms 3.421 ms 3.170 ms
...
64 * * *
NYTimes.com from home
1 [AS198949] openrg (192.168.1.1) 1.884 ms 1.055 ms 0.941 ms
2 [AS0] 10.240.161.169 (10.240.161.169) 9.701 ms 15.744 ms 23.435 ms
3 [AS0] 67.59.235.204 (67.59.235.204) 26.740 ms 8.853 ms 13.977 ms
4 [AS6128] ool-4353dd1e.dyn.optonline.net (67.83.221.30) 14.162 ms
[AS6128] ool-4353dd1c.dyn.optonline.net (67.83.221.28) 17.290 ms 13.755 ms
5 [AS0] 451be068.cst.lightpath.net (65.19.99.104) 12.795 ms
[AS0] 65.19.100.0 (65.19.100.0) 13.459 ms
[AS0] 451be0f8.cst.lightpath.net (65.19.99.248) 17.317 ms
6 [AS0] 64.15.0.196 (64.15.0.196) 14.624 ms
[AS0] 64.15.5.68 (64.15.5.68) 13.020 ms
[AS0] rtr3-tg11-3.in.nycmny83.cv.net (64.15.0.78) 17.914 ms
...
64 * * *
Interesting Hops
[AS6128] ool-4353dd1c.dyn.optonline.net (67.83.221.28): AS6128 Cablevision Systems Corp. - this is an ISP.
rashidakamal.com from Toronto
1 [AS0] 10.14.0.1 (10.14.0.1) 4.728 ms 4.730 ms 4.607 ms
2 [AS577] 207.164.85.241 (207.164.85.241) 3.609 ms 7.680 ms 5.421 ms
3 [AS0] 10.55.55.109 (10.55.55.109) 4.380 ms 4.036 ms 6.861 ms
4 [AS577] bx6-chicagodt_0-7-0-0.net.bell.ca (64.230.79.87) 15.796 ms 22.822 ms 15.008 ms
5 * * *
...
64 * * *
rashidakamal.com from Washington SQ
1 [AS0] 10.17.0.2 (10.17.0.2) 216.102 ms 13.347 ms 95.044 ms
2 [AS0] coregwa-te5-8-vl901-wlangwa-wwh.net.nyu.edu (10.254.2.32) 1.701 ms 1.787 ms 1.549 ms
3 [AS19905] 128.122.1.36 (128.122.1.36) 1.668 ms 1.961 ms 1.693 ms
4 [AS0] ngfw-palo-vl1500.net.nyu.edu (192.168.184.228) 2.824 ms 2.213 ms 2.172 ms
5 [AS19905] nyugwa-outside-ngfw-vl3080.net.nyu.edu (128.122.254.114) 2.286 ms 2.302 ms 2.270 ms
6 [AS0] nyunata-vl1000.net.nyu.edu (192.168.184.221) 2.355 ms 5.357 ms 2.601 ms
7 [AS19905] nyugwa-vl1001.net.nyu.edu (192.76.177.202) 2.611 ms 2.835 ms 2.540 ms
8 [AS19905] dmzgwa-ptp-nyugwa-vl3081.net.nyu.edu (128.122.254.109) 3.163 ms 3.096 ms 3.254 ms
9 [AS19905] 128.122.254.72 (128.122.254.72) 2.660 ms 2.987 ms 2.671 ms
10 [AS3356] 6-1-30.ear3.newyork1.level3.net (4.28.130.117) 3.119 ms 3.065 ms 3.140 ms
...
64 * * *
Interesting Hops
AS3356] 6-1-30.ear3.newyork1.level3.net (4.28.130.117): Level 3 Parent, LLC - this is an ISP.
rashidakamal.com from ITP
1 [AS0] 10.18.0.2 (10.18.0.2) 2.084 ms 1.700 ms 1.521 ms
2 [AS0] coregwd-te7-8-vl901-wlangwc-7e12.net.nyu.edu (10.254.8.44) 1.696 ms 63.931 ms 1.496 ms
3 [AS19905] 128.122.1.36 (128.122.1.36) 1.743 ms 1.672 ms 2.089 ms
4 [AS0] ngfw-palo-vl1500.net.nyu.edu (192.168.184.228) 3.244 ms 2.522 ms 2.521 ms
5 [AS19905] nyugwa-outside-ngfw-vl3080.net.nyu.edu (128.122.254.114) 2.605 ms 2.481 ms 2.350 ms
6 [AS0] nyunata-vl1000.net.nyu.edu (192.168.184.221) 2.351 ms 2.750 ms 2.346 ms
7 [AS19905] nyugwa-vl1001.net.nyu.edu (192.76.177.202) 2.493 ms 3.229 ms 2.394 ms
8 [AS19905] dmzgwa-ptp-nyugwa-vl3081.net.nyu.edu (128.122.254.109) 3.295 ms 3.205 ms 3.136 ms
9 [AS19905] 128.122.254.72 (128.122.254.72) 2.761 ms 2.836 ms 2.947 ms
10 [AS3356] 6-1-30.ear3.newyork1.level3.net (4.28.130.117) 3.050 ms 3.218 ms 3.389 ms
...
64 * * *
rashidakamal.com from home
1 [AS198949] openrg (192.168.1.1) 3.293 ms 2.233 ms 2.696 ms
2 [AS0] 10.240.161.169 (10.240.161.169) 9.650 ms 11.431 ms 10.098 ms
3 [AS0] 67.59.229.26 (67.59.229.26) 21.027 ms 11.953 ms 12.123 ms
4 [AS6128] ool-4353dd1a.dyn.optonline.net (67.83.221.26) 12.589 ms 15.003 ms
[AS6128] ool-4353dd18.dyn.optonline.net (67.83.221.24) 21.088 ms
5 [AS0] 451be0f6.cst.lightpath.net (65.19.99.246) 14.633 ms
[AS0] 65.19.100.0 (65.19.100.0) 13.615 ms
[AS0] 451be070.cst.lightpath.net (65.19.99.112) 16.379 ms
6 [AS0] 64.15.5.68 (64.15.5.68) 18.992 ms
[AS0] 64.15.0.76 (64.15.0.76) 15.663 ms
[AS0] 64.15.2.94 (64.15.2.94) 17.179 ms
...
64 * * *
Further Discussion
I was particularly intrigued by why so many of the hops in my routes were seemingly timed out: “***”. Interesting, it seems that traceroute results can be unreliable enough that there is a market for other network diagnostic tools. Part of the complication stems from the protocols that traceroute uses to probe the network, and how those routers along the way respond to each one. By default, traceroute uses Internet Control Message Protocol (ICMP) echo packets. According to its man page, the -P flag will allow the user to specific a desired protocol, the choices including UDP, TCP, GRE, and ICMP.
ICMP is a network-layer protocol that is used by routers (or other network devices) to send error information or updates. Not unlike other protocols, ICMP consists of datagrams, including an IP header. The protocol standards were first definied in 1982 in RFC 777, and then updated in RFC 792. A ICMP packet is sent in response to an incoming packet when the transmission of that message fails. It’s important to note that ICMP is not a transport protocol like TCP or UDP, meaning that it isn’t really used to send data between systems. Diagnostic tools such as traceroute make use of its error reporting function by using the Time To Live (TTL) IP header, which specifies the number of routers a packet can pass through. By starting at a TTL of 0, waiting for an ICMP message, and then incrementing TTL until the destination is reached, traceroute can reveal the nodes along a network route.
Another interesting aspect of the traceroute results were the round trip times (RTT). I also noticed that the RTTs for some of the hops from my home were much longer than any from either NYU networks (though the hops to Amazon had longer RTTs than the other sites from all locations). In some cases, it seems that the RTTs may correspond to a physical distance (but then, surely my apartment isn’t so far from everything!), and possibly the processing delays of routers along the way. Given the speed of light, 1ms represents about 100km, some of the difference must be due to particularities of my network route from home (as I am not 100s of km away from NYU).
Helpful Resources