How the Internet Works... haha

[mhammett]

I spun this out of the Chicago thread as it was OT. It started with me saying that the physically closest speed test site isn't always your best site.

 

Due to how TCP functions, latency plays a big role. Latency, jitter (worse on mobile networks), packet loss, etc. all affect what you can get.

 

Just because you are physically located in say Bozeman, Montana doesn't mean the Bozeman server would offer the best performance. Your provider may haul you all the way back to Westin in Seattle before interconnecting with other carriers and thus to the speedtest site. To bring a bit more localness to it, if he is in Arlington Heights, his signal will travel to the MSC (or LTE core) responsible for that site (LTE cores do make it more interesting because they're dynamic). From there it travels over Sprint's network (this part I'm fuzzy on) to wherever Sprint interfaces with the rest of the world. In Chicago, these locations are likely 600 S. Federal and 350 E. Cermak. If a speedtest server sitting on a network that BGP currently prefers is in Joliet, that is where you're likely to get the best performance. It is a lot further away than an Arlington Heights server, but how the Internet is connected is a much bigger factor than physical location. My credentials for this are that I have my own ISP with microwave backhaul and I have equipment in the major Internet exchange points in Chicago.

 

Very true, if your provider's transport went back to 350 E. Cermak Chicago to pick up something cheap such as Hurricane Electric and the speed test in the town next door's transport went to say 56 Marietta in Atlanta, your speed test to next door could have gone to Chicago then NYC then Atlanta then to your neighbor.

 

Another example of this? If you're on any Boingo hotspot, your traffic is hauled to Austin first. I know this because when on a Boingo hotspot served by my network, my traceroutes go through Austin when going to another location on my network.

 

 

First of all, my post sounded more than a little arrogant, so I apologize for that. I had no doubt that you knew what you were talking about; despite your low post count (at the time of the post), it was pretty obvious that you knew a thing or two about network technology. You may have resurrected a lot of old posts, but at least you made use of the “MultiQuote” feature here and obviously have a genuine desire to contribute and not artificially increase your post count.

 

As for your explanation, thanks! I think I have an idea what you’re saying, but I’m still a little hazy on how the whole thing works. I was under the impression that a latency test was very distance-dependent (hence my “can’t beat physics” comment). As an example, here’s how I thought a latency test worked (and this is probably totally wrong):

 

Imagine a very long-distance, rudimentary network is set up between New York City and Los Angeles (straight-shot distance: ~2500 miles). If the network hardware itself somehow didn’t impact the test, the latency would still be a minimum of 27 ms (2500 mi / 186000 mi/s)*2. That number would go up because of the time it takes to change the signal from one form to another (wired -> microwave, for example).

 

As I wrote that, I began to realize that if you don’t take distance into account, you don’t get a true measure of how fast the network can get a response, which is what I think your explanation covers, right? If I’m in Arlington Heights and run a latency test to a server in Joliet, would the packet travel like this? Phone->Tower->MSC->BGP->Server->BGP->MSC->Tower->Phone. If so, is the latency a measurement of the time it takes to go from the BGP to the server and back to the BGP (bolded section)? Just to be clear, is the BGP the Sprint/Internet interface that you referenced?

 

Well, BGP is just a protocol. More appropriate there would be "Internet". Latency is from you to the server you're talking to, then back to yourself. It is called the Round Trip Time (RTT). Each device along the line adds a bit of latency. These devices include amplifiers, routers, switches, radios, xWDM gear, etc. Everything that touches the signal adds latency. Actually, even the fiber itself adds latency compared to microwave.

 

This is assuming everything is running smoothly. Congestion can add tremendous latency. It can also add jitter. It obviously reduces throughput. Not all carriers connect to all other carriers. http://fixedorbit.com/stats.htm shows how many peers the top ten networks have. A peer is another provider running BGP (the protocol of how providers talk to each other). Sometimes carrier A has to go through B, C and D to get to E. I will post some traceroutes from my border router to different Chicago area SpeedTest.net sites.

 

SilverIP Communications

 

1 65.182.160.153 1ms 1ms 1ms

2 ge-11-1-2.mpr2.ord6.us.above.net 1ms 1ms 1ms

3 xe-0-3-0.cr2.ord2.us.above.net 2ms 2ms 2ms

mpls-label=400430

4 xe-0-0-0.cr1.ord2.us.above.net 2ms 2ms 2ms

mpls-label=475547

5 xe-1-1-0.er1.ord7.us.above.net 2ms 2ms 2ms

6 above-ntt-2.ord7.us.above.net 3ms 2ms 2ms

7 ae-8.r06.chcgil09.us.bb.gin.ntt.net 4ms 2ms 2ms

8 xe-0-3-0-6.r06.chcgil09.us.ce.gin.ntt.net 2ms 3ms 6ms

9 ip46.208-100-32.static.steadfastdns.net 3ms 2ms 2ms

10 mail.pixelfxservers.com 3ms 4ms 3ms

11 108.160.193.27 2ms 3ms 2ms

 

That goes my network, (well, another one not named), then AboveNet, then NTT, then Steadfast, then the SilverIP server.

 

Blast Communications

 

 

1 65.182.160.153 1ms 1ms 1ms

2 ge-11-1-2.mpr2.ord6.us.above.net 20ms 1ms 1ms

3 xe-0-3-0.cr2.ord2.us.above.net 2ms 2ms 2ms

mpls-label=530078

4 xe-0-0-0.cr1.ord2.us.above.net 2ms 2ms 2ms

mpls-label=621163

5 xe-4-3-0.er1.ord7.us.above.net 2ms 2ms 2ms

6 0.xe-9-3-0.br3.chi13.alter.net 2ms 2ms 2ms

7 0.ae3.xl3.chi13.alter.net 2ms 2ms 2ms

8 tengige0-4-4-0.gw2.chi13.alter.net 3ms 17ms 2ms

9 internap-gw.customer.alter.net 4ms 5ms 2ms

10 border6.po1-bbnet1.chg.pnap.net 3ms 3ms 3ms

11 blast-10.border6.chg.pnap.net 5ms 6ms 5ms

12 gi03.core1.blastcomm.com 5ms 5ms 5ms

13 66.151.18.110 8ms 6ms 6ms

14 216.152.48.62 6ms 6ms 6ms

15 216.152.52.250 7ms 6ms 6ms

 

That goes my network, (well, another one not named), then AboveNet, then Verizon, then InterNAP, then Blast.

 

GigENet

 

 

1 65.182.160.153 1ms 1ms 1ms

2 ge-11-1-2.mpr2.ord6.us.above.net 54ms 14ms 1ms

3 xe-0-3-0.cr2.ord2.us.above.net 2ms 2ms 4ms

mpls-label=530078

4 xe-0-0-0.cr1.ord2.us.above.net 2ms 2ms 2ms

mpls-label=621163

5 xe-4-3-0.er1.ord7.us.above.net 2ms 2ms 2ms

6 above-level3.ord7.us.above.net 2ms 2ms 2ms

7 vlan52.ebr2.chicago2.level3.net 2ms 2ms 2ms

mpls-label=1457

8 ae-5-5.ebr2.chicago1.level3.net 2ms 2ms 2ms

mpls-label=1259

9 ae-22-52.car2.chicago1.level3.net 2ms 2ms 2ms

10 te3-1.er1.chi1.servernap.net 2ms 2ms 2ms

11 po-11.20g.csr1.chi3.servernap.net 4ms 4ms 5ms

12 te-13-1.ddr3.chi3.servernap.net 4ms 4ms 4ms

 

That goes my network, (well, another one not named), then AboveNet, then Level 3, then GigENet.

 

 

 

To make it more interesting... Just because that's the path my data takes to get there, it may take a completely different path on the way back.

 

Blast's Aurora server is physically closest to me. It's actually only about 3.1 miles away from me at one point. However, my best performance latency wise is to SilverIP. However, my best performance throughput wise was GigENet.

 

Take a look at http://bgplay.routeviews.org/ and plug in an address block advertised on the Internet to see how connections between carriers change.

[jhtechservices]

Just had to write a short post to say thanks! This is an AWESOME post!!

[CriticalityEvent]

This is assuming everything is running smoothly. Congestion can add tremendous latency. It can also add jitter. It obviously reduces throughput. Not all carriers connect to all other carriers.

 

Is this the kind of jitter that you’re referring to? http://en.wikipedia....delay_variation

 

http://fixedorbit.com/stats.htm shows how many peers the top ten networks have. A peer is another provider running BGP (the protocol of how providers talk to each other). Sometimes carrier A has to go through B, C and D to get to E. I will post some traceroutes from my border router to different Chicago area SpeedTest.net sites.

 

So, if I understand this, if I were to run a latency test while on a Level 3 network to a given server, it would be more likely that I would see a better result than if I ran the test on Sprint’s network to that same server, right? Since Level 3 has 2,703 peers compared to Sprint’s 1,316, is Level 3 likely to have a more direct connection to the test server than Sprint?

 

To make it more interesting... Just because that's the path my data takes to get there, it may take a completely different path on the way back.

 

Is this because the network determined that the path the data came in on was no longer the optimal path to return it on? Could the conditions of the original path have changed in that short amount of time?

 

Take a look at http://bgplay.routeviews.org/ and plug in an address block advertised on the Internet to see how connections between carriers change.

 

This looks really interesting! I’m just having a hard time understanding exactly what it’s telling me… What are the address blocks that I’m supposed to plug into it?

[mhammett]

Is this the kind of jitter that you’re referring to? http://en.wikipedia....delay_variation

 

Yes

 

So, if I understand this, if I were to run a latency test while on a Level 3 network to a given server, it would be more likely that I would see a better result than if I ran the test on Sprint’s network to that same server, right? Since Level 3 has 2,703 peers compared to Sprint’s 1,316, is Level 3 likely to have a more direct connection to the test server than Sprint?

 

Likely, but not guaranteed. That server may very well be directly on Sprint's network. That server may also reside on Company A's network. Company A may connect to Companies B, C and D, but neither Level 3 or Sprint. They may connect to Company E, F and G where Company G has a connection to Level 3. Not the case in real life, but Level 3's 2,700 network peers may have zero overlap with Sprint's 1,300 peers. You would then rely on the intermediary networks to connect.

 

Is this because the network determined that the path the data came in on was no longer the optimal path to return it on? Could the conditions of the original path have changed in that short amount of time?

 

Each network that originates the communication (whether the request from the client or the reply from the server) has complete discretion over the direction of the first hop of their outgoing traffic. The network the server is on may have 15 peers, but only send data out of two of them. When determining what network path to take, the major factor is network hops (as opposed to router hops). There is additional tuning that each network can do to affect the flow of traffic into and out of their network. Few networks actually determine on a path-by-path basis what way is best. Companies that do this are InterNAP (hardware and network), ArbiNet (network) and Route Science (since deceased and was a hardware product).

 

 

 

This looks really interesting! I’m just having a hard time understanding exactly what it’s telling me… What are the address blocks that I’m supposed to plug into it?

 

Whatever address block you want to see how it is connected to the Internet. It won't graphically depict all connections they have, but it will show how that address block is viewed by the Internet in terms of what network path to take to get there. These changes happen administratively for whatever reason and a failure of a circuit\device\provider.