I spun this out of the Chicago thread as it was OT. It started with me saying that the physically closest speed test site isn't always your best site.
Well, BGP is just a protocol. More appropriate there would be "Internet". Latency is from you to the server you're talking to, then back to yourself. It is called the Round Trip Time (RTT). Each device along the line adds a bit of latency. These devices include amplifiers, routers, switches, radios, xWDM gear, etc. Everything that touches the signal adds latency. Actually, even the fiber itself adds latency compared to microwave.
This is assuming everything is running smoothly. Congestion can add tremendous latency. It can also add jitter. It obviously reduces throughput. Not all carriers connect to all other carriers. http://fixedorbit.com/stats.htm shows how many peers the top ten networks have. A peer is another provider running BGP (the protocol of how providers talk to each other). Sometimes carrier A has to go through B, C and D to get to E. I will post some traceroutes from my border router to different Chicago area SpeedTest.net sites.
That goes my network, (well, another one not named), then AboveNet, then NTT, then Steadfast, then the SilverIP server.
That goes my network, (well, another one not named), then AboveNet, then Verizon, then InterNAP, then Blast.
That goes my network, (well, another one not named), then AboveNet, then Level 3, then GigENet.
To make it more interesting... Just because that's the path my data takes to get there, it may take a completely different path on the way back.
Blast's Aurora server is physically closest to me. It's actually only about 3.1 miles away from me at one point. However, my best performance latency wise is to SilverIP. However, my best performance throughput wise was GigENet.
Take a look at http://bgplay.routeviews.org/ and plug in an address block advertised on the Internet to see how connections between carriers change.