Internet Infrastructure AMAA (ask me almost anything)

mhammett · June 23, 2015

(I got an initial okay from Robert for the topic, he'll weigh in later to give final approval.)

As some of you know, I operate a WISP and therefore spend time on towers delivering fixed Internet to homes and businesses. This past winter, I started an Internet Exchange (IX) with some partners. These are at a very different point on the Internet, more like the center.

Internet providers use largely fiber optics (although ocassionally microwave) to connect the local access points throughout the country back to facilities called carrier hotels. Carrier hotels are typically older buildings in major cities that have been repurposed for the Internet age. Dozens or even hundreds of networks converge on these carrier hotels to exchange traffic amongst each other. Often at these facilities you'll also find data centers where web hosting companies, corporations, etc. have located their servers. Why? That's where they have the most diverse supply of Internet for the lowest cost. Examples of carrier hotels are 60 Hudson, 350 Cermak, 1 Wilshire, 56 Marietta, etc.

The large networks such as Level 3 and Cogent would be directly interconnecting with each other with dedicated 10, 40 or 100 gigabit/s connections. However, those companies will only exchange traffic with other titans and now more often with the likes of NetFlix, Google, Facebook, etc. Everyone else joins the Internet Exchanges located within these buildings. A network purchases a port on the exchange, which then permits them to exchange traffic with any other network on the exchange. This is where you will most often find local and regional ISPs, web hosting companies, CDN giants like Google, Facebook, etc. Usually they'll have route servers. If a network connects with a route server, they automatically connect with all other networks on those route servers. This enables small networks to connect to people that might otherwise pass them by due to the work involved in manually configuring your router for each other network.

The largest IXes in the US barely break 300 members. The largest IX by members is AMS-IX, with 800 members connecting via several data centers in Amsterdam. AMS-IX has daily peaks of about 3.75 terabit/s. DE-CIX in Germany (I think largely Frankfurt) is the largest by traffic. They only have about 600 members, but they have daily peaks over 4 terabit/s.

The major Chicago facilities

-----

350 E. Cermak: Most connected building not on a coast. Largest single building non-government data center... in the world. Former RR Donnelly print house (Sears catalog and Ameritech phone books).

427 S. LaSalle: Former Western Union (telegraph and telephone)

520 S. Federal: AT&T CO that hosted AADS, one of the first exchanges after the Internet went public

600 - 700 S. Federal: Heavy data center and interconnection presence, likely due to 520 S. Federal being next door

Indianapolis

-----

Indy Telecom Center: Twelve to fifteen buildings hosting data centers, long haul POPs, AT&T MSC and an IX. Formerly a major rail facility.

I had something more majestic in my mind when I thought this up a week or two ago, but this is what I ended up banging out on the keyboard.

https://fortune.com/2015/06/08/cloud-computing-buildings/

https://www.peeringdb.com/private/exchange_list.php?s_name=&s_city=&s_ipaddr=&s_country=US&s_media=&s_region_continent=

http://long-lines.net/

http://www.cablemap.info/

http://www.telecomramblings.com/network-maps/

http://www.telecomramblings.com/metro-fiber-maps/

I'll now open this up to Q&A.

mhammett · June 23, 2015

[Reserved for future use.]

mhammett · June 23, 2015

[Reserved for future use.]

belusnecropolis · June 23, 2015

Thank you for this.

I have a ton of questions, yet unfortunately just finished physics homework and have to squeeze in some sleep before work. The willingness of others to share experience in industry is greatly appreciated and I look forward to this thread.

I'm pretty sure what you do is along the lines of what I would like to do. Once again S4gru is the spot. I learned more useful info in that first post than my whole day at work.

Thanks again everyone and great idea. Have a great day.

S4GRU · June 23, 2015

Approved for all audiences! :tu:

Using Nexus 6 on Tapatalk

red_dog007 · June 23, 2015

Wow, an IX! What does one specifically look for if they wanted to work at one? IX job postings?

Whenever an ISP gets their backbone, do they connect at an IX, or do they "splice" into a cable that passes somewhere nearby? My ISP (EPB, I am sure you have heard of them), uses Level 3, Century-tel and Sprint for their backbone. How does EPB connect intothose backbones?

Any good sources on starting a WISP? Hardware, software, spectrum, licenses, who to talk too, etc. What is the minimal capital one should plan on having to get started? There are a lot in my area that have zero landline option (800 residential households), and I have been thinking it might be cool to start a WISP in my local area. Plus neighboring counties still have a few thousand here and there not served by landline.

Care to go into some details on your WISP? I'm sure you have a website?

irev210 · June 23, 2015

Could you talk about capacity constraints?

Obviously there has been a lot of news about ISPs limiting interconnections against potential competitors (most recently a small streaming company in San Diego and TWC popped into the news).

As a WISP - how do you provide ample capacity for your customers? Say you were buying capacity from cogent who was overloaded with netflix traffic - are you protected by SLAs to prevent that from being an issue?

Million questions... but it's a start...

mhammett · June 23, 2015

Wow, an IX! What does one specifically look for if they wanted to work at one? IX job postings?

Whenever an ISP gets their backbone, do they connect at an IX, or do they "splice" into a cable that passes somewhere nearby? My ISP (EPB, I am sure you have heard of them), uses Level 3, Century-tel and Sprint for their backbone. How does EPB connect intothose backbones?

Any good sources on starting a WISP? Hardware, software, spectrum, licenses, who to talk too, etc. What is the minimal capital one should plan on having to get started? There are a lot in my area that have zero landline option (800 residential households), and I have been thinking it might be cool to start a WISP in my local area. Plus neighboring counties still have a few thousand here and there not served by landline.

Care to go into some details on your WISP? I'm sure you have a website?

Unfortunately, IXes are typically really lean environments. An IX with a couple hundred networks may have only a couple employees and they may even be part time. IXes are pretty simple to operate once the supporting infrastructure is built. Just keep an eye on area IX's web sites, social media pages, etc.

Some ISPs are so small that they're not even present in these carrier hotels. They're just buying services from whomever is in their area. I've been working to get as many as I can into these carrier hotels.

EPB appears to be in TelX in 56 Marietta Atlanta. https://www.peeringdb.com/private/participant_view.php?id=7007 They are likely obtaining wavelengths (xWDM systems can take dozens or hundreds of circuits and put them on a single fiber (or pair of fiber)) and riding that into Atlanta. xWDM is part of the reason for the fiber glut of the early 2000s. A few networks will let you hop on anywhere, but most will only do so at defined POPs. https://radar.qrator.net/as26827 It also looks like they're using Cogent, Level 3 and Qwest as upstreams. I'd diversify that quite a bit more if I were them. They may be taking delivery of those different networks right there in Chattanooga, but they could be getting them from Atlanta as well.

I have a web site, but it sucks. It's ten years old. I started to update it, but I got busy with other work (the IX). Check out www.wispa.org for beginner WISP stuff. Join their public mailing list and people will likely just point you to a few resources to go over, but when there are specific questions, they'll be glad to answer.

Could you talk about capacity constraints?

Obviously there has been a lot of news about ISPs limiting interconnections against potential competitors (most recently a small streaming company in San Diego and TWC popped into the news).

As a WISP - how do you provide ample capacity for your customers? Say you were buying capacity from cogent who was overloaded with netflix traffic - are you protected by SLAs to prevent that from being an issue?

Million questions... but it's a start...

An IX likely has no capacity constraints. They're simple platforms that aren't trying to go hundreds or thousands of miles to the middle of nowhere. Usually the IX would be more than happy to sell more ports. That said, the big operations like AT&T, Verizon, Comcast, TWC, etc. likely have direct connections to the major content networks. I didn't read into the details of that little streaming company, but to me it looked like a startup that doesn't know how the Internet works trying to game these new rules into getting stuff for free. (I'll try not to rant too much more on Net Neutrality.)

At the height of the Comcast - NetFlix - Cogent problem, the problem was between Cogent and Comcast. Netflix bought service from Cogent, so there wouldn't be any issues for other networks off of Cogent going to Netflix. There would be issues, however, with my network on Cogent going to anyone on Comcast, given that's where the congestion was. I would certainly be angry at my immediate upstream for having congested ports, but if the other party is unwilling to participate, there's only so much to do. Ultimate, it would be my situation to resolve for my customers. Maybe an SLA would play into things, but that's not going to prevent an issue from happening, only provide compensation when there's an issue. An SLA probably wouldn't even apply in that situation as it would be off-net.

I'd just obtain service from someone that didn't have the problem or get to the IX myself where I could work around the problem myself.

I don't have congestion issues in my WISP as I have built the network to have ample capacity.

dkyeager · June 27, 2015

Question on your links or hypothetical links in your network.

Let us say you wish to order a 1Gbps fiber link, but it is not available in a time fashion / has excessive costs. What would be your equivalent order in Gbps over a microwave link operating at approximately 11,000Mhz?

Let us say you get pushed further up the scale into rain fade territory at 23,000Mhz, would there be a compensating increase in designed bandwidth to compensate?

Would any compensation for microwave (with or without rain fade) be based on any formula, let use say based on Mhz and distance?

Would a bidirectional microwave ring design or feed from two directions be an effective alternative?

I am trying to get a solid understanding of Columbus's Clear Huawei network to see what they might do in terms of site conversions and am done with about 75% of the raw data collection phase (our microwave thread excludes Clear). Of course additional tests will be done and additional information included.

Many thanks!!

radem · June 28, 2015

Would a bidirectional microwave ring design or feed from two directions be an effective alternative?

I am not as well versed with network infrastructure as mhammett is, but I know a few things about it and have been around it for many years so I will attempt to answer this question.

In almost every network design, having at least two completely separate paths for the data to follow is considered to be better than one. This is why nearly all major data centers are located very near an IX or have an IX inside of them. Multiple data paths not only increase the total data throughput available but add redundancy for when a network link is cut by construction, congested, rain fade blockage or misalignment affects one of the microwave transfer paths, or for any other reason one data path is not working optimally. It is not uncommon for important data networks to follow a ring pattern or a ring of data rings so that is there are always at least two paths for the data to follow. Hardwired and wireless can both contribute to the redundant paths as long as they really are redundant and separate. Redundant links for the internet and local networks were designed and implemented by the US Defense Department DARPA and research institutions well before home and business users were ever allowed on the internet. See https://en.wikipedia.org/wiki/Self-healing_ring for how redundant network links work.

A data center that I have worked with in the past has backbone interconnects (both fiber and microwave) for AT&T, Sprint,and Comcast, plus it has several smaller carriers on either dedicated fiber, microwave, or both. It is used partially as a backup data center for one of the large Chicago data centers listed in the original post by mhammett. It has multiple ways to get data to and from its primary data center. If any data path is down (for example AT&T), the redundant nature of the networks causes it to automatically route around the problem to stay functional by possibly routing through Sprint or Comcast to a different IX and then back into AT&T's network. See https://en.wikipedia.org/wiki/Router_%28computing%29 for how enterprise routers work.

In my opinion, Sprint should have rapidly deployed microwave in addition to the already in place slow wired connections (multiple T-1s) for any site it was having trouble getting the enhanced wired back haul to. Obviously this would not have worked for locations where the antenna location could not support the weight or positioning of direct line of sight microwave equipment. This would have allowed Sprint to have the bandwidth available quickly in many more locations. They could have then rapidly deployed Network Vision and it would have given them some connectivity redundancy. Some effects of this additional microwave use would have been a higher installation cost, more power usage, and possibly higher lease costs due to the additional tower weight for the microwave equipment. I believe it would have been worth it to get NV done quickly but I do not make decisions for Sprint. It appears that Sprint went with a less expensive option that took longer, left them with no redundancy, and made it look to others that they could not execute quickly. Marcello and Son appear to be changing this type of decision making within the senior ranks of the company.

afazel · June 29, 2015

I'm a Check Point firewall engineer. Is there any work for me using their carrier-grade equipment at these IX facilities, or is it all dominated by Cisco and Juniper? Do they even have much firewalling going on? I have a feeling my IDPS skills wouldn't have much use there.

mhammett · July 3, 2015

Question on your links or hypothetical links in your network.

Let us say you wish to order a 1Gbps fiber link, but it is not available in a time fashion / has excessive costs. What would be your equivalent order in Gbps over a microwave link operating at approximately 11,000Mhz?

Let us say you get pushed further up the scale into rain fade territory at 23,000Mhz, would there be a compensating increase in designed bandwidth to compensate?

Would any compensation for microwave (with or without rain fade) be based on any formula, let use say based on Mhz and distance?

Would a bidirectional microwave ring design or feed from two directions be an effective alternative?

I am trying to get a solid understanding of Columbus's Clear Huawei network to see what they might do in terms of site conversions and am done with about 75% of the raw data collection phase (our microwave thread excludes Clear). Of course additional tests will be done and additional information included.

Many thanks!!

Well, there are a variety of platforms to choose from, depending on how firm the throughput requirements are. The increases in frequency (and corresponding increases in rain fade) would dictate an increase of antenna size or switch to a platform with high enough Tx (transit) power or increase in Rx (receive) sensitivity to compensate for the losses.

There are formulas for rain fade by frequency, rain rate and distance. There are also formulas for dish size and frequency giving you it's gain, so you could make one formula to do all of that... or just use the tools that are available and just choose a bigger dish and see what it gets you.

Here's an example of some links done at different frequencies and dish sizes. I haven't done much work at 23 Ghz, so I may simply be seeing a limitation of the product\tool. 23 GHz also appears to only be slated for 50 MHz channels as opposed to 80 MHz channels in 11 GHz and 18 GHz. I haven't verified if that's an FCC thing or a product thing. 2048QAM only seemed to be available on the 23 GHz radio and not the 11 or 18, so it made up a little bit for the smaller channel with increased modulation complexity.

https://www.dropbox.com/sh/3smdt2g1ih0jg0b/AADie--OZ7vQNmTOHQSXLu3pa?dl=0

Having multiple links that go in different directions is good. For example, having a N-S link and an E-W link. Storms are typically lines and traveling along lines. A storm that went from West to East on an E-W link would experience a less intense fade for a longer period of time, while a N-S link in that same area would experience a more intense fade for a shorter period of time. If you have both and a layer 2 system that effectively measures available capacity dynamically and uses it in aggregate (such as Accedian or OpenFlow), then you are the least mitigated. Such fancy boxes are fairly new. Building these into a complete ring maximizes availability.

Nice maps! I wanted to generate that for all Clear\Sprint nation-wide, but I only got part way through setting up that system. I've been crazy busy the last year or two, so that hasn't been done. When Clear built their network, Fiber To The Tower wasn't a term yet, so they made do with what they had available. It was obviously being done, but it wasn't embraced like it is now. Many of those sites may have fiber available now. Many of those sites may be existing Sprint sites. Many of those sites may be close enough to build a backhaul to an existing Sprint site.

I would be using microwave a lot more if I were these guys. Every tower that is microwave-only would have at least two paths to fiber. Fiber drops would be to completely diverse fiber routes and providers. Fiber fed sites would still have a microwave path out to another fiber provider.

mhammett · July 3, 2015

A data center that I have worked with in the past has backbone interconnects (both fiber and microwave) for AT&T, Sprint,and Comcast, plus it has several smaller carriers on either dedicated fiber, microwave, or both. It is used partially as a backup data center for one of the large Chicago data centers listed in the original post by mhammett. It has multiple ways to get data to and from its primary data center. If any data path is down (for example AT&T), the redundant nature of the networks causes it to automatically route around the problem to stay functional by possibly routing through Sprint or Comcast to a different IX and then back into AT&T's network. See https://en.wikipedia.org/wiki/Router_%28computing%29 for how enterprise routers work.

Where? o.0 ;-)

I'm guessing something government or financial given the degree of diversity and the carriers chosen. Then again, generic enterprise makes similarly odd decisions. ;-)

mhammett · July 3, 2015

I'm a Check Point firewall engineer. Is there any work for me using their carrier-grade equipment at these IX facilities, or is it all dominated by Cisco and Juniper? Do they even have much firewalling going on? I have a feeling my IDPS skills wouldn't have much use there.

Typically an IX is completely open and very basic. An IX may have increased measures on its management or operations networks, but the business end is not much more advanced than basic switching. Measures are typically in place to protect the IX operation, such as a single MAC per port. Route servers usually have route filtering of some kind in place to protect the Internet in general from operator (mess) ups.

dkyeager · July 3, 2015

Well, there are a variety of platforms to choose from, depending on how firm the throughput requirements are. The increases in frequency (and corresponding increases in rain fade) would dictate an increase of antenna size or switch to a platform with high enough Tx (transit) power or increase in Rx (receive) sensitivity to compensate for the losses.

There are formulas for rain fade by frequency, rain rate and distance. There are also formulas for dish size and frequency giving you it's gain, so you could make one formula to do all of that... or just use the tools that are available and just choose a bigger dish and see what it gets you.

Here's an example of some links done at different frequencies and dish sizes. I haven't done much work at 23 Ghz, so I may simply be seeing a limitation of the product\tool. 23 GHz also appears to only be slated for 50 MHz channels as opposed to 80 MHz channels in 11 GHz and 18 GHz. I haven't verified if that's an FCC thing or a product thing. 2048QAM only seemed to be available on the 23 GHz radio and not the 11 or 18, so it made up a little bit for the smaller channel with increased modulation complexity.

https://www.dropbox.com/sh/3smdt2g1ih0jg0b/AADie--OZ7vQNmTOHQSXLu3pa?dl=0

Having multiple links that go in different directions is good. For example, having a N-S link and an E-W link. Storms are typically lines and traveling along lines. A storm that went from West to East on an E-W link would experience a less intense fade for a longer period of time, while a N-S link in that same area would experience a more intense fade for a shorter period of time. If you have both and a layer 2 system that effectively measures available capacity dynamically and uses it in aggregate (such as Accedian or OpenFlow), then you are the least mitigated. Such fancy boxes are fairly new. Building these into a complete ring maximizes availability.

Nice maps! I wanted to generate that for all Clear\Sprint nation-wide, but I only got part way through setting up that system. I've been crazy busy the last year or two, so that hasn't been done. When Clear built their network, Fiber To The Tower wasn't a term yet, so they made do with what they had available. It was obviously being done, but it wasn't embraced like it is now. Many of those sites may have fiber available now. Many of those sites may be existing Sprint sites. Many of those sites may be close enough to build a backhaul to an existing Sprint site.

I would be using microwave a lot more if I were these guys. Every tower that is microwave-only would have at least two paths to fiber. Fiber drops would be to completely diverse fiber routes and providers. Fiber fed sites would still have a microwave path out to another fiber provider.

Thanks for the maps compliment. I have used maps in our Microwave thread with more detail and offered my spreadsheets to others but gotten no takers. http://s4gru.com/index.php?/topic/6836-microwave-backhaul-map/page-7&do=findComment&comment=422792. Another like would be appreciated.

Due to the complexity of some of the FCC data I started putting it into a spreadsheet to make sense of it. I try to cover a complete geographic area that the FCC uses so I can better keep track of changes (state or county). The Columbus Clear microwave map is a test of a simpler version of the spreadsheet.

I guess the 23,000Mhz microwave Mbps equivalent to fiber first depends on the level of risk that is acceptable. Now Clear and Sprint appear to use much smaller dishes than your examples -- about 12 to 18 inches in diameter. Thus my guess for 23000 @90% is about 92% of maximum capacity -- a more exact formula for rough estimation or better guideline would be welcome. In practical terms, it appears they would bump it up one level for the transmitter manufacturer, which is what seems to be happening. Obviously there are exceptions for the spurs, and all of the loop links would need to be calculated.

Looks like no short-cuts except to go out in the early morning and measure pings to find the fiber sites (trace routes did not seem to previously work - must be using VPN or virtual LAN type tech). My feeling is the stand-alone Clear sites using fiber are the most likely to be converted.

I was involved with negotiations with Clearwire when the network was setup. They wanted the speed of using microwave over waiting for fiber. Unfortunately a site that lies in a coverage hole in Upper Arlington Ohio was unable to be built because even though it got zoning approval, its connecting site did not (Clearwire did not want our assistance). Sprint also has weak coverage in that area today.

I should be posting the Columbus Clear map with more detail and stats soon.

Thank you very much for your help (and examples).

:thx:

mhammett · July 3, 2015

Thanks for the maps compliment. I have used maps in our Microwave thread with more detail and offered my spreadsheets to others but gotten no takers. http://s4gru.com/index.php?/topic/6836-microwave-backhaul-map/page-7&do=findComment&comment=422792. Another like would be appreciated.

Due to the complexity of some of the FCC data I started putting it into a spreadsheet to make sense of it. I try to cover a complete geographic area that the FCC uses so I can better keep track of changes (state or county). The Columbus Clear microwave map is a test of a simpler version of the spreadsheet.

I guess the 23,000Mhz microwave Mbps equivalent to fiber first depends on the level of risk that is acceptable. Now Clear and Sprint appear to use much smaller dishes than your examples -- about 12 to 18 inches in diameter. Thus my guess for 23000 @90% is about 92% of maximum capacity -- a more exact formula for rough estimation or better guideline would be welcome. In practical terms, it appears they would bump it up one level for the transmitter manufacturer, which is what seems to be happening. Obviously there are exceptions for the spurs, and all of the loop links would need to be calculated.

Looks like no short-cuts except to go out in the early morning and measure pings to find the fiber sites (trace routes did not seem to previously work - must be using VPN or virtual LAN type tech). My feeling is the stand-alone Clear sites using fiber are the most likely to be converted.

I was involved with negotiations with Clearwire when the network was setup. They wanted the speed of using microwave over waiting for fiber. Unfortunately a site that lies in a coverage hole in Upper Arlington Ohio was unable to be built because even though it got zoning approval, its connecting site did not (Clearwire did not want our assistance). Sprint also has weak coverage in that area today.

I should be posting the Columbus Clear map with more detail and stats soon.

Thank you very much for your help (and examples).

I was following a guide (that didn't seem to work perfectly) to import the FCC's daily transaction log, dump it into SQL, parse it for the links, then dump out a KML. Getting the system together has been pretty manual, but the end result would be fairly easy to run again and again for anywhere.

I have exported 1' versions for 18 GHz and 23 GHz. I don't believe 1' antennas would be allowed in 11 GHz. These are just samples of one vendor's implementation. There are a variety of factors that go into what you use. I picked... the easiest thing to generate pretty reports for you guys. ;-)

Keep in mind that the sample link I used is 4.1666 miles. Something half of that length would perform much better at 23 GHz. Also, if throughput demands for a site are smaller (as such is the case for NV1.0 and Clear towers), the smaller dishes may provide the required throughput at acceptable reliabilities. The max throughput on even an NV2.0 tower (with one channel per band) is around 500 megabit. Now factor in how much you're likely to see with the varying modulations for the active users and you may never see anything close to that. They're probably okay with less.

Microwave ping times are going to be nearly indistinguishable from fiber ping times. Modern licensed microwave gear is going to have fractions of a millisecond per hop. The load on the management interfaces of the testing devices would play more of an impact. No clue on the stability of mobile LTE latencies.

There is going to be a layer 2 tunnel from the eNodeB all the way to the core. Traceroutes wouldn't help (unless maybe it was an MPLS transport and the gear was configured to respond with MPLS information... unlikely.

mhammett · July 3, 2015

I'm a Check Point firewall engineer. Is there any work for me using their carrier-grade equipment at these IX facilities, or is it all dominated by Cisco and Juniper? Do they even have much firewalling going on? I have a feeling my IDPS skills wouldn't have much use there.

and actually, Cisco isn't overly prevalent in the IX market. It is usually Jennifer, brocade or extreme. Extreme seems to be picking up market share as there switches have a good time pls stack and the large exchanges use a lot of MPLS for scalability. Brought to you by voice to text, so hopefully it makes sense.

radem · July 4, 2015

Where? o.0 ;-)

I'm guessing something government or financial given the degree of diversity and the carriers chosen. Then again, generic enterprise makes similarly odd decisions. ;-)

Your guess is correct and that is all I will say on it.

mhammett · January 29, 2016

This didn't take off nearly how I had hoped...

dkyeager · January 29, 2016

This didn't take off nearly how I had hoped...

I am certain your comments will be highly valued once Next Generation Network gets moving. Just start or contribute to a new thread then.

Sent from my LGLS991 using Tapatalk

bretton88 · January 29, 2016

I'm hearing a lot about dark fiber being a big deal. What is the difference between normal fiber and dark fiber?

lordsutch · January 29, 2016

Dark fiber is fiber that isn't in use; either the fiber was never activated because it wasn't needed after it was installed, or it was previously live but has been deactivated since.

Lots of fiber was put in place during the dot com boom for needs that never materialized, or by competitive providers who went bust, or to provide additional capacity that didn't turn out to be needed because of advances in transceiver technology (getting more bandwidth out of the existing fiber links). Hooking up your equipment to dark fiber is cheaper than building your own, but of course it needs to go where you need it to go to be useful.

mhammett · January 29, 2016

Dark fiber is fiber that isn't in use; either the fiber was never activated because it wasn't needed after it was installed, or it was previously live but has been deactivated since.

Lots of fiber was put in place during the dot com boom for needs that never materialized, or by competitive providers who went bust, or to provide additional capacity that didn't turn out to be needed because of advances in transceiver technology (getting more bandwidth out of the existing fiber links). Hooking up your equipment to dark fiber is cheaper than building your own, but of course it needs to go where you need it to go to be useful.

There were some crap business models, but I like to attribute a lot of the .com crash on DWDM. There was no longer a need for most of the fiber that was built. xWDM, though, was good for the long term.

Sent from my Nexus 6 using Tapatalk

mhammett · January 29, 2016

I'm hearing a lot about dark fiber being a big deal. What is the difference between normal fiber and dark fiber?

In the nineties there was a lot of fiber being built to simply be dark fiber to provide to other providers. During the consolidation years afterwards, a lot of that dark fiber was removed from the market. The fire was still for the president, but the companies that purchased it or not interested in selling dark fiber. The past few years, however has seen an increase in people willing to sell dark fiber again. This is partially due to the lack of companies that are willing to provide dark fiber. Some of the uses include new us networks that are needing to be built due to the consolidation of previous US networks, international networks arriving on new submarine cables that do not have any or a significant us footprint, as well as increasing market depth for existing providers. A lot of companies build out to the major metro areas and largely skipped the areas between. Those areas have seen a lot more attention the past few years.

Sent from my Nexus 6 using Tapatalk

mhammett · January 29, 2016

Here's a post I made last night about backhaul circuits and dark fiber.

Going from 3 20 mhz carriers to 6 20 mhz carrier requires double the backhaul requirements (1gigabit minimum per site to 2 gigabit minimum) which most of the time entails nearly twice the backhaul costs. We're about paying $1000-2000 to $5000-$7000 a month.

I don't buy in the volumes Sprint does, but they're not going to get a ton of discounts if there are build costs rolled in. On the contrary, it is easier to absorb the build of a random circuit here and there vs. tens of thousands of them. Your random GigE of backhaul will likely go for $1500 - $4k/month, depending on build requirements. A second GigE on the same route is going to be significantly less. A 10GigE is going to be $3k - $8k. I've had 10GigEs com in under that. If they were smart, they'd go from GigE to 10GigE and use some microwave backhaul to aggregate sites.

Verizon has been moving to raw dark fiber, allowing them to light it themselves, be it 10GigE, 40GigE, Nx10GigE, etc. Obviously those are all that the very high end of what is needed for cell backhaul outside of a venue, but Verizon will have more control over their network and their pricing by going dark.

Dark in a 20 year IRU format will range from $800/strand mile to $10k+/strand mile, all depending on routes. Difficulty and competition play the big parts here. I thought I saw it was a $145M deal, but I can't find that now.

Sent from my Nexus 6 using Tapatalk

Sign In

Internet Infrastructure AMAA (ask me almost anything)

Recommended Posts

mhammett

mhammett

mhammett

belusnecropolis

S4GRU

red_dog007

irev210

mhammett

dkyeager

radem

afazel

mhammett

mhammett

mhammett

dkyeager

mhammett

mhammett

radem

mhammett

dkyeager

bretton88

lordsutch

mhammett

mhammett

mhammett

Join the conversation

Posts

Recently Browsing

The Hub

Activity