Welcome to the second entry in this series on “wargaming” WAN scenarios to make sure you’re getting the most bandwidth bang for your buck.
If you didn’t read the set-up to the series, please go back and check that out. This will all make a lot more sense if you do.
The first actual scenario we’re going to tackle is adopting local internet breakouts in an MPLS-DIA hybrid WAN. Let’s dive in.
Adding Local Internet Breakouts to Your MPLS WAN
Many WAN managers we talk to have already begun implementing local internet breakouts, even before SD-WAN adoption.
A recent interviewee from our 2019 WAN Manager Survey—which is coming soon to a browser near you—explained that their adoption of local breakouts with DIA was a result of changes in compute and security. “[W]e used to be a Citrix house connecting a hosted desktop environment to the data center—and the internet was in the data center, in that case,” they said. “But we changed to Windows 10 desktops and mobile devices, which meant [that] backhauling to the data center didn’t make sense anymore. So we implemented breakouts in all of the offices. The other component of that is Zscaler, which provides our security layout for that.”
"We changed to Windows 10 desktops and mobile devices, which meant [that] backhauling to the data center didn’t make sense anymore. So we implemented breakouts in all of the offices."
Another 2019 interviewee mentioned that network service to their sites in the U.S. frequently went down. According to this interviewee, an auto accident might take out an entire site or local construction will cause an outage. Similarly, carriers will relocate circuits and fiber pairs to be more diverse. "This has happened a lot with [carriers]," they said. "They say they're mandated by a bridge law to move circuits."
While compute and physical diversity are concerns, getting the most bandwidth for the lowest price is certainly a motivator, as well.
"I didn’t feel our telecom [spend] before MPLS was out of whack, but we more than tripled our intranet bandwidth, got our internet up 40 times, and we are doing it for six figures less a month.”
One of our interviewees from our 2018 survey indicated that moving to local internet breakouts with DIA was mostly a most-bandwidth-for-the-best-price move. “We were dedicated to tripling our intranet bandwidth, our internet capacities would go up 40 times, and we would do it at the same price,” said this interviewee. “We had always been pretty tight with telecoms. We would threaten an RFP every half a year and used benchmarking services like TeleGeography. I didn’t feel our telecom [spend] before MPLS was out of whack, but we more than tripled our intranet bandwidth, got our internet up 40 times, and we are doing it for six figures less a month.”
What Does This Mean?
To distill these ideas into some quick lessons about why we see WAN managers shifting from all-MPLS to MPLS-DIA hybrid WANs:
- Enterprises have adopted SaaS applications, many of which grew out of consumer applications running over the public internet and have been optimized to work over local internet connections rather than through the data center.
- Security used to be located in the data center with the centralized internet breakout, but now local breakouts can be centrally secured from the cloud out to each individual site.
- Even beyond business apps, most large organizations have a lot of traffic that is bound for the public internet rather than the intranet.
- DIA is often cheaper than MPLS, so why pay for best efforts traffic to go over expensive private links?
- Many enterprises are squeezed for capacity and adding additional MPLS capacity comes at a much greater cost than increasing with internet.
So, here in our first scenario, we’re assuming that local internet breakouts are DIA, not business broadband. DIA is the carrier-grade internet answer to MPLS VPN and, unlike business broadband, it’s always symmetrical, uncontested, and comes with carrier-grade SLAs.
But is it actually cheaper?
We get this question a lot, especially from customers who might have heard that—at least at a best efforts CoS—MPLS VPN and DIA prices had largely converged. I won’t make you wait, DIA is still cheaper than MPLS, though the amount varies by location and port size.
Let’s see how it works in a realistic WAN scenario where we boost the available site capacity.
Comparing Costs for a Bigger Hybrid WAN
To test out the MPLS-DIA hybrid WAN we made five changes to the Original MPLS WAN:
- We lowered many MPLS ports sizes, sometimes by as much as 50%. These adjustments tended to be in the middle ranges of 5-10 and 11-50 Mbps. Some small-capacity sites were kept as-is; we couldn't make them any smaller.
- For the large hub sites at 1,000 Mbps we replaced MPLS with DIA entirely.
- We adjusted the CoS levels accordingly. Instead of 10/30/60, we shifted to 30% voice/video, 50% real time data, and 20% best efforts so we could push more CoS-sensitive traffic over the smaller MPLS ports.
- We added DIA to all 140 MPLS sites. Usually, the DIA port size at least matched the original MPLS port size, but in some instances we added capacity beyond the original MPLS port size.
- We lowered all access line speeds accordingly, and added an access line for each new DIA port. For this reason, most sites now have three access lines: one MPLS primary, one MPLS backup, and to connect the DIA port.
These changes resulted in a network that had many MPLS and DIA ports in the 5-10 Mbps range, but are added together for double the site capacity.
For example, if a site has 10 Mbps of MPLS VPN and 10 Mbps of DIA, the total site capacity is 20 Mbps. There are still quite a few ports in the 51-100 Mbps range, as well. The average total available capacity per site ended up almost doubling from 125 Mbps in the original network to 241 Mbps in the new one.
How Did We Change Capacity Ranges from the Original MPLS WAN?
Original MPLS WAN Site Capacity Range
MPLS DIA Hybrid WAN Site Capacity Range
Much of the capacity increase was in the U.S. & Canada, Western Europe, and East Asia, where the headquarters and data center sites are located.
Only sub-Saharan Africa had the same average site capacity.
How Does Average Site Capacity Change by Subregion?
MPLS-DIA Hybrid WAN Average Total Site Capacity by Subregion
That increase in capacity came at a price.
The resulting annual total cost of ownership (TCO) is 48% higher than the original MPLS cost. Keep in mind this represents a nearly 100% increase in capacity. The chart shows that, while MPLS costs go down and DIA costs increase slightly, it is access that really contributes to the higher TCO.
How Does Adding Capacity With DIA Affect the Total Cost of Ownership
Original MPLS WAN & MPLS DIA Hybrid WAN Annual TCOs
What if your DIA service was on-net and did not require an access line?
When we first started tracking DIA in 2006 carriers exclusively sold it as a “port-based” product with an additional charge for local access. However, in the past several years, many carriers and ISPs have started to sell DIA as a bundled service when it is “on-net,” or in a building in which they have a presence. Particularly for those customers that are in downtown areas, multi-tenant office buildings, office parks, or other locations where there are likely to be competitive carriers, it is increasingly common to be able to get on-net DIA service without any additional local access charge.
Particularly for those customers that are in downtown areas, multi-tenant office buildings, office parks, or other locations where there are likely to be competitive carriers, it is increasingly common to be able to get on-net DIA service without any additional local access charge.
Also, in some cases, carriers source on-net DIA from an aggregator and pass on that pricing. The port charge we show is equivalent to the total charge in these cases. The access portion of a DIA site cost can range from 25% to 75%—or more—so if you can buy on-net DIA service, you will likely save considerably.
With that in mind, we re-ran this exact scenario, but took out the access lines in regions and locations where it seemed feasible. We left access lines in place in Africa and remote sites (16-50 km access distance band or above) and removed the others. About 25 sites ended up keeping access lines to DIA.
What Happens When We Increase Capacity Using Mostly On-Net DIA?
Original MPLS WAN, MPLS DIA Hybrid WAN, & MPLS On-net DIA Hybrid WAN Annual TCOs
It shouldn’t be surprising that this network, while still nearly doubling available site capacity, increases the TCO much less at 13%. While increasing site capability to handle SaaS, streamlining security by moving it out of the data center, offering sites increased protection from outages, and significantly increasing available site capacity we’ve only increased the TCO moderately. Not bad!
Next time we’ll look at local internet breakouts using cheaper (but perhaps less reliable) business broadband to see how we might do these things while still lowering costs.
Next up:
- Wargaming Your WAN: MPLS-Broadband Edition
- Adding SD-WAN, Keeping MPLS
- How to do SD-WAN Without Totally Losing MPLS
- What Happens When You Leave MPLS Behind?
- Aggregator Broadband & SD-WAN
- SD-WAN Management Costs, Wargamed
Greg Bryan
Greg is Senior Manager, Enterprise Research at TeleGeography. He's spent the last decade and a half at TeleGeography developing many of our pricing products and reports about enterprise networks. He is a frequent speaker at conferences about corporate wide area networks and enterprise telecom services. He also hosts our podcast, TeleGeography Explains the Internet.