On Rad's Radar? - outage Archives

Are Outages The New Normal?

2013-04-24T04:12:15Z

This is a guest column from Peter Eisengrein, SVP Network Operations & Design, Evolve IP

Last week one of the nation's largest carriers experienced an outage that affected tens, if not hundreds of thousands of Voice over IP users, maybe more. At least one carrier employee dubbed the outage "catastrophic" yet the news media shrugged. While not exactly a reliable news source, even social media, which is at least a quick indicator of newsworthy events, hardly noticed.

How can it be that a "catastrophic" outage that is so far reaching never made the news? Perhaps it is because it was a busy news week covering an actual catastrophe, the tragic Boston Marathon bombing. If this had been Google or Facebook or Twitter, however, it probably would've made headlines. People that were impacted by the outage certainly noticed, though. Maybe we've just become jaded to "typical" outages that are not caused by nefarious acts of hacking, and maybe vast network outages are the new normal.

The unofficial cause of the outage (the official reason for outage (RFO) has not been released, at the time this is being written) was "the result of a DNS issue" which prevented calls from the carrier's PSTN gateways from completing for nearly two hours. The same source that called the outage "catastrophic" also suggested that this DNS issue may have actually been a denial of service attack; it seems unlikely, even if this is true - and at the moment it is pure hearsay - that it will be included in the RFO. Why? If you were to Google that carrier + DDOS you would find that there is a complete business practice focused on DDOS protection.

DNS is a particularly curious cause since some (many?) of the carrier's customers and service providers connect via IP addresses, not hostnames, and therefore DNS services are not needed. So, perhaps this had more to do with routing of calls within the carrier's network as opposed to access routes to competitive VoIP providers and enterprises. Whatever the root cause is determined to be, it is clear that there is still work to be done to prevent these kinds of problems. Is this the new normal? I don't think so. While it is still not infallible and problems with core components and services, such as DNS, can have a significant impact, a distributed VoIP network offers a greater level of fault tolerance than traditional services ever could. And it will only get better as we learn from these outages.

Will Sandy Rain on Cloud Adoption?

2012-11-01T17:09:19Z

Storm Sandy has flooded NYC and taken out power, which has resulted in quite a few data centers to have operational issues (i.e., stop working). The data centers experiencing outages include 75 Broad Street; 33 Whitehall (Cogent); 882 3rd Ave (nLayer, XO, Cogent, Verizon, Sidera and AT&T); 111 8th Ave (Voxel, Internap); and 121 Varick. [Paetec/Windstream are having outages but no idea what data center these use. Info about outages from slashdot, CNN and here.]

This is a massive outage - and I have to wonder why the lessons of both 9/11 in NYC and Katrin in New Orleans have yet to be applied.

That said, will this disaster cause people to more likely adopt cloud or shy away from it?

Some responses from the VoIP community:

"It could encourage companies to pay attention to their Operations team and build geographically diverse POP's that can handle a disaster in another in a different region." Shouldn't they have learned that in 9/11/2001 and again in 2005 for Katrina?

"Overall I predict a huge uptick in cloud migration. The people whose buildings and servers are under water or cut off from the world will see the value of having that off-site." That may lead to collocation sales, not cloud sales.

"The cloud companies who only had one data center and are now flooded will be out of business shortly. Just like the VoIP companies that are in that situation." Good point.

"People will be attuned to the conceptual irony of the fact that clouds are bad for cloud computing?" Which was one of the survey results from Citrix - consumers think cloudy days affect cloud services. well, 75 mph winds and high tide certainly do! Hard to believe in 2012 that data centers still have required gear in the basement!

The irony is that some of the companies with outages sell disaster recovery!

The CLEC Space is Changing

2012-09-21T19:03:51Z

The FCC has removed its rules against CLEC-Cable mergers. Right now the M&A department at Comcast, flush with cash, is picking out its next target. "Hey, Guys! Call me if you need help with valuations or choices!"

Sprint and VZ and 42K other companies are selling Office 365. This consultant says to short that stock. "Resting on existing revenue streams, they are becoming good at expense management, but aren't finding meaningful growth." Which is True. "Telecom are generally stuck in the legacy thinking that their role is to choose vendor technologies, operationalize them, and sell them to the masses." Which is also True due to Monopoly Mindset or what I call Bell-Head Thinking. "It does not cost that much money for new entrants to come into the marketplace and replace telecom in nearly every sense at a fraction of the cost structure. Hope is not a strategy. Real disruption is on the horizon. Great leaders will embrace the chaos and create new markets for their companies, and will assemble a team of people who have fun doing it." (I work with some of the smart ones!) Razorsight says that Pipe revenue will always drive most Duopoly revenue, since their margins on it are high.

On the flip side, CLEC business hasn't turned out as expected, says Gary Kim. A majority of the CLEC business was UNE-P until 2004 - and most of that went to consumers of AT&T and MCI. Cable owns that consumer voice business of the CLECs today. Cable is turning its attention to small business, which it will dominate to the chagrin of CLEC's and ILEC's alike. Cloud and managed services will be important for CLEC's and ILEC's for a couple of reasons: (1) staying relevant to the marketplace; (2) a non-facilities revenue stream; (3) an avenue of differentiation (if they actually take it); and (4) a way to reduce churn and increase ARPU with customers.

Script out Change by Dan Heath for Comcast Business

AT&T Southeast experienced a huge Metro Ethernet outage in Atlanta and South Florida yesterday. AT&T reported that it was a core Cisco router. This is the third outage for AT&T in the last few months. See here and there. What is formerly BellSouth's Metro Ethernet surprisingly is not resilient.

Final piece of news in the CLEC space: Inteliquent Announces Changes to Its Management Team. Inteliquent is the new name of Neutral Tandem and Tinet. Inteliquent's President/COO, Surendra Saboo, and the CFO, Robert M. Junkroski, are both stepping down on October 1.

What's Up With Level3?

2012-07-17T02:42:05Z

What's up with Level3? Not its income or free cash flow. (But its long term debt is up to $8.426B).

The acquisition of Global Crossing was going to fix all of that - or so they said. The brains behind this move forgot a few things:

Level3 is awful at integration! Didn't we all learn thsi before?
Far too many customers had both GX and L3 for redundancy. They are moving to other providers, so L3 is losing the combined revenue.
Repeatedly turning networks and routes up and down creates a bad overall network that customers complain about and leave.

I was going to say that at least L3 did not play around with its VoIP networks or its CDN. I can't. L3's customers suffered a huge outage in Florida recently that blackholed all the VoIP numbers and Internet traffic for over 4 hours. I know because I had customers complain - then ask for quotes on replacements.

The two sales forces plus the Channel could have equaled more, but integration; network issues; and replacement quotes have kind of squelched that fire.

Is bankruptcy now an option?

Has Verizon Stopped Repairing Copper?

2012-02-17T21:08:18Z

Over and over, I am hearing that Verizon has given up on copper. From repair issues to DSL to stripping copper out when FiOS is installed, the story seems to point to VZ looking to forget its copper plant.

in a discussion on LinkedIn about SLA's, one agent had this to say, "The absolute WORST cases I have seen have all been in the northeast where Verizon's copper is concerned. Verizon seems to have made the decision to put all efforts and funds behind their fiber build out (a good thing) but have completely sacrificed the quality behind their copper services such as T1. If your copper T1 goes down in New York, you might has well throw your hands up in prayer, because that's the only thing that will get it fixed."

Another commenter wrote, "Verizon in some places is actively ripping up copper as they lay fiber because they are not required to resell fiber to CLECs and ISPs at wholesale rates." This has been widely reported, because VZ doesn't want the expense of running to networks - copper and fiber. Plus the fiber doesn't have to be shared and the copper does. The copper means competition. Fiber means they just have to worry about cablecos, who quite frankly are kicking their butt.

Wholesale used to be a healthy business for ILEC's. Today, neither cablecos nor ILEC's want to wholesale anything. In fact, clients of mine in VZ regions have a lot of issues.

For example, "We had an outage about 3 weeks ago that lasted more than three days. This also affected [another local ISP] as I spoke him last night about the current outage. We [both have] a bunch [of customers still] out of service as well. They have been out of service since Monday. The last outage caused an exodus of customers and this one will do the same. Our guys have put in tickets, called to escalate many times. .... no one at VZ will listen. Ever. They simply close the tickets that we open."

It's a systemic problem - widespread - from the C-Suite down - the story has been that every company -- even wholesale customers - are the enemy and the Union and on-union workers must do everything they can to make it uncomfortable unless you are a direct VZ customer.

We have the case of a BK CLEC who had recorded conversations with VZ employees soliciting a customer who was down saying that it wouldn't happen if they were with VZ. [This has been a problem with both RBOC's since I got into telecom in 1999.]

Verizon faces up to $400,000 in fines after New York's Public Service Commission accused the company of not making service repairs in a timely fashion.

What do you do when the RBOC doesn't want to wholesale, doesn't want to repair, and just looks at the bottom line and the few metrics that Wall Street analysts can understand??

Many states don't even regulate the ILEC any more, so what do they do? It becomes the job of the FTC, the FCC and the court system. Talk about a deck stacked against the customer!

When our underlying telecommunications structure suffers, so too does our economic growth.

here's 2 problems with a fiber only strategy for an ILEC:

One, fiber goes out with power, so no 911 or dial-tone when the lights go out.

Two, the installation period for fiber is wicked long. Copper can be installed within two weeks. Fiber takes months. That hurts businesses. I have one moving in 3 weeks and to get 20MB of bandwidth he has to wait months. That won't work.

Ever think that just nothing in this country makes sense any more?

In the discussion about SLA's, the conclusion is to convince your clients to buy redundancy: 2 pipes. That's nice in theory but not in reality. The thing is that you have to set the expectation that if Internet or VoIP is integral to their business operations, no SLA is going to save them, redundancy and business continuity planning will. Otherwise, an outage will be a disaster that they have not planned for. It is not IF, it is WHEN.

Privacy versus Security

2012-01-25T20:51:22Z

"Concerned about privacy? Maybe you should be concerned about the economy instead. That was the subtext of a keynote speech by Sheryl Sandberg, Facebook’s chief operating officer, at a technology conference in the heart of Europe." [Benton]

Not to get political, but privacy is a ship that has sailed. Google, Facebook and other sites track online activities. Credit card companies and Paypal can track both offline and online activities. CarrierIQ software on cell phones is able to see every keystroke. GPS tracking on devices, including your TomTom. Cellphone and Internet logging by ISP's for government agencies track you.

I'm not suggesting we just give up on privacy. I am suggesting that we take the advice of Sandberg and start focusing on the important stuff. Maybe Security is the Important stuff.

When Wikipedia went dark in the face of SOPA and PIPA bills, it caused many supporters in DC to publicly back away. Privately, however, they will back TransPacific Partnership, ACTA and other bills. Why? Hollywood and content companies want it. Just another example of industry influence over voters or common sense in DC.

In the midst of the privacy talk, why isn't there more talk about security? The latest breach at Zappos affected 24 million. Laptops and cellphones are lost daily. Experts agree that mobile threats and breaches are inevitable. We don't need more rules or laws, we have plenty now, including HIPAA and PCI DSS. We need enforcement and monitoring - or really big fines.

There are simple methods for security available, it might be time to do so. "

""I am angry. There are real problems facing the world, and we, as a society, are not doing enough to address them in the right ways, not the ways we know are possible. The old way isn't working, and we know it." This is the opening paragraph to a new ChangeThis manifesto titled Shift & Reset by Brian Reich. "What might be possible if we were really committed, as individuals and as a society?"

From another ChangeThis manifesto, "The real business opportunity is to become more relevant and meaningful to customers in ways that create sales."

Verizon's Bad Month

2011-12-30T20:54:45Z

VZW has had three outages this month. The third outage on its LTE network today (12/28/11). There was another LTE outage on 12/21 and VZ Long Distance in Florida was down on 12/8/11.

Then yesterday VZW announced that it would charge a convenience fee of $2 for not having auto-pay. Just a few minutes ago, it rescinded that idea. The power of social media and a slow news cycle means that VZW will have to wait to implement that one. Even this guy agrees.

There was also some scam that Phone Plus reported with VZW and revenue reported by Verizon Comm. Inc.

Not a good month for the giant's PR department at all.

Juniper, TWC Have a Bad Day

2011-11-07T21:05:38Z

TW Cable had a nationwide outage this morning as reported on twitter and TheVerge. Internet slowdowns and outages were still being reported by Noon Eastern time.

According to the NANOG chatterbox, Level3 & TWC are having the same problem: it appears that Juniper routers are doing a core dump due to a BGP advertisement that triggers a bug. BGP peering causing a number of crashes on the Juniper platform for the last year or so. Unsure if that's related to this problem or not, but the saying one hears frequently is "IPv6 bgp peering isn't as widely tested as the vendors would have you believe"."

I'd use this outage to sell redundancy!!!

InterOp InterNAP Rumor (is False)

2011-05-12T19:19:35Z

At InterOp in Vegas this morning, there is talk that InterNAP shut down its network. It has closed its peering points and going to focus on its cloud strategy of data centers and managed services. I'll have to work on confirming it.

UPDATE: My clients - at least 3 - who use InterNAP say that it is up and running still - even the peering points. So the rumor is false.

UPDATE from email: Internap, AS22212, will be leaving NOTA on May 6, 2011. Please feel free to remove any peering sessions with Internap, AS22212, at your convenience.

Internap, AS22212, will be leaving NYIIX on May 6, 2011. Please feel free to remove any peering sessions with Internap, AS22212, at your convenience.

Thanks and it was a pleasure peering with you. - INTERNAP Network Services

Data Center Certs

2011-05-06T16:50:20Z

I am working on a data center project for a client. Evaluating data centers is kind of funny, especially the certifications. SAS70, ISO, LEEDS and PCI Compliance.

I don't think people even understand that SAS70 means that a CPA has audited the procedures and processes to ensure that there are indeed written processes in place and that at least some of them are followed. Now some of you will argue you with me about this, but the reality is, my evaluation is pretty accurate. It's a goofy deal to spend between $15k and $35K for the cert. Why?

Well, most of the regulations - like HIPAA, SOX, and PCI Compliance - are based around Data Security, not transport security. Let me give you an example: HIPAA doesn't distinguish between a paper file that is left on a desk and gets misplaced or read by the wrong eyes and the electronic file on a server in a data center. PCI Compliance has to do with stored data - much like how merchants have to protect against employees stealing carbon copies and properly disposing of credit card imprints. We just concern ourselves more with the electronic theft. Why? Probably because it is more intangible.

That being said, why is SAS70 so important? I think it is about appearnace more than anything. If there is a data breach, and your servers were at a non-certified facility, the executive that made that decision would look like an idiot. It's about appearance. And to some extent Marketing.

It's kind of the same with LEEDS. How energy efficient is a data center? It's sole purpose is to provide space, ping and power. Any data center wants to be energy efficient, because it drives the bottom line. Now it is also a marketing bit.

Physical security is important, of course. A locked cabinet at the minimum, but that won't stop someone from throwing coffee in the cabinet and burning out your server. There's always a loophole.

Think about how important Internet Access and Data Backup are to businesses. The lessons of Katrina and other disatsers have prominently demonstrated that water damage will destroy all your files and you will be out of business.

Internet Access is a utility - and the cost to some businesses to be out of Internet Access is tens of thousands of dollars. Some businesses have even experienced that but still refuse to buy redundancy.

I guess that redundancy, security, encryption, and backup are seen as unnecessary - until it happens to you - and more like insurance - good to have if you want to pay for it, but if it isn't a legal requirement (like email archiving is for financial planning firms), then most companies forego it.

So why would the data centers need to worry about the certs then? Marketing for sure; but also because it is a check box on many RFP's. But the story they have to start telling is about process and security instead.

Even uptime as a check box is funny. Most consumers are used to TDM uptime of five nines (99.999%) and with line powered POTS service for voice. Even today, most consumers have not come to grips with the fact that VoIP and cell service are NOT five nines. The tower is not a central office. At four nines that means about 1 hour of downtime per year. At three nines that's almost 9 hours per year of downtime. Advertised 100% uptime is great, but we have seen many outages in the last year from even the big names like Rackspace, Amazon and Google. (Not to mention Facebook and twitter).

As we move to Cloud services, outages will increase. Why? More vendors in it means more competition, which means commodity pricing will drive down margin and providers will cut costs in staff, equipment and redundancy. As one CEO put it, Resiliency and redundancy are nice, but very expensive. Even security is an expensive deal. I don't know what kind of security Epsilon had, but all that data was hacked -- just like TJ Maxx and many others. So what it was a SAS70 data center? If you don't secure the data properly (and back it up), the server might as well have been in the hallway.

This was just food for thought as I diligently research for my project.

The Inexpensive Cloud

2011-04-21T14:20:11Z

Image via Wikipedia

Amazon Web Services including EC2 is down today. When Gmail has any failure my twitter stream goes nuts. Facebook collapses often.

There is one thing people should keep in mind: Building Resiliency and Survivability into a Cloud Platform is not cheap.

For Agents, used to selling TDM with five nines reliability, moving to VoIP with less than that will be a shock, especially when they find out after the first outage.

For VAR's, who have been running their own servers, and like visitors to Vegas who tell me they are up money, will tell you about never having any down time, moving customers to the Cloud, where you lack control and transparency, will be unsettling.

Everything breaks or burns out - that's the nature of commodities (and servers, power supplies, hard drives and NIC's ARE commodities). Outages will occur. The Cloud Provider has to provide updates (honest updates) - and the Channel will need to communicate with the customer. It will just be a shock the first time.

Let's just say that a cloud provider doing less than $8M in revenue is not going to have the finance or propensity to design in high-tech security, survivability, and resiliency, because it will be just too expensive. (This might explain why the finance industry is taking slow steps toward cloud use).

Let me explain. Security is a full-time job. Administering password management is just one aspect. The network gear is expensive - IDS, firewall, router, switch, etc. And in redundancy?! Twice or four times as expensive. (Two per data center as a back up and two data centers equals 4.) Add monitoring all those parts. Then securing the data, encrypting it, backing it up, off-site storage, etc. Not only expensive, but a challenge to manage.

And for Cloud Communications providers, dual data centers is more expensive when you consider that the softswitch is $250K and rides on a blade server (x2 for redundancy at each data center). Plus other gear like SBC's which aren't cheap either.

Even Google and Amazon can't keep everything up and running. Even these two don't have enough survivability and failover built-in to takeover after an outage.

This will be a problem for Agents selling Cloud. Agents and Customers will have to come to grips with the fact that the Era of Five-9's is ending. Luckily, consumers are so used to dropped calls and other glitches on the Internet, that perhaps this will lighten the blow. (Much like how inferior cellphone call quality allowed VoIP to take hold).

Expectations in 2011 with Jon Arnold

2010-12-16T01:43:24Z

Jon Arnold joins me for a chat about what to expect in 2011. We touch on Acme Packet, SBC, security, outages, IPv4, Cloud and Social CRM. It's a full 25 minutes.

Verizon Starts Blocking

2010-06-09T18:52:09Z

Comcast won in court over the FCC on the issue of traffic shaping, so Verizon started blocking ports this week.

First, VZ started blocking port 25 yesterday on their FiOS and DSL networks. Now VoIP Traffic is being blocked.

From an ITSP, "We started getting calls today, and it seems that Verizon has started blocking ports, traffic shaping, something, but it is interfering with voice service, email, and some web traffic. One example is a customer that we are no longer able to ping, but we are polling their router via SNMP. For a few minutes I could not access our switch due to the FiOS at our office."

So it looks like an attack on over-the-top services. Welcome to the walled garden!

This is why you need competition -- and competition is not a Duopoly!

Otherwise you have to try to regulate Net Neutrality.

Imagine if this blocking was preventing access to email servers, tele-presence, or your applications in the cloud. Companies would be totally stuck.

A Big Outage Day!

2010-06-09T18:39:34Z

Twitter is down. (And a few million people have nothing to do).

Facebook is experiencing issues as well. I guess people will just have to work today.

AT&T's cellular network is down in Tampa Bay. (And a few thousand iPhone users are cursing.)

A point: you can't count on another network - twitter, Facebook, Google or any other - for your business. It's outside your control - and at the price of free, well, you can't have expectations of four 9's uptime. (As you can see, AT&T can't even keep its cellular network up four 9's.) NING should have been a lesson to folks: when Ning stopped its free option, many networks had to move or pay. You should expect the same in the future -- or build it yourself or have a back-up plan. But foremost the content you build should be on your own site (or backed up).