"Only two months into 2017 and already 13 million people have had personal information compromised. Attackers breached 15 companies in February alone. Among them, the biggest names included popular music festival Coachella, restaurant chain Arby's, and the InterContinental Hotel Group."
I was one of 393,430,309 people pwned in the River City Media Spam List data breach. The Compromised data consisted of Email addresses, IP addresses, Names, Physical addresses.
"In January 2017, a massive trove of data from River City Media was found exposed online. The data was found to contain almost 1.4 billion records including email and IP addresses, names and physical addresses, all of which was used as part of an enormous spam operation. Once de-duplicated, there were 393 million unique email addresses within the exposed data." [source]
This warning came with the notice: "When financial information is in question, it's important that data breach victims monitor their bank accounts and credit cards for fraudulent activity. Contact your bank or credit provider if you see anything that looks odd." But I didn't get a notice of the breach from River City Media or anyone. I got the notice from a monitoring system I signed up for.
Also with the notice: "Why are you only hearing about this now? Whilst the breach occurred in January, sometimes there can be a lengthy lead time of months or even years before the data is disclosed publicly." Like Yahoo or any of the EMR systems.
I just found out about the September 2016 breach to NetProspex. "In 2016, a list of over 33 million individuals in corporate America sourced from Dun & Bradstreet's NetProspex service was leaked online. D&B believe the targeted marketing data was lost by a customer who purchased it from them. It contained extensive personal and corporate information including names, email addresses, job titles and general information about the employer." The Compromised data: Email addresses, Employers, Job titles, Names, Phone numbers, Physical addresses. Did D&B or NetProspex contact me? NO!!
Currently there is a successful GMAIL phishing scam going on. See more at LifeHacker. We need more user training on how to handle email and data.
One Tampa firm, BayCare, did an assessment of their workers' knowledge of scam email. It went sideways. But you SHOULD be checking to see if workers follow safe email procedures, since that is the Number 1 way that hacks occur!
There is more:
Privacy hawks in Congress call on Homeland Security to warn Americans of SS7 hacking threat.
You Won't Believe How Many Organizations Have Experienced Large Cyber-attacks by Rich Tehrani
For its 2016 Verizon Data Breach Investigations Report (available for download here), Verizon used a final data set of 64,199 security incidents and 2,260 data breaches.
The problem with selling cyber-security solutions is (A) the cost; and (B) no one thinks it will happen to them. You have to sell it like Life Insurance. Sir, one day you will die. Then what?
No one is immune.
From VZE: "Take a look through the list of published data breaches and one thing will immediately strike you: no location, industry or organization is immune from attack. Even with the strongest defenses, you can't bank on not being breached. But you can deter the criminals."
]]>CenturyLink just sold off its data center business that was a combo of Qwest Cyber Centers and SAVVIS to a group of PE firms for $2.15B in cash and C-Link keeps a minority stake worth $150M in the new company. CL bought Savvis for $2.5B in 2011. Buy High; Sell Low. Bell-Head Mentality.
The PE coalition that bought the data centers also grabbed 4 cyber-security firms in order to announce this global security co, to be run by Manny Medina, former CEO of Terremark Worldwide.
Wired's headline says it best: The World's Telecoms Are Under Threat From All Sides.
Broadband, cellular and voice are all flat or declining markets.
IAAS and PAAS are ruled by Amazon, IBM and Google. Microsoft only got into the game recently and is doing better than all the telco's combined.
PE firms are buying up data centers as the world adjust to cloud computing, an app market and streaming TV and radio.
DDoS attacks are happening too often. So are Hacks. There are not enough fingers to fill all the holes in this dyke.
UCaaS is ruled by 8x8, Vonage Business, RingCentral, Fuze and a bunch of other providers that are not a telco. The PBX market may be shrinking but not fast enough for the other Hosted VoIP players. Cisco and Microsoft have chunks of the enterprise UCaaS business that the telcos don't.
Comcast Business is at $6B in annual revenue, which makes it a bigger CLEC than almost all that are left. WIND does $5B. EarthLink less than $1B. Birch and TelePacific are private. Level3 does $8B. CenturyLink does $17B (much of it ILEC revenue). Zayo is $2B.
Apps like Messenger, WhatsApp, Skype and Slack are replacing voice and SMS and even email. It is a topsy-turvy world. What's a telco to do? Well, merge! Get bigger because bigger solves nothing, but it makes money for top execs in the C-Suite and the Board room and on Wall Street.
Our economy spins on e-commerce and the Internet. When the companies that provide that Internet are too clunky to do it properly, what happens to our economy?
We went from a five nines voice network of reliability to cell phones and VoIP that quite frankly can't be more than three nines. Have you noticed the number of outages lately by telcos and cablecos?
There is a lot going on. There are many areas of opportunity, but the fall back from these guys is "more of the same", "do what I know" and "one more quarter!". None of these transactions is good for the industry, the economy or the consumers. They are stop gap, short term money movers. We are going to wake up shortly and realize that it is 1970 all over again. It makes the NSA job easier when there are few players, but what about the customers?
In the data center space, one master agency contacted me after the C-Link announcement to tell me that the folks at CenturyLink have no details about the sale. How can that be when Monroe has been trying to sell the DC division all year? Great planning, guys!
Whose customer is it? Will the agent still get paid? Will the customer see a price increase? Who is the billing entity? Who will the customer be paying? These are good questions that bothered some TELX customers when Digital Realty took over.
I keep seeing executives at master agencies say these deals are good. Do they say that in print because they have to?
Don't forget that you can leave a public comment with the FCC on any of these mergers. You can voice your opinion here. You will need a docket number but you can google it after the filings are in the system.
]]>Krebs experienced an IOT botnet attack earlier this month. An ISP client was under two DDoS attacks in August.
These attacks are increasing in frequency -- and are not going away. This will be normal business soon.
Email and iPhone hacks are in the news.
What are you doing to protect your clients?
Quite a few data centers offer a DDoS Mitigation service. (So does Level3).
There are a number of managed security service offerings - from firewall to IDS to UTM* - available from a number of providers.
In a time when bandwidth pricing is decreasing -- and customers want to spend less -- someone needs to bring up the topic of security and redundancy. Why not you?
*Intrusion Detection Service and Unified Threat Management
]]>Do business consider this? Not often enough. As a telecom consultant, it is my job to point it out to the business decision maker. How much is an hour of down time worth?
Last week, Comcast suffered a massive voice outage nationally. Windstream, CenturyLink and others have all had widely reported outages this year. Outages happen more often now because of a best effort mentality. No more five-nines.
Hacks happen every day. Unprotected computers are infected almost immediately. No one thinks it will happen to them. It is the hurdle in selling security (and insurance). Why worry? It won't happen here.
Ramsonware is scary. It is occurring more often. A smart backup can alleviate some of the hardship.
Business Continuity (BC/DR) is becoming more important for businesses every day if they recognize it. One indicator: do they utilize battery backups everywhere? Then start the conversation there.
SD-WAN will solve some security, fail-over and BC issues. It might be time to learn about how.
On the carrier side, AT&T is going deep into SDN and NFV. AT&T is releasing their SDN software, ECOMP, to open source. They are talking about being able to use white box CPE globally. Via 4G that CPE will download the necessary software to be a router or firewall or what-have-you.
TelePacific Rolls Out Advanced SD-WAN Connectivity in Nationwide Pilot.
SD-WAN is fueling 150% growth at Aryaka.
Verizon is partnering with Viptela to offer a hosted SD-WAN service.
Masergy, XO, CenturyLink and Mettel - even Vonage - have all added SD-WAN overlay services. Don't you think you should know what they are when your mid-market clients ask?
Join us for another Blab! about SD-WAN on 8/3 at Noon ET.
]]>We went from TDM to VoIP to Hosted PBX to UCaaS to UC&C.
We went from T1 to cable broadband to Gigabit.
The consolidation of cable will tighten the market in 15 to 18 months. (It takes that long for integrations to take hold.) Now if the integrations are not a big fail, then cable - New Charter/Spectrum, Comcast, Altice - will ratchet up the competition in the small business market for triple play.
"Cable/MSOs are the fastest growing providers in the business services market, with much of their recent success in the mid-size business space," reported MarketResearch. Think about that: the mid-sized space - not just the small business segment of the market.
Of the $104 Billion total businesses spent on telecom services in the US in 2014, AT&T had the largest share (33%), followed by Verizon (22%) and the rest of the LEC band of brothers (Level3, CenturyLink, Sprint, Windstream). MSOs have more than $12 Billion of that pie, with the lion share - $5B - going to Comcast coffers alone.
SIP anyone? 54% of business cable subscribers also use cable for voice, the report states. That means less than half the businesses using cable are buying voice from another provider. That is a shrinking opportunity for the 2000 Hosted VoIP players in the US.
"Last year the Cable/MSO share of businesses with 100+ employees rose to 17%, reports TNS. "The main driver behind this growth was a heavier reliance on internet service and the need for greater bandwidth; two areas where larger cable providers excel."
Telco broadband has not kept pace with cable in speed and price. Egged on by Google Fiber - and a declining market share of businesses - ILECs have started tentatively rolling out faster fiber based broadband - 100MB to 1Gigabit depending on the ILEC (Windstream versus CenturyLink or AT&T).
UPDATE: Google just rolled out Gigabit Fiber to small business starting in Charlotte in July of 2016.
The ILECs have made a tremendous CAPEX investment in TV - just as OTT TV is hitting its stride. They spent big to supply triple-play, when they could have spent the money on FTTx projects for faster bandwidth. That was just uncreative thinking. [More of that Me-too mentality ingrained in telco.]
All of this will stress ILECs, some CLECs and even some OTT VoIP players. When cable takes about 35% of the SMB market, there won't be much room left for anyone else.
In March of 2016, "During the fourth quarter, Verizon reported that total broadband connections dropped to 2.1 million as it lost more DSL subscribers after losing 94,000 DSL customers," according to Fierce media.
Verizon is transitioning. Verizon is now betting on mobile ads (AOL acquisition and Yahoo bid); 5G fixed wireless broadband replacement for wireline services; and IoT (including connected cars) to add to its coffers.
A point I make often is that the debt that the ILECs carry is crippling with flat revenues.
Think about this: Vonage has taken $800M worth of voice revenue. Twilio gets $240 million in voice revenue. This is revenue that typically would go to Level3, Verizon and AT&T (and it probably does terminate to them eventually for a smaller percentage of that money).
WebRTC is being used in so many apps to allow for video and voice calls - bypassing the traditional voice network. [And bypassing the cellco text system and dollars.]
Then, we have Cable beating Telco in broadband bandwidth. Always has in fact. Gigabit fiber will be the real winner if the telcos decide to pursue that route for real (versus in just press releases).
We have telco getting in the data center - and now we have telcos looking to get out of that business without embarrassment.
There is a Talent problem, too. There are too many musical chairs. Not only can't you set a strategy when you shift personnel that much, you can't execute on a strategy either if the cogs are constantly being replaced. (And I don't mean cogs in a bad way. It takes a lot of talent to keep the wheels spinning.) The talent drain has also resulted in a domain knowledge drain as well. Quite frankly that means they don't where things are and how things have been done to keep things working. It isn't all documented, especially fiber maps!
Let's face it, for many companies that started with an A Team, they are now running with a B or C team. Why? As Steve Jobs said, "A Players hire A Players, B players hire C players. Get it?"
People move from company to company in teams. The same routine and team may work once, but it is not often a repeatable experience. There's a reason the Cavaliers recruited LeBron back to Cleveland - and didn't hire the whole Miami Heat starting line up.
The telco organizations harbor stifling factors: monopoly mindset, legacy systems, federal accounting and regulations, departmental silos and competing internal interests. These factors do not lend themselves to attracting more A Players.
There is also a surprising lack of talent for the new services and skills needed for omni-channel marketing; omni-channel customer service; cloud, managed services, migration and integration. This lack of skill will choke growth and brands.
We see outages and hacks every day. The worry is only about getting a customer. There is little concern for retaining that customer; data security; or a resilient network (4 Nines is good enough).
Many people are choosing smaller organizations to work for. The reasons are numerous but I would think that impact and voice play a major part. In smaller businesses, any one person can have a voice and can see the impact that they are having on customers, culture, and the company. That isn't the case in larger organizations.
Flat organizations (and smaller companies) have less meetings, fewer silos, maybe more transparent governance.
Most financial experts are predicting an economic slump in 2017. It won't matter which candidate wins the Presidential election, a slump is coming. We have under-employment; increasing number of freelancers; and a stagnant wage. None of these components inspire an economic engine that is fueled by consumer spending.
ARPU for cellular, cable and VoIP segments have been fairly constant over the last 4 years worth of data I could find. Bandwidth and voice revenues are actually shrinking. Total telecom spending from 2013 to 2014 shrunk $6 Billion dollars according to MarketResearch.
Growth will be hard to find. We are seeing a price war in cellular accompanied by escalating customer acquisition costs.
Hosted VoIP is experiencing a similar battle for customers that is increasing the cost of customer acquisition. Rising SPIFFs and other compensation are being used to grab both market share and channel partner attention.
PBX vendors are NOT crashing and burning as many had predicted. Premise PBXs are still being sold and installed by a robust band of vendors - Mitel, Shortel, Avaya, 3CX, Fonality, Zultys, Panasonic, NEC, Siemens and more.
We are half way through 2016. No big winners. The Twilio IPO was a surprise. Vonage spending all of its acquisition money for the year on Nexmo, Twilio's competitor, seemed strange, since there were Broadsoft clients they could have picked off instead to take a big step forward in the race. Slack and all the Skype4B hype are little surprises.
2016 is half over - and so many companies have either done M&A or played musical chairs that I expect nothing magical to happen in the rest of 2016. And I look at all of this and wonder what 2017 holds.
ASIDE: telco versus cable consumer data.
]]>Last week one of the nation's largest carriers experienced an outage that affected tens, if not hundreds of thousands of Voice over IP users, maybe more. At least one carrier employee dubbed the outage "catastrophic" yet the news media shrugged. While not exactly a reliable news source, even social media, which is at least a quick indicator of newsworthy events, hardly noticed.
How can it be that a "catastrophic" outage that is so far reaching never made the news? Perhaps it is because it was a busy news week covering an actual catastrophe, the tragic Boston Marathon bombing. If this had been Google or Facebook or Twitter, however, it probably would've made headlines. People that were impacted by the outage certainly noticed, though. Maybe we've just become jaded to "typical" outages that are not caused by nefarious acts of hacking, and maybe vast network outages are the new normal.
The unofficial cause of the outage (the official reason for outage (RFO) has not been released, at the time this is being written) was "the result of a DNS issue" which prevented calls from the carrier's PSTN gateways from completing for nearly two hours. The same source that called the outage "catastrophic" also suggested that this DNS issue may have actually been a denial of service attack; it seems unlikely, even if this is true - and at the moment it is pure hearsay - that it will be included in the RFO. Why? If you were to Google that carrier + DDOS you would find that there is a complete business practice focused on DDOS protection.
DNS is a particularly curious cause since some (many?) of the carrier's customers and service providers connect via IP addresses, not hostnames, and therefore DNS services are not needed. So, perhaps this had more to do with routing of calls within the carrier's network as opposed to access routes to competitive VoIP providers and enterprises. Whatever the root cause is determined to be, it is clear that there is still work to be done to prevent these kinds of problems. Is this the new normal? I don't think so. While it is still not infallible and problems with core components and services, such as DNS, can have a significant impact, a distributed VoIP network offers a greater level of fault tolerance than traditional services ever could. And it will only get better as we learn from these outages.
]]>This is a massive outage - and I have to wonder why the lessons of both 9/11 in NYC and Katrin in New Orleans have yet to be applied.
That said, will this disaster cause people to more likely adopt cloud or shy away from it?
Some responses from the VoIP community:
"It could encourage companies to pay attention to their Operations team and build geographically diverse POP's that can handle a disaster in another in a different region." Shouldn't they have learned that in 9/11/2001 and again in 2005 for Katrina?
"Overall I predict a huge uptick in cloud migration. The people whose buildings and servers are under water or cut off from the world will see the value of having that off-site." That may lead to collocation sales, not cloud sales.
"The cloud companies who only had one data center and are now flooded will be out of business shortly. Just like the VoIP companies that are in that situation." Good point.
"People will be attuned to the conceptual irony of the fact that clouds are bad for cloud computing?" Which was one of the survey results from Citrix - consumers think cloudy days affect cloud services. well, 75 mph winds and high tide certainly do! Hard to believe in 2012 that data centers still have required gear in the basement!
The irony is that some of the companies with outages sell disaster recovery!
]]>Sprint and VZ and 42K other companies are selling Office 365. This consultant says to short that stock. "Resting on existing revenue streams, they are becoming good at expense management, but aren't finding meaningful growth." Which is True. "Telecom are generally stuck in the legacy thinking that their role is to choose vendor technologies, operationalize them, and sell them to the masses." Which is also True due to Monopoly Mindset or what I call Bell-Head Thinking. "It does not cost that much money for new entrants to come into the marketplace and replace telecom in nearly every sense at a fraction of the cost structure. Hope is not a strategy. Real disruption is on the horizon. Great leaders will embrace the chaos and create new markets for their companies, and will assemble a team of people who have fun doing it." (I work with some of the smart ones!) Razorsight says that Pipe revenue will always drive most Duopoly revenue, since their margins on it are high.
On the flip side, CLEC business hasn't turned out as expected, says Gary Kim. A majority of the CLEC business was UNE-P until 2004 - and most of that went to consumers of AT&T and MCI. Cable owns that consumer voice business of the CLECs today. Cable is turning its attention to small business, which it will dominate to the chagrin of CLEC's and ILEC's alike. Cloud and managed services will be important for CLEC's and ILEC's for a couple of reasons: (1) staying relevant to the marketplace; (2) a non-facilities revenue stream; (3) an avenue of differentiation (if they actually take it); and (4) a way to reduce churn and increase ARPU with customers.
Script out Change by Dan Heath for Comcast Business
AT&T Southeast experienced a huge Metro Ethernet outage in Atlanta and South Florida yesterday. AT&T reported that it was a core Cisco router. This is the third outage for AT&T in the last few months. See here and there. What is formerly BellSouth's Metro Ethernet surprisingly is not resilient.
Final piece of news in the CLEC space: Inteliquent Announces Changes to Its Management Team. Inteliquent is the new name of Neutral Tandem and Tinet. Inteliquent's President/COO, Surendra Saboo, and the CFO, Robert M. Junkroski, are both stepping down on October 1.
]]>The acquisition of Global Crossing was going to fix all of that - or so they said. The brains behind this move forgot a few things:
I was going to say that at least L3 did not play around with its VoIP networks or its CDN. I can't. L3's customers suffered a huge outage in Florida recently that blackholed all the VoIP numbers and Internet traffic for over 4 hours. I know because I had customers complain - then ask for quotes on replacements.
The two sales forces plus the Channel could have equaled more, but integration; network issues; and replacement quotes have kind of squelched that fire.
Is bankruptcy now an option?
]]>Over and over, I am hearing that Verizon has given up on copper. From repair issues to DSL to stripping copper out when FiOS is installed, the story seems to point to VZ looking to forget its copper plant.
in a discussion on LinkedIn about SLA's, one agent had this to say, "The absolute WORST cases I have seen have all been in the northeast where Verizon's copper is concerned. Verizon seems to have made the decision to put all efforts and funds behind their fiber build out (a good thing) but have completely sacrificed the quality behind their copper services such as T1. If your copper T1 goes down in New York, you might has well throw your hands up in prayer, because that's the only thing that will get it fixed."
Another commenter wrote, "Verizon in some places is actively ripping up copper as they lay fiber because they are not required to resell fiber to CLECs and ISPs at wholesale rates." This has been widely reported, because VZ doesn't want the expense of running to networks - copper and fiber. Plus the fiber doesn't have to be shared and the copper does. The copper means competition. Fiber means they just have to worry about cablecos, who quite frankly are kicking their butt.
Wholesale used to be a healthy business for ILEC's. Today, neither cablecos nor ILEC's want to wholesale anything. In fact, clients of mine in VZ regions have a lot of issues.
For example, "We had an outage about 3 weeks ago that lasted more than three days. This also affected [another local ISP] as I spoke him last night about the current outage. We [both have] a bunch [of customers still] out of service as well. They have been out of service since Monday. The last outage caused an exodus of customers and this one will do the same. Our guys have put in tickets, called to escalate many times. .... no one at VZ will listen. Ever. They simply close the tickets that we open."
It's a systemic problem - widespread - from the C-Suite down - the story has been that every company -- even wholesale customers - are the enemy and the Union and on-union workers must do everything they can to make it uncomfortable unless you are a direct VZ customer.
We have the case of a BK CLEC who had recorded conversations with VZ employees soliciting a customer who was down saying that it wouldn't happen if they were with VZ. [This has been a problem with both RBOC's since I got into telecom in 1999.]
Verizon faces up to $400,000 in fines after New York's Public Service Commission accused the company of not making service repairs in a timely fashion.
What do you do when the RBOC doesn't want to wholesale, doesn't want to repair, and just looks at the bottom line and the few metrics that Wall Street analysts can understand??
Many states don't even regulate the ILEC any more, so what do they do? It becomes the job of the FTC, the FCC and the court system. Talk about a deck stacked against the customer!
When our underlying telecommunications structure suffers, so too does our economic growth.
here's 2 problems with a fiber only strategy for an ILEC:
One, fiber goes out with power, so no 911 or dial-tone when the lights go out.
Two, the installation period for fiber is wicked long. Copper can be installed within two weeks. Fiber takes months. That hurts businesses. I have one moving in 3 weeks and to get 20MB of bandwidth he has to wait months. That won't work.
Ever think that just nothing in this country makes sense any more?
In the discussion about SLA's, the conclusion is to convince your clients to buy redundancy: 2 pipes. That's nice in theory but not in reality. The thing is that you have to set the expectation that if Internet or VoIP is integral to their business operations, no SLA is going to save them, redundancy and business continuity planning will. Otherwise, an outage will be a disaster that they have not planned for. It is not IF, it is WHEN.
]]>Not to get political, but privacy is a ship that has sailed. Google, Facebook and other sites track online activities. Credit card companies and Paypal can track both offline and online activities. CarrierIQ software on cell phones is able to see every keystroke. GPS tracking on devices, including your TomTom. Cellphone and Internet logging by ISP's for government agencies track you.
I'm not suggesting we just give up on privacy. I am suggesting that we take the advice of Sandberg and start focusing on the important stuff. Maybe Security is the Important stuff.
When Wikipedia went dark in the face of SOPA and PIPA bills, it caused many supporters in DC to publicly back away. Privately, however, they will back TransPacific Partnership, ACTA and other bills. Why? Hollywood and content companies want it. Just another example of industry influence over voters or common sense in DC.
In the midst of the privacy talk, why isn't there more talk about security? The latest breach at Zappos affected 24 million. Laptops and cellphones are lost daily. Experts agree that mobile threats and breaches are inevitable. We don't need more rules or laws, we have plenty now, including HIPAA and PCI DSS. We need enforcement and monitoring - or really big fines.
There are simple methods for security available, it might be time to do so. "
""I am angry. There are real problems facing the world, and we, as a society, are not doing enough to address them in the right ways, not the ways we know are possible. The old way isn't working, and we know it." This is the opening paragraph to a new ChangeThis manifesto titled Shift & Reset by Brian Reich. "What might be possible if we were really committed, as individuals and as a society?"
From another ChangeThis manifesto, "The real business opportunity is to become more relevant and meaningful to customers in ways that create sales."
]]>According to the NANOG chatterbox, Level3 & TWC are having the same problem: it appears that Juniper routers are doing a core dump due to a BGP advertisement that triggers a bug. BGP peering causing a number of crashes on the Juniper platform for the last year or so. Unsure if that's related to this problem or not, but the saying one hears frequently is "IPv6 bgp peering isn't as widely tested as the vendors would have you believe"."
I'd use this outage to sell redundancy!!!
]]>