tag:blog.tmcnet.com,2013:/blog/tom-keating//4/tag:blog.tmcnet.com,2008:/blog/tom-keating//4.37048- 2013-02-22T21:04:46Z Comments for <![CDATA[Level 3 & Packet8 major outage]]> VoIP & Gadgets blog - Latest news in VoIP & gadgets, wireless, mobile phones, reviews, & opinions Movable Type 4.38 tag:blog.tmcnet.com,2008:/blog/tom-keating//4.37048 2008-07-31T15:37:09Z 2008-07-31T17:41:47Z Level 3 & Packet8 major outage BREAKING!A source is telling me that Level 3 Communications and Packet 8 are having a system-wide network outage for the last 30 minutes. Level 3 is a huge provider of VoIP termination and I believe Packet8 uses their network for... Tom Keating http://blog.tmcnet.com/blog/tom-keating/ A source is telling me that Level 3 Communications and Packet 8 are having a system-wide network outage for the last 30 minutes. Level 3 is a huge provider of VoIP termination and I believe Packet8 uses their network for termination. Other VoIP service providers (Vonage?) will also be affected if indeed Level 3 is having an outage. I'm trying to confirm this as I type this.

I'm currently on hold (5min so far) for Packet8's technical support and sent my contacts at 8x8 an email. I'll update this post when I find out more.

Update: Just talked with Packet8 technical support and indeed they are experiencing outages. They also confirmed they use Level 3, so it appears Level 3 is the cause. Contacting Level 3 next. Stay tuned!

Update: 1:13pm
This from my source of the outage who I asked to keep me aprised of when service was restored. The outage seemed to have occured from approximately 11am EDT to 12:15pm EDT (1hr 15min long). He makes a good point about some VoIP customers being medical offices, which are obviously greatly affected by any sort of extended outage.
Yes... I am an IT Management and Communications consultant - a few of my customers are Packet8 customers (mostly medical offices, which poses many potentially dangerous situations when phones go out).

OK - As of 12:15pm EST Level 3 has restored their servers and service has been restored.

PS - great blog, thanks for putting this out there... Looks like a few companies need a crash course on Crisis Communications Management and Planning - "keep you publics informed"... (feel free to blog that too.)
Update: 1:34pm
Just spoke with Packet8. The issue was due to a Packet8 database crash at Level3's data center. Level3's NOC itself did not experience an outage. Also, stuff was failed over and business customers experienced NO outage - only residential. I'll upload a recording of the call that explains everything in just a minute.

Here's a mp3 recording of my conversation with Packet8 representatives.

Highlights of the conversation:
Packet8 was able to failover their business customers immediately. They were not affected at all. They prioritize the failover of their business customers over their residential. The residential customers would failover after 15 minutes or if they rebooted their device.

Medical offices should not be affected if they use the business plan unless they use the softphone application or if they violate the ToS (terms of service) by going with the cheaper residential plan. (which shouldn't be used for business)
]]> tag:blog.tmcnet.com,2008:/blog/tom-keating//4.37048-comment:38595 Comment from Craig on 2008-07-31 Craig Um. I have 6 phone lines here, and a fax on the East Coast, and all work fine. I didn't go down once. We were up and running all day long.

]]> 2008-07-31T16:59:50Z tag:blog.tmcnet.com,2008:/blog/tom-keating//4.37048-comment:38597 Comment from Carolyn Schuk on 2008-07-31 Carolyn Schuk http://www.voipprincessblog.blogspot.com I've seen some Packet8 problems over that last week - calls not connecting or just getting a busy signal.

]]> 2008-07-31T17:25:06Z tag:blog.tmcnet.com,2008:/blog/tom-keating//4.37048-comment:38598 Comment from Alex on 2008-07-31 Alex This is a very interesting piece of info, since the medical office in question is on a Virtual Office business plan and was out for nearly 75 minutes - no dial tone, dead-air on pickups - on all 12 extensions.

The auto-attendant answered calls, but caller choices were only routed to voicemail - no live operators. Having tested it myself the entire time of the outage, I guarantee this.

In addition, I find it interesting that my first 8 calls to tech support were all transferred to dead-air, eventually being disconnected after approximately 2 minutes of wait-time.

]]> 2008-07-31T17:53:17Z tag:blog.tmcnet.com,2008:/blog/tom-keating//4.37048-comment:38599 Comment from Craig on 2008-07-31 Craig There has been absolutely no problems with QoS from Packet8 Virtual Office over the past week, or past month, or ever for that matter.

I can't speak of Residential (we are a business using several business lines and fax).

]]> 2008-07-31T18:01:20Z tag:blog.tmcnet.com,2008:/blog/tom-keating//4.37048-comment:38601 Comment from Mike on 2008-07-31 Mike RE a crash course on Crisis Communications Management and Planning - "keep you publics informed"

it was great watching this unfold. I am currently putting together a report on crisis communications at my company. this was a valuable lesson in both what to and what not to do

]]> 2008-07-31T19:01:19Z tag:blog.tmcnet.com,2008:/blog/tom-keating//4.37048-comment:38602 Comment from Eric on 2008-07-31 Eric We have many Packet 8 phone lines. Didn't experienced any issue.

]]> 2008-07-31T19:32:42Z tag:blog.tmcnet.com,2008:/blog/tom-keating//4.37048-comment:38603 Comment from C on 2008-07-31 C For the most part this guy can make calls out but nothing is coming in.

Location = Mass

]]> 2008-07-31T19:52:47Z tag:blog.tmcnet.com,2008:/blog/tom-keating//4.37048-comment:38605 Comment from Robert Basye on 2008-07-31 Robert Basye I just contacted customer service has no idea when they will be fixed

Location KY

]]> 2008-07-31T20:29:53Z tag:blog.tmcnet.com,2008:/blog/tom-keating//4.37048-comment:38606 Comment from Kevin on 2008-07-31 Kevin http://www.infodefense.com We have the new 675x SIP phones. We are still down. Our phones have been out since about 10:00 CST. Our location is Dallas, TX.

I was able to reach Packet8 Tech support immediately both times I called today (from a different phone). They were helpful, offering to call me when the service is restored. I think it would be helpful if they offered updates from their web site, however.

]]> 2008-07-31T21:30:45Z tag:blog.tmcnet.com,2008:/blog/tom-keating//4.37048-comment:38628 Comment from Alex on 2008-08-01 Alex I am utterly disappointed by Packet8's response to this situation. Having listened to the audio recording and reading all subsequent blogs and articles written on the matter, I cannot grasp the reasoning behind Martin's remarks.

As the IT consultant for THE medical office in question, I sincerely resent the statement that, "if you were one of our business customers, you were not affected.” This office, in addition to others who commented on this blog, did lose connection for extended periods; ours lasting for a total of 75 minutes. For an office of approximately 20 employees in a critical medical capacity, this was unacceptable. Furthermore, to insinuate that my client "opted – as they shouldn’t – for a cheaper residential line" is an outrage and a blatant disregard or even an acknowledgment of our communications loss.

]]> 2008-08-01T15:02:55Z tag:blog.tmcnet.com,2008:/blog/tom-keating//4.37048-comment:51996 Comment from Dan bohn on 2010-01-12 Dan bohn We are a small business in Oakland CA and still have no service at 1:38 pm

]]> 2010-01-12T21:33:29Z tag:blog.tmcnet.com,2008:/blog/tom-keating//4.37048-comment:52000 Comment from hr on 2010-01-12 hr We have been out and at almost 5 we are still out of all service. We are on a business plan so I dont get that
We are in Dallas Texas. Tried to go to customer service no response and phones there must have been overwhelmed
Still down

]]> 2010-01-12T22:53:23Z