A source is telling me that Level 3 Communications and Packet 8 are having a system-wide network outage for the last 30 minutes. Level 3 is a huge provider of VoIP termination and I believe Packet8 uses their network for termination. Other VoIP service providers (Vonage?) will also be affected if indeed Level 3 is having an outage. I'm trying to confirm this as I type this.
I'm currently on hold (5min so far) for Packet8's technical support and sent my contacts at 8x8 an email. I'll update this post when I find out more.
Update: Just talked with Packet8 technical support and indeed they are experiencing outages. They also confirmed they use Level 3, so it appears Level 3 is the cause. Contacting Level 3 next. Stay tuned!
This from my source of the outage who I asked to keep me aprised of when service was restored. The outage seemed to have occured from approximately 11am EDT to 12:15pm EDT (1hr 15min long). He makes a good point about some VoIP customers being medical offices, which are obviously greatly affected by any sort of extended outage.
Update: 1:34pmYes... I am an IT Management and Communications consultant - a few of my customers are Packet8 customers (mostly medical offices, which poses many potentially dangerous situations when phones go out).
OK - As of 12:15pm EST Level 3 has restored their servers and service has been restored.
PS - great blog, thanks for putting this out there... Looks like a few companies need a crash course on Crisis Communications Management and Planning - "keep you publics informed"... (feel free to blog that too.)
Just spoke with Packet8. The issue was due to a Packet8 database crash at Level3's data center. Level3's NOC itself did not experience an outage. Also, stuff was failed over and business customers experienced NO outage - only residential.
Here's a mp3 recording of my conversation with Packet8 representatives.
Highlights of the conversation:
Packet8 was able to failover their business customers immediately. They were not affected at all. They prioritize the failover of their business customers over their residential. The residential customers would failover after 15 minutes or if they rebooted their device.
Medical offices should not be affected if they use the business plan unless they use the softphone application or if they violate the ToS (terms of service) by going with the cheaper residential plan. (which shouldn't be used for business)