Interesting VoIP report from Keynote Systems I thought I'd share. It states Vonage ranks first in reliability while AT&T CallVantage ranks #1 for best audio quality. Some other interesting insights as well - check it out...
Today Keynote issued the results of the first of its kind study of the service quality of top providers of Internet telephone service.
Where Is the Dial Tone? Asks Keynote Study of Internet Telephone Service
• Vonage Ranks First in Overall Reliability Among VoIP Providers; Time Warner Cable Far Ahead as Most Reliable Network Carrier
• AT&T CallVantage Provides Best Audio Clarity; Time Warner Cable Ties with UUNET for Top Spot for Audio Clarity Among Networks
• AT&T CallVantage, Packet 8, Primus Lingo, Skype, Verizon and Vonage Included in Study; Calls carried over AT&T, Sprint and UUNET Business-Class Networks, as well as Residential DSL and Residential Cable Networks.
• Lack of Availability or Garbled Information Could Spell Trouble for 911 Calls
San Mateo, Calif., — July 12, 2005 — New competitive intelligence from Keynote Systems (Nasdaq: KEYN), The Internet Performance Authority®, reveals the need for considerable improvement in overall service quality from the leading providers of Internet telephone service (VoIP). The results of the Keynote VoIP Competitive Intelligence Study, the first of its kind to benchmark and rank the quality of VoIP as perceived by end-users were released today. For the study, Keynote ranked six leading VoIP providers on critical performance factors that influence the end-user experience using Keynote’s VoIP Perspective measurement service.
Included in the study were AT&T CallVantage (NYSE: ATT), Packet 8, Primus Lingo, SkypeOut, Verizon Voicewing (NYSE: VZ) and Vonage in the New York and San Francisco metro areas. To understand the impact of underlying network performance on call quality, the VoIP telephone calls were carried on three business-class networks: AT&T, Sprint (NYSE: PCS) and UUNET. In addition, the study captured the impact of the last-mile on call quality by measuring each of the six providers on residential DSL lines from SBC (NYSE: SBC) and Verizon and residential cable lines from Comcast (Nasdaq: CMCSK) and Time Warner Cable (NYSE: TWX) as well.
Vonage ranked first overall in the Keynote study as the most reliable VoIP service provider with a normalized rank of over 80 points on a scale of 0-100. Keynote found a significant gap in reliability between Vonage and the lowest ranking providers. In spite of its high average, Vonage still has room for improvement on the Dropped Calls performance factor.
AT&T CallVantage led all other providers in audio clarity with a noticeable gap between AT&T CallVantage and its competitors. Here, as well, AT&T CallVantage has room for improvement on geographic variability on the audio delay factor – a leading cause of conversational disruption. The importance of conversation disruption grows with the importance of the information being imparted during a conversation. Overlapping conversation breaks down human ability to understand when to stop or start talking resulting in disruption of conversation and missed information, which could be critical, for example, during a 911 call.
“VoIP reliability and audio clarity are important factors limiting the widespread adoption of VoIP in consumer markets. Consumers are unsure whether VoIP can live up to the dial-tone reliability and crystal-clear communication quality they have come to expect with traditional phone service over the years.” said Dharmesh Thakker, senior product manager for service level solutions at Keynote. “Our study provides an objective assessment of these critical performance factors that affect end-users’ perception of a VoIP service. While several providers and networks did well in certain areas, no single provider or network dominated the study in all metrics considered.”
A key take-way from the study is that Internet telephone service is not yet up to the standards to which users are accustomed when using standard ‘plain old telephone service’ (POTS) and VoIP providers have some work to do to capitalize effectively on the growing consumer adoption of VoIP service.
Moreover, not everything impacting the VoIP service quality is under the control of the provider. The Keynote study found that the underlying network carrier could make a dramatic difference in service reliability and audio clarity. Specifically, Keynote recommends that VoIP service providers address the audio delay in their calls to minimize conversational disruption and reduce the variability in performance experienced by callers in different cities.
The data for the study was collected over a five-week period from May 21 to June 25, 2005 to assess market readiness, compare leading providers to each other and analyze the impact of the underlying network by comparing the call quality of leading VoIP providers in San Francisco and New York to traditional Public Switched Telephone Networks (PSTN).
With this study, Keynote continues to extend its brand to include emerging technologies such as VoIP, streaming and wireless, which increasingly are core technologies of the enterprise. As Voice over Internet Protocol emerges as an influential technology that promises to cut consumer phone bills and enterprise communications expenses, the Keynote rankings help assess VoIP readiness and highlight market leadership among the various providers.
How the Study Was ConductedThe study objectively ranked leading providers on critical factors that influence service quality for end-user experience when making Internet calls. Moreover, the study also analyzed various diagnostic metrics that constitute the service performance factors and evaluated the impact of the underlying network carriers on the VoIP quality perceived by end-users.
Keynote placed coast-to-coast calls from VoIP to standard telephones (PSTN) between San Francisco and New York once every 30 minutes. Additionally, local loop VoIP to PSTN calls were placed every 30 minutes for each of the provider and network combinations to emulate a contained environment and accurately track the network impact on call quality. Calls were also placed from and to traditional phones every 30 minutes between the two metro areas to understand what residential customers can expect when switching from traditional phone lines to VoIP.
To quantify the end-user experience with call quality over each VoIP provider and network combination, Keynote identified ten key performance factors based on multiple network and audio fidelity metrics that are collected during the call placement. These performance factors, to which every VoIP provider and network carrier should pay attention to ensure quality calls include:
• Service Availability – indicating the reliability of the service
• Outage Hours – how many minutes the service was unavailable during the study collection period and on a daily basis
• Average number of call attempts – how many times, on average, a user must call to establish a connection
• Dropped Calls – the number of times a call is dropped in the midst of a conversation
• Audio Delay – once the call is established, the average lag between utterances. High audio delay leads to overlapping and unpleasant conversations in real-life.
• Listening quality (or Mean Opinion Score – MOS) measured using the Perceptual Evaluation of Speech Quality scale (PESQ) - the internationally recognized ITU-T P.862 standard
• Audio delay consistency over time
• Audio delay geographic uniformity between New York and San Francisco
• MOS consistency over time
• MOS geographic uniformity between New York and San Francisco
Keynote aggregated these ten factors into two overall quality indicators – Reliability and Audio Clarity. Reliability includes service availability, the average number of dial attempts and dropped calls. Audio Clarity, which was computed based on a weighted average of the Average Mean Opinion Score (MOS) and Average Audio Delay including adjustments for percentage of calls below minimum acceptable thresholds and high geographic variability.
The study highlights which areas up-and-coming providers need to pay special attention to in order to provide high service quality to end-users and best capitalize in their respective markets. Moreover enterprises can also learn industry best practices they can implement during their branch-office VoIP deployments and when rolling out converged customer-care applications, to maximize employee productivity and retain high customer satisfaction.
The full study is available for purchase from Keynote. For more information on the study or to obtain an abstract, please contact firstname.lastname@example.org.
Founded in 1995, Keynote Systems (Nasdaq “KEYN”), The Internet Performance Authority®, is the worldwide leader in e-business performance management services. Over 2,100 corporate IT and marketing departments and 16,000 individual subscribers rely on Keynote’s portfolio of measurement and monitoring, service level and customer experience management services to improve e-business performance by reducing costs, improving customer satisfaction and increasing profitability.
Keynote is viewed as The Internet Performance Authority due to the company’s global infrastructure of over 1,600 measurement computers in more than 50 cities worldwide that capture and store on a daily basis over 60 million Internet performance measurements, frequent media citations quoting Keynote's Web performance data and analysis, the company’s market-leading Web performance indices for vertical markets and leading customer research that provides critical business insight into online customer experiences, industry trends and competitive Web strategies.
Keynote Systems, Inc. is headquartered in San Mateo, California and can be reached at www.keynote.com or by phone in the U.S. at 650-403-2400.
Keynote, The Internet Performance Authority and Perspective are registered trademarks of Keynote Systems, Inc. Other trademarks are the property of their respective owners. © 2005 Keynote Systems, Inc.