• The "Can You Hear Me Now?" form of NGN testing is a useful method of determining if common business voice services are functioning as needed but only on a small scale

• Making a two-way or 3-way conference call, placing a call on hold, call transfer, call forwarding, and listening for good two way voice quality can easily be tested by dialing phone numbers and pressing keys (flashook, or star codes, i.e. *3) to actuate typical business VoIP services or features.

• The "Can You Hear Me Now?" form of NGN testing does not scale to effectively maintain continuous high quality, business class reliability, and assure service agreements to today's corporations and enterprise customers doing business across NGN networks.

• Test Equipment must be deployed to perform bulk call generation and voice quality measurements - Empirix is a leader in this arena

• Effective operation of this test equipment requires a data driven solution to effectively generate the correct instances for all network city pairs, call types, and service parameters

• Ensuring that calls signal properly and provide a business class voice path must be performed during installation and continuously thereafter to detect and correct issues that can negatively affect the customer's quality of experience and satisfaction

• Network congestion, equipment problems, or software errors are just a few of the defects that can contribute to poor sound quality over NGN VoIP networks
  • One way speech path, delay, echo, choppy voice (you're breaking up!) can render business conversations difficult and frustrating to users

• Failure to signal calls properly may indicate a network component is failing or service provisioning may not be correct for the feature in use

• Test equipment can detect many of these intermittent problems as well as and hard outages if executed continuously and automatically across the networks

• Managing all the test equipment and scripts, endpoint data, and large volume of tests results is challenging. Making this information accessible across the organization and keeping in sync with customer data requires an innovative solution

• The number of tests that must be executed routinely may number in the thousands requiring a more automated and repeatable solution -- one that is easily available across the entire service provider's organization

• The "Can You Hear Me Now?" form of NGN testing must be augmented with systems and the service providers recognize this and are working towards solutions. The following explains some of the major enhancements that today's service providers are now deploying.

• Operations mangers want to simply "push a button and test the entire business VoIP network"
• Lab managers need to identify problems with equipment, configuration, design boundaries, stress conditions, and much more to ensure that problems don't make their way into the network

• They want tests run automatically and auto-generated reports delivered to users on completion in formats they can easily analyze and share with the proper level of detail needed
  • Auto generated reports emailed to selected users -- summary, detailed
  • SMS alerts sent to selected users if failures occur and human intervention is needed
  • SNMP traps sent to operations NMS systems when selected failures occur
  • SOAP Web Services interfaces to generate trouble tickets in the operations trouble management systems and quality management systems

• They want to test across disparate communications networks and measure quality end to end
  • Data driven, end-to-end tests that cover all service locations and service permutations
  • Tests Run on demand after a network change to verify they didn't introduce problems and then run tests continuously as a network diagnostic and measurement probe
  • Scheduled test suites and recurring suites of tests run as needed to monitor network performance

I've brought to light issues of multiple vendors and their proprietary scripts, lack of a single repository for test results and inadequate staffing. Next on my radar is accessibility.

I discussed the concept of a single repository that can house all test results for all equipment and all engineers.  But, what good is that repository if it is not easily accessible?  Often, the people who need this database are offsite, around the world, in a different time zone or maybe just caught up in meetings.

Another issue I've highlighted is the equipment. It typically lies idle or in static operation for about 16 hours a day, plus weekends and holidays.  Seems like a giant waste of resources.

But, the Internet has changed that. With just a web-enabled interface, you have access to the world! Now you can get to anything, anywhere with just everyday tools.  7x24 access means better resource utilization, which test labs desperately need.

Test labs need a tool that lets developers and testers access, manage and automate multiple test scripts for VoIP, IMS, IPTV, PSTN, and wireless devices, at any time, from anywhere. It is not uncommon for test organizations to develop test automation tools in some form or another using TCL, Perl scripts, bash scripts, etc., however developing and maintaining a fully featured test automation framework and environment is often beyond the core competency, budget, and/or resource capacity of many test organizations.

How about one that runs on any windows or UNIX platform and connects to any device (Telnet, SSH, serial port, or command shell)?  Or, one that offers multi-user Web GUI access using any browser, like Firefox, IE6, IE7, Safari, Opera, Chrome or Netscape, from any Internet browser-capable device, including a PC, MAC, UNIX or Linux servers, iPhone, PDA, or G1 mobile device.  This approach should require no software to be installed and maintained on the user's device, thus eliminating the pain of keeping all the user computers up to date.

Once you can access, monitor, and control your test lab from anywhere, and manage all the devices under test (DUT) and devices involved in test (DIT), you can design and build test cases, and execute simple test procedures or create more complex test scenarios limited only by the testers' imagination. 

And, these tests can run 7x24, which maximizes the equipments' ROI and certainly speeds up the test processes to speed time to market, which is really what management is measured on!

So, I'm anxious to hear any ideas you may have on this topic!

Terry

The next critical issue that I have run up against, in my years in the field of testing, is the data....not the just the high-level test status themselves (Passed/Failed), but the easy access and availability of those detailed test cases, test procedures and test results. 

Many test labs rely primarily on skilled test engineers to prepare and repeatedly run hundreds or even thousands of test cases for each major test cycle, manually gathering and interpreting the results, and consistently reporting problems and progress.  So, the

question arises, how does one take the results of many tests, from multiple vendors' test equipment, which are often in separate silos and locations, and consolidate them into a meaningful report combined with logs and other resulting details from all the devices in the test?

Test organizations must collect and manage important test information manually, typically using Microsoft Office tools or some form of internally developed and maintained test management tool. Manual methods can drastically limit collaboration and reuse, and certainly restrict knowledge base sharing. Unless a concerted, manual effort is made to coordinate and distribute updated and timely test information to everyone on the team, it can be difficult to track and manage valuable test information. Test information can be lost, requiring testers to recreate the test setup from scratch. This can be extremely frustrating and time consuming if accurate and detailed test procedures and associated scripts aren't easily retrievable and usable.

Automation of these consolidation techniques is one part of the equation.  The other is to have a central repository that everyone can access, draw information from, and post reports and information into, thus building a knowledge base over time.  This database can centralize valuable test suites, test cases and test results in an online repository, allowing the data to easily be organized, searched, shared, reused, copied, modified, and compared to previous tests and results.  The single data repository allows testers to catch inconsistencies and bugs that may crop up in new or updated software, no matter when or where the product was tested. This is particularly important when members of test groups are geographically dispersed.

Reports should be able to be easily exported in standard formats, such as PDF, XLS, CSV, XML, and HTML, particularly for managers who are not as close to a test process as an engineer.  This allows them to be able to evaluate and oversee test results that may be taking place in labs down the hall, or overseas! 

A common, centralized database and file system is an idea whose time has come.  Any thoughts or experience with this? Do you have a central repository for this valuable test information in your lab? I'd love to hear from you!

Terry

I have been identifying some critical issues in the field of testing.  So, we need sensible ways to address the need to ensure product integrity and quality while meeting aggressive market deadlines and coping with dynamically changing test environments. 

Today I'm going to discuss the use of equipment in test labs. Test labs have a variety of equipment to do a variety of tests.  The equipment often is from multiple vendors, each requiring proprietary scripts. To be able to automate the multiple scripts from multiple machines into one program would be a test engineer's dream come true.

No one piece of test equipment does everything well, thus many labs utilize equipment from more than one vendor. In fact, companies may spend as much as $300K or more on just one piece of test equipment that can sit idle more than half of the time if used ineffectively. Also, test equipment, while homogenous at a protocol level, each has its own unique test case scripting, control interface, and test result storage method that must be managed independently by the tester. In-depth knowledge of each piece of test equipment including the content and location of test scripts is required to perform each repeated test correctly. A test organization ends up with stovepipe or vertical instantiations of manually intensive test regimens that must be repeated properly and frequently for every release or version of the device or system under test. These individual solutions do not lend themselves to scaling or reuse across multiple projects.

To date, none of the equipment vendors offer a comprehensive automation solution that overlays this multi-vendor, multi-device hybrid lab environment. Automation of these tests and equipment would allow testers to run tests all day, every day, not just during an 8 hour.

Test automation can fully utilize equipment that may be idle overnight, on weekends and on holidays, for a 300% improvement on hardware usage and efficiency. Labs can avoid the purchase of additional equipment to support increased demands on limited test teams during their 8-hour day. Instead, managers can invest a much lower amount in commercial third party test automation solutions to optimize equipment usage and speed testing. This requires a commitment to keeping test setups physically stable to avoid wholesale test case rework.

Your comments and experiences are welcome!

Next time, I'll discuss the data repository for all these tests.

Terry

As the title of this blog implies, the focus will be on the future of testing, and in particular, how to incorporate the knowledge and results of legacy testing with next generation products and services and yield substantive results. We need sensible ways to address the growing demand for better ways to ensure product integrity and quality while meeting aggressive market deadlines and coping with dynamically changing test environments. 

In order to understand the future of testing, it is important to identify some the key problems and challenges that are facing many of today's telecommunications software testers and managers. We'll discuss these challenges in some detail, and then offer some practical and effective means to deal with many of them.

My team and I have many years of hands on involvement in development and testing of advanced telecommunications products and systems. We've experienced a host of problems on almost every project we've encountered.

Today's posting will be on adequate staffing for testing environments.

The economy drives business decisions, and today's economy has resulted in reduction of staff. Not only is the number of engineers cut, but their expertise and knowledge base are gone, as well. The remaining engineers still have their tasks, but the other tests still have to be run, the delivery schedules have likely not changed, and the number of hours in a day to be on site to run these tests is finite.

Many test labs rely primarily on skilled test engineers to prepare for, and repeatedly run hundreds or even thousands of tests cases each major test cycle, manually gathering and interpreting the results and consistently reporting problems and progress. Running hundreds or even thousands of test scripts per release is challenging. People tire from running the same manual tests every test cycle, yet they cannot be eliminated.

The answer, at least in part, is automation. In fact, at least 50% of manual tests can and should be automated.

Many tests, such as regression tests, are good candidates for automation. They must be repeated exactly for each release. Running tests the same way each time can ensure nothing obvious slipped through the cracks and fewer embarrassing deployment problems arise that must be isolated and patch in the field.

Not all good test engineers are software developers and not all software people are good testers. Test Automation solutions should enable the creation and executions of tests with zero programming required. In-depth knowledge of the device or system under test is of course crucial to effectively applying any test automation solution.

Manual testing and human coordination go hand in hand. Both will always be needed in test environments to be truly effective. Test Automation can have an immediate and significant impact when applied appropriately. Automated test procedures targeted at solving specific project pain points can be extremely effective and have lasting benefits.

I'd love to hear about your experiences in a test lab, and any comments,

My next blog will focus on limited test equipment.

Terry