Tomorrow’s Testing Today

As you've seen from my blogs, I am passionate about testing, and its future.  But, barriers can make a testing environment hard to navigate.

I've brought to light issues of multiple vendors and their proprietary scripts, lack of a single repository for test results and inadequate staffing. Next on my radar is accessibility.

I discussed the concept of a single repository that can house all test results for all equipment and all engineers.  But, what good is that repository if it is not easily accessible?  Often, the people who need this database are offsite, around the world, in a different time zone or maybe just caught up in meetings.

Another issue I've highlighted is the equipment. It typically lies idle or in static operation for about 16 hours a day, plus weekends and holidays.  Seems like a giant waste of resources.

But, the Internet has changed that. With just a web-enabled interface, you have access to the world! Now you can get to anything, anywhere with just everyday tools.  7x24 access means better resource utilization, which test labs desperately need.

Test labs need a tool that lets developers and testers access, manage and automate multiple test scripts for VoIP, IMS, IPTV, PSTN, and wireless devices, at any time, from anywhere. It is not uncommon for test organizations to develop test automation tools in some form or another using TCL, Perl scripts, bash scripts, etc., however developing and maintaining a fully featured test automation framework and environment is often beyond the core competency, budget, and/or resource capacity of many test organizations.

How about one that runs on any windows or UNIX platform and connects to any device (Telnet, SSH, serial port, or command shell)?  Or, one that offers multi-user Web GUI access using any browser, like Firefox, IE6, IE7, Safari, Opera, Chrome or Netscape, from any Internet browser-capable device, including a PC, MAC, UNIX or Linux servers, iPhone, PDA, or G1 mobile device.  This approach should require no software to be installed and maintained on the user's device, thus eliminating the pain of keeping all the user computers up to date.

Once you can access, monitor, and control your test lab from anywhere, and manage all the devices under test (DUT) and devices involved in test (DIT), you can design and build test cases, and execute simple test procedures or create more complex test scenarios limited only by the testers' imagination. 

And, these tests can run 7x24, which maximizes the equipments' ROI and certainly speeds up the test processes to speed time to market, which is really what management is measured on!

So, I'm anxious to hear any ideas you may have on this topic!

Terry

In my earlier blog I said I want to put the spotlight on the future of testing.  As next gen products and services evolve, the lessons from legacy testing need to be incorporated. 

The next critical issue that I have run up against, in my years in the field of testing, is the data....not the just the high-level test status themselves (Passed/Failed), but the easy access and availability of those detailed test cases, test procedures and test results. 

Many test labs rely primarily on skilled test engineers to prepare and repeatedly run hundreds or even thousands of test cases for each major test cycle, manually gathering and interpreting the results, and consistently reporting problems and progress.  So, the

question arises, how does one take the results of many tests, from multiple vendors' test equipment, which are often in separate silos and locations, and consolidate them into a meaningful report combined with logs and other resulting details from all the devices in the test?

Test organizations must collect and manage important test information manually, typically using Microsoft Office tools or some form of internally developed and maintained test management tool. Manual methods can drastically limit collaboration and reuse, and certainly restrict knowledge base sharing. Unless a concerted, manual effort is made to coordinate and distribute updated and timely test information to everyone on the team, it can be difficult to track and manage valuable test information. Test information can be lost, requiring testers to recreate the test setup from scratch. This can be extremely frustrating and time consuming if accurate and detailed test procedures and associated scripts aren't easily retrievable and usable.

Automation of these consolidation techniques is one part of the equation.  The other is to have a central repository that everyone can access, draw information from, and post reports and information into, thus building a knowledge base over time.  This database can centralize valuable test suites, test cases and test results in an online repository, allowing the data to easily be organized, searched, shared, reused, copied, modified, and compared to previous tests and results.  The single data repository allows testers to catch inconsistencies and bugs that may crop up in new or updated software, no matter when or where the product was tested. This is particularly important when members of test groups are geographically dispersed.

Reports should be able to be easily exported in standard formats, such as PDF, XLS, CSV, XML, and HTML, particularly for managers who are not as close to a test process as an engineer.  This allows them to be able to evaluate and oversee test results that may be taking place in labs down the hall, or overseas! 

A common, centralized database and file system is an idea whose time has come.  Any thoughts or experience with this? Do you have a central repository for this valuable test information in your lab? I'd love to hear from you!

Terry

As my earlier blog said, we're here to get a discussion on the future of testing, and what it means to legacy testing as it melds with next generation products and services. 

I have been identifying some critical issues in the field of testing.  So, we need sensible ways to address the need to ensure product integrity and quality while meeting aggressive market deadlines and coping with dynamically changing test environments. 

Today I'm going to discuss the use of equipment in test labs. Test labs have a variety of equipment to do a variety of tests.  The equipment often is from multiple vendors, each requiring proprietary scripts. To be able to automate the multiple scripts from multiple machines into one program would be a test engineer's dream come true.

No one piece of test equipment does everything well, thus many labs utilize equipment from more than one vendor. In fact, companies may spend as much as $300K or more on just one piece of test equipment that can sit idle more than half of the time if used ineffectively. Also, test equipment, while homogenous at a protocol level, each has its own unique test case scripting, control interface, and test result storage method that must be managed independently by the tester. In-depth knowledge of each piece of test equipment including the content and location of test scripts is required to perform each repeated test correctly. A test organization ends up with stovepipe or vertical instantiations of manually intensive test regimens that must be repeated properly and frequently for every release or version of the device or system under test. These individual solutions do not lend themselves to scaling or reuse across multiple projects.

To date, none of the equipment vendors offer a comprehensive automation solution that overlays this multi-vendor, multi-device hybrid lab environment. Automation of these tests and equipment would allow testers to run tests all day, every day, not just during an 8 hour.

Test automation can fully utilize equipment that may be idle overnight, on weekends and on holidays, for a 300% improvement on hardware usage and efficiency. Labs can avoid the purchase of additional equipment to support increased demands on limited test teams during their 8-hour day. Instead, managers can invest a much lower amount in commercial third party test automation solutions to optimize equipment usage and speed testing. This requires a commitment to keeping test setups physically stable to avoid wholesale test case rework.

Your comments and experiences are welcome!

Next time, I'll discuss the data repository for all these tests.

Terry

I'd like to thank TMC for the opportunity to share some thoughts with you on the subject of testing and test automation. It is a pleasure to have this space to talk about this important topic.

As the title of this blog implies, the focus will be on the future of testing, and in particular, how to incorporate the knowledge and results of legacy testing with next generation products and services and yield substantive results. We need sensible ways to address the growing demand for better ways to ensure product integrity and quality while meeting aggressive market deadlines and coping with dynamically changing test environments. 

In order to understand the future of testing, it is important to identify some the key problems and challenges that are facing many of today's telecommunications software testers and managers. We'll discuss these challenges in some detail, and then offer some practical and effective means to deal with many of them.

My team and I have many years of hands on involvement in development and testing of advanced telecommunications products and systems. We've experienced a host of problems on almost every project we've encountered.

Today's posting will be on adequate staffing for testing environments.

The economy drives business decisions, and today's economy has resulted in reduction of staff. Not only is the number of engineers cut, but their expertise and knowledge base are gone, as well. The remaining engineers still have their tasks, but the other tests still have to be run, the delivery schedules have likely not changed, and the number of hours in a day to be on site to run these tests is finite.

Many test labs rely primarily on skilled test engineers to prepare for, and repeatedly run hundreds or even thousands of tests cases each major test cycle, manually gathering and interpreting the results and consistently reporting problems and progress. Running hundreds or even thousands of test scripts per release is challenging. People tire from running the same manual tests every test cycle, yet they cannot be eliminated.

The answer, at least in part, is automation. In fact, at least 50% of manual tests can and should be automated.

Many tests, such as regression tests, are good candidates for automation. They must be repeated exactly for each release. Running tests the same way each time can ensure nothing obvious slipped through the cracks and fewer embarrassing deployment problems arise that must be isolated and patch in the field.

Not all good test engineers are software developers and not all software people are good testers. Test Automation solutions should enable the creation and executions of tests with zero programming required. In-depth knowledge of the device or system under test is of course crucial to effectively applying any test automation solution.

Manual testing and human coordination go hand in hand. Both will always be needed in test environments to be truly effective. Test Automation can have an immediate and significant impact when applied appropriately. Automated test procedures targeted at solving specific project pain points can be extremely effective and have lasting benefits.

I'd love to hear about your experiences in a test lab, and any comments,

My next blog will focus on limited test equipment.

Terry