Is Test Automation THE Answer?

Sorry, no such luck! Test Automation is AN answer, to be sure, for the right types of tests. But I don't think it can ever replace manual testing and the critical thinking the goes into testing all the permutations that a complex software product or service presents.

So, it is important to understand that test automation is one of many valuable tools that can, and should be applied to the testing effort to ensure that fewer defects reach the deployment stage. When applied properly, test automation has been extremely helpful, but too often test automation projects fail to deliver on expectations. The reasons are many:

·         Inadequate organizational responsibilities:

o        Everyone on the test team should have a stake in the test automation effort

·         Limited management support, understanding, and experience

o        Management cannot dictate success but should facilitate success

o        Test managers typically know what works and what doesn't, and has knowledge of prior test results, so upper management needs to trust the judgment of their lab managers, and support them

·         Overzealous objectives 

o        It is not reasonable to expect that 100% of testing can be automated. Automating 50% or more of tests is achievable and reasonable

·         Unrealistic time and cost expectations

o        Practical expectations for time and costs savings must be established at the outset

·         Inadequate tools and training

o        With power and flexibility comes complication. So, test automation must be simple to use and everyone who uses it, must be trained

·         Poor communication and overlapping efforts

o        Automation stakeholders all must start on the same page

o        There is no substitute for regular communication and updates on automation efforts!

In my earliest post, I promised to provide some practical approaches towards test automation based on experience and feedback from many of the engineers I've worked with over the years. All of them will tell you that test automation for certain aspects of the test effort is a must. Repetitive, manual steps will wear anyone down over time.

Let's take a straightforward example of where automation can help and also save time and costs immediately. I'm going give you my personal experience with the daily build and smoke testing process in a typical telecom development and test lab environment. This test process is a good practical example of where automation can help. 

In a typical scenario, software developers code by day and check in changes to the source control system before going home. Later that night, some automated build process kicks in and builds the product executable software so that it is ready for testers to install and test in the morning. Once testers arrive and start their day, they will typically download the new build to one or more testbeds and begin the process of looking for problems with the new build. They will run a series of commands and test scripts to ensure the product or service functions as expected, and that the software engineers didn't miss anything in their unit testing. Typically within a few hours, engineering is given feedback on the health of the new build. This cycle repeats itself with the hope that nothing will break from day to day, and that one or more problems are detected in this manner.

Here's the math:

Let's assume that there are three testers on the team that perform daily smoke tests on a product under active development, and let's assume it takes these testers four hours per day to run their series of manual smoke tests to ascertain the health of the build. Let us also acknowledge that fully automating this process the first time will take a few hours of someone's time, but once it is completed, very little time should be required to maintain it using an automated testing framework.

Now, let's automate that same process and have this work run at night right after the build is completed at 3 am. We can have the automation framework wake up at 3:15 am and download the software to three testbeds, provision the systems or devices under test (SUT/DUT), run a series of regression scripts, modify the SUT/DUT while the tests are running, run another series of scripts, etc., just like the manual testers do. Then upon completion, a test result report is generated and emailed to the appropriate team members and managers so that when they arrive at 8 am, they know the current health of the daily build and can focus on testing the new features that were added.

The savings that can be achieved from automating this basic process can be as shown below:

(((3 Testers * 4 hours per day = 12) * 300 days per year) * $30 per hour) = $10800 in savings. And this is for only one type of test! Multiply this for other types of regression and Q/A tests and the savings grow with each automated process.

Of course, all of the parameters above may differ based on actual lab operations, but as you can see, the cost savings and improved productivity can be quantified and are quite significant over time.

In my next post, I'll offer more practical examples of how test automation can be implemented to achieve real savings and process improvements. I'd be interested in hearing your thoughts on this topic.

Terry