System and Device Testing

Is Testing Getting Harder

October 22, 2009 12:09 PM |

The other night I had a conversation with my niece about college. I remarked that my experience was that my upper division classes were much more challenging than the earlier courses. I didn't feel, however, like they continued to escalate from there. Sure they were challenging but I always thought they were different as opposed to harder.

I find myself wondering the same about testing: Is it getting harder? Converged applications ... Cloud ... SOA ... SAAS? Is it harder, or just different?

Do the tools we have now and the experience we have gained neutralize the growing complexity?

Will that be true when 4G is here with "more sophisticated mobile applications"?

What do you think? Harder? The same? Different?

Cheers,

David

Testing Next Generation Networks (NGN, LTE, 4G, WiMax)

August 31, 2009 4:19 PM |

0 Comments

Just posted a blog article on CIO magazine's online site. Short discussion about getting ahead of the next generation network testing, because what just worked for 3G isn't working well for 4G.

http://advice.cio.com/fanfare_software/next_generation_networks_require_next_generation_testing_strategies

Will you run the right POC or get the answer you want?

August 25, 2009 8:25 PM |

0 Comments

We had a good discussion internally and wanted to share a provocative email from Andreas Dharmawan, Director of Customer Experience Group, who runs our AEs and our Consulting team.

Enjoy, don't agree?

Many business leaders and technical architects often places such a thorough process in new technology selection. They mandate the vendors to go through such a high bar. More often, during the proof of concept (POC), none of the vendors can satisfy the detail requirements in exactly the same way that the organization has imagined. This is due to several underlying problems that many organizations are not aware of. Here are the two important ones.

The requirements or the success criteria for the POC were driven by the engineers who had been trying to build such technology or infrastructure in house for the last many years. When the requirements are analyzed carefully, they represent the solution needed to solve the problems caused by the deficiencies of the home brew solution instead of the solution needed to solve the root cause business problems. The requirements or the success criteria for the POC were driven by very junior engineers or senior engineers who are behind in the new technology curve. They are too comfortable with the existing processes and skills sets. When the requirements are analyzed carefully they represent the solution needed to justify the status quo of processes and skill sets instead of the solution needed to advance the organization efficiency and leadership.

If the organization is not aware of the above underlying problems, the process of technology selection becomes murky as the POC unfolds. At the end, an emotion-driven decision is made instead of a data-driven decision. Vendors are often asked to bend backward, throw a sword in the air, flip, and catch the sword as it falls. At this moment, an inexperienced vendor would go out of their ways to customize its solution. It is a short term win for the inexperienced vendor and the organization because this approach is only addressing the symptoms and not the root cause business problems. It is like treating a brain tumor by taking ibuprofen. In another case, the organization chooses the vendor who they have been working for many years based on relationship knowing for a fact that the solution is inferior.

Related Entries:

What is automated testing? - Jul 16, 2009

Diagnostics versus Testing

August 3, 2009 6:07 PM |

0 Comments

Just wrote an interesting article and posted it here..

http://www.telecoms-mag.com/article.asp?HH_ID=AR_5525

Do you think we are already there? Do you think Ops will have more respect for testers than developers?

What is automated testing?

July 16, 2009 1:43 PM |

0 Comments

I have to admit I got into a religious war the other night; the debate raged something like this:

Automated testing is any testing that is performed and repeated by a computer -- it could be one step or action or ten thousand. How do I measure it? It saves me time each time I run it.

The counter-argument is that automation has to include analysis. Meaning not only can I run it but the test tells me whether it passed or not -- e.g. the number of packets was great than 10 as expected. The image, named "success" was present. The measure? It saves "more" time and is more likely to occur in a regression system or plugging into a test management system. Of course we went on to 'documentation': Does an automated test have to have documentation as an output? ... and on into even more esoteric points.

I have to admit that a couple of beverages were involved, but it is comical in hindsight that we are arguing the shades of automation. Of course I am right, just look at the dictionary. J

Related Entries:

Will you run the right POC or get the answer you want? - Aug 25, 2009

TMC 2008 Communications Solutions Product of the Year Award

May 22, 2009 11:53 AM |

0 Comments

Good news and a little plug here, but I TMC just awarded our Company, Fanfare, the 2008 Communications Solutions Product of the Year Award, for iTest.

We have a lot of momentum right now for a testing solution focused on Device and System testing that is changing the game in the communications industry. This is our 3^rd such award in just 12 months.

I want to thank TMC for acknowledgement and the award.

Here is a link to the awards page here in TMC: http://www.tmcnet.com/voip/

Here is a link to our press release: http://www.fanfaresoftware.com/news/pressReleases/20090518-communications-solutions-product-of-the-year.aspx

David

Next Generation Testing Tools

May 22, 2009 11:50 AM |

0 Comments

So, one aspect of the down economy is doing more with less, and unfortunately that happens to management as well. I had hoped to have more frequent posts but it just seems to be a very busy time for everyone to just to make ends meet. I happen to be a good busy, our company is doing well, but the same business takes more effort than a year ago.

However I did write a blog for Xchange Magazine, Next-Gen Services Test Testing's Boundaries, to talk about what next generation testing tools should be providing.

Check it out: http://www.xchangemag.com/blogs/guest/blogdefault.aspx

David

What is testing really for?

April 17, 2009 4:53 PM |

0 Comments

OK, obvious question, it is clear that testing is the key to quality. And while testing is about finding bugs and defects, testing is also required for the rapid resolution of defects. So this blog is about a testing best practice. A best practice that helps ensure tests are just as good at pinpointing defects as they are at confirming the existence of a defect.

The best practice is simplicity and modularity.

I will start at the opposite end. I see test cases or scripts hundreds of lines long doing all kinds of conceivable operations, configurations, and clean up. So the test script runs, and guess what? It fails. So where is the issue? Is it the script? Is it the setup procedure? Is it an actual defect in the product? Well, that is hard to say because that "mother of all test cases" is very complicated. So now, you need to instrument each command, procedure, and configuration to isolate the fault. Yes, good work, but it makes the script even longer, more complicated, and harder to maintain than it was in the first place.

The best practice is simplicity and modularity.

Make each section and test as simple as possible. Say you need to set up a testbed or device, test a feature, and then reset the lab. That is easily three tests in my book--one for each phase. If you need a lot of things going on at the same time, consider individual test scripts and run them in parallel or multithread them.

Why? Deduction, my dear Watson.

When you have a problem, you can simply remove one of the procedures or not run as many tests in parallel. In this way, it becomes easy to figure out which portion of the test or setup is actually causing the problem. Removing a section, procedure, call , etc.: Deduction.

Secondly, these smaller scripts are more leverage-able and can be used in other tests, and are easier to maintain. But I hear all the time, "It is easier to have this killer test script 'cause it does it all." Well I'll argue that it took longer to build, debug, and then find the problems than to build the same test via components. Oh, and the other statement; "It's simple because it is just one file" just doesn't fly. My components are all in the same folder, with an appropriate folder name.

In my engineering discipline, one valuable creed that comes up over and over again is the KISS principle. It works.

Testing Worldwide

March 26, 2009 1:17 PM |

0 Comments

I just got back from a trip in Europe where I got to speak with 15 journalists about testing. It is always a pleasant surprise to hear how well testers are accepted and respected in Europe and how colleges actually have programs that graduate professional testers-- a career and a good one. Careers backed by solid network, device, design, and programming skills. I could laud more , but I will stop.

I will also point out that quality (at least in the communications space) is becoming a big deal in Europe and that, with a bad economy where more is done with the same or less, testing is a bit cooler than normal. For years, carriers believed that loyalty to a national company would keep customers from wandering (at least virtually) to companies across the border. Today that is not true and fear of customer loss is rising.

I found one interesting comment from a European that, in respect to phone service, they expect so much for free compared to the US where consumers expect to pay for extra services. But in the semi-regulated European Union, they find little service difference between carriers, and price differentiation is just as thin. But he went on to say that what is being talked about is quality, and that customers are switching carriers for reliability more often. Perhaps loyalty is just a dropped call away. That reminded me of a couple of carriers here in the US (one of which uses "it's the network" as their marketing message). Perhaps a little more respect and acknowledgement is due testers in the US (as their brethren garner in Europe).

Expert Abstraction

March 6, 2009 12:57 PM |

0 Comments

In the last blog I spoke about abstraction, and discussed user credentials and device abstraction in the examples. However those examples are not the most valuable aspects of abstraction that automation experts think of. The first item that usually comes to mind is response abstraction and software GUI changes.

Here is an over-simplified example, but bear with me. Let's say I test a GUI, and today the dialog box I am testing is in the top left corner. I build a test and everything works fine. But tomorrow, development moves the box to the bottom of the screen. Well now my test breaks and I have to rebuild that test case. If I had built 100 tests for different aspects of that GUI and now it has changed, how many of them are broken and need to be fixed or rebuilt? Chances are the answer is a significant percentage -- and that is just frustrating and feels like wasted work. However, if you abstracted that response or GUI screen to another file and then pointed the test case to that file, the file would now be a "map" of the GUI. Now if I ask for a link or box, it tells me where it is. So now my hundred test cases refer to that "map" and if development makes that GUI change, I need only update my "map" file -- and all 100 test cases are instantly updated.

Now hopefully this makes sense. The same story holds true for those who test devices or embedded systems. Here's another example. Last month I built a test that sends command line interface (CLI) commands, and I spent a lot of time parsing the unstructured DOS-like data to make the test work. (BTW, there were 42 lines to this command response, so lots of work.) But now, development has changed line two -- and all my tests are busted. Rework!

If you are testing and trying to figure out how to get a product to market faster, we all know automation is part of the answer -- and those who have been doing automation know that abstraction is key. But beware of the marketing hype on abstraction for the automation product you may be considering. In fact, if you are doing an eval or POC and don't have an entire abstraction solution as part of the eval, you are making a mistake. I can tell you that most don't do it, and later find the abstraction features to be weak or hard to use. The ROI for why you were buying the tool just went poof. Powerful and easy to use abstraction features are more than marketing buzz words; they are the key technology that makes testing easier and more efficient.

Do you know what abstraction is?

February 26, 2009 3:01 PM |

0 Comments

Today I want to talk about abstraction. Abstraction is a requirement to make testing maintainable and portable. Perhaps more importantly abstraction is critical to make test automation work, and to achieve a lights out regression system.

Abstraction is not a new term and from my research on the technology side has long been used in engineering teams. But now it is also firmly in the hands of test automation and regression teams. However we find that most feature tester and manual testers don't know what abstraction is. However this is one of the few concepts I have yet to find an analogy or even a comparison to something you do in everyday life.

Before I attempt to explain abstraction let me state the reason I bring this up is, I see the future of testers in all roles needing to know abstraction. Testing is changing and maturing as a career and the value and benefit to the company is being more widely recognized. That means more want to see testing and they want to see automation as a big contributor to raise quality, reduce time to market and control costs. So where abstraction was something applied at the end of testing assembly line, I see it moving upstream to the very beginning.

So here is abstraction. If I make a test case, and that test case requires a log-in into a software application I generally put that information into the test case. If I am functional tester this is fine, I test run the test case, uses my credentials and performs the test. What if I transfer that test case to another tester or put it into regression. I really don't want others to know my username and password. So I want to abstract that username and password. One common way to abstract is to create a reference file somewhere, where the test case can pull the data. Thus, now when I pass the test case I am not passing my credentials anymore, but the test case will still need to look for a file to run and if it is in regression, that team will need to create a file with another set of credentials. In this scenario we see that I protect my credentials but have created a dependency on a separate file. But let's look at a bigger benefit lets say I have 100 test cases that require an username and password, and I have abstracted them. Tomorrow IT tells me they are moving to a more robust password mechanism and my weak 6 letter password must be changed? Historically this means hours of waste time just changing a password in 100 test cases. But with abstraction I just need to change it once, yes just change it in the file and all those test cases that refer to that file are instantly updated. So there is a true benefit of abstraction for automation but also real value to the tester himself or herself .

Abstraction is the only way to make automation a reality without tremendously burdensome and constant maintenance for each and every test case. Abstraction is not new to automaton teams or regression teams and in the past is has been a very specialized skill because tools really didn't help those make abstraction easy. However technology has marched forward and tools are making abstraction easy, this can mean taking more coffee breaks for the automation teams, but what it really means is if you want to deliver the most value to the business , You need to accelerate testing and that means abstraction needs to happen when you first make the test case, this saves growing work downstream and means those who build and understand the test can build it right the first time. Besides, now you can keep your credentials private.

Related Entries:

Moving Up. Time for a Real Testing Career.

January 16, 2009 7:14 PM |

0 Comments

There is a lot of change in the market; budget cuts, RIFs, bankruptcy, reorganizations and blind slashing. Too often QA and testing receive an unfair brunt of it. Ironically I would say this is in direct contrast to the actual business benefits. And that cuts to testing can delay time to market for revenue producing products and services, can impact quality which drives the cost of support up, wastes developer cycles as they fix defects, and risks losing customers all together. So why does it happen to testers?

Very few understand and even fewer can articulate what they do in relation to the profit and loss of a company's business. They can't talk the talk, they don't get in the club and they don't get a vote.

In America most don't go to school to be a tester, but many find their way into the career and they love it. In Europe and Asia it is different as there is established curriculum for testing and software quality. But, like their American brethren, as they rise in ranks they rarely get the business education. In fact, most on the engineering side of company are not encouraged at all in this area.

Leaders and managers in testing, please take the time to learn the ways of the business. Whether it is a few classes at the local college or a full MBA, please think about it.

I work with many organizations that have thousands of testers around the globe; people, technology and process at staggering levels. To run and manage that effectively takes great skill. These organizations are currently run by those who rose up from the ranks. But if you compare a CIO today with one ten years ago, you will find today's CIO is a business thinker with knowledge of the technical subject matter. If testing wants to control its destiny it needs to have a seat at the table. The table is the business.

What will make the difference in 2009?

January 6, 2009 11:21 AM |

0 Comments

As we approach New Year I started to reflect, as cliché as that sounds, on the conversations I had during the Thanksgiving and Christmas holidays. Here in the United States most of the conversations in eventually meander to the economy. That usually spurs the questions of whom our tax dollars should be bailing out, followed by general sentiment that taxation is not the answer. I live in California and the governor has proposed the solution for our state budget crisis is to raise taxes. Most everyone I had cheer with over the holidays just about screamed "stop spending", is the answer. Not more taxation to solve a spending addiction.

Now I have friends and relatives in all walks of life, from the state government, medical, retired, tech, paint business, auto body, car parts, green cars and green products, water deliver drivers, students, horse ranchers, and even a couple of mechanics. So what does this have to do with testing? Little.

But their sentiment has me thinking about how this translates into business. One topic that these conversations led to, was the genuine feeling that customer service was coming back, and that even their Starbucks, Wal-Mart, Barns and Nobles were friendlier, and they were receiving coupons and offers that were not "just spend more", but free stuff, and items that made them feel valued as patrons. Heck last week I went to lunch and when leaving received a business card from the restaurant for a free calamari appetizer next time I was in.

Sure we can interpret this as desperate attempts to win our business and extract the last few dollars in pockets; that is, if we are pessimistic. But what I read is the obvious, next year will be tight, individuals and business will be conservative with their $$$. So as you think about your business and your customers in '09, you need to think about customer service and you need to act on it. And believe it or not, much of your customer service revolves around the quality of your product. Ask yourself this question, are you contacting your customers because of poor product or support reasons or can you be proactive to just say you are appreciative of their business?

Happy New Year

David

Does a downturn economy mean less testing jobs?

December 19, 2008 5:51 PM |

0 Comments

I had just written an article on perftesting.org, titled "Doing More with the Same--How to Make the Most of Testing Resources" when these questions arose. "What do I think will happen in the testing industry in the down turn? What will happen to testers who focus on system and device testing?"

Let's look at it from the business standpoint. I think we will see some reducing demand for products, but emerging countries will keep both Network Equipment Manufacturers(NEMs) and Service Providers (SPs) busy as they are still forecasting growth in those areas and I read press releases with big dollar contracts awarded each and every week. So it looks like at this point any massive pull back will be delayed, even with Cisco $1+B cuts it is very small in relation to their overall budgets.

So in generally we should see slowing of product development and thus reduced number of new products in the market which requires less testing. Then that is a reduction of focus and size of testing teams? No, and I admit it seems illogically, but over the last few years it has been a little known fact that testing couldn't keep up development often forcing the time to market versus testing coverage question, thus rolling the quality dice. With insight into 70+ companies over the last couple years; only one was reducing their testing team size. In fact, almost every team I met was hiring. Hiring was hard, balancing hard to find testing skills, with product and segment knowledge. So, lots of open head count in the testing team even over the last year when America was already in the recession.

But even though management has been rolling the dice, they know that quality is a dangerous gamble. If you sacrifice quality you not only risk initial business, and follow on business from existing customer but you also incur costs. Costs to support, and higher costs to fix and redeploy. So what does that mean? It means that in general I think we will see test teams staying the same size, some may lose open headcount, and some will get a reduced slightly. There will be some ongoing debates once again about outsourcing to cheaper resources or letting contractors go to retain employee headcounts. Some companies have testing operations in emerging countries so they are both less expensive and employees. But that seems to be spurring the thoughts around centralizing testing versus having groups co-located with engineering.

But in general organizations other than testing will suffer more in this down turn economy. This can lead to engineering mantra of "keep your head down". However, QA should not lean back in the easy chair, they should figure out how to get more done with same. QA can really take a leadership position and set an example in the company. Realize QA really does affect the top-line and bottom-line and in a downturn as uncomfortable as it may be, it is an opportunity. There is market share to be had, as the weak pause the strong will consume. QA can give their company the advantage to go with the appetite.

David

Welcome to the blog on Quality and all things testing.

December 10, 2008 12:39 PM |

0 Comments

Welcome to the blog on system and device testing.

I am excited to be writing about a space that is really growing in many ways. This testing discipline helps keep those iPhones and blackberries working, ensures that cable box delivers, and that email has 5 nines reliability. Thus those who test these devices keep the networks up and the devices that run on them working smoothly. This group of testers have also been doing yeoman's work over the last 10 years often testing with manual brute force and home build tools and systems. They often could not benefit from tools and advancements we saw in the IT world, but that is changing.

In this blog I will often focus on the business impact and decisions around quality and testing. We will certainly talk about the best practices and methodologies to improve QA and other departments. These discussions will be focused on getting more testing done faster, to drive costs down, and deliver products to market faster. After all, this is where QA and testing needs to contribute to the company, and if successful should provide QA a seat at big boy table and not in closet. So we can talk about really getting a regression system and process working, how to build a resilient web testing solution but seldom we will dive down into how do test a HTML pop up failure dialog box.

We will also stray outside the pure system and device testing to talk about application testing, and frankly some significant overlap as consumer network products are using web GUI and other applications. Debate, is valued. This space is changing and maturing rapidly and while there some market leaders, there are many right ways to solve quality problems. Some of which have nothing to do with technology. For example; Is an assembly line QA philosophy superior to a cradle to grave approach? I look forward to dialog and hope to see system and device testing take more of a spotlight like the devices they test.

David

The opinions and views expressed in comments, blogs, etc. are those of the authors alone and not necessarily those of TMC, TMCnet, or its editors. TMCnet reserves the right to edit, delete, or otherwise make changes to the content that appears on these pages at its own discretion and as it deems necessary.

System and Device Testing

Is Testing Getting Harder

Testing Next Generation Networks (NGN, LTE, 4G, WiMax)

Will you run the right POC or get the answer you want?

Tags:

Diagnostics versus Testing

What is automated testing?

Tags:

TMC 2008 Communications Solutions Product of the Year Award

Next Generation Testing Tools

What is testing really for?

Testing Worldwide

Expert Abstraction

Do you know what abstraction is?

Tags:

Moving Up. Time for a Real Testing Career.

What will make the difference in 2009?

Does a downturn economy mean less testing jobs?

Welcome to the blog on Quality and all things testing.

Search

Recent Entries

Around TMCnet Blogs

Latest Whitepapers

TMCnet Videos

Tags:

Tags:

Tags:

Search

Recent Entries

Subscribe to Blog

Around TMCnet Blogs

Latest Whitepapers

TMCnet Videos