Practical measurement of Outsourced Product Testing

Abstract

The success of an outsourced product testing effort is a function of a number of factors, not all of which are easily measurable. Traditional metrics used for this have tended to focus on objective attributes such as “productivity per tester” but have generally ignored more subjective factors such as “quality of communication between client and vendor teams”. This paper describes a methodology used at Aztecsoft named "Service Measurement Methodology" that seeks to address these shortcomings of traditional approaches. We outline limitations of traditional approaches and describe the “Test Quality Index (TQI)”, a metric that addresses many of these limitations.

Over the past few years we have collected large quantities of data to test the efficacy of the approach described here. We conclude by presenting this data and comparing the efficacy of our approach with that of tra ditional approaches such as “Customer Satisfaction Index”.

Introduction

The last few years have seen an exponential growth in the level of testing services outsourced to India. Simultaneously there has also been a drastic change in the nature of test services offered from India. These trends are a result of a number of factors that include:
- Lower cost
- Closing of gap between the abilities of onshore and offshore capabi lities
- Faster release cycles as a result of round-the-clock product development achieved through distribution of work across multiple geographies

Managing this exponential growth in levels of business without di luting the quality of service is a major challenge that vendors in this space have had to grapple with. One prerequisite for even attempting to address this challenge is to be able to measure the quality of service. Indeed, it could be argued that this ability may be the differentiator between companies that have “made it” versus those that have fallen by the wayside.

There have been a number of attempts at designing such metrics. Most of these have tended to focus on objective metrics such as:
- Productivity per person day for activities such as test case design and/or test case execution
- Number of defects identified during the test case review
- Percentage of test coverage

These metrics have been useful in improving specific processes and in precisely quantifying deliverables but have been lacking in the abi lity to capture the quality of service delivered. In addition, they fail to ca pture important “soft” attributes such as “quality of communication”.

At Aztecsoft we designed “Service Measurement Methodology”, an assessment methodology that attempts to factor in tangible as well as intangible observations about a project in attempting to measure the quality of service provided on the project. This methodology was developed over the past couple of years and is currently used extensively in Aztecsoft Independent Testing projects.

Click here to read the complete paper