|
Abstract
The success of an outsourced product testing effort is a function of a number of factors, not all of which are easily measurable. Traditional metrics used for this have tended to focus on objective attributes such as "productivity per tester" but have generally ignored more subjective factors such as "quality of communication between client and vendor teams". This paper describes a methodology used at Aztecsoft named 'Service Measurement Methodology' that seeks to address these shortcomings of traditional approaches. We outline limitations of traditional approaches and describe the "Test Quality Index (TQI)", a metric that addresses many of these limitations.
Over the past few years we have collected large quantities of data to test the efficacy of the approach
described here. We conclude by presenting this data and comparing the efficacy of our approach with that of traditional approaches such as "Customer Satisfaction Index".
|
Introduction
The last few years have seen an exponential growth in the level of testing services outsourced to India1. Simultaneously there has also been a drastic change in the nature of test services offered from India. These trends are a result of a number of factors that include:
- Lower cost
- Closing of gap between the abilities of onshore and offshore capabilities
- Faster release cycles as a result of round-the-clock product development achieved through distribution of work across multiple geographies
Managing this exponential growth in levels of business without diluting the quality of service is a major
challenge that vendors in this space have had to grapple with. One prerequisite for even attempting to address this challenge is to be able to measure the quality of service. Indeed, it could be argued that this ability may be the differentiator between companies that have "made it" versus those that have fallen by the wayside.
There have been a number of attempts at designing such metrics. Most of these have tended to focus on
objective metrics such as:
- Productivity per person day for activities such as test case design and/or test case execution
- Number of defects identified during the test case review
- Percentage of test coverage
These metrics have been useful in improving specific processes and in precisely quantifying deliverables but have been lacking in the ability to capture the quality of service delivered. In addition, they fail to capture important "soft" attributes such as "quality of communication".
At Aztecsoft we designed "Service Measurement Methodology", an assessment methodology that attempts to factor in tangible as well as intangible observations about a project in attempting to measure the quality of service provided on the project. This methodology was developed over the past couple of years and is currently used extensively in Aztecsoft Independent Testing projects.
Section 'Conceptualize the "Service Measurement Methodology"' describes how we went about designing this methodology, Section 'Design a tool to measure' enumerates the elements of this methodology and describes artifacts making up this methodology in detail. Section 'Structure of TQI' explains actual structure of TQI. We conclude by looking at the methodology in action - section Case Study: Customer Response' contains a case study of how this methodology was applied on a particular project.
Click here to download the white paper
|
