Agile methodologies were created to provide an alternative means for software product development that would avoid “plan-driven” methodologies’ long development cycles, high expense, and sluggish response to changing requirements.    They have been hugely successful.   Recently, Microsoft publicly announced that it was encouraging its development groups to use Scrum project management.

One disadvantage of Scrum is that, depending on the balance struck between sharing tacit knowledge through collocation of all members of the Scrum team, and documenting project features and requirements through artifacts such as use cases, acceptance test cases, and backlogs, a Scrum project may lack sufficient documentation to be understood outside the Scrum team, not only as regards technical specifications, user requirements, and test cases, but also its processes.  Someone may need to know what happened inside a sprint.  How thoroughly was this code increment tested?  Can we confidently refer to test cases and user requirements when planning integration testing?

    Scrum’s primary measure of success (following the Agile Manifesto ) is working software.  Scrum practices continuous integration, so, if a Scrum team has completed its sprint, it has delivered working software.   The main question for test managers is whether Scrum can scale.  If the ideal Scrum team’s size is 5 to 9 people (cf. Schwaber), and a project is large enough that it must run three or more sprints in parallel, common standards and processes will need to be coordinated between multiple groups to achieve the major quality attributes of process reliability, consistency, and predictability.   Is there a way to adapt traditional capability/maturity models like CMM to achieve that end?
There is.  It is so similar in structure to Scrum that we have called it Scrum QA Assessment.

Our company was asked by the IT division of an American state government agency (“The Agency”, in this paper) to assess their Quality Assurance (QA) capability and maturity levels, and develop a plan for improvement that would enable them to support the development and delivery of a Service Oriented Architecture version of their case management, records management, and data exchange systems.  They planned to use Scrum as their project management methodology.

Figure 1: Scrum Process Overview

Early in the discovery phase we realized that QA maturity and capability levels were radically inconsistent across The Agency’s product groups.   While one group practiced statistical process management, another relied upon automation that had not been maintained for months; still another used only unit tests and bug-fixing.  What caused the diversity?  Three factors were responsible:

• The Agency did not have a separate QA organization, so there were no common policies.
• Several “latest and greatest” technologies had been abandoned, leaving behind random procedures and checklists that didn’t fit into anyone’s plans.
• Each Scrum Master (Scrum team leader), believing that he or she alone had implemented Scrum correctly, stubbornly resisted efforts to align their team’s procedures with others

 We wanted to find a guideline that would enable us to target specific improvements for recessive groups; foreground and preserve solid QA practices in others; and, last but not least, provide an overall roadmap for moving the entire IT division up a couple of levels.

Fig. 2:  Frameworks Quagmire

We also wanted to find a way to bring about process reliability, consistency and predictability, and do so without violating the Scrum principle that you don’t want anyone outside your team telling you how to run your sprint.

None of the many standards, models and methods that were available (see Figure 2) met all three of these criteria.  We could, it was proposed, “cherry-pick” standards and practices from several different frameworks, a sort of cafeteria approach to assessment.  We could, for example, resolve many problems at The Agency by clearly and comprehensively defining their QA processes, but the procedures recommended by frameworks for process definition assumed that other processes, such as governance, had already been introduced and established, which was certainly not the case at The Agency.   Since we found this tight coupling in many areas, we decided that, rather than spend our limited time in manipulating an existing framework, we would construct our own.

We developed a point-by-point matrix consisting of 52 criteria for software quality engineering and software testing that we selected for their relevance to SOA project plans, to legacy maintenance that would continue during SOA development, and to the organizational structure we thought would be able to support those projects.  We rated The Agency’s practices and procedures in two ways:

• The Current Capability Index measured the potential realized by QA practices and processes in terms of The Agency’s present resources, organizational structure, management practices, and understanding of QA.   We scored practices and processes for each of the criteria on a scale of 1 thru 5 points (5 being the most desirable score).   We expressed the sum of all Current Capability scores as a percentage of the total possible points for all 52 criteria, and called it the Current Capability Index.  A Current Capability Index of 100 would indicate that The Agency had fully realized its QA potential for its present level of maturity.
  
  
• We calculated the Readiness Index in a similar manner.  A Readiness Index of 100 would indicate that The Agency had fully realized its QA potential for the level of maturity required to execute their anticipated SOA project.  For example, Figure 3 shows  was the 22nd item in the matrix:

The CC  (Current Capability) score was 4 (out of 5).  The Agency did not review all test cases for completeness (which would be scored as 5); nor did all groups review test cases irregularly (which would be scored as a 3).   Many but not all groups reviewed test case regularly; so we assigned a score of 4.  We favored an upward interpretation because we knew that, given a choice, The Agency was more likely to adopt best practices than bad practices.   The R (Readiness) score was only 2, because failure to review test cases for completeness would carry much more risk in The Agency’s proposed SOA project.  

The Agency’s total Current Capability Index was 64; their Quality Readiness Index was 40.  Even if their total Current Capability Index had been 100, their Quality Readiness Index would have been much lower, because The Agency’s present resources, organization structure, etc. were inadequate to realize its business objectives.  

In the end, The Agency elected to “pause” their project for several months because they understood that their project management, software development, and quality assurance processes and methodologies were too immature to proceed without critical risk.

Our assessment model had several interesting features. 

• It combined aspects of both staged and continuous improvement models.  Staged improvement provides a road map that an organization can follow, step by sequenced step, toward its chosen goal.  Continuous improvement emphasizes growth and improvement in individual process areas.  
• Secondly, although we assessed each process area by itself, we developed capability profiles for the division as a whole.  This can be represented graphically, by group, process, or process area:

Figure 4: Capability profile (illustration only )

• Third, we achieved assessment specificity, because the criteria we used to measure their capacity were based on specific business objectives (the SOA project and all that it entailed) and their demonstrated process management structures and skills.   

This approach does not violate the spirit of Scrum, nor does it violate the discipline of continuous process improvement: “SW-CMM tells what to do in general terms, but does not say how to do it. … The implementation of those methodologies must be aligned with the spirit of the agile philosophy and with the needs and interests of the customer and other stakeholders.”

It has become fashionable to claim that agile methodologies like Scrum do not plan; in fact, both Figure 1 and Figure 3 illustrate that planning is a constant activity in agile methodologies.  With daily scrums to review status and with daily updates of the backlog, Scrum is always in a planning phase, ready to adapt to emergent circumstances.   Scrum QA Assessment, because of its assessment specificity and integration of continuous and staged improvement, can be updated as each process improvement goal is achieved, or as each new impediment emerges.  It is not necessary to bring in outside experts on a periodic basis; the expert is very local – it may be the test manager coordinating with Scrum Masters, or it may be a Scrum Master assessing his or her own group.

Fig 5: Waterfall and agile models of development

This approach to assessment, and to planning for continuous improvement, can be called Scrum QA Assessment because it is consistent with the problem-solving technique that inspired the development of agile methodologies (including Scrum) in the 1990s.  Christopher Alexander, a professor of architecture at the University of California, wrote two books, A Timeless Way of Building and A Pattern Language, that inspired a generation of architects.   The same books, and preparations for those books, inspired the Silicon Valley software community of the early 1990s.   Alexander introduced the use of “patterns” in problem-solving.  A pattern is a recurring structural configuration that solves a problem in a context, contributing to the wholeness of some whole, or system, which reflects some aesthetic or cultural value.  By adding structure (patterns) that encapsulates our professional and personal experience, we can gradually bring systems into balance through piecemeal changes, judging their effects by empirical observations (cf. Schwaber’s insistence on empirical process control).

 “Conway’s Law”, for example, is a pattern that ties the organization to the architecture of the system under test: 

If the parts of an organization (e.g., teams, departments, or subdivisions) do not closely reflect the essential parts of the product, or if the relationships between organizations do not reflect the relationships between product parts, then the project will be in trouble….Therefore: Make sure the organization is compatible with the product architecture.  An organization will have periodic reviews of the architecture and potentially of project management strategies [as in daily stand-up meetings].  At each of these meetings care should be taken to align the structure of the architecture with the structure of the organization by making piecemeal changes to one or the other.

One would consider using this pattern if a product was undergoing frequent changes, or if a QA group had been assigned to a new project whose architecture was very different from their previous project.    The need for applying such a pattern is determined through Scrum QA Assessment.

The best thing that a project manager or test manager or Scrum Master can do in agile project management is “to lead a culture where it wants to go”.   That requires assessment.    It may be assessment within a sprint:

• What is this group capable of?
• What additional resources do I need?
• What techniques will optimize our work?
• What statistical measures and controls can I introduce that will increase the team’s desire to succeed?
• Is this project sufficiently critical to justify enforcing traceability?
• Should I engage external teams for work that is not tightly coupled to the sprint backlog?

Or it may be assessment of the entire QA function within a company.  
From a less technical, more human perspective, consistent with Alexander’s instruction in the use of patterns (context and inherent forces must drive configuration), the test manager should constantly strive to understand the team’s or organization’s current strengths and weaknesses, gauge the capacity and interests of all the people (developers, company culture, customers) who will constrain or enable the work to be done, and continue to build a personal library of patterns, including those learned from others (including from books such as those listed in the footnotes of this paper).   Those patterns should be applied, in Scrum QA Assessment, as needed to understand and meet quality and business objectives.