Managing QA for Software Patches

Introduction

Whereas version updates of software may include feature enhancements as well as several bugfixes, patches typically include only one bugfix or at the most a cluster of bugfixes addressing a single defect.   Vendors are willing to undertake comprehensive quality assurance for version updates.   But the same vendors assume that a patch requires much less effort, because its scope of impact is presumed to be restricted.   They may only require that new code pass its unit tests, or a few hours of ad hoc testing.  From the standpoint of best practices in QA, though, this is very risky.  We propose that risk mitigation be as carefully considered for software patches as for version updates, up to and including the identification of software metrics by which the quality of software patches can be confidently assessed.

Difference between Version Upgrade and Patch

Version Upgrade

A version upgrade is a collection of updates that fixes issues in the product. The criteria for including updates is based on customer feedback and internal testing since the last release. The goal is to deliver a well-tested, comprehensive set of fixes that is suitable for use on any customer's computer. The version upgrade addresses a wide variety of bugs. Version upgrades may addresses not only security bugs, but also bugs affecting stability, performance, application compatibility, proper operation of product features – in fact any area for which there is customer demand or internal requirement. These updates are cumulative, in that each update is a "roll-up" of all previous updates for that product.

Version upgrades are quality-driven and require extensive test cycles. That is, Company won't release a version upgrade until it meets the same quality standards as the product itself. Version upgrades should be constantly tested as they're built, and then undergo rigorous final testing that includes end-to-end testing, integration testing, and if necessary testing of compatibility with hundreds or even thousands of third-party products. If testing reveals bugs that prevent the product from meeting the quality standard, Company may postpone the patch’s release.

Patch

Patches are mission critical fixes and enhancements to the released product on a per customer basis. Patch refers to the individual component fix. In most cases, patches are usually single-issue fixes released specifically to the single customer who requested the fix. 

Risk Assessment

Risk assessment and a study of technical viability are required before the patch is created.   All of the following questions affect not only development but also QA planning around the patch:

• Is there a workaround for the defect?
  
  
• What is the impact of the defect we propose to correct?   Is it so severe that it cannot wait for the next version upgrade?  
  
  
• Should the patch be made available to all customers, or does its impact depend on how customers have deployed it?  
  
  
• What is the estimated cost of developing and testing a fix for the problem?  Is the cost justified in terms of risk mitigation?
  
  
• How stable is the code in which we propose to make the fix?   What degree of confidence do we have that a fix will not introduce additional problems? 
  
  
• What is the worst case for collateral damage of this bugfix?  What is its likelihood? 

Each change in code may lead to a separate problem.   The impact of the patch cannot be measured by the number of lines of code that are changed or added.  Changing a single operator in one line of code may have greater impact than several changed lines of code elsewhere.  Development should report the result of their risk assessment to QA; QA should review and sign off when understood and accepted.   QA then creates tests to verify technical implementation of the proposed changes.   If users will see changes in system behavior, the user requirements should be reviewed and revised to reflect that impact, and test cases altered or created to validate the new requirements.  

Once a decision has been reached to develop a patch, the project should be treated as a software release, and subjected to the same discipline and rigor as other sustained engineering projects.

Aligning Patch Testing with the Best Practices of Sustained Engineering

Since every bugfix included in a patch must be rolled into future product releases, its development must be conducted within the constraints and regulations set by the vendor’s configuration management program.

Its progress should be monitored efficiently through status reports standardized within the vendor organization.  Typically, vendors decide to develop patches only when they correct highly visible bugs, or bugs of high severity.  Patches should therefore be given the highest priority by both development and QA.   Any delays should be escalated.   The high priority of the patch is inherited by any internal or external components on which it depends. 

Breadth of impact of the patch should be considered not only in terms of functionality, but in terms of hardware and software compatibility where applicable. 

Early in the planning of QA for the patch, tests should be evaluated in terms of viability for automation.  Although automation will probably not deliver sufficient ROI during the period of patch development, QA should take into consideration its long-term payback in regression testing for future patches and releases.s

An example of a well-known patch that Microsoft had to release:

In 2003, Eye digital network found a critical defect (Security loophole) in Microsoft Products dealing with RPC service running in Windows OS. The defect  involved a buffer overflow condition in RPC service in all versions of current Microsoft operating systems. Since it was a security issue, a remote attacker could exploit this vulnerability and gain access to the system. Someone did, resulting in the virus known as w32.nachi and blaster worms that caused loss of ~1$ billion world-wide.
Microsoft responded with a fix (patch) called MS03026. The patch was thoroughly tested, considering all test factors, in a short period. This was possible due to the mature QA and Dev processes at Microsoft.
However, that wasn’t the end of it. Since Microsoftanticipated that there could be several other ways to exploit related vulnerabilities,  they developed and tested fixes for those vulnerabilities in the next version operating system upgrade (service pack, or SP).

Common challenges that might be encountered during patch development and testing:

• Patch doesn’t fix the problem
  
  
• Patch causes regression in another area
  
  
• Patch is claimed to be available in build X but is actually available in Build Y
  
  
• Build X is missing files on which bugfix in patch is dependent
  
  
• Chronically low QA resources; non-QA resources are called on for testing
  
  
• Analysis of patch’s impact shows that an entire regression test cycle will be required
  
  
• Spikes in resource needs arise unexpectedly in two or more projects simultaneously
  
  
• Due to inefficient or incompetent use of configuration management
  
  
  • There is no way to restrict release of a patch, so all patches tend to be propagated on whatever the next build is; or
  
  
  
  • Test cases are not updated to reflect new or changed requirements; or
  
  
  
  • Test environment is not updated to reflect new or changed requirements
  
  

• Tester is not familiar with the area that s/he is required to test for the patch
  
  
• No method is available to verify whether all earlier fixes get propagated to the next version of the product
  
  
• No method is available for comparing QA effort estimates against actual QA effort.
  
  
• No reports are regularly compiled showing patch recidivism (failure of released patches in the field)
  
  
• No reports are regularly compiled showing elapsed time between start of patch development and release to test
  
  
• No reports are regularly compiled showing elapsed time between release to test and sign-off for release

Best practices to be followed in order to mitigate the above risks/challenges:

Work Flow:

• Understand the impact of a code change and prioritize test efforts accordingly.



• Code coverage tool can be used to analysis the code changes. The tool should provide the following data:

• The coverage rate of each binary, provided by one test or by combining all tests 
  
  
• The coverage rate for each function in each binary, provided by one test or by combining all tests
  


• Which lines of code are not covered by any tests?
  


• Path of functions to call before you can hit one specific function.

• Such information will help pinpoint test holes, prioritize test cases to run, and improve test case code
  
  
• Accelerate the coverage of new and changed code by executing the right tests at the right time



• If there are code changes in “A” area, its always better to start the testing with that component and then run regression passes for “B” and “C” etc
  


• Build a regression test suite for all components so that before releasing a Patch, respective regression cases can be run effectively.

• Through smart grouping and sequencing of multiple test cases, reduce test pass time. This helps reduce actual time required for testing.
  
  
• Automate all test cases (assuming that they are well-formed, meaningful test cases) that can be automated.

• Reduce test bed setup time using Platform images and a dedicated infrastructure for complex setups. MS Virtual Server or VMware images can be used for this purpose.
  
  
• Use a sophisticated defect management tool for tracking regression bugs and their fixes. Before closing a bug, test managers should fill the following analysis report to ensure that proper action has been taken. Here are the report’s mandatory fields:
  
  
  • How was this bug verified? List the steps, manual tests or tools etc to verify this bug.- - - How will the new test cases / tools ensure future coverage? Provide pointer to updated test plan / tools.
  
  
  
  • List test areas that would be affected by the bug.
  
  
  
• Test and Dev managers should meet regularly to review and verify that the analysis has been completed on all closed bugs.
  
  
• Use a sophisticated result-reporting tool for tracking test results. Here are some common expectations from such test reports :
  
  
  • Reports must show test status for the patch under test.
  
  
  
  • Reports must show the amount of time spent in various stages during the lifetime of any patch
  
  
  
  • Reports must always be available for generation on demand
  
  
  
  • Reports must support color coding entries that meet specified criteria for better visibility- Every report must include summary statistics of the bugs reported
  
  
  
  • Reports must show patches by category where the categories are pre-defined and chosen by the user
  
  
  
  • Reports must support sorting on all fields shown in the report
  
  
  
  • Reports must support HTML and XML, and can be exported to Excel (because it is widely used for viewing results).
  
  
  
• Define defect triage, defect management, and patch release criteria completely, and review them periodically to formalize or reject changes in practice.
  
  
• Analyze regressions and devise and formalize procedures for preventing them. Postmortem reports should be prepared by test/dev managers to analyze the regressions that occurred in the patch. Reports should at-least contain the following information:
  
  
  • Detailed explanation of the problem resulting in the regression
  
  
  
  • How was the issue found?
  
  
  
  • If Test or BVT did not catch it, then explain the reason
  
  
  
  • What are steps being taken to prevent this from happening again?
  
  
  
• Devise vendor-specific comprehensive metrics for patch management. The checklist helps test teams to verify the patch’s contents and accuracy.
  
  
  • Have you verified that the bug or bugs targeted by the patch have been fixed?
  
  
  
  • Did you verify that the test procedure you used to test a bug fix can reproduce the bug on the build on which the bug was originally reported?
  
  
  
  • Has the fix been code reviewed by another developer?
  
  
  
  • Has the fix been code reviewed by a tester?
  
  
  
  • Are there any dependencies? If yes, have appropriate actions been taken?
  
  
  
  • Have you verified the checksum/versions for each binary in the Patch?
  
  
  
  • Have you verified the setup (installation) variations?
  
  
  
  • Did you complete functional testing for all affected components?
  
  
  
  • Have you verify the fix on all supported platforms? General rule is to focus testing on the most recent version update of the product, and conduct at least a sanity pass on the next most recent one
  
  
  
  • Have the release criteria for each affected component been met?
  
  (Release criteria must be measurable; metrics used to verify release criteria may include defined levels of stress, test tiers completed, bug-free internal deployment, etc.)

Conclusion

We have reviewed some of the general challenges in testing software patches, and sketched guidelines for systematic testing of patches. 

We have described approaches to risk assessment, rigorous processes that will forestall errors of commission and omission in testing, provisions in infrastructure that will foreground defects, and means for reducing the time required for, and therefore the cost of, patch testing. These guidelines can serve as the basis for your organization’s standard procedures for testing its patches (and, with modifications necessary to accommodate the testing of multiple patches, for testing patches included in version upgrades).