Testing Metrics & Measurement

Metrics & Management
You are conducting testing for a product and in middle of testing, Product Owner (PO) would like to know the quality of the product. What to answer? How to measure the quality of the product?
If number of open defects are huge, especially the high severity defects, than it is easy to answer this question - "At a moment product sucks". But how about if number of open defects are very less in number, especially if no high severity defect exists? Is the quality of the product good? It depends on how long high severity defects are not being found and other important aspect is to check the coverage of testing. What percentage of requirements tested? What percentage of code has been covered during the testing? If coverage is less than quality of product can't be measured rather only quality of features can be explained.
If number of high severity defects are less in number and not being raised for a longer time and coverage is also almost more than 70%. Does this mean quality of the product is qood? At a moment, I will say yes but reality will be known only when the product will be released to market and number of defects being identified by customers. If they are more in number, then it is a time for retrospective to find out where and what exactly got missed and what can be done to fix this in future. Does it require different testing strategy? Are there any skill gaps present in test team? Was quality of testing sufficient? Does it requires any process in place e.g. review process?
Is this possible to target zero defects found by customers? Answer would be NO but manager should strive for that as this improves productivity and process. TQM, ISO 9000, Six Sigma are answer to it but remember this increases cost of quality as well. It is important to know how much process is enough for team.
Once product owner is satisfied with your answer, project manager comes and ask that how much more effort will be required to complete the testing on schedule? Or, the question can be are we on schedule?
It can be answered well only if you know how many test cases are left for execution? How much effort is being put for defects re-testing? What is the rate of incoming defects? Effort required in re-testing of bug fixing? How much planned testing efforts are left?
If testing team meets the schedule than good, have party :) and try to improve the productivity or process to achieve the same goal with less effort. In can be automation, usage of tools etc. If schedule is missed, than have retrospecitve meeting and find it out - What extra effort than planned were put during testing and why? If extra effort were put than find it out why? Why it got missed? How much time were missed for waiting for build or for testing? Were defects fixed late in cycle and re-testing took extra effort? Was re-testing planned? Is this a team capability issue?
Product Quality is being measured correctly and project is also on schedule, the next thing to fix is - ROI, an important aspect for business and where your promotion is dependent. You can measure product quality well and always on schedule but how much it cost to project? What is the size of team being used for this? How many high cost resources are being utilized? Are licensed tools being used? Can it be replaced with open source or free tool? How about the lab cost? Can it be reduced by using virtulization? Will conducting testing on cloud be helpful? Do we require to conduct full regression cycle or partial regression testing will be able to address all major risks? Can testing time or effort be decreased without increasing business risk? Do you have numbers with you, suggesting total cost of testing of the product? Cost of team salaries? Cost of hardware resources required? Cost of application licenses? Cost of infrastructure need for testing team? Cost of support personnel required for testing team? Cost of management overhead due to team size? Cost of process implementation? Cost of measuring and analyzing?

Comments

Popular posts from this blog

Performance Test Run Report Template

Understanding Blockchain

Bugs Management in Agile Project