We continue to share the MMTR Technology experience and methodology. Anastasia (Quality Assurance engineer), from our QA department, outlines the Pairwise Test Method and shares her experience on how MMTR Technology employs it for running tests. She also shares examples of the method in use.
“All test techniques can be divided into two classes.” Anastasia, Quality Assurance Engineer
First are techniques focused on test data (test case) creation and second are techniques focused on test scenario creation which utilize operations sequences performed by testers.
“In our case, we are focused on techniques surrounding data creation.” Anastasia, Quality Assurance Engineer
This category can be further broken down into techniques that utilize separate parameters and those that use a combination of parameters.
“Parameter combination techniques are more effective when compared to other techniques.” Anastasia, Quality Assurance Engineer
Pairwise is one of the many combinatory test techniques that exist.
Why Combinatory Techniques and Pairwise?
What makes combinatory techniques the best choice and why Pairwise in particular?
“They make the process better at finding bugs caused by either obvious or implicit dependencies between parameters.” Anastasia, Quality Assurance Engineer
It is necessary to understand the main problem for testers who begin to combine parameters. The main problem is ‘combinatory explosion,’ which occurs when the option quantity or the combination quantity multiplies. With every new parameter, the combination quantity increases exponentially.
“Compared to polynomial functions, the exponential function growth rate is much faster.” Anastasia, Quality Assurance Engineer
You need to test a system with a large number of parameters that influence performance. Examples are system performance tests with different operating systems or website stress tests on different browsers. Each quality assurance engineer is aware of the impossibility of verifying all of the combinations. To illustrate this, checking all combinations of 10 parameters with 10 values each would require 10,000,000,000 tests. One tester or even a few testers can not execute this number of micro-tests.
“This is why there exists specially designed techniques meant to help resolve this issue.” Anastasia, Quality Assurance Engineer
The primary objective of these techniques is to reduce the number of micro-tests. Fewer tests mean less time required for testing. At the same, it is essential to keep the scope in mind. It is necessary to find the ‘golden mean’: minimizing the number of tests while covering the whole scope.
“The professional quality assurance engineer has to be able to kill these two birds with only one stone.” Anastasia, Quality Assurance Engineer
Pairwise testing: test data set formation technology for interacting values. (separate parameter testing is more suitable for non-interacting values)
Our example is a car rental website. We start simple with users able to choose the type of car as well as the color. We are asked to perform testing to ensure that everything is operating effectively. Testing requires picking the vehicle type and color, filling in the application form and then receiving the correct confirmation via email. There are approximately six test cases, the combination of which covers the scope. This approach is also called combination theory. You cover every parameter and, theoretically, it guarantees 100% test-case coverage. In this easy case, the testing is quick, easy, and manageable.
We now raise the difficulty by adding insurance deductibles, “credit card” or “cash” payment parameters and “days of rent” from 1 to 30. Now we have approximately 1800 test cases that need to be written and tested. It is possible to run tests, find and fix bugs, but the number of test cases makes it impossible to test every single possibility.
“In this scenario, we start to receive bug reports from clients.” Anastasia, Quality Assurance Engineer
It appears that if a user selects the option for a zero deductible and pays with a credit card, a bug appears.
“And we have 17% of clients renting their cars with these parameters.” Anastasia, Quality Assurance Engineer
By chance, we have not covered this combination with our test-cases and have not found any bugs. Among 1800 possible test cases there are only 180 tests (30 * 2 * 3 ) which contain this pair. 1620 tests fail to find the bug.
“This is what we call excess testing.” Anastasia, Quality Assurance Engineer
There are many tests, but there is only one that is required. Combination theory results in a resources over-expenditure.
According to research conducted by IBM in the seventies, they conclud that the majority of mistakes are caused by a combination of only 2 parameters which negatively coincide and result in undesirable results. Combinations of 3 or more parameters are rarer. Pairwise methodology states that you need to check each parameter at least once with another parameter. The success rate for Pairwise testing is 90%. By checking each possible pair, this type of testing discovers all but the rarest of bugs.
There are 2 conventional approaches to Pairwise testing: equivalent values and boundary values.
- Equivalent values (class): a group of values which are each the same as the other. (i.e., input parameter groups that have identical results)
We unify equivalent values into one group and only check one value from this group and equate this with checking all of the other values in the same group.
- Boundary values (class): where test cases are generated using the extremes of the input domain, – maximum, minimum, just inside / outside boundaries, typical values, and error values.
Let’s try to include boundary values and equivalent classes here. We unify as equivalent values all of the “Days of rent, ” and we leave the boundary values 1,2,29, and 30. “Color type,” “Car type,” “Payment,” and “Deduction” parameters cannot be unified into one equivalent class. We need to write our tests to cover all possible pairs of these values. We first find the parameter with the biggest number of values. In this case, it is the “Days of rent” parameter. There are two parameters that tie for second largest with three values and are “deduction,” and “color type”. It does not matter which one we pick. Therefore, we shall start with the two largest classes of parameters in a pair unification. For example, “Days of rent” and “Deductions.” For each day of rent, we include all of the possible deductions. Then, we add any another parameter. For example, we add a “Payment” parameter. It is also necessary to check that the credit card and cash payments types be included in every day of rent and the credit card and cash with different deductions as well. Now all three parameters with every value among them touch at least once. We then add the “car type” parameter. We check that every value has a pair. We also add the “color type” parameter. We can enter any parameter into empty cells.
“Our test design is ready with only 12 tests, unified with 77 pairs. Personally, this would take me only 20 minutes to complete writing the test cases.” Anastasia, Quality Assurance Engineer