Rapid Testing Intensive - Day 4

Day 4 of Rapid Testing Intensive with James and Jon Bach was another fun-filled (and full) day.

Here are some of the things we discussed today:

We started the with them showing us a graph of all of the bugs we have found in the last several days testing eBay Motors.  One of the important things for something like a bug graph in Rapid Software Testing is to ask yourself if the graph is truthful (filter out duplicates, etc.). If not, it's just noise.  Something like a bug graph may not always seem like something that provides value to your testing, but it may provide value to stake holders, so don't have it be misleading by not cleaning it up first.  This seems obvious, but it's easy to be misleading sometimes without meaning to.

Another thing James talked about was using documentation, don't look for some magical document that has the secrets to the universe.  Behind every document there needs to be someone you take seriously.  This also a very good thing to remember when you put together your own documentation.

Modeling:
I've always been a visual person, so drawing out how a product works when I'm new to it has come pretty natural to me. But I've really learned from James how important modeling and modeling the product's interactions is. For instance:

• If you're doing code-based testing, model around elements of the code.
• If you're doing state-based testing, model around elements of the states.
• Learn enough about the product to identify interacting variables.
• Before creating your models, survey the prodcut by reading, interviewing people, reviewing field data, doing exploratory testing, scenario testing, user testing, or otherwise keeping your eyes open during any other kind of testing.
• All testing is based on models. Without models, you can't test systematically. And you cannot systematically test anything important unless you first know that it is THERE.
• Turn options off/turn options on, play/test, see which ones actually interact.

State-based testing:

• Create a state-based model.
• Check transitions.
• All the ways to get into states and out of states (input and output).
• Any variables that get modified.
• Look for any sub-states.
• Combination testing.
• Tours.
• Flows of states.
• Understand how other variable effect state model.
• This is all part of deep coverage testing instead of just looking at product and wondering "what does this button do".

Combination-based testing:

• What if you have a lot of variables and want to test - this is combination-based testing.
• First step in combination-based testing: identify the interacting variables. Find variables that "might" interact and any "potential" risks. There are a LOT of variables in your product.

After having a representative from uTest (Testers, check it out), we ended the day doing some combination-based testing on eBay Motors.  Here is the combinations I came up with along with the results.
     

SECTION	COMBO	RESULTS
Motor -> Cars/Trucks	No fields (unclick New & Used)	43,699 matches found
Motor -> Cars/Trucks	New only	5,232 matches found
Motor -> Cars/Trucks	Used only	38,470 matches found
Motor -> Cars/Trucks	New & Used only	43,701 matches found
Motor -> Cars/Trucks	New & BMW only	11 matches found
Motor -> Cars/Trucks	Used & BMW only	2,064 matches found
Motor -> Cars/Trucks	New & Used BMW only	2,075 matches found
Motor -> Cars/Trucks	New, BMW, 3 series only	4 matches found
Motor -> Cars/Trucks	Used, BMW, 3 series only	600 matches found
Motor -> Cars/Trucks	New, Used, BMW, 3 series only	604 matches found
Motor -> Cars/Trucks	New, Used, BMW, Zip (80301), 3 series only	604 matches found