Saturday, July 28, 2012

Rapid Testing Intensive - Day 5

Today was the final day of the Rapid Testing Intensive from James and Jon Bach.

This has been a great experience.  It really makes me want to take their full Rapid Software Testing class the next time it comes up.  My mind is full!  I am looking forward to using many of the things I learned this week.  And it's been great getting to hang out with a lot of Testers from all over the world for 5 days of learning, talking, and challenging ourselves about testing.  And one of the best parts throughout the whole thing was the humor and interaction between the Bach brothers.  It made the experience even sweeter and I feel very fortunate that I was able to participate in the first Rapid Testing Intensive. 

Friday, July 27, 2012

Rapid Testing Intensive - Day 4

Day 4 of Rapid Testing Intensive with James and Jon Bach was another fun-filled (and full) day.

Here are some of the things we discussed today:

We started the with them showing us a graph of all of the bugs we have found in the last several days testing eBay Motors.  One of the important things for something like a bug graph in Rapid Software Testing is to ask yourself if the graph is truthful (filter out duplicates, etc.). If not, it's just noise.  Something like a bug graph may not always seem like something that provides value to your testing, but it may provide value to stake holders, so don't have it be misleading by not cleaning it up first.  This seems obvious, but it's easy to be misleading sometimes without meaning to.

Another thing James talked about was using documentation, don't look for some magical document that has the secrets to the universe.  Behind every document there needs to be someone you take seriously.  This also a very good thing to remember when you put together your own documentation.

I've always been a visual person, so drawing out how a product works when I'm new to it has come pretty natural to me. But I've really learned from James how important modeling and modeling the product's interactions is. For instance:
  • If you're doing code-based testing, model around elements of the code.
  • If you're doing state-based testing, model around elements of the states.
  • Learn enough about the product to identify interacting variables.
  • Before creating your models, survey the prodcut by reading, interviewing people, reviewing field data, doing exploratory testing, scenario testing, user testing, or otherwise keeping your eyes open during any other kind of testing.
  • All testing is based on models. Without models, you can't test systematically. And you cannot systematically test anything important unless you first know that it is THERE.
  • Turn options off/turn options on, play/test, see which ones actually interact.
State-based testing:
  • Create a state-based model.
  • Check transitions.
  • All the ways to get into states and out of states (input and output).
  • Any variables that get modified.
  • Look for any sub-states.
  • Combination testing.
  • Tours.
  • Flows of states.
  • Understand how other variable effect state model.
  • This is all part of deep coverage testing instead of just looking at product and wondering "what does this button do".
Combination-based testing:
  • What if you have a lot of variables and want to test - this is combination-based testing.
  • First step in combination-based testing: identify the interacting variables. Find variables that "might" interact and any "potential" risks. There are a LOT of variables in your product.
After having a representative from uTest (Testers, check it out), we ended the day doing some combination-based testing on eBay Motors.  Here is the combinations I came up with along with the results.
Motor -> Cars/Trucks
No fields (unclick New & Used)
43,699 matches found
Motor -> Cars/Trucks
New only
5,232 matches found
Motor -> Cars/Trucks
Used only
38,470 matches found
Motor -> Cars/Trucks
New & Used only
43,701 matches found
Motor -> Cars/Trucks
New & BMW only
11 matches found
Motor -> Cars/Trucks
Used & BMW only
2,064 matches found
Motor -> Cars/Trucks
New & Used BMW only
2,075 matches found
Motor -> Cars/Trucks
New, BMW, 3 series only
4 matches found
Motor -> Cars/Trucks
Used, BMW, 3 series only
600 matches found
Motor -> Cars/Trucks
New, Used, BMW, 3 series only
604 matches found
Motor -> Cars/Trucks
New, Used, BMW, Zip (80301), 3 series only
604 matches found

Thursday, July 26, 2012

Rapid Testing Intensive - Day 3

Today was day 3 of Rapid Testing Intensive.  Unlike the other two days, today the "onsighters" and "onliners" split up.  Us onliners had the pleasure of working with Scott Barber, another master tester.  He gave a very interesting talk on Performance Testing.  We talked about TCO's (Test Coverage Outline), tools, and risks.

Here is the eBay performance TCO:

In the afternoon, James talked to us about Test Reporting and the use of Testing Tools. 

  • Heart of testing.
  • Structure which helps to manage testing.
  • A test report is any description, explanation, or justification of the status of a test project.
  • A comprehensive test report is all of those things together.
  • A professional test report is one thoughtfully designed to serve the clients of testing in context.
  • A test report isn't just the facts, it's a story about the facts. Learn to tell the testing story!
  • Make choice of which facts matter.
  • Practice test reporting. Even if your management doesn't want it, practice anyway.  It's one of the hearts of Rapid Testing.

  • Tools (and automation) are important for many aspects of testing the product, but they do not take the place of what humans can find in testing.
  • Do not get so dependent on a tool/automation that you miss things.
  • If you are spending so much time keeping the tool/automation running, you may not be thinking about what needs to be tested today.
  • Are you spending a lot of effort/time/money on tools/automation that keeps breaking?
  • Use free tools!
  • If your company buys an expensive tool, then you have to use it because the money was spent, but not if you're using a free tool.
  • They can work if your product is very easy to test and it doesn't change much.  Does that describe your product?
A nice mix in teams that James like:
  • Not all testers should be programmers.
  • Have at least one person on the team that loves tools/programming.
  • Someone that absolutely hates tools.
  • Someone good in math.
  • A good writer.
  • Liberal art majors (yay for us liberal arts majors!).
  • Musician.
  • The toolsmith should be directed by testers.