Tip:
Highlight text to annotate it
X
>> Hello all. Thank you for coming and I--I appreciate, especially because some of you
had to learn about to follow the back channels. So, try and get that straightened out on the
future tech log if they come to the regular system. My name is Harry Robinson, I work
up in Kirkland, I report to high-tech. And one of the jobs I had is to bring in, speakers
and one of the purpose of this, is I get to bring in speakers that I really enjoy and
heard from. So, John is one of those people. How many people have used Exploratory Testing
in someway? Okay. So have I, but part of the frustrations with it is to really affect what--how
do I know I've done a good job? How do I learn to do this better? But John, is actually one
of the people who's answered those questions in large way for me? So, now, he will answer
them for you. So, John, please. >> BACH: Thank you Harry. Thank you all for
being here. I want to correct something right away, for those who, who didn't raise your
hands, you've done Exploratory Testing. You just don't know it yet. Okay. How am I first
sound? Am I good, one? Okay, good. My name is John Bach and I've been testing for 10
years. My focus is Exploratory Testing or unscripted, unrehearsed, improvisational testing,
okay. And I want to show you, it felt like as well, I'll show you that you've all done
it. Because I have a lot of paradigmatic examples, okay. All right. I'm going to talk today about
some definitions and maybe this is why you didn't raise your hand, maybe you don't know
you're doing it, that is what--this is what I suspect. So, what I love--I love this magazine
by the way, I love Wired, anybody hear Wired? Okay. This is the latest issue I got at the
airport, I slipping through trying to find something, as I often do, accidentally in
Exploratory fashion for this talked. And I found something, page 30. It's titled, let's
see, "Your lousy Tinggo of you". And it's--it's--these are linguistic quirks around the world, for
example, "Nico, Nico". I don't know if I'm saying it right, that's the Indians nation
word of--oh sorry. Does anybody know what that is? Other than being an Indonesian word,
because I've given it away a little bit. The Indonesian word for someone with a novel idea
that actually make the situation worst. So, that's what they called that, all right. Isn't
that interesting. How about, "Exorapak", anyone? Exorapak, I maybe saying this wrong. So, though--well,
It's the Indian way of describing it, active repeatedly going outside to check, if someone
is coming, outside the igloo I would presume, I don't know what. Anyway, so these are interesting
words from, from different cultures. Well, I want to talk about--the main thing that
I want to talk about today is, what is Exploratory Testing and what are the words, we can use
to describe what it is we are doing, okay. I might actually going to show a curling video.
Does anyone know what curling is, the sport of curling? Okay. If you don't, you're about
to, okay. Because I want to take something, that looks really easy and say, "Oh, that's
easy anyone can do that". And see how many of you resonate with that. And then talk about
the terminology that they use. Okay. But first, first the Haiku. "Oh, masters of test, describe
to me your methods," "We launch in an hour". Okay, does this sound familiar? To anyone
when you have, like one hour to test or one day to test and you go, "Hey can we ship yet?"
And wait a minute you pry this out of my cold dead hands. Okay. That may be a sign that
you're doing Exploratory Testing, no test cases you know, just your wits, your intuition,
okay, no spec, okay. Okay. Here are my assumptions. Well, some of you have said you didn't do
Exploratory Testing. We'll get to that, but it's okay, if you haven't, if you--well, like
I said, worth it. But maybe, you might need to do it. So, you're got to know how it's
done, so you can at least care about it enough. And more importantly, do you know if you're
doing it well as Harry said. Okay. I going to flip my monitor, for one second, so, I
can see my screen. Okay good. Okay. Here's the problem that I face. Wasn't it just ran--random
pounding on the keys? You know, can any toddler, any child do it? What's--what's the big deal?
Okay. That's problem number one that I face. Problem number two, I don't know how this
tester does it? She's just amazing, give her 30 seconds, she can find a blue screen. I
mean all, I don't know how she does it, I just keep doing, keep doing it and you've
do what you do and I'll just report, the bugs you report. Camp A, camp B, right? My talk
today is attempting to bridge these two camps to say, you know, you people are in number
one, it's harder than it looks". And I'll explain, why it thinks it harder than it looks.
And you people in number two, don't worry, you don't have to be like a Zen master to
do this well. I'll show you some skills and tactics, 14 of them to be précised, okay.
So, hopefully we'll meet in the middle and then in my ultimate crusade is to evangelize,
if it's not apparent already, my candor, my diction or my tone. I want credibility for
Exploratory Testing, okay. I wanted to be--I wanted to be brought from the alleys in the
shadows and into the light, okay. Okay. This are common sentiments I've heard so, this
is my counter argument, this is how it's--Exploratory testing is absorbable, evaluatable and teachable
skills, okay. All right. Let me just demonstrate for those of you who said you weren't sure,
if you're doing your Exploratory Testing. Let me find the right talk here. Well--well
that's launching. Let me try an exploratory exercise right now, just off the cuff. Here
are some of ideas, I can think of with that little white that you saw before. I think
this going to be really slow to launch, so, I apologize for this. Okay. Let me toggle
us back, okay good. Okay. So, take three identical numbers right that's an equilateral triangle,
okay. Three different numbers makes a scalene triangle or three unequal sides. There's a
triangle called an isosceles triangle that makes two equal sides, okay. So, one of those
three triangles is what this program will create, okay. So, I don't have a spec, it's
in my head, you just heard the spec and I could think of ideas to, to test this, is
this working. So, I tried conformant test. Okay. Three, three, three equilateral, one,
two, three scalene, two, two three, isosceles, what about large numbers more than a million
character's, yes? >> One, two, three is not scalene.
>> BACH: Oh, thank you. So, one, two--so, three, four, five. Okay. This is important
now, I just heard some--some domain knowledge here be--the reason this is not a triangle
by the way, it's because the two's smallest sides have to be greater than the third side
for the triangle be legitimate. Okay. So, I don't expect to be trigonometry majors to
know this, but it's--it's--will help with the oracle's problem. Okay. The oracle's problem
is, do you know a bug when you see one? Okay. So, you may have to know some math, okay.
So, three, four, five is a scalene, okay. So, I try large numbers, I try large letters
in there, I try tab borders, let's just see, if I could get it to crash, holding the tab
keys, it's called a shoe test. What if I took my shoe and put on the space bar you know,
also known as the Ham-Sandwich Test. Because allegedly some are left there ham sandwich
on the return key, once upon a time in our folklore and it caused the blue screen, okay.
All right. There's a--I think, I can cut and paste into these fields you know, you see,
one--kind of scattered, right? So it does seem like a toddler could do it and then just
randomly pressing on the keys. But what if I gave myself a mission, and what if that
mission was still unscripted. But the mission was to get it--to draw a triangle outside
of the boundaries of the box, okay. What would you do? Okay. Anyone have one idea that they
would try? To try to reproduce whether or not a triangle can be drawn outside the boundaries
of the box, negative numbers. Okay. So, maybe it'll draw down here or out here or something
like that. Okay? So that's one idea, any other ideas?
>> Larger numbers. >> BACH: How large is large?
>> So let's say its one, two and fifteen or something like that.
>> BACH: Okay. >> Does it make a triangle?
>> BACH: Okay. >> Try to make something that which will launch
[INDISTINCT] >> Okay. Try to make something that doesn't
launch as a triangle. Okay. All right. Let's talk more about this. Okay. Here are some
questions that come up, that you maybe asking yourself. That come up from me. Is it unstructured
and unmeasurable? I'm here to say no, it's not. The structure for--the structured part
of exploratory testing can be giving yourself a mission, like I just gave you. I gave you
a very loose charter if you will. Okay. What about the measurable part? Oh, what if we've
spent an hour trying to reproduce that one buck that tax reporters reporting that the
customer saw a triangle outside the boundaries of the box. And what if in that one hour we
kept track three activities. Okay. Miles placing key strokes or testing activities. Anytime
we stop doing that and found something we were suspicious of, like a bug. Bug investigation
and reporting lets call that activity number two. And the third activity like anything
we had to do that kind of set-up to begin testing in the first place or times we had
to stop to maybe get more information about our mission. Like call tech support and say
what operating system were they running? So what if we just kept track of those three
activities in terms of our percentage in a given hour, how much time we spent? Oh, that's
measurable. It's unscripted. It's unrehearsed. It's intuitive. It's improvisational. But
I just gave you three different metrics if you will that we could do in an hour which
is another measurable unit, time. This by the way is something that my brother and I
invented called Session-Based Test Management. You can Google it, I'm here I'm talking to
Google I get that--it's no longer just an ordinary word. Session-Based Test Management
is a way to manage and measure exploratory testing and with time and with metric system.
These are couple of the things to it. But I'm here to refute number two with that--with
that the one example. Okay. Not too long ago, last February, several colleagues and I got
together and we've been doing exploratory testing for a long time. And we wanted to
talk about exploratory testing the way we knew it, because maybe there was some important
problems and disagreements that we had about this. Let's really study what if its exploratory
testing is about. Here is our mission. Teaching styles, what teaching styles that did we have
in our teaching arsenal? Could we come to a consensus of what exploratory testing is?
Are there any particular models or exercises that are fun or useful to teach, like, the
triangle program is a classic one. What about disagreements that's the fun part. Maybe there's
some important disagreements. So that was our mission, so several conversations starters.
What makes exploratory testing effective or ineffective? Are their particular behaviors
that we've seen or whether course of our careers? People we've managed that kind of go away
from making exploratory testing kind of a accepted legitimate approach. And I say approach
and not technique on purpose. Okay. Because scripted testing, were you have the test steps
written down. Okay. Is one approach. Exploratory testing is the opposite approach. But it's
not the technique because within both of those approaches, you can do several kinds of techniques.
For example, you could do scripted automations, scripted domain testing, scripted flow testing,
scripted stress testing or you could do exploratory stress testing, exploratory flow testing,
exploratory baby testing. Okay. Those are all techniques, two different approaches.
Okay. So, this are conversations starters we had for the exploratory testing research
summit, to give you an idea of how people who call us experts are talking about what's
the next big thing? How can we understand this a bit more? Okay. All right. So, here
are some definitions that people at that summit came up with. Robert Sabourin, awesome speaker,
Canadian fellow who has a company called AmiBug. He says continuous test design is testing
continuous--continuous testing as designed continues and continuous test planning as
testing continues, so planning, execution, design all done simultaneously. There's no
designing phase and an execution and you can never go back to design. Okay. Elizabeth Hendrickson,
a style of testing which you explore the software well simultaneously, there's that word again,
designing and executing test using feedback from the last test to make the next test more
robust or better or provide information about it. So, she mentioned this to a developer
and the developer said, "That's like test driven testing", as opposed to, you know,
test driven development, test driven testing, I thought that was pretty clever. Michael
Bolton, operating and observing the product with the freedom and mandate to investigate
it in an open-ended search for information. Freedom, that started his thinking, hey is
flow chart testing is about freedom, the extent to which you are in control of the test, the
extent to which the idea of the test comes from you. and the control of the test comes
from you. It wasn't written for you and you have to follow the script exactly, Freedom
is a big part. And lastly, Cem Kaner, author of our testing computer software, one of the
great books of our industry. Simultaneous, again is that word, learning designed execution
with an emphasis on learning--emphasis on learning, I'll come back to that. Oh well,
we'll come to it right now. So, what about this word ad hoc testing. Okay. Ad hoc, there's
some definitions, ad hoc testing doesn't necessarily care if you learn. So, it is kind of like
pounding on the keys. You're divested from the information clinging from the test. Okay.
So, that's why you say, "Oh, ET is ad hoc" because that means those definitions but not
all ad hoc is exploratory testing, because ad hoc doesn't care if you learn or not. It's
the way--the way I define it. Which is a--I'm going to take this time now, to invite you
to challenged me on anything your going to hear today, by the way. I meant to do that
up front, not only questions but problems you have with this definitions. Okay. I invite
to refute any of this because I want to make our craft better. I want to help, bring this
into the light as I mentioned and that--I want you to feel free of--to give me feedback
about that. Okay. Exploratory testing paradigmatic examples or analogies. Think about this, just
pick your favorite the one that resonates with you most. A job interview, let's pick
that one for a second. Ever been on a job interview where you sat down in front of the
interviewer, they had a script or a checklist and no matter what you said they did not deviate
from that checklist. You could really say, you know, I really got to think about it,
I really don't know if I want to work at Google. And like, okay, so tell me about when you
were an 8th grade. They don't even look up on their checklist, it's like, "Excuse me?"
So, when they say, "Excuse me, can you say that again?" That's exploratory testing. Because
the day that they just got, changes their tests--their test plan. Okay. They're adapting,
driving a car, same thing. There's cruise control, yeah. We can set cruise control.
We don't have to do any thinking. It just depends where we're setting cruise control.
Okay. So, driving a car, you get constant input that you have to adapt to. Any sports
team or sport analogy, they start off with a game plan, they call it a game plan, right?
Then they adapt when things change. Well, this team's tougher than we thought, we got
to bring in the special gold, you know, colored game plan. Okay. All right. All of those and
lastly, I think one more up there. Even going to a testing conference you start it off,
looking at the speakers. Here's who I want to see, here's who I don't want to see. Here's
what I think it's interesting, here is--here's my plan, then you go see that speaker you
really want to saw--see and you go, "That's still his talk from 10 years ago. I know.
And he just put a different title on it." So you get up, and you walk out. So, you adapt
as things come to light. Okay. These are useful because I find that people still don't understand
exploratory testing. They see of it--they see it as random and they see it as purposeless.
Okay. Here I am with the paradigmatic examples. So, at extras, the Exploratory Testing Research
Summit, Mike Kelley, offered this paradigmatic example. What happens if you have a certain
defect like the Triangle Program and drawing it outside the--Triangle drawing outside the
box? Scott Barber, ever had a developer walk up to your desk and say, "I could use your
help. Can you just whip something up?" Okay. Michael Bolton again, who is an excellent
colleague and consultant in Canada working on a new built of an existing product, checking
for bug fixes by using old tests. Here's my brother, James. Please investigate this puzzling
situation. Or, what will happen if you're asked to test something that doesn't exist
yet? It's just a sketch on a napkin, or it's a--the first spec of your meeting and it's
just--they're still flushing out what it's going to be. Can you think of test--at it,
ways that might fail even before you see it? My answer is, yes you can. And here's another
one from Kaner, test from a quick test list. Okay. So, these are loose scripts. Here's
another one from James Lindsley, once the script is executed choosing different tests
and re-executing. So, even if you--when you have a scripted case, change some of the variables.
Okay. So, there is the bug that I--that I repro, okay. And deleted the input. Someone
said negative numbers, that's exactly what happened. If my pro-programmers working on
it I'll show it you, I'll try to get it working once we hit another slide here, maybe the
link will work. Okay. In fact this may--let me try this real quick, just to see if it
launch, yes. This might work. It worked. Okay, great. I just need to see it--okay. Negative
numbers, so that's a clue. But it may not be the only way to reproduce this, okay? So,
let's try it, repro. Ideas, please just shout them out and I'll try to keep up with you.
>> Fifteen, fifteen, three. >> BACH: Fifteen, fifteen, three. Okay. Isosceles
>> Zero, zero, zero. >> BACH: Zero, zero, zero, another try.
>> 0.3, 0.4, 0.5. >> BACH: Point, decimals. 0.3, 0.4, 0.40 [INDISTINCT]
check, what do you expect? >> Draw something...
>> BACH: Do you expect it to draw something? >> A small?
>> BACH: A small? Okay. All right. Does that meet your expectations? Okay, so is this a
bug? Hands up, it's a bug. >> It's a feature.
>> BACH: It's a feature? Who said? I forgot my audience for a second, who am I?
>> Is the software smart enough to scale according to?
>> BACH: So she says the software is smart enough to--smart enough to scale. So, hands
up who thinks it's a bug and you report it? Okay. Ariane, and you, okay two, out of like
600--no, 3, 4, okay. All right. All right. I won't ask why. I can understand why they
might report it because they say, "Well, it should be--it's not following a scale or should
be skipped." Whereas she's saying, "It calculates for you." It's all--so it's a feature, okay?
Okay. All right, still no repro, any other ideas?
>> A, B, C. >> Leave it empty.
>> BACH: A, B, C, leave it empty. So, I'm hear--I'm continuing to hear ideas, so that's
that a--so you're adapting to each other because you're not content that you've seen it yet,
right? That's what I'm guessing. All right. >> One, two, three.
>> BACH: Yes. The one, two, three. That is an excellent test. Okay. It's an excellent
test, but keep going. So, this is called the Rumble Strip Heuristic. I borrowed this from
my brother. A Heuristic is a useful way of solving a problem by the way that sometimes
works, sometimes works. For example, when you're in a strange city, and you want to
find the Burger King, find the McDonald's. Because the sign is higher, and brighter,
and yellow. And they're usually across the street from each other. Okay, but not always,
not always. So, it's a rule of the thumb. Okay. Heuristic, and one of the definitions
of testing. So, the Rumble Strip Heuristic says, well, you know, when you're driving
and you're--it's getting late at night and you kind of go too far off the road. And there's
a Rumble Strip, there's cruise on the road, and your car goes [INDISTINCT] that's meant
to wake you up so that you don't go off the road. So, when you hear the Rumble Strip,
it saying something bad may continue to happen if you don't take action now. Okay. So, the
Rumble Strip here, says something bad may continue to happen if you don't action now,
so don't take action. Remember your testers, you don't have to take any action to correct
it. You take the opposite action. Okay. So, one, two, three, any way you would improb
off of that, now that you're close, you got it to draw outside. But it's technically not
a triangle and text support says, "No, no that's not what they saw." [INDISTINCT] okay.
>> Just out of curiosity, can you reverse them do three, two, one [INDISTINCT]
>> BACH: Three, two, one, excellent. I'm glad that came up. Three, two, one, what does that
tell you? Not a triangle, no line. Anyone? Does that tell--does that--hopefully your
brains tingling a little bit saying, "Wait a minute, it shouldn't matter."
>> Go for three, one, two. >> BACH: Three, one, two. All right. Three,
one, two, not a triangle but it's there. Okay. I want to--I want to help you out a little
bit. Using a done idea I already heard okay, the decimals. 1.9, check. Okay. How about
2.1? Check. That's interesting. 2.2? All right. What does it think about 2.9? Okay. What is
it going to think about 3.2? That's interesting, 2.9. Where was that again? 2.9? Okay. So,
it seems to kind of adapt. 3.1, that's interesting. Okay. So, I'm just--I'm just doing some improvisation
so, still no repro. So, we can play with this, okay. So, now we're--now we're on the trail.
And by the way, this particular technique is called OFAAT. or one factor at a time.
We're only changing one thing because we're trying to narrow in like a laser beam. Okay.
I think I've spend a little too long on that because I have some more content, we'll come
back to it. Okay, I'm into scripted versus exploratory before, hopefully this will help
you explain the difference to anybody that you feel may care about this specially, you
know, as you work with it too. Here is purely scripted, here is free style exploratory,
you know there is a continuum here. Have you ever run a test script were you said, "Oh
set four, oh, I know what they mean". And you decided to tweak one little thing, well
are you exploring or are you scripted? Well you are in control of that particular line
item. You weren't in control of the whole test perhaps but you were in control of an
important part of the test perhaps. So it's not fully scripted so it maybe a vague script,
okay? On the opposite role, have you ever assumed the role of a user, a particular user,
maybe your sister or your brother, a very particular persona? Okay. With the full history,
age, date or birth date, social make up, heritage, all that stuff. Okay, that's a--that's a role,
okay? So it's not a full script but it's not completely free style. Okay. So to know where
you are, to what extent am I in control of the test? Some people don't think exploratory
testing is an art. I just looked up the definition from Merriam-Webster.com. It seems to me like
it's an art, it's a skill that takes cultivation, you cultivate your experience and takes study,
okay? Yes, Curling. I promise to--so who has seen--who has not seen Curling? Okay. A couple
of people from in the--Okay. So you're asking your self, "Why Curling John?", "What is this
about?" I got infected with this commentary, color commentary and play by play for a sport
that hacks rocks down in a sheet of ice with brooms, are you kidding me? But I saw this
double take up and I'm like, "That's going to be hard to do." And so I went to the web,
I started learning web, the Curling terminology. And what a hog line is, what a tagline is,
what a house meant, how the points are scored. And I suddenly appreciated the skill of these
guys, so I could have--I suddenly saw an exploratory testing parallel. This is my life, my career
as a tester. I see exploratory testing everywhere. The camp that says, "Oh, how hard could it
be?" you know. And the camp that says, "I'll never be that good." Somewhere we meet in
the middle hopefully, so I started to think about what I would do to become a Curling
master. And--oh, this is a--by the way up on this site, there's some great flash animations
of the--of what Curling is about? And this video shows two rocks throwing with equal
weight or equal speed. One is swept and one is not, and it shows how one starts to curl
or changes trajectory directly a little bit because the ice is not normal ice, its special
ice. I didn't know that. There--it's sprayed with like kind of a mister so it creates little
pebbles on the ice, little bumps. So that when the sweepers come in they're actually
melting those little pebbles enough where it starts to change the direction of the rock,
that's a very strategic thing. So sweeping is a really important facet of Curling. So,
when the--when the thrower is saying "Hard, hard." I mean sweep hard melt those pebbles
fast because I want that thing to sneak in there right behind that other stone. Okay,
so there some great little videos up there. Okay, so stages in my Curling mastership.
Okay. So I defined a master as a--as an artist or performer in a play or consummate skill.
Okay. So on your way to becoming a great exploratory tester, just think of these as I've related
them to Curling. Okay. So you have to know that there's something called Curling so now
some of you do, you have to watch it being done, check. SO that's the red. Okay. Have
some curiosity hopefully, you know, maybe not, maybe this is a total experiment. I've
have never done this talk with Curling before, so I think some of you were engaged, and that's
all I need. It's just the few that feel like I'm not wasting your time. Learn basic terminology,
the hog line, the hammer, the last rock in the end and try it. So it's not red because
I haven't done it yet. Okay. Here are some more things. Okay. This ice I--put these in
there. Oh, I need to the audio so in case you can't see. So it's never [INDISTINCT]
to realize I can never be good at it, practice, get over the fact that I never be good at
it and compete in the Olympics. Key note at a Curling conference and moved to Canada and
become a commentator for CBC. This is--this is kind off how I started my testing career
and focus on exploratory testing. I knew there's something called exploratory testing, I watched
it being done, I had a lot of curiosity, I learned some terminology, and back 10 years
ago there wasn't a whole lot frankly. So I tried it and then I realize I can never be
good at it, and then I practiced, and here I am, you know, after a lot of things in that
I list other than moving to Canada, I haven't done yet--that yet. But wouldn't it be cool
if there was a tester competition with color commentary and play by play? This is something
my brother and I wanted to do at the Software Testing Analysis and Review Conference last
November. Is give a demo to people like you and say, if you're confuse about exploratory
testing, let us show you how it's done, okay. Let us show you the 14 different skills and
tactics. And I'll do play by play and my brother will do color commentary. And we invite you
to out think us, in fact it was, the title of the talk was called, Taking--Testing Outside
the Bachs, B-A-C-H-S, because I'm John Bach and my brother is James and after that. So
I was like, "Oh, look at the cheesy title that come up--came up for us James." He goes,
"I thought of that." I was like, "Oh? Sorry man." But it worked--it worked because we're
known for being passionate exploratory testers. And so we wanted to do this color commentary.
Well, to do that we needed to come up with a language. What is it that we are asking
people to look out for when we're testing--to watch for? I shouldn't say look out for. We
want them to notice it. Okay. Okay. So what would a Curling master might do? Okay. They
might study games. They may compete or play for recreation, maybe write articles about
their experiences, okay. Same for exploratory testing and, you know they may know and accept
that they won't win every game. When you think about what a master is, to become a master
at this, this is what comes to my mind. Okay. Here's how I got that thing to repro. Okay.
So negative numbers--it was a combination of everything I've heard so far. The 321 test
was a brilliant test because I thought, "Well, that's interesting." And then I tried the
same thing you did. I'm so glad it happened here. I did not script that by the way. Someone
said, "What about 231 and 213?" That's exactly what I did. And I got to see that there was--I
mean, at really close phases. There were some interesting disparities. So I tried decimals,
right? And then I tried negative decimals and got a triangle to draw all the way up
there. So the sides, the order of the numbers did matter. I didn't think it mattered. So
I had to break myself out of the assumption which I did by coming up with some test. Okay.
So all right, enough talk. What about this exploratory testing skills and tactics? Okay.
I remember it as Mr. Q Cap, crabsey, R&R. It's a--it's a mnemonic, sounds like Jar Jar
Binks, I guess. Mr. Q Cap, crabsey, R&R. Okay. Here is what that stands for. Mr. Q Cap, crabsey,
R&R. These are different ways that I used in the lab where I work. I work for an outsource
testing live in Seattle. My job is to help us compete without showing frankly and to
find better barks and to find it faster because we're more expensive, dollar for dollar. So
what that means is we have to provide better value. Now this isn't a vendor talk, this
is a technique talk, an approach talk, right? So this are different skills that I trained
for at the lab to help make us better, okay? All right, let me talk about each one really
briefly and I don't know what's going to be done with the slides but if you email me I'll
be happy to send them to you. Okay. So I don't expect you by the way to remember all this,
I just--when you leave maybe remember something about Curling, the guy how talked about Curling
and how it's like under appreciated and how now you got to do Curling lessons or something,
I don't know. But don't feel compelled to remember all of these, but challenge me on
any of this or if I'm missing something especially if there's a skill or technique that you have
done and you've got a name for it, tell me about it. Okay. So, modeling. All right. This
is the--how are you going to know what the product is going to do, how it works, what's--for
example, what's in the file menu if you're testing a business application or something
that has a file menu. Whether all--what's on all the menus? Okay. What are the files
that get installed? Okay. Having a feel for the product, okay. All these kinds of things.
Okay. Resourcing, to what extent can you gather and obtain the tools and information to support
the very active exploratory testing, right? That's also getting people to help you, you
know. If you don't know, ask. Go into the web looking for more information. Okay. Different
things that's called resourcing. Questioning, the biggest thing. All testing is questioning.
In fact the test is just a question in disguise. All right. If it's-if you do not--if you don't
have a question in your head when you're testing, you're not testing. Okay. But there's an art
to it. What about loaded questions, okay, questions you think you know the answer to.
Rhetorical questions, okay, sometimes I've been in meetings where I ask a rhetorical
question just to see if someone will answer me, like why are we shipping this thing? It's
a remembrance on that--it's always these particular, "Oh, yeah. That's interesting, what purpose
do you think it solves? Because we may not be on the same page. Okay. Asking questions
in a way that elicits different kinds of information. Okay. Chartering, this is crucial in session
based test management. Remember the technique that uses time boxes to do is to frame exploratory
testing, every session, every time box has a charter or a mission statement. For example,
let me give you an analogy, Lewis and Clark, the explorers who set up from the Missouri
River in 1802 I think it was, and their mission was to get to the pacific northwest, somehow
right, and the--well, their specific mission from Jefferson was to get to the Pacific Northwest,
get to the Pacific Ocean by using rivers. Okay. And established a trade route, okay.
So it's important to know that they actually failed on that mission but they're still called
explorers. They've found a lot of information about that territory, okay. All right. Observing,
okay, well how hard can observing be? Well, there's a lot to observing, in fact there's
something called inattentional blindness which happens to scientist. It all happens to all
of us. But sometimes, when were looking so closely at something, we missed the periphery,
okay. So at the lab, I teach classes with some objectives that involved how to observe
and look out for particular behavior that you maybe missing. Okay. Manipulating, making
and managing contact with the object of you're study, configuring it. Okay. There maybe something
you have to do with the software, to what extent you are doing--are you interacting
with it. Okay. Pairing up, this is a technique you can do exploratory testing alone or you
can do with someone else. Okay. You can let someone drive while you take notes, work together
distributed cognition, okay, also known as cheating--some people call it cheating. It's
like "No, I'm just distributing cognition around, so" Generating and Elaborating, working
quickly, refactoring, adapting, revisiting a situation later on, okay. Kind of like a--well,
I--before I get back into Curling. Of the video you saw, is there anything that you
can recognize in here? What they were doing or what we were doing at the triangle program.
Okay. Like refocusing, once I heard, 123 and then 321 that was kind of"Oh, wait. I'm not
just trying any numbers here. I'm trying single digit numbers less than five. I'm deciding
to focus in there, okay, so skills and tactics. Alternating, switching among contrasting,
different activities or perspectives. Let me give you some examples. Have you ever been
testing something where you're just gathering data just to see what's happening, and you
may do some analysis later on but you're just gathering data. And then you'd say, "Well,
I'm tired of gathering the data. Now, I want to look at what the data is telling me and
look at patterns." So, you may alternate from data gathering to data analysis. Okay. I was
trying to test an IP address program. All I did was check the syntax of the four octets,
really simple. Is it from zero to two fifty-five? Are you chopped at, you know, in that range?
Well, it was giving me invalid results for valid numbers but I couldn't figure out the
pattern, so I just tried hundreds and hundreds of IP addresses that I could think of. I just
tried test after test, after test, after test, after test and I couldn't see a pattern. It
didn't make any sense to me. Sometimes it would fail with 111 in the first octet, someone--sometimes
it would pass with 111 in the first octet. And I'm like, "What is going on? So over a
period of eight hours, I alternated between analyzing the data. I was like, "Oh, this
interesting. I'll try this test. This is what the data is suggesting. Now, it did--it didn't
work, so I need to gather more data. Okay. These are polarities, what about solo work
for versus team effort. There are times that you may want to pair up with somebody who
has a particular technique or tactic you can use to exploit an alternating skill. Okay,
More polarities. Sometimes you want to really dig in there and investigate and sometimes
you just want to get a feel for the product. You just want to be more of the tourist, okay,
just--but maybe in the first session that you do, for example. If you give yourself
90 minutes to explore something, give yourself a mission, you may just want to do some conformances
form assess Just to get a feel for how it works and try your devious stuff later on.
So that's testing versus touring. You may want to try one feature now and then take
a break from that and try this other feature, just to give your mind a break. Okay. This
is--now all of these things I've been mentioning are things that I've seen testers do and seen
myself do as an exploratory tester. Branching versus Backtracking, to what extent do you
take an idea and then get inspired to try an idea off of that idea, and then come back
to the original idea later on. Conjecturing, what--oh, here's what I think is going on
here. Let me--let me see if I can corroborate or refute my own conjectures. Is the moon
made of green cheese, something like that? How would you corroborate or refute that?
Recording, to what extent do you take notes or what kind of notes do you take, what do
you decide to write down, do you use a recording tool? Okay. This is some people call any--use
of any tool automation. I call it Computer Assisted Testing. There's a product called
specter by the way, that records mouse flicks and key strokes, another one called BB Test
Assistant that allows or even doing bad audio and to your recording, so you can show the
developer what your--what your thought pattern was. That's recording. And lastly Reporting,
you know, there's a difference between recording and reporting, right? Because recording is
something you do for you perhaps, reporting is something you deliver to someone else.
So you may not want to deliver them your raw notes. So it's a--it's the separate skill.
Okay. How we doing on time, okay, good. I will send links to whoever is interested here,
some useful links for check list and ways to help you think of test ideas when you're
stuck. This is created by my brother. Elements of a test plan, quality criteria there's another,
there's a better one, Heuristic Test Strategy Model. I'm going to scroll down to this. Function
testing, scenario testing, claims testing, all different kinds of techniques remember?
That you can use with either a scripted approach or an exploratory approach. Okay. These documents
are freely available at satisfies.com, my brothers' website. Flow Testing, random testing,
user test--all kinds of things. Costumers and team, equipment and tools, all the factors
that may decide what kind of test you're going to run. Okay. So, where are you at? Do you
know there is something called exploratory testing? Yes, you do. You watched it being
done? Yes. You have some curiosity. Create some definitions, ask some questions, participate
in discussion threads, study, others experiences and finally, you know, we're all going to
move to Canada eventually to become a commentator for CBC. So, we better just do it now. And
I'll leave you with this about gaining skill or becoming better at this. A word from Ernest
Hemingway "It's none of their business that you have to learn to write. Let them think
you were born that way." I like that one. A quote from Bradbury and then a quote from
my dad who wrote a book called "The Illusions. The adventures of a reluctant messiah." It's
about a guy, an airplane mechanic, who happens to be a messiah and he doesn't want to be
a messiah anymore. So, he just becomes an airplane mechanic, that's only what he wants
to do but people keeps asking him to heal them and everything. Its--so that's one of
the--one of the quotes from the messiah's hand book. They give you a hand book when
you're a messiah, apparently according to dad. And that's one of the quotes in there.
So I believe that's it. I'll just end with a haiku. Okay. And gauge the skill set, 14
different skills and tactics. What do you think, am I missing something, was something
confusing, can I elaborate on anything? I know I didn't leave many time for questions
but now is the time to ask them. Any questions--any questions? Yes.
>> What's your email address? >> BACH: Oh, thank you. Oh, I don't know.
I don't know. I was so afraid I would get a question that I couldn't answer. I've got
like eight E-mail addresses. No, let me give you Jonb, J-O-N, B as in boy, @quardev, Q-U-A-R-D-E-V.com.
Okay. Johnb@quardev.com. I do have a G-mail account but I'm embarrassed to say I haven't
used it in awhile and I forgot the password and I haven't got around to getting it. So,
forgive me. I was bestowed upon that honor and I haven't used it wisely. Yes.
>> How many do Test Driven Development do you consider exploring directly?
>> BACH: I don't know enough about Test Driven Development to wage a good answer about that.
What's--tell me more about the spirit of your question though.
>> Because when I was using agile in my old job. We would give story points, first by
having the testers develop test cases that we're going to run without any specs, whatsoever.
So we can estimate a lot of--it would take us to test a certain product and also the
type of test that we were going build to it. The developers build the product...
>> BACH: Yeah. >> ...base on those tests.
>> BACH: Uh-hmm. >> So, I see some of the skills you said you
had that sort of go in to that. >> BACH: Yeah, okay. So, let me see if I found
the parallel here. For Test Driven Development, which is--its an agile way to create test
in--while its still in development and do more unit testing sooner and do more of the
testing--a front loaded? >> Before it's even developed.
>> BACH: Before it's even developed. Yeah, I--I don't know how I would give an answer
that would be a perfect integration to what you're looking for. Other than to try some--try
a session, try to come up with 20 or 30 charters of loosely scripted, you know, a paragraph
of here is what I want to explore. And give yourself an hour or ninety minutes, take some
notes. My--the quick, uh; tactic is notes, bugs, and issues. So, write it up in your
notes. Write up whatever bugs you find in any issues, things that maybe bugs but you're
not sure. And see if after you've done like a days worth of that three or four sessions.
See if that helps unearth some more charter ideas or ways to build cases if that's what
you need to do on the unit side. That's the only way I think I can do justice to your
question, so. Okay. Yes. >> What's the biggest weakness of exploratory
testing? >> BACH: What's the biggest weakness of exploratory
testing? I think the biggest weakness is not knowing how to do it well, because if you
don't know how it's done and how--what the skills and tactics might be. And you don't
do them on purpose, and you can't explain the story of your testing. It only fuels a
mystique, the mystique of it and it only contributes to managers who may say "Oh, okay. Then anybody
could do it. So, lets just offshore it." Or "Okay, you know, I'm just confused as to what
you really did there." Where as if we came up with a language, it doesn't have to be
this one but a language that works. That may strengthen exploratory testing's ability to
be perceived as valuable. Kind of like if Curling didn't have any nomenclature to it,
it was just--and the commentators were just able to say, "Well, that looked like a good
shot." And you weren't able to describe why? I think that's--that would actually harm the
sport. But it was because this guys knew how to get excited and why they where getting
excited that I got excited about Curling and saw the parallels with exploratory testing.
So, I think the biggest weakness is exactly what I'm trying not to do today, is to give
you guys some weapons to describe what it is you're doing? And to maybe pick out some
of these skills and technique--tactics and say, "Uh-huh. I need to be better at note
taking." Or any "I want to try, see if I can do some of this chartering stuff." Okay. Or,
"Man I missed a couple of bugs last round, I want to see if there's any particular observational
tactics and techniques I can try." Okay. Good question, thanks for that. Yes.
>> Where is the future direction towards the exploratory testing? Like, where's the research
going and when is it being done? >> BACH: You're looking at it. Where is the
research going next in exploratory testing, he asked. Where is the future of it? This,
exploratory testing research summit is a new thing. And, what I'm excited about is working
more with this language and trying to see if do some more research on how long does
it take a tester to come up to speed or be considered a, you know, testing expert, you
know, or savant or what is it that testers do that makes them the guy or the gal you
want to go to, when you've got one hour to ship or one hour to test. What is it about
them? And then talking about it, then use it like this and at conferences. And saying,
"You know, its not random, it's very thoughtful. You know, it's not unstructured it can be
structured." Dispelling the myth that if you put two exploratory testers back to back,
there going to do the same thing anyway because there's, you know, it's pretty obvious what
needs to be done. And also the myth that, "Oh, if you put two exploratory testers side
by side they'll do completely different things. So, what difference does it makes?" How--you
know, both myths actually have some validity to them. But, they can't exist in the same
space which tells me that there's more research to be done. Okay. So--and were going to--I'll
probably have another--there's a workshop in heuristic and exploratory techniques in
May where a bunch of colleagues and I, where its going next. Good question. Okay, anything
else? >> One last question?
>> BACH: One last question. I think my time maybe up anyway, my time is up. Great. Thanks
everybody. I appreciate it.