Tip:
Highlight text to annotate it
X
bjbj< Brandon Schauer: Coming up next is Tomer Sharon, perhaps one of the longest fought
battles in the management of UX is of course over research. People say We know our customer
or We did a survey last year or That s not a statistically valid sample size. I guess
that s one we hear a lot. I think we ve all heard various versions of this kind of stuff,
so next up is someone who I think frames the problem and the solutions very, very crisply.
Tomer has led the user experience research effort for Google s online advertising management
platform since 2008 and now he s gonna help us get a leg up on that elusive buy-in for
UX research and apparently is a Sox fan. Tomer Sharon: All right, so recently I noticed that
I should use this form of saying winning. So UX practitioners, researchers, and leaders,
are constantly looking for ways to get this buy-in from the stakeholders, and the reason
is they re trying of course, they know it informs design, it helps prioritize development,
it helps shape product road maps and it helps initiate company-wide change of direction
sometimes. Many of them are frustrated and feel isolation because they need to deal with
very difficult people who don t understand or don t respect the research process. What
I want to do today is share five stories and lessons learned based on my career and I m
gonna start immediately. This is a URL for you to ask questions. I m using Google Moderator.
This is the first time ever that someone is trying to use the Internet during a presentation,
so I hope it works. So Google Moderator if you don t know allows you to submit questions
and allows everyone else who doesn t have a question to vote for questions. So you get
to get me to answer the questions that are most important for you, so please use that
during the presentation. You re gonna see the URL on the slide. Very briefly about myself,
I work on a product called DFP, Doubleclick for Publishers. It s for managing online advertising
for companies who have websites in which they have ad space to sell. In my spare 20 percent
time I m supporting the Honeycomb project with research. I m a senior vice president
at the Sharon residence, and I also offer a book for Morgan Kaufmann Publishers called
It s Our Research: Getting Stakeholder Buy-in For User Experience Research Projects. Something
I wanted to show you, which is nice, I work in a company called Google. I heard that there
are some problems. Today you can buy two Googles for $3.98 at the Home Depot. So let s jump
into the first story. The first story happened when I was an in-house practitioner a while
back and I was experimenting with measuring UX metrics and I generated very cool charts
I think, I wanted to believe. There was this one software engineer that constantly poked
holes in my presentation while I was giving it in front of the entire team. His main problem
was that I was misrepresenting results and I m dramatizing the findings by using different
tricks in my charts, and he said that I m probably right but I don t need to do that,
and I think I m being very gentle. He was really nasty and I felt terrible. I was insulted.
I didn t wanna see this guy, I didn t want to work with him. I was clearly a very young,
inexperienced, unconfident practitioner so I just completely ignored him for two months,
and then I decided to take a step back, reflect, and understand that I was doing the wrong
thing, and I was looking for a way to turn things around. What I did, not rocket science,
what I did is I just asked him if he was willing to poke holes in my presentation privately
before the team meeting and he s happy to poke holes, so he agreed to do that and we
made this a habit. So this was excellent, why? Because I got to get the feedback and
improve my presentation. He got to poke holes. He was satisfied, and most importantly he
felt appreciated because he got to see the results first and we made this a habit. We
did that each time when results of studies came in. I felt that I can do more with this
guy, so when the right opportunity presented itself I invited him to join me on a large
international field study and we worked on this together for a couple of months and one
of the preparation meetings he suddenly looked at me and said, You know, wow, I wasn t realizing
that research involved so many detailed methodology and planning , and at this point I knew that
this person is becoming a champion for user experience research. So the lesson learned
here is that there is a way to turn things around even with difficult people. Having
said that, I m going to talk about the hostile stakeholder, the one that no matter what nothing
is gonna turn them around. You know these people. I m sure you ve worked with these
people. They are the ones that disregard research. They know better. They laugh at your sample
size. They d rather count on their intuition rather than on observed behavior. So to explain
what I sometimes, it s not a thing I m doing every time, what I m doing is I m gonna take
an imaginary military conflict between the blue army and the red army. So let s assume
that the blue army has three divisions, one in the east, one in the west, and one one
step back, and this is the reinforcement division. So the attack started. The two divisions are
attacking, but the east division is very successful while the west division is stuck. The red
army is blocking the attack for the west division and now the general is thinking where should
they send reinforcement. So for that I need you, Jay, mission control. [Laughs] If you
are that general, where do you send reinforcement? Jay: I wasn t looking at the first slide I
have to confess. Tomer: No first slide. Two forces attack. One is very successful, one
is not. This is reinforcement. Jay: So I m sending where they re not successful, right?
Tomer: re sending it here. All right. Thank you. The art of war, if we ignore the irony
for a second, is saying that the general should send reinforcement to the succeeding forces.
Why is that? The reason is that if you send reinforcement to the succeeding forces, that
changes everything for the red army. They need to take care of serious developments.
There are dangerous developments with your succeeding forces and they have to change
things completely. If you re gonna send reinforcement to the stuck division the second division
would probably get stuck even more. So how is that related to the hostile stakeholder?
So what I m doing with stakeholders is pretty much the same. I have stakeholders that are
very accepting of research, they are the successful division, and I have these hostile people
that are not cooperating no matter what. So what I do I leave these people alone. I leave
them alone. You handle it by yourself. You re not getting research love. And I m working
even more. I m sending my reinforcements. I m working even more with the stakeholders
that accept research and I do do that. I ve been working in the field as a researcher
for 12 years now. I think I did that three times and as opposed to the battlefield, which
things change in a couple days, with hostile stakeholders it takes about a year for them
to come back, and when they do I accept them with open arms and bring them back to the
research highway. So that s the second story. Third story I call it, and many people who
speak the Queen s language here of course I think, there is at least one, you never
walk alone. What I wanna say by that is I m asking the question of when is the best
time to collaborate with stakeholders when you talk about research, and to demonstrate
that I m using a chart and I ll explain it. So these are the phases of a typical research
project and red means that you don t collaborate with stakeholders in that stage, green means
that you do, and this is the level of engagement of uptake of research by stakeholders. So
let s take the extremes. If you want to be extremely successful, you want your stakeholders
to be fully engaged with research, you need to collaborate in each and every step of the
research. If you want to completely fail you do research for research not for stakeholders.
Don t involve them at all. Don t even tell them about it and fail completely, usually
associated with academia. Yeah, I know. [Laughs] The gray areas are where most people are.
If you wanna be somewhat successful you should involve them in planning. Don t involve them
in recruiting, execution and analysis, and show them the report and follow up and so
on and so forth. So this is a good chart for you to evaluate how much uptake and consumption
you have in your organization, so ask yourself what is the level of update you usually have
and can you make it better or change it somehow? The next story is about what happened when
the only person who knew what UX was left the company. So I was hired by a mature start-up
with about 200 people by the CEO. He was the only person who knew what UX was. Very appreciative
of research, very understanding, and I love the day I joined the company. Four weeks into
my new job I learned two disturbing facts. One is that out of the 200 people who worked
there he was the only person that knew what I was doing and why I was hired, and the second
disturbing fact is that he announced he was leaving. ll make a long story short. I stayed
there, everything was fine, and my point is that the one thing that helped me thrive and
survive there are the meetings that I attended, the meetings that I initiated and the meetings
that I avoided, and I wanna tell you about these meetings. So meetings I initiated were
meetings with senior product managers or engineering leaders. Probably I did and still do that
every month or quarter. This is a conversation usually 30 minutes, not more, in which I try
to understand what questions are bothering them, and for all of these meetings I m going
to talk about I m trying to identify opportunities for research. The second meeting that I initiated
were meetings with one level down, the product managers or maybe the team leads, the engineering
team leads, and with them what I m trying to do is just the day-to-day research needs.
Who are we going to recruit? When is the study gonna happen? Things like that. The second
type of meetings are meetings that I attend. I m not saying a word. I m staying quiet.
I m just listening. These are product management meetings, engineering team meetings or engineering
leadership meetings. In most cases I don t understand a lot of what they re talking about,
but it s important for me to be there so they can see my face. s important for me to be
there because I m starting to understand the language they re talking. It s important for
me because I understand the relationships between people in the room and I know what
I should suggest to whom. The third type of meetings are meetings that either I m completely
avoiding or meetings that I m being very careful with. I should be careful here. [Laughs] Meetings
I avoid are meetings where a bunch of well paid non-UX people are criticizing designs.
It happens a lot. I m sure you know these meetings. I get so annoyed that I just can
t be there. I m a person who prefers observed behavior of users rather than opinions of
people who don t understand design, although I acknowledge the fact that sometimes you
hear good feedback, but I try to avoid these meetings. At least at Google we don t have
to show up to each and every meeting. Meetings I m being careful with are meetings where,
and I m sure you all attended these, the company-wide vision, the one time a year that the CEO is
describing the vision for the next year or quarter or whatever. Why I m being careful
with that? If you sit back and listen to strategy and think that this is how you re going to
identify research opportunities you are dead wrong. It s too late when you hear that. If
you wanna think big, you are the one that should help set this strategy with your research,
so I m being very careful with these meetings and what s going on there. Last story and
the one I personally really like. When stakeholders join you or observe you, moderate studies,
whether a field study or lab study or whatever, it could be extremely overwhelming for them.
A lot of things are happening. Sometimes they re stressed because this is their baby that
s being tested. Users discuss what they did. Users actually use the product, so a lot of
stress is involved and a lot of information is passing by. So what I m doing when I moderate
sessions in which stakeholders join me or I know they observe from far away or in the
next room, I m doing something that I call coloring the experience for them, and I learned
the term when I was a young cadet in the Air Force Academy in Israel. This is the plane
I flew. This is not me in the picture. I want to tell you a little bit about that. I was
17. I was extremely ambitious. I thought I m gonna be a pilot and it took about three
months from the day I was drafted until I was airborne with this small plane, and during
these three months we learned everything we could about the plane, the systems and everything,
and about the maneuvers we are expected to demonstrate in the air. So when the time to
fly came I took about three weeks. Every morning we had one hour in the air and the instructor
was sitting behind us in the plane and he was telling us what to do, and we needed to
demonstrate our performance. About 90 percent of the people were to be screened out of the
academy including myself, and I demonstrated some high performance there, and the purpose
is not to see if you re gonna be s not to teach you how to fly. It s to see if you have
the potential of becoming a great pilot because as you can imagine the Israeli Air Force does
not use these planes in war. So let s say we re in the air and the instructor is asking
me to make a left turn and I m so stressed out and confused and I m making a right turn
instead. So what I needed to do is to call my mistake out loud, so I would say Turn right
instead of left and just move on to the next maneuver. Now during debrief on the ground
thankfully we were analyzing our performance and mistakes and the trick was that you vividly
remembered your mistakes because you called them out loud. This was called coloring your
mistakes. So what I m doing today with stakeholders during sessions is pretty much the same, and
I color the experience for them in three different ways, two small ways and one big important
one. The small way, the first one is with body language during the session. So if something
important happens I turn to the stakeholder. If they re in the room with me I turn to them
suddenly so they understand that something important is happening. You know that, with
a blink of an eye or two-second lack of focus they can miss the most important thing in
a one-hour session, or if I know that this is being broadcasted to another room or location
I turn my head to the camera and wink or raise my eyebrow or something, or I turn to the
one-way mirror or things like that. Of course this is all behind the back of the participant
so they don t get scared. The second way I m coloring the experience is with written
language during the session. So if I m interviewing a participant on the phone or if it s a remote
session and the stakeholders are with me in the same room I use a Post-It note or piece
of paper and show it to them and we talk about something. We have a very important research
question and it was just answered in front of our eyes. I m writing something like Important
[laughs]and putting that in front of their face, so that s how I color the experience
with written language. The last one is probably the most important and effective one. I color
the experience with spoken language after the session. So you probably all do that yourselves.
When we re out of a field visit immediately after that in the train, in the street, in
the park, wherever it is, we talk about what happened because stakeholders sometimes see
different things than what you see, than what you know is most important. So we talk about
surprises, we talk about what we saw, we talked about what is it that we re gonna ask tomorrow
and things like that. I make sure that I tell them, The anecdote you just watched is not
really important. What s important is this big thing that happened or that was said,
so take good care and good notice of that instead of anecdotes. Same thing with lab
studies. I just walk into the observation room immediately after the session and I m
pretty much doing the same thing. These are all my stories. So remember the engineer that poked holes in my presentation,
the way to use reinforcement, when is the best time to collaborate, what do you do and
which meetings you initiate, attend, participate, and avoid, and how to color the experience.
Shameless plug for my book that is still due in a year. These stories that I just told
you will be a part of the book. I m collecting stories from all over the world, so if you
have a story about a terrible or a very successful engagement of stakeholders please talk to
me. I m still collecting these stories. I have currently about 50 stories from all over
the world and if your story fits well then I might use it. If I wanna follow me on Twitter
I m providing book statuses. One more important thing, and I m being selfish here. We re recruiting
for designers in the ads domain, so this is the area that brings money to Google. If you
re interested we re always looking for great designers and let s try the magic now. So
if you have questions that you didn t submit do it very quickly and vote for questions
and let s see what s going on there. Okay, it s working. What are your views on the claim
that having a sample size that is as low as three to five users is enough to identify
80 percent of the critical issues? Do you do guerilla testing often? So I think that
for the low-hanging fruit three to five users is pretty much enough. I m doing the same
thing, maybe not three but not less than five, and I m trying to not highlight this as the
first thing in my methodology slide in the presentation. It s not that I m lying about
it, it s just that You used five users! And I do do a guerilla user testing, maybe not
very often but it depends on the situation. If you re looking for the smaller things,
sometimes maybe the things that you wanna be more confident about then I m using larger
samples, maybe different methodologies. It s not that I m doing a usability study with
or a lab study with 25 participants. I never do because either wise I wouldn t be here
today, but we re doing maybe surveys or these online studies, remote studies on moderated
studies. s see if it works. What s the single biggest factor that determines buy in? I would
say, and it s gonna sound weird, contribution. If you come to your stakeholders and they
understand that you want to contribute, you don t want to do their job, you just want
to make them heroes, and I m referring especially to lead engineers and product managers, you
should say that you are there to contribute, to help them do their job better. You re not
looking for anything for yourself. That s what I think is the biggest factor that determines
buy in. All right, quantitative research. So quantitative research works like magic.
It was said here before, numbers do it for certain people, and seriously sometimes you
really need that. If you come back from a field study with 12 participants and let s
say 11 of them said that they re not using something or some kind of feature, are you
really confident that this is what s happening? I would go to blogs and see what s happening
with this feature to support this finding. Maybe I m completely wrong and that I m not
even gonna consider that as a relevant finding. So yes, quantitative research is definitely
worth it. It s a lot of work. You need a lot of cooperation with your team, especially
the engineering team, but it s definitely helpful. If you are not getting this help
you can measure things in lab studies and I think we re gonna hear about that right
after the break. Aside from waiting until things change, is there a way to address complete
buy in blocked by budget or priorities? I would start small. I think I mentioned that to June, yesterday. They didn t let me but
in a previous company I really, really wanted to work on the application that everybody
orders food in the company and then show my benefit, my contribution there, and then everybody
else would know, but they didn t let me do that. Yeah, but waiting until things change,
I mean I wouldn t wait. What does winning look like? I think we can see that on TV every
day when it goes wonderfully right. So I saw this question in advance and I wanted I thought
about it a little bit and I think I have ten answers to that question. Signs of engagement.
Users are happier, researchers consumed, changes are made in design, findings are long and
lasting, trust is established between you and your team, you re invited to important
discussions, teams want to work with you, business decisions are made based on research,
product roadmaps are reshaped because of research, new products are born, staffing is changed,
people are allocated to take care of things that you find and researchers are recognized
with awards and just good words. So these are all signs that something is going very,
very well. If you re not getting that, that s terribly wrong. That s it. Thank you. [End
of Audio] Tomer Sharon Page PAGE of NUMPAGES MX 2011 www.verbalink.com Page PAGE of NUMPAGES
hODD gdQ5 gdQ5 gdQ5 gdQ5 gdQ5 gdQ5 gdQ5 gdQ5 gdQ5 :pQ5 [Content_Types].xml u$Nw @8Jb _rels/.rels
theme/theme/themeManager.xml sQ}# theme/theme/theme1.xml G$$DA :$BR si-@R r,[L bX*x KfN1 ,tV@ .EML
M .c =