Tip:
Highlight text to annotate it
X
Scott: This afternoon, we have a featured keynote from IBM Watson with Jeff Javlonowski.
He is one of the original IBM Watson commercialization team members from 2011. This is the team that
productized the IBM Watson Jeopardy Game Playing System. He is an IBM Watson core technology
development manager that has done a considerable amount of hiring and recruiting to grow his
team. He will share his experience in their technical interview process and the role of
the HireVue tools for building out the IBM Watson team and hiring the best of the best.
Thank you for being with us today, Jeff.
Jeff Javlonowski: Thanks for the introductions, Scott. I appreciate the time today. I'd like
to spend a little bit of time...again, my name is is Jeff Javlonowski. I'm going to
take you through our use of the HireVue CodeVue tools and how we're using it here in the IBM
Watson organization to help us hire some of the best of the best in the industry and hope
you get something out of the...how we're doing it here in Watson. So let me start with kind
of where we're at with Watson and some of the challenges facing us today. So what we're
dealing with is, how do you deal with a candidate pool in excess of 1,000 or even 10,000 applicants
when you look at our intern pools of people that wanted an internship with Watson this
summer? How do you best spend that precious time of yours and your candidates' time with
the right folks? And most importantly, how do you drive the offer acceptance rate up?
Something north of 80% with some of the best of the best in your candidate pool. So I'm
going to spend some time, the next few minutes with you, going through our practical experiences
using the HireVue CodeVue tools and its important role and its use in IBM Watson's talent recruiting
program.
So let me start with the technical screening process. So the way to do this is it's absolutely
essential and critical to have an effective and efficient technical screening process.
Here at IBM Watson, we're using two of the tools from the HireVue organization, what's
called the CodeVue OnDemand tool and also the CodeVue live tool. So I'll take you through
later in the presentation, specifically our own experience but let me show you in the
flow here where it fits in the recruiting and talent process for IBM Watson. So the
IBM Watson team uses CodeVue OnDemand as a tool to screen our technical positions by
giving the candidates an on-demand coding challenge exercise. And followed by that is
an electronic whiteboarding set of interviews done if a candidate passes the first set of
on-demand autoscored and evaluated exercises. Don't get me wrong, we still value and leverage
on-site interviews in our decision process. But these CodeVue tools are very important
for us to help us ensure we're spending the right time with the right folks.
If you look at the graphic on the right, you will see what I would call a pass through,
which is kind of a representative number and how candidates would pass through a technical
screening process. So if you think of the total set of people starting with the on-demand
exercise, that would be the full set of a 100%. And then moving next to the on-demand
screening process, 75% of the candidates actually took and completed the exercises. Of that...that's
when the evaluation of the technical screen process begins and we looked at passing that
in the 45 to 50% range is something that you could expect. And after that process...now
that is getting you a screen candidate pool that you are comfortable with, has the right
skill set, and interests to match your posted positions. Next we move to a live phase, where
we're actually doing live electronic whiteboarding interviews. So this could be the first time
your technical team engages with the candidates. So that could be...so you have a good solid
body of candidates at this point in the process, now we're looking at going through and interacting
with the candidate and we'll be seeing experiences maybe in the 70% range of people that will
pass that and then proceed into being considered for an offer.
So what I'm showing you is kind of a representative process here and do not think that this on-demand
starting point here is your entire candidate pool. There is a talent assessment done at
this point. So there is a bit of resume screening and skills matching to ensure those candidates
are matching well to your needs in the organization. So let me proceed and give you a little bit
of what we've seen and some of the experience we've seen with the Watson teams. So be prepared
to be surprised. You will find that candidates will be able to stand out that perhaps didn't
stand out on paper and on their resume. Not just resume bloating by people bragging but
really looking at the results and evaluating a submission from a candidate. You have more
than that one or two pages of paper to really get a feel before you're even interacting
and spending your time with the candidates. Look for and find the passion and the valued
experiences of your candidates by being able to ask them questions and seeing video responses
returned back to you. See the innovation in their coding and problem challenges. So give
them challenging exercises and look for that innovation and those creative problem solving
skills.
And you will find that candidates are really able to differentiate themselves and find...I
was very much surprised by some of our candidates results that we've hired this year in 2015.
And we've been able to use this tool to effectively evaluate a larger candidate pool. We were
able to send the exercise out to a larger candidate set. Perhaps some of those candidates
that you would say they're on the edge of whether or not you want to spend the time
with that candidate, you could send them the CodeVue on-demand exercise and let them differentiate
themselves and see if they're the right candidate for you to spend time with. So done right,
this will allow you to spend the right amount of time with the right candidates to match
the needs of your organization.
So how is this done? Done in volume, this is certainly not a one man or one woman exercise.
So what we've done here in Watson is enabled a relatively large team of technical screeners
to do and review the CodeVue results of the candidates. So we've done enablement across
our team, shared what I would call our answer key with our reviewers so that people understand
what's to be expected and typical results are and have a larger pool of technical screeners
doing these CodeVue on-demand reviews to see if we see the skills and interests match with
our candidates for IBM Watson. Next, we move to the...when we move to the live interview
phase, we will do a CodeVue live electronic whiteboarding session and we will do that
with a hiring manager along with a team of technical leads for that part of the organization
where we see the best fit for that candidate. So this will be a combination interview of
a fit interview as well as a skill set match and interest interview. So we've kind of combined
the conversations between the management team and the technical lead team with the candidate
in a very engaging conversation.
Also very importantly, we have the role of what we call a hiring champion by major job
function that will assist in matching candidates, perhaps the example would be that the CodeVue
whiteboarding exercise saw a very capable candidate but wasn't an exact match for the
needs of the team doing the interview. The team doing the live interview can engage the
hiring champion and perhaps recommend another part of the organization that would benefit
from and just consider that candidate. So we do have the ability to do very focused
interviews but we do also utilize the ability to match that candidate with a larger part
of your organization. Another large benefit you will see from leveraging these CodeVue
tools is flexibility, flexibility built in by design.
You will see that the CodeVue on-demand exercises are asynchronous meaning you send and invite
a candidate to take a coding challenge. They take it on their own schedule, when they have
time and energy to focus on your interview. You also have the same benefit on your end
as the reviewing hiring team to review that candidate submission and see if it's a good
match for group. The net result of this, it provides...it's really providing you a much
more professional experience where you're not playing phone tag, you're not having people
rushed in and out of a phone screen interview to see if this candidate is the right skills
match for your group. You will have a much more professional, polished experience and
that's the feedback that we receive with Watson and how we're using the tools today. So let's
get into some of the practical experience and practical uses of the tools. I mentioned
the tool and now you can see one of the screens that you would see as a reviewer. The CodeVue
OnDemand tool today that's available, allows you a combination of the ability to record
a video introduction to your candidate explaining to them a little bit about your organization.
And then, we can get right into a series of questions and challenges presented to your
candidate. We've done use of today. Our use has been a combination of thoughtful video
response questions, posting a set of text to the candidate and allowing them time to
contemplate an answer, provided back as a video response.
So again, you're not seeing a recorded or an audio over the phone response from your
candidate. You have a webcam recorded interaction with your candidate to see and get a little
bit of a feel for the capabilities and abilities of that candidate to match your needs. You
have the ability of submitting coding exercises, coding challenges. So these will be timed
exercises. You pick the time to give your candidate, the amount of time deemed necessary.
As well as the ability to autoscore and evaluate those exercises. What I'm showing in the image
on the right is a picture of what you might see as a candidate looking at a job...a technical
exercise that we gave out. So again, you have the ability to give example test set to the
candidate, amount of time to code it up, and as well as when you compose this coding challenge
question, you also have the ability to give what I would call blind questions where you
have test steps to actually run where those are not visible to the candidate. So your
score is a combination of the public viewable as well as the private blind question sets.
But be careful. Do not get hung up and too focused on absolute, empirical results.
Again, coding and problem solving skills is not always the end result and calculated result.
We do spend the time to look at the quality and the approach taken from our candidates.
I'm just cautioning. Be careful. Don't look for the absolute scores and try to assess
candidates because as we know in the technical industry, it's not all about coding up something
that comes out with the expected result. It's also the approach and the creative problem
solving and how do you do that effectively is looking and reviewing their code. And we
also utilize the combination for the coding challenges and video responses. And what we
call video responses to the coding challenge where we actually ask the candidate to describe
their approach. So we get a little of the ability to the thought process. We see that
as a big benefit here and an innovative way of using the tools together to look at the
right candidates and see their problem solving skills. Next for candidates that pass an on-demand
autoscored exercise, we will move that candidate forward into the live electronic white boarding.
Again, you still have not needed to bring a candidate on-site.
This is where we would have a combination interview with our management and our technical
leadership team to have an electronic whiteboarding screen sharing session with the candidate.
So this is a very engaging experience and you will see...if you can see the image on
the right, you will see that in the background, we will see the video webcams of not only
the candidate but the participants as well. So it is a very engaging experience allowing
you to flip focus between the whiteboarding session and the video webcam conversation
you had with the candidate. So it is very engaging and we find it effective. We're using
this across some of our development sites so we're not even needing to be all in the
same room to do this interview. We'll have the IBM team at multiple sites having the
right people talking to the candidates. So very receptive. We also found a practical
approach here was to be prepared and paste the question or exercise into the whiteboard.
So when we engage with a candidate, you're still conversation over the phone rather than
not losing anything in the verbal quality of the communication. We'll paste the actual
exercise.
It's just another way to be more engaging with the candidate. And done right, again,
efficient and effective assessment of problem solving skills. So very engaging and at this
point, we would be moving...candidates that pass this point, we would consider for coming
on-site and evaluating them. So again, a good combination of innovative tools to get to
talk to the right people with your team. So achieving a high acceptance rate. There's
not a silver bullet here. This is a combination. This is going to be a combination of a set
of offerings, your technology itself obviously is an important part but many factors are
involved here of...the offer itself, the innovative experience you provided with the candidate.
And look at constantly innovating your approach on how you do this. It's innovation in terms
of how you interact with your candidate and also how you leverage your team, your extended
team. Remember, the candidates here are also interviewing you and looking to see, "Is this
an organization I want to join?"
So have the passion and the commitment on both ends of the conversation. Look for that
and convey that to your candidates. Remember, they're going to be assessing you and your
abilities to meet their needs and their growth needs in their career as part of the interview
process. If we look at this, in the process here and obviously a key part of this is a
pipeline of talented individuals for your organization. I talked about a process for
how to screen those candidates but this process only works if you have the right candidates
in evaluating people with the right skill sets for your organization. So what we've
done is empowered our team where our current college hires from 2013 are playing a large
role in our recruitment process for bringing on our 2015 team. So look to them, empower
them, and leverage them in your process. Our on-campus interviews where we're building
our pipeline, obviously we're sending our alum to those events but also have your team
and reward them to go out and attract new and talented individuals to your team. Build
your team with wave...your wave two team should be your wave one team that's hiring and doing
the screening as part of the process. Remember, talent will attract talent and you get the
right combination of folks, wave two is going to be a very talented and capable team added
to your organization.
Remember, be humble. This is my own personal experience going on-site campus recruiting.
The candidates did not want to talk as much to me as they did our recent graduates from
that organization. So be humble and remember that the candidates will want to and relate
to these new hires from you organization as well. You are the decision maker and the key
part of the process but again, my message here is leverage your extended team and attract
new members through your existing team. And IBM's commitment, so just in summary wrapping
up here, IBM is committed to an equal opportunity hiring process. I've talked about just one
aspect of that process here with our technical screening process. Again, this is part of
a larger talent recruiting process here at IBM but I hope you've enjoyed some time. I
know I have, summarizing how I've used the IBM...IBM Watson team has used the the HireVue
CodeVue tools today and continue to use them in our screening process. And I hope you see
an angle and perspective and ability to use these withing your organization. With that,
I'd like to wrap up my session today and I'd invite you to attend two breakout sessions
starting at approximately 12:30 today and hope you enjoyed the time. I know I did and
thank you.