Tip:
Highlight text to annotate it
X
people's problems. Pick -- that was good to increase power output we found that we suck
community members get interested and with a new members joining and that was a different
problems and so those things are kind of important to understand that the difficult -- the problem
could actually induce and there aren't always interested it in solving the same problem
over and over again those other three motivating factors that I found as we win the survey.
Going back to the actual experiment itself, participants were randomly assigned to three
different types of incentives on this cash, the about status and the other one is Jeff
market and that one has to think through this where is explore this and that you should
get a good sense of you are challenged stakeholder challenge owner. Agency, what other incentives
could you possibly offer other than cash. If think your platform manager is incentive
and [ indiscernible ] they can steer you to what the right cash amount is but there are
other things that we always discuss at the lab they can increase or boost with the community
members are looking for. To put it simply this was the cash price structure where participants
were divided into rooms of 20 like a sudden this was what was offered to them into additional
cash point and you can see the breakdown there. The status prizes actually really interesting
because it's important to community members that their status is reflected on the public
site of the platform. What this was the case for top quarter in you can see a screenshot
there that you have ratings in you all kinds of ratings for marathon matches or studio
competitions and things like that. It's important that they get recognize them but we did in
this spirit was if you chose not to participate in of silo view and if you chose the extra
status prize you didn't get the extra cash your name would be given on the leaderboard.
If you were in the other rooms then you wouldn't have that choice. So glory within the community
is really really important terms of their parents and their i nteraction. Remember those
of the community is really important McKenna related it to some sense to use a basketball
analogy it's not just good enough for Kobe Bryant come back and play after think of injury,
he needs to come back and to not just the world that he is a basketball players but
the rest of the peers he plays within his level of play needs to be higher than what
they're saying it is. That would drive him a lot. Employees really important. Another
thing we see is the job market price this is really interesting. But we did was we offered
letters of recommendation from Harvard in NASA to these competitors and what we would
do is we have given them the option to send this therefore the sign to national GPL. This
really to a lot of folks who like solving problems and they think that when you take
a look at these results they didn't necessarily do this for the sake of gaining more prizes
in terms of money but I think they found that if my record is good enough for my resume
to be forwarded that to this kind of draw me in, expertise to these areas where I can
kind of be able to give my score over to these kind of highly selective companies and see
what we get. >> We have another pull question. -- pull
question -- pull questionpole question -- poll question. Basically is your agency a barrier
to challenges? >> You can select yes or know t here. The
results here are mixed. But 5050 and not just jumped up a little bit that's great that means
okay. Some of your agencies are doing everything that they can to facilitate challenges. For
those of you that answered yes there are barriers but go to the next one of basically we will
ask what is the biggest barrier of entry to challenges at your agency? People are bringing
in a boat -- they're both here. Okay. With that about a third of the people responding
and it straight across the board. These are things that we kind of the generally a challenge.gov
but it looks like number one right now is legal concerns. That something good to know
is that some events planned in the future and hopefully we will help you with that great
thank you very much. Jim, back to you. >> That is something that a lot of stakeholders
that I work with often come into issues with so I do hope that we will be able to organize
more around that and give you some information around it. In terms of actually winning a
challenge, whatever like to suggest this kind of goes off the competitor think -- seem we've
got going today maybe we can address this more thoroughly in the future is definitely
start with a challenge that you can run that's interesting and also if there's a technical
challenge I would have the right stakeholders in your organization engaged at the front
and I would also kind of recommend that there are ways to devise challenges that make it
interesting for the competitor. You may have a compelling technical challenge that you
would like to get soft but you might not know how to frame it so that others would be interested
in it. One thing to think about is how does this challenge have a broader impact outside
of my agency and what does that mean for my agency and the work that we're doing and what
doesn't mean and a broad scope of how does this benefit humanity? You'd You would be
surprised because remember these competitors are not domain experts, none of them are.
And if you pulse a challenge your winners are not going to be folks with Fortress in
that type of field but they have an interest in going back to motivating in dealing with
communities, these communities do want to feel at-large kind of a sense of participation
and they want to make a difference. I would suggest that if you have a challenge in mind,
one of the exercises that would be great to do is to walk through how is this important
to my own organization and how does this impact on a broader scale. I think folks who have
done that in the past a little bit more successful in framing and marketing the challenge.
>> I'm going to move on here. A couple of other possible motivators and again this is
kind of redressing this -- redressing prestige but in a slightly different way. What we've
done with Center for excellence is we've -- we have offered batches which sounds silly to
some people but these aren't actual patches that you ship out or anything but community
members have profiles you can decorate them with medals and badges and so on so we came
up with our own federal government batch that you see here this is actually the patch itself
is produced in the studio contest. Those folks can witness if they won a competition or a
challenge and it ends up being something that's very lucrative and going back to not just
the rating for external view but internally they want to be able to say that they've done
many things that are good and if your organization and if this is something you want to think
about how can you contribute to the platform that makes the platform a little bit more
lucrative and a little bit more on innovative side in terms of how can the platform engage
with you to bring forth new ways of rewarding participants because prestige does matter.
And here you another one. Toggle is a video production site where they solicit responses
in regards to the whole video production and advertising boat if what they do is they will
often highlight a person. Usually I think once a month and this is called the toddler
of the month. And they will have a Q&A session with a member the loss of a video of this
person's work and it helps this person get the word out on how they went about doing
this challenge. Not all the platforms do this but that's another interesting thing that
US stakeholders can explore is how did this person go about solving this problem and what
do we learn from that in terms of the challenge and that might help you think more about your
ongoing challenges that you have if you haven't thought about those it might give you an insight
as to how competitor thinks and feels about these challenges. Once again I think this
will tell you a lot about how the challenges presented to them and how they see themselves
fitting into the grand scheme of open interpret -- open innervate -- open innovation and crowd
sourcing. They said the vendors don't all offer something like this but I'm sure they're
willing to engage with you in some way if it's some s ort -- something that you thought
through. >> I think we're ready for questions. Does
anybody have any questions? >> Good. If anybody has any questions please
recommend. While we're waiting for more question's to come in, I wonder to myself, Jim.
>> My first question is in regard to sort of a were talking about the communities and
the drivers and things like Steege and -- recognition within the community that the -- tournament
lab deals a lot with top coder and seems to me have the community that is -- Technik in
their the quarter community and have you guys found that these types of communities exist
for all types of challenges outside of encoding challenges as well?
>> Sure. The short answer is yes. They do exist. We don't partner with some of these
other vendors or at least not as frequently but are researchers have had done work with
them and each community is different at the same time people don't -- people can meet
-- people who compete in these communities value the fact that their colleagues respond
to them and they value that there is somebody out there that loves loves challenges is much
as they do. It's important to consider that this is a people community, this is it just
-- we don't have robots on the other and just cranking stuff out the special if you go into
some more creative ones are things that to do with ideation and design and video and
you find that you see that people learn from one another and that learning experience is
really important to them as well. >> Rate. Thanks. Another question we have
here is we have one of the participants is a big proponent of patches and he wanted to
ask are you working are collaborating with Mozilla's open-source patch project?
>> We are not. As far as I know we are not, at least not in the Harvard section of it
but I think that could be very interesting. >> That might be something to look into.
>> Gets lost in open innovation is rewards at that kind of talk about that there could
be a lot more rewarding things. Often when we run challenges with NASA or other federal
government agencies the ask the question what else could you guys offer other than the price
amount. Again you have to operate within the perimeters of what you got to know that a
lot of private sectors can offer a lot of things sometimes they offer internships were
the offer Q&A sessions with some corporate officers and that sort of stuff. We've done
things and we've run experiments in the past we've offered resume writing to help resume
writers and subscriptions to an online magazine the sort of stuff. I think the more you get
in you get into this White house can meet offer that's not solely monetary and again
you have to deal with that legally with your agency but I think if you kind of get boxed
into well money is the only thing we can do think you are not exhausting the full resources
of the community. You would be surprised on top coder and some of these other ones, they
are really into the T-shirts. You be surprised a 20-year-old competitor will do for pool
NASA T-shirt. They compete the heart for it. >> That's great to hear. Thanks within answer.
Okay. We've been another question that's a little bit more broad but I think you can
get some insights basically this person just ask how often do you recommend running a contest?
>> There sort of anything that NCL has created a drop-off in entry?
>> There is a couple of things. We don't have any substantial data yet to support it but
we're kind of a firm believer in not running expansive algorithm contests back to that
-- but to but mostly because it's mentally exhaustive for a lot of competitors but we
have not done too much research around that so I couldn't say for sure it does happen
but I think with the pool it depends on the pool of entrants that you're dealing with
and again the idea of crowd sourcing it least for the stakeholder of the client is that
you're going to get the best possible outcome so I would be wary of running multiple challenges
began we don't have any substantial data yet on that side of want to get ahead of myself.
It's just in opinion at this point. When it comes to non-technical algorithms a software-based
challenges, I don't think there's anything wrong with running multiple challenges. If
think you can get a lot of people to contribute a lot of good ideas. So just to sum that up
I think it depends on your type of challenge in the platform you are using but from what
I've seen so far I think because the corporate sector kind of doesn't really well. The private
sector they have people kind of thinking of challenges and how they tie in and that sort
of thing. I would run them as frequently as I can. It least the week is by the our community
right now, more challenges doesn't hurt. At least at this stage.
>> Great. >> There is a great question that just came
through here. The Netflix drives which was won by a team that was formed during the competition
I'm hardly encourage that kind of open collaboration and challenges in the future?
>> Good question. We've done some studies actually around team formation that's actually
a separate topic that we can address at another time but I will oftentimes the way these things
work and appendix closely speaking to -- actually I will use Tongal for an example remember
Tongal is a video production site and they will have several phases you can join in on
a pitch phase or an idea phase where you can say something and you may not even have the
expertise to be able to create a video. If it's on like me you don't have any expertise
to do that and you don't on software and are not very familiar with it you might have some
great ideas around the topic so you could contribute that way and in another face of
the will take that and run with it and actually have something and more of the story and then
from that there will be an actual person will take that and make it a video and so we've
seen that in the past that you don't have to necessarily do things in a way that -- in
the traditional way where one person comes up with the same from the beginning of a way
to the end. In terms of specific team formation we've done experiments were we actually put
people in teams and for them to work together. It is effective. It does have it's upside
and downside of kind of making people work with each other but they -- again I think
people come up with some good ideas and good ways to solve problems that you may not have
originally thought of. >> I think that's great insight. The phase
model is something that we see being adopted more and more to try to break up those areas.
Okay. Is another question. Has anyone link competitions with learning events or that
earning certifications as a way to motivate article thinking?
>> Yes this is a question about earning a certification during the challenge process?
Have you seen that all, Jim? >> We have not. To date online all about kind
of what we're doing and open gov of crowd sourcing, it's not quite yet an established
practice where you can get certified to do it. I think we have folks who kind of are
examining that and taking a look at offering some things and structuring it not necessarily
like a 12 week course type of thing but making it a little organic. Today I don't know anybody
that does that. >> Okay. Thanks.
>> Maybe you do. If you do, feel free to please contact me.
>> Personally, I don't either. If anyone on the call right now has a challenge that involve
that type of work we put it in the comment box and we will be able to read it off.
>> Okay, I think we have time for a couple more, a few more questions in the queue here.
About How about this one. Hi week motivate solvers not only to participate but submit
high-quality solutions especially outside of the outgrowth of the knee.
>> I'm sorry, can you repeat that? >> Sure. Tackling motivate solvers not only
to participate but to submit high-quality solutions especially outside the algorithm
community? >> That's a really interesting question. One
of the things that we found from the signals experiment is that more money did not necessarily
yield better results it just yielded and we were able to track their progress. They just
work harder at it it didn't necessarily produce higher results again I kind of go back to
one of the things as you're experimenting, how are you engaging the community and how
are you engaging that part of the community -- and it's a bit of a marketing ploy but
I think it goes back to what is the core foundation of your challenge what problem are you trying
to solve what is that means your organization and what does that mean to the world at large a think if
you can answer those questions I think you can get some great participation.
>> Rate, thank you. I think we are down to about our last question here. As a final question,
this person asks do you have any advice on determining how wide to cast the involvement
net in terms of the number of offices or agencies to get involved in the challenge. This is
more like a partnership question. Given insight to that?
>> Well we've done some things where we had multiple partners, we have also had not sure
if I'm asking for asking the right question but we've had partners who have experimented
on multiple platforms and they know this one right now was actually using two different
types of platforms. Answer the same question we've had people, the tech challenge for example
for [ indiscernible ] prevention Dave used different platforms and a phase kind of setting
where they did step one into was one and then three through five with another. In terms
of engaging, I think you need to configure out what the purpose of that would be than
I think you need the right parties. That's what I would suggest. Are often than not from
my experience from having organized and run some of these things, if you kind of start
off without the right parties in the room it tends to drag on the process of being able
to formulate the challenge. If you're dealing just with your organization that's one thing
and I certainly understand how complex it is but if you have to include another agency
or another partner does make it more difficult. With that said we've had lots of folks express
interest in some of our public sector challenges that aren't from the government. We've done
some collaboration and it does take effort and again if your dot your I-letter's and
cross your T's about that means a few legal department and put that means in terms of acquisition and IP
what that means in terms of actual and how much time you had from stakeholders. I would caution you to do that due diligence
before you dive right in if you're looking to
partner with other agencies to do something. Other than that I think so is helpful to have more challenge champions in each organization
that's kind of what we are hoping for
is the community at large and the more we have the more easier it is to explain things.
>> Thank you, Jim. I think this was a fantastic webinar if nothing dealt -- if nothing else
than to buck the trend that these competitors are going and just for
the money. Money it sometimes is important that you highlighted a bunch of really interesting
factors that lead to into the motivating factors. Thank you very much and thank you so much
everyone else of the participants right now for joining us today we hope that
you will return in
and join us for another webinar in the future. >> Thank you very much.
>> [ Event concluded ]