Tip:
Highlight text to annotate it
X
Sheena Goss is an interaction designer and user-experience analyst at Wells Fargo. She
researches web users and applies findings to simplify processes and solve business problems.
Formally trained in graphic design, Sheena combines her knowledge of layout and typography
principles to design and evaluate internal websites and applications. Her recent accomplishments
include developing a content strategy for the Wells Fargo Law Department, redesigning
the online experience for requesting and tracking work orders inside the corporate properties
division and conducting her department's first remote usability test to compare products
for multimillion dollar identity and access management project. Sheena earned her MLIS
in 2012 from San Jose State's School of Library and Information Science where she concentrated
on research methodologies, online information seeking behavior and web usability. Sheena
Goss, thank you so much for coming to speak tour Library 251 course here and go ahead
and take it away. Well, hi everybody. I'm really excited to be here and speak to you
tonight and to share my experience as a user-experience and interaction designer and also talk a little
bit about how some of the course work that you're learning in this class in the program
can help you explore other careers in the library and information science field. So
Caroline feel free to ask questions along the way if I miss something in the chat window,
just let me know. I'm really open to questions as we go through this presentation if you
have anything that you want further explanation on or needs a little bit more information.
So I'm going to start out by telling you a little bit about my background and I'll explain
that user-experience design process in the department where I currently work and then
I'll show you some examples of sketching and usability testing that I employ with some
of the projects that I work on. So Jeremy mentioned that I am a graduate from the MLIS
program at San Jose State University. I graduated in May and I've been working as an interaction
designers since October of 2011. I have a background in professional communications
and mostly graphic design and how I got to my place--my current place of employment right
now is largely in part due to a lot of the skills that I've learned in this program and
in this class specifically. My current career as an interaction designer, I found this opportunity
after completing an internship with the San Mateo County Library looking at redesigning
their internet and I'll show you some examples of what I did in that project later on. While
I was completing my internship and finishing my course work in the program, I was employed
as a communications consultant with my current company Wells Fargo. And as I was working
through the program, I started to explore alternate career path that has to do with
the Library and Information Science field and it really turned me into interaction design
and usability testing. So a lot of what I learned in this course really helped prepare
me for my current career. A little bit more about how I gained my experience, a lot of
the heuristic review skills, the sketching, and the usability testing assignments that
I completed in my Library 251 really helped prepare me for my internship at the San Mateo
County Library system. I applied to work on general projects with the library and because
of my background and my involvement with the internet, when the library was redesigning
theirs, they placed me and my internship project on helping the library system redesign and
evaluate their internet. And that project largely involved understanding the current
state of the internet and helping them understand what might be working well and might not be
working well using the vocabulary and the skills that come from interaction design and
usability testing. Then a lot of the project works that I completed in my internship really
helped when I was applying for an interaction designer position at Wells Fargo because I
was able to talk about and find heuristic reviews, usability testing, sketching, and
designing. And so it put me on a good path to be able to explore careers and user-experience,
and interaction design. So, here's an example of my--one of my projects from this course.
This is a sketch of the San Francisco public library's e-library site and I chose to look
at this site and evaluate it for my project that involved sketching and user-testing,
and heuristic reviews. And what I thought was so great about this project and what really
helped me in the internship and in my career is that it gives you a wide variety of applications
for sketching. And this particular one that I've got up on the screen was completed using
Adobe Illustrator and it's a little but more polished than most of the sketches I initially
started with. What it can get so great about sketching when you're working as a designer
is it gives you a lot of good ways to get ideas down on paper and it lets you approach
design from an iterative methodology where you're constantly looking at ways to refine,
and update, and adapt to your design based on the feedback that you're getting from your
users and from things like usability testing. Here's an example of a record that I completed
for my internship as one of the first project that I started on the San Mateo County Library
System called their internet FIDO which is an acronym for Finding Information Digitally
Online. And when I was getting started in the internship, the librarians were really
looking for a way to talk about what wasn't working well with the current state. They
knew that people were having problems with it but without something like a heuristic
review it was more difficult to articulate what exactly wasn't working with the design
and what's the interaction. So by being able to do something like a heuristic evaluation,
you can give a project team or an organization the vocabulary and the language to talk about
what they can out in their current system to create a stronger user-experience. Some
of the other deliverables that came out of my internship that really helped me when I
was interviewing for my current position were doing things like creating a site map because
that shows that you can digitally think about how to organize information for something
like a website or an online information system. I talked a little bit about the heuristic
review. I do many heuristic reviews now in my current position. They generally aren't
written up as formally as this particular report or as formally if they were in some
of my assignments that I completed for this course. Oftentimes, I don't share the full
research with my client because really what they're looking for is a way to understand
what's not working well and how it can move forward from that point going forward. But
for me as a designer, heuristics review--heuristic review has given me a really good way to level
that on what's happening in the online environment and what I wanted to do to adapt the interface
and make it more usable. I also completed some user personas in my internship and that's
a user-centered design tool that is extremely helpful not only in website design but also
in marketing and a lot of the information that you do to understand a user group or
any kind of style--of people that might be coming to you and looking for services. And
so before we get into more examples of some of the work that I've been completing I just
wanted to talk to you really briefly about career opportunities in user-experience and
obviously through taking this course, you've got a bit of an interest in this area. Library
and information science students are really well-qualified for a lot of positions because
of the skills that you are learning in this program. Many of them are transferable and
many of them are directly applicable like this usability testing course. But just some
of the areas where LIS students can look for areas of employment are in positions like
content strategist and those are designers that are working on projects to help understand
what information needs to be placed on the website to help users complete their goals.
So it's things like the text that makes up the information on a webpage, the documents
that are included in the website, the naming conventions for URL and subject headings,
and any of the other headers within a webpage. Information architecture is also another area
where library and information science students are really what qualify because you have the
skills for taking large amounts of information and finding logical ways to group as based
on the needs of your users. Some other career titles that you may hear and some fancy job
description names are things like user researchers, where you are employing different methodologies
to learn about the users who come to a website, how they digest and consume information and
their mental models for when they go to websites and applications and how they look for information.
Of course you're familiar with usability testing and usability analyst now, those are emerging
roles where your skills are definitely applicable. And then interaction design is like website
design but it's the actual discipline of figuring out how users move through a website or a
web application and interact with it when they're trying to perform very specific tasks.
And again LIS students are really well-qualified for those positions. A lot of the skills that
are employed in this career field are things like audience assessment, information organization,
program evaluations, and research. And these are all things that you're learning in the
program. That you have to do well in these kinds of careers. So let me tell you a little
bit about the user-experience process at my group. I work in a department of Wells Fargo
that specializes in technology and operation. So it's all of the data and the system that
helps function. And our particular group is a practitioner of something we call ,It's
a website discipline that really focuses on iterative design and constantly updating website
and designs based on user feedback. We operate almost like an in-house consulting group and
so different areas of the company will come to us with either a website or a web-based
application, or an online workflow that even looking to design from scraps or they have
something in place already that maybe isn't operating as optimally as they would like
and they want our help evaluating and figuring out what whether you certainly can update
them. So generally, what our process out is we stick with the client and we get to know
the particular issues that they're trying to resolve. We write out a working agreement
so they have a clear expectation of the kind of work that they can expect from us and the
things that we'll be able to do to help address their problems. And then we spend a lot of
time understanding their business needs and this is a really fundamental process in web
design because one of the key points of having a website is understanding what your users
are going to come to the site to do. And space website design where you're really figuring
out the kinds of things people need to do on a website. It really helps to build more
successful sites down the road. So we spend a lot of time at the beginning hoping that
this is--understand and articulate what they're trying to accomplish. Then we also set up
timelines and just get a general understanding of what they can expect from us and what we
should expect from them in terms of--if they have a particular timeframe in which they
need to get something developed and deployed. After we've got that information solved, we
took of our user to research. This involves doing things like interviewing users of a
website or an application to figure out, maybe what's working while right now, or what isn't
working well and we employ a lot of user-centered designed techniques at this stage where we
do things like, I've gone to people's offices and watched them work throughout the day so
that I can observe how that things like and print out lots of documents and take them
up around their office that help needs, that they're having a hard time finding the information
that they need in an online environment. So we look for little ways that we could help
figure out what users are trying to accomplish and how to include that in our designs, in
our deliverable. We have to do a lot scenario and task modeling where again we take those
business needs and the goals of the website and figure out what steps need to be taken
in an online environment to accomplish that goal. At this point in time when we're getting
a baseline understanding of who the users are and what kind of information they are
looking for with might develop high level site maps. We generally don't go much farther
than the first or second level of a site at this point because this is where our concepts
of comes into play. What we do is we start with a general idea of what we're building.
And then because we want to involve users and constantly test and incorporate their
feed back, we don't develop the entire site map at the beginning. We wait until that portion
of the site so that we can design a little bit, get a prototype developed and then put
it in front of a user and watch them try to use it and make sure that it's meeting their
needs and that they are able to move through the environment in the way that we expect
them to. And then it's a constant process of iterating based on that feedback. If we
observe them doing something with the website that maybe takes them down a path they weren't
necessarily anticipating or we observe that they're having some trouble using the application,
we figure out where they're getting hung up and then iterate our design to address those
issues. And then it's a constant process until we get to a point where we feel like the user
is successful in accomplishing their goals and they're able to carry out those tasks
that they want to accomplish when they come to the site. So I'll show you some examples
of how things like sketching in usability testing can be applied in--for environments
and some project setting. Again, the way that my team works is very heavily project-based.
So we'll get an assignment from a client and then work in team where we may have an interaction
designer and an engagement manager who is focused on building the relationship of the
client and the web developer who is responsible for creating the code of the product. And
so in my role as an interaction designer, I'm generally involved at the beginning stages
of a project to understand the business problem. Conduct the user research and then start to
sketch and design possible solution and evaluating those sketches with website users and making
sure that they are functioning properly before we invest a lot of time and resources and
things like development and actually coding the final product. So a little bit of information
about sketching and how we use it in my department. I mean sketching is one of the best things
that you can do at the onset of a project even just to document ideas or things that
you hear from users, or ways that you think could incorporate a design pattern into some
kind of solution. Some of the things that I sketched and I've got around my workspace
are things like for a project that I'm working on at a given point in time. I will always
sketch out a website wire frame before, and then do an application to draw it out. And
I'll show you some example of what that looks like in the next slide. I've also sketched
out page lay outs, navigation elements. And my preference is to always start with pencil
and paper, I feel like it's a lot less intimidating and a lot less finalized if I can just get
my ideas down on paper first. Because often if I put something down on paper when I move
it into an application that will start my first round of iteration because I'll think
about new ideas or new ways of presenting information based on what I've heard from
users that I may not have necessarily known when I started my initial sketch. There's
a few different ways that you can sketch or really classify a low fidelity prototypes
and a high fidelity prototypes. And the difference is a low fidelity prototype is just a sketch
with no logic behind it. A high fidelity prototype could be something that is electronically
based that a user could sit down in front of and actually interact with like they would
function in website. So, low fidelity prototype generally are pencil drawings on paper, but
they could also be wire frame sketches using a wire framing application or drawing application.
The difference is that high fidelity prototypes have that logic implemented in them, so that
users can actually function with the document whereas the low fidelity prototype that the
user to interact with it generally for means like touching it with their finger or pointing
somewhere with a pencil. Generally reference but my team does remote usability testing.
And because I'm in one geographic location and my clients are in another one, the way
that we handle low fidelity prototype testing is all set up a screen image of something
that are done in Photoshop or another wire framing application. And I'll ask the tester
to move their mouse and show me where they would click it so it's an actual functioning
website. So here is a comparison of low fidelity and high fidelity prototype for a project
that I'm currently working on. This is for a department within Wells Fargo that has got
an internal website in intranet. They've actually got 5 or 6 of them. I'm trying to bring them
under one URL that better organizes that information and better market search services and explain
who they are to the entire company. And so when I started out with the project, the picture
that you see in the back left is a picture of an index card that I'm using to sketch
out wire frame of websites. This particular one is an idea for the homepage of the new
intranet. And you can see it is pretty unsophisticated. It's very quickly done. It's just a way for
me to get an idea of things like how they want the global navigation to appear. Some
of the tools that I've heard from some of the users and the department representatives
that I've talked to have mentioned things like, you know we have such a hard time figuring
out who we are suppose to call in certain situation. So I made sure to carve out a space
to put in a search function for people to find subject matter experts. So the goal at
this point with the low fidelity prototype was to be able to document whatever appearing
problems of the users, about what they are trying to accomplish and make sure that I
have a point of reference to go back to with my sketch. This one I actually didn't do any
usability testing with it yet. What you see on the upper right hand corner is a high fidelity
version of the sketch. This is done in an application my group uses. It's called the
. And this is an application that lets us create clickable prototypes and wire frames
where we can actually include our business requirements and with the sketch. So if there's
annotation that our developers need to be aware of, we can bundle those in with our
sketches in one program. And with this version this is what I'll actually use to conduct
some usability test with the users later on in the project. And I see Jeremy has got a
question. I'm going to turn this off for one moment and hand it over to him. I'm sorry
if you had a chance to look at the SOLVIT from Google and if this is a similar kind
of an application? Yes, this is a similar application I've not used SOLVIT but I've
met a number of designers who use it and I believe there's another one called that's
similar. They are all kind of in the same vein and just produced by different companies
but it is exactly the same concepts as us. And Eric looks like you've got a question?
We'll do you have another one or ask one? Oh go right ahead? Okay, Sheena I was wondering
the second diagram, the one that looks more refined that was unclear is that actually
a website or was that another sketch that you drew with that program you were talking
about I missed that part. The second more refined image is not yet a website. It's a
high fidelity prototype that was created with that application that's called . So, there's
no HTML coding in this. There are no style sheets in the sense of how a website is created.
There's no scripting or programing. It's a lot like if you use an application like Photoshop
or Illustrator any kind of graphic layout application that lets you put images or shapes
under document that same kind of concept. I haven't employed any kind of logic or interaction
to this prototype yet. I might a little bit later on in the process so that when I have
someone test it they can see what it's actually like to try and search for somebody's name
and see a result window that pops up. But right now this is just strictly sketching
with text tools and layout tools. Well I think its brilliant because, you know, you are starting
out that basic one then you go to this other one and to me this is just a personal observation
that would make it easier then to use the coding and then say well, you know we could
make a table in the middle or three rows or whatever is so. I like that the program. I'm
going to investigate that. Thanks for letting us know. Absolutely and a lot of those kinds
of applications will let you download free trials, so I'm not certain if has one but
if you go to the website that Jeremy provided it will give you more information about the
product and show you some demos. And if you see those ones or and things like this and
they do have those trials and would encourage you to--if you are interested download them
and then play around with them a little bit and see what kinds of things you can create
using those application. Okay so, I'm going to talk to you a little bit about our process
for usability testing. I think you are either approaching an assignment where you're going
to experience usability testing or you may have already completed some. But I think usability
testing is an essential part of being a designer and creating website and online applications.
It's really hard as a designer to sometimes separate yourself from the process and make
sure that you're keeping your users top of mind and design things in ways that not only
seems right to you but also makes sense to the user. And so usability testing gives you
an opportunity to step out of your role as a designer and watch someone interact with
the solution that you've created and determine how easy it is for them to navigate through
a website or an application. So some of the things that we've expressed to our clients
when we tell them that we want to include usability testing in a project or engagement
we tell them it's a process for them to get an opportunity to observe users actually interacting
with the website that we've created. There's a number of reasons why usability testing
is incredibly important to businesses and companies and if you go to usability.gov I
believe I've got some resources at the end of this presentation. They'll tell you all
kinds of reasons why businesses and organizations should invest the time in usability testing
to save later on down the road when they've actually deployed the application that includes
things like, cost save and better customer experience, better reputation management.
So there really is a strong business case for including this in any kind of web design
process. One of the most important parts about usability testing and something that we remind
our clients of when they see you participate in observing usability test is that--this
is the time to watch the tester navigate through the website and listen to their reaction.
It's not like--many of our clients come from a computer software background where they
do things like user acceptance testing and they try to follow functional script to make
sure that an application that they can complete the instructions that they've been giving
and usability testing is almost the opposite of that where you just want to set up people
use and help them explore the interface as they would on their own if they were interacting
with it in their normal environment? So as a usability tester, one of the most valuable
things that you can do is to guide your user in giving them tasks to complete but then
step back and just observe and watch what happens. If you notice that somebody is struggling
with something or if they comment on something often it's an opportunity for you to ask pertinent
questions and figure out what's happening and really get into the user's state of mind
so that you can be able to articulate later on why something may or may not work well
in a particular design. So my team's process for usability testing follows a pretty similar
path to what we do with our design projects. It's really when we start a process, we define
the study with our clients and our project partners to help us understand the goals that
they have for conducting the usability test. In the project that Jeremy referenced in my
introduction we were working with a client that was comparing two different applications
and before they made the investment in either application, they wanted to understand what
few members would feel like when they use these systems and instead of investing the
money in purchasing both of them, they wanted to create a proof of concept environment where
people would get an opportunity to actually try and conduct their day to day work using
this application. Then we help them to understand what's--how to--the best user experience out
of the two. So I'll show you some examples of that project in the following slides but
it's really essential in the first stage of your usability test to define your study in
terms of what you're testing. Who's going to be your test participant? How they're going
to conduct the test, whether it's going to be in person or using remote tools which we'll
talk about. Then you want to design your test and that includes a lot of the exercises that
you go through in this course where you develop a testing script and you come up with the
template for taking note. You figure out the kinds of activities that you are going to
test. You're going to have the user test and that you're going to observe. Then of course
you conduct the test and there is all kinds of information available about how you can
facilitate usability tests in unbiased ways and make sure that that information that you're
collecting really does represents the user's behavior. And then finally, you--of course
always want to analyze and share your results so you can help people understand what you
learned by conducting these tests. And I'll show you some examples of our usability test
and how we packaged the information for our clients to be able to make a decision about
which application will work best for them. So, here is an example of when we were defining
this usability test about how we approach--and hearing out what people were going to--essentially
what they were going to test in the usability scenarios. We always start with an audience
assessment, user interviews, developing profiles and then deciding tasks that users conduct
in those scenarios. So in this example, you'll see a big table of five unique user groups
that we identified and the kinds of activities that they would conduct in relation to this
application. And the reason why it was so critical to have this defined with our partners
in the study was because they have a general sense of associated activities that were going
to take place with this application but they didn't really have it documented anywhere
And because the process was so different across the companies one representative may have
done things a certain way while another one in a different department may have a completely
unique approach for accomplishing these kinds of things. So, when we work with the clients
to define the kinds of tasks that we're going to test and made sure that all the usability
test facilitator and the client have a shared understanding about what we are trying to
observe and what was important to have us ask the usability testers to try and do during
their test. So here's an example of the usability test that we used for this project. It was
developed using the resources on usability.gov and using a lot of their templates and suggestions
for how to a usability test, but you can see a few distinct components of this test group.
And this is the first page from the documentation that we used in the test. I spread a lot of
this word for word when I was working with the testers. And what I wanted to do was set
their expectations for what was going to happen over the next hour during the usability tests.
I wanted to explain my role to them that I was here to evaluate and record their reactions
but then I also wanted to make sure that they were completely aware that we were testing
the application and it was not a test of their performance or any kind of judgment on them
about how they used the application. Sometimes people go into a usability test and if they're
not familiar with the process they may think that they're being tested, they're evaluated
in some way or they may not be using the application correctly. And in fact, that's exactly the
opposite. We want to watch real users conduct it in the way that they naturally would so
that we can make sure that the application is designed in a way that facilitates their
normal behavior. So we also make sure that some participants knew some things about the
test like it is not a test of them. There weren't right or wrong answers. We were recording
the audio from the test and also, capturing the screen images of the test since we are
working remotely. And any time you do that it's always important to make sure that people
know that you are recording and using the information. And depending on--you know if
you work for a contracting company or any kind of corporation they may have additional
requirements about gaining people's approval or consensus before recording their voice
or using their name in these kinds of studies. So it is always important to check that out
before you conduct the usability test. And then finally I made sure that I had prompts
for myself throughout the script to remind me for when I needed to do certain things
to make sure that the equipment was recording the test appropriately so you'll see some
bold text and italics at the bottom and throughout the script. Those were just to make sure that
I did things like start the recording application and start the recording of the conference
call. And like I mentioned our usability test was done remotely so that me, the test facilitator
and my test participants were not in the same room at the time of the test. The way that
we set it up in our department was we have an application that we use that lets us view
another persons computer screen, call a Live meeting and it's a Microsoft product but its
very similar to something like meeting or if you're familiar with those and what it
does is lets me view another persons computer screen on my own computer so I can watch do
things like use the website or try and use an application to conduct a certain activity.
And then finally since we did not have audio bundled in with our screen sharing application
a separate conference call set up so that myself and the usability test participants
and the clients could call in and listen to a tester as they were going through the test
and giving us feedback. So, here's an example of the Note Taker guide that was used for
this particular study. There's another designer on my team who is based in Minneapolis and
she plays the role of the Note Taker during the usability test so that I could focus on
asking questions to the participant and observing what was happening when they were trying to
complete test. And so, I created a template for her to take to record information about
the test and make sure that some of the metrics that we wanted to evaluate at the end of the
test to qualify and quantify the result or capture it somewhere. So, we wanted to know
things like the lengths of time it took them to complete a task. We wanted to know whether
or not the participant was successful in completing certain tasks. And then I have some follow
up questions that I ask at the end of the any scenario that we gave, and I gave a place
for the note taker to be able to record the test participant's responses to that. So,
the way that we set this up is there were at least seven or eight scenarios that I issued
to each test participants, and you can see an example at the top of the screen here where
I read out a scenario and then set the tester loose on the application to try and figure
out how to complete it. And so, this is the guide that the note taker was using as we
move through those scenarios to make sure the important information was captured so
that when we are analyzing the results we have this kind of guide that we could work
off of. And then of course the final step in conducting the usability test is finding
a way to share your results and there is a lot of different ways that you can summarize
the result. A lot of the decision just depends on what makes sense for you and your client,
and how much time and investment you can make in summarizing and capturing the result. usability
tests where consultants put together these really impressive presentations with video
clips and sound bites and recordings and other definitely helpful in recreating specific
issues that took place in the usability test and helping clients understand where users
may have struggle during the test, but those do take a lot of time and if you don't have
that kind of time there are other ways that you can still summarize your results and help
people understand what happen in a usability test without spending as much time on creating
a report like that. Some of the things that you can do is put together a spreadsheet of
quantitative results if that's one of the matrix that you're using in your usability
test. And quantitative results are anything that's got a numerical value. So it could
be things like the time it took to complete a task or a scenario. It could be something
like the number of participants who are able to complete a task, and the number of participants
who are not able to complete a task. What I usually put together is a combination report
that includes things like a summary of quantitative results, screen shots from things like remote
screen captures so that I can show what's happening in an interface when usability tests
are either struggling or encounter something that works really well for them. And, I almost
always will walk my clients through this reports by actually giving them a presentation. It's
not often that I just hand something over to them without scheduling some kind of meeting
to explain what happened, and that's because as the usability consultant it's helpful to
explain what's happening in a way that you could articulate what you're seeing and help
the client understand, this is what's causing problems for your users and there is a way
that you he can fix that. That's really where you can add a lot of value as a usability
consultant. It's not only point out what's going wrong, but to help them understand how
they can implement design changes that will remedy those situations. And one of the tools
that you're learning in this class that I thought was extremely valuable but understanding
things like design patterns and user interface design patterns, things like how the action
buttons, the navigation menus. Because when you understand those concepts, you are able
to articulate what's happening in an interface that either works well or doesn't work so
well. So I've got a couple of slides here where I can show you examples of slides from
the PowerPoint presentation that I put together for my clients, explaining what happened during
their usability test, because of non-disclosure agreements, I had to take out the images that
showed the screen shots of the tester using the application. So there will be a few places
in here where it says, image omitted. I'll try and talk to what was happening in the
test as best as possible. But what I wanted to share with you is there are some things
that I knew as a library and information science student in addition to usability testing,
that really helps me articulate what was happening with these applications and help the client
make a better decision because of somewhat the design patterns and actions that were
used in these applications. And, the first one that you will see on this image if we
were looking at the actual presentation that was given to the client, those on this interface,
there were about four different design patterns embedded into one particular point in this
scenario. What was happening with the user was moving through an interface and trying
to assign a new permission to a user's online account. And the way that the interface represented
this action was by putting in a drop down menu in the action button that was embedded
in the table cell. And so, they are trying to accomplish way too many things with all
of this very different design patterns that really confused the user. So it not only made
the page really difficult to look at but it made it difficult for the usability testers
to figure out what they were supposed to be doing at that point in time when they are
trying to add new permission to the user account. There are some other things that happened
with the page where it was just really difficult to scan and you could tell that the user was
having problems at this point because she plugged down and was reading through all this
information. She paused quite a bit. You could see her mouse cursor moving all around the
page and she just wasn't sure where she should go whereas if this was crafted from the user's
point of view there would have been a really clear call to action that would have guided
the user down the passage. She knew she should have taken to conduct this activity. In the
next couple of slides, reference, what I was talking to you about in terms of things that
library and information science students would be well-equipped to evaluate in these kinds
of usability tests. I've seen a lot of occasions where an application or a website will use
things like search tools or search catalogues in ways that are not really developed with
the user in mind. And as a library and information science student, you have a good understanding
of what they're trying to accomplish but coming from a usability background and having this
course work behind you, you can really help developers and designers understand how they
can facilitate these kinds of searches in a way that's more meaningful for the user.
So in this particular example, this was the second application that we were evaluating
and there was a catalogue of different kinds of permissions you could add to a user account.
But there was no real prompt for the user when they got there. They had to come up with
the search terms on their own. And because there weren't really any hints or suggestions
given to the users, they all kind of stalled and got scared when they got to the search
catalogue and didn't really know how to progress forward. There's also some training and instructions
on the interface screens that instructed the user to use the search technique and they
felt pretty advance for a lot of the office users so we have people putting that asterisk
at the beginning of the truncated search term. They were trying to give people that put in
a full search term and then also include the F search so it was just a little bit more
complicated that it needed to be for this user base. And then finally, another example
from the first application that we were evaluating that included search fields where there were
so many different icons and buttons that were used to combine terms from searches that it's
very overwhelming to the usability tester. A lot of people had difficulty starting their
searches because they felt so overwhelmed right away. So that when they got to this
point in the applications they all kind of stalled and I think we only have about 50
percent completion rate for this scenario because the interface for conducting something
like a search was so complex. So those are just some examples of ways that you can package
information and help recreate for people that did not observe the usability tasks. What
your testers encountered and why it was problematic? Like I mentioned that we tend to ever include
a site for suggestions for design patterns that might mitigate some of the issues. That
really helps you scan out as a usability consultant because you're not only identifying the problem
you're helping people understand how they can overcome it and how they can design more
efficiently to still accomplish those goals. So finally, here are some references that
I find helpful and that I can quote pretty regularly, usability.gov. It's a great place
to go to help get templates that you can use for your own usability test. It's also a great
place that you can send clients who want to get up to speed on the principles of usability
testings and this is the website that has information about the benefits of usability
testings. I've used that section of the website a couple of times in projects where we've
had to helped the clients agree to studying the extra, you know, three or four weeks in
a project to do usability testing because they can see that it's going to get them better
results in the design later on down the road. If you're interested in the careers in usability's
and usability testing and design, the Usability Professionals Association is a great association
to become a member of. They have a conference I believe in April or May each year with presenters
from around the world and they're just--they are really great associations to help you
get an understanding of what the profession is like and also make connections with career
practitioners. The third and fourth bullet are some books that you can look into that
will help you get a better understanding of what it's like to participate, the user experience,
designer and a project environment. And also understand how to apply user-centered design
skills in addressing design challenges and business problems. Jesse James Garrett the
author for the first book is pretty prominent in the community and that's a really helpful
book for coming up to speed on a lot of the term out--terminology. They are only about
usability and user centered design principles but also about the different roles that come
into play in a project. And then, the fourth book was the there a project guide to US design.
That's a great resource for getting a better understanding for what actually happened on
a project when you're playing a role of an interaction designer or a usability tester.
Boxes and arrows is a great online community that has resources and articles posted. I
put one up that it's probably of interest to people that are looking into an information
architecture career path or user researcher career path. But coming from the program,
one of the activities that I found that I've really enjoyed most was a card sort. This
is a user centered designed techniques where you can get a better understanding for how
your users think about things like navigation menus and information to architecture by doing
things like writing out all the contents of your website on index cards and laying them
out and having users organize them in a construction that makes sense to them. It's a really fun
process. It helps you get insight into how users think about things like navigation and
website organization and maybe be can be fun for you to try out in your projects too. And
finally my contact information of if you have any questions about career or about working
in a field about making the e-mail address there, I'm on a and if you want to connect
to we have a profile, feel free to reach out. And again this is my e-mail, so if you have
any questions or just want to talk more about the profession, feel free to reach out. And
does anyone have any questions about any of the concepts that we talked about or anything
about the applications and tools? Okay Eric, I'll go ahead and turn it over to you. It
sounded like this something you were doing but I was watching one of Jeremy's videos,
her professor's videos we don't want to get too informal here and it was close to the
beginning of the class and what it would be with the students that in I guess, graduated
or had already done the class and what he was doing was flipping a card for each step
that he was taking so that he have essentially the number of cards maybe 10 or 12 that were
just about the same and he would flip one and then it would show button to press so
that he would flip another one and would go to the next screen. And so it was essentially
all the steps you would be taking and then look like it was towards the end of the design
and they're just wondering if that might have been something you do too? It sounds like
that using some kind of iterative design approach and I have not used a process where I flip
cards necessarily. But I have done things like printed out a wire frame and sat down
with the user and asked them questions like "if you are trying to find information about
this particular group where would you go on the website?" And it sounds like it's a similar
goal but what that student was doing, trying to create an iterative approach to refining
a design so that of the user could craft us down there model or excuse me down their top
they have all of the design is supporting that kind of task progression. Any other question?
I was just going to add real quick it sounded like he already had it finished but then it
was sort of the way, we know, like if Paul already had the drawings but then he was sort
of putting into practice in way. But I'm not certain that was the method but that's kind
of what I get out of it that he had the drawings done and it was like a way of navigating around
in the sense, Yeah, interesting this one is a video or a website or something that would
be interested in. But yeah sounds like a way that at least a designer was printing their
information down on paper before moving it into development. What do you do with people
doing website and people doing coding and all that who have no value for usability who
really just want to impose a vision or a sort of a view of things--actually it's not the
secondary, the user's secondary concern. What do you do about that? That is something that
we still encounter frequently and in fact one of the project that I'm working on right
now is they've got the sense that they wanted to skip through all of these research and
just get down to development. And what we tried to do in those situations is to help
educate clients and their partners about the value of user centered design. And the fact
that even though you are investing a few extra weeks in the project you are saving yourself
millions of wasted dollars in redeveloping your site later on down the road when you
find out that it may not necessarily be working for users correctly or, you know, coming up
to a point where you've lost it and you are not getting the results that you are looking
for because you had a completely different understanding of what you were trying to accomplish
versus what your users are actually looking for. So in business especially in our company
we've got a pretty firm commitment from leadership on the value of user centered design. We do
still encounter issues where people don't necessarily want to employ as much time as
we'd like them to. But in those situations we try as often as we can to help them understand
the value of what it means to invest a little bit of time upfront for what actually happens
later on down the road. And especially in redesign projects if you can do things like
take their existing website and gather some informal feedback it can help the business
start to understand why maybe their current solution is not working well. And from that
point on if you can get some feedback you can start to have some discussions about whether
it's an issue with the way the navigation is built or you can try and move through it
yourself and do something like a heuristic review and figure out if it's a way that the
content on the website is set up or you can talk to some users. Maybe there's a mismatch
between the business' strategy and what the users are actually looking for, so really
just a process of education in helping people to understand why it's a necessary step in
website development. This becomes pretty common in a lot of industries now, but there is still
occasion where you'll get people that may not necessarily see the value right away.
So it really helps if you can articulate why these kinds of things are important. Okay
thank you very much Sheena. So the students are starting their third seminar. So they
are starting to do some user testing, they started to do some paper prototyping and user
testing. They're going to be doing a presentation very similar to this one that you've just
done in terms of showing some testing process that they've gone through talking about the
script that they've used, talking about task that they've given the users and then watched
them fail at those task. What advice do you have to the students who are starting this
process of an iterative design in getting users involved with that design but they're
not necessarily using even--something that's like or rise? Do you have any advice for these
folks just starting in their first round of iterations? Even usability testing with pencil
and paper sketch is extremely valuable. Sometimes it might work to your benefit to start with
those tools because when people see a sketch or they see something in paper and pencil
they get the sense that its not formal and sometimes they feel more comfortable giving
you feedback or constructive criticism about something that is very clearly still in an
iterative design process. Sometimes if it's more formalize people feel like it's too late
for whatever reason to offer any kind of feedback that could be incorporated into the final
development of the process. So probably the first piece of advice is, you know, just to
you know that there is a value in doing that kind of testing and that it doesn't necessarily
have to be as slicked and polished tool really talking to users and getting their input on
the things that they are trying to do and the way that they think about using websites
and looking for information is going to give you a lot of value. And that's very easily
accomplished with the kinds of tasks that students are going to undertake in this assignment.
Another piece of advice that I would give is when you are sitting down with the users
and talking to them about what's happening it can be really easy when you see someone
struggling to just kind of suggest, you know, why don't you try clicking over here or what
do you think about this button? And its really helpful if you can avoid that as much as possible
because you don't want to accidentally influence the user into doing something else because
all those corrupts integrity of your test if your guiding the user on how to complete
a task. So even if it does feel a little uncomfortable sometimes it can be okay to sit in silence
while the user thinks through what they are trying to accomplish. There are a couple of
times in the usability test that I showed you examples from we had testers that were
just completely confounded by this application. And I so wanted to help them figure out how
to use it but I did everything I could to stay silent and let them work through the
issue and that brought us a lot of really good feedback in the end that we could give
to the clients about what users would actually experience if this application were deployed
and how they can go about remedying it because the user was able to think on their own and
tell me what they would be looking instead of what they were actually seeing on the screen
at that point in time. What advice do you have for folks who would present a design
to a user and then ask the user what they think? So do you think I'm doing it right?
Do you think this good, you know as oppose to giving them a task and watching them fail
at the task. But you, do you ask the general public the users what they think about in
the interface, what do you think about that? It's--I really don't ask them subjective things
like, do you think this is good unless it's followed up with questions about tell me why?
I try not to ask them questions about, am I doing this right either? I think that information
that you're trying to gather from those kinds of questions are things like does this interface
suit you and would it help you achieve your objective when you come to this website? And
there's more of what you think you can ask that without making it so much about you as
the designer and instead focusing on the feedback that you're looking from the user about their
experience if they were to interact with that tool. So generally what I do in a project
when I sit down with a user is I probably talk to them during an initial phase of user
research where I've asked them things like, tell me about your job, tell me about your
responsibilities, tell me what kinds of things you've do in your day to day work. And when
they tell me what they do in their day to day work I make note of that so that when
we get to usability testing I can come back and say," you know, you've told me that as
human resources manager, you issue time cards to employees, looking at this interface where
would you go to issue a time card?" So instead of tying the questions that you're asking
the usability testing fact to things like design critique, we are tying it back to objectives
the user undertake, the user would undertake on the website. And what are the best ways
that I got up to speed about conducting user interviews and focusing on asking more open
ended questions. First of all the library 210 reference course is a great course for
learning more about asking open ended question. But I also look at a lot of resources online
if you Google something like the questions to ask during user interview. You'll get a
lot of good information about what kinds of open ended question you can ask to get, to
know your users. And what kind of open ended question you could ask during something like
a usability test or design critique. Okay. Great. Thank you very much Sheena Goss, interaction
designer with Wells Fargo bank and a previous student in the SLIS and in this course. I'm
not going to take credit for your rise here. I think you brought a lot of background to
the program in visual design and also you got a really good internship and I think all
those things definitely joined together to get you some great opportunity. So thank you
so much for coming and sharing your experience with the students in the course now. And I
wish you the best wishes going forward. Thank you very much. I'm so happy I got a chance
to come back and talk to the students. I'm really passionate about helping students understand
career paths in user experience. So please if you have any question. And just a plug
for the internship, if you are looking for opportunities and I was at a situation where
I needed to work on an internship outside of business hours the program at the San Mateo
County Library have many opportunities for distance learners so if you have questions,
feel free to e-mail me and I can tell you a little bit more about my experience and
how to get in touch with the librarians there. Did you have a question ? Oh, I was just going
to see at that video that I was talking about earlier I just posted it. It wasn't one of
yours but somebody that did homework for 251. I'm not sure if you were teaching it at the
time then but it if--Sheena if you wanted to know what I was talking about that was
the video I spoke of. Great, thank you. I'm going to copy that link and watch it after
the presentation tonight. Well that was a wonderful presentation by Sheena Goss. I really
appreciated her hands on concrete recommendations for web usability and she's really in the
thick of it, she's really in the very narrow field. I think a lot of people in the SLIS
program, any MLIS program are going to have a lot of opportunities outside of just you
know, this very specific job of web usability, web analyst, usability analyst. So anyway
today, I wanted to go over the rest of the class, we're going to deal with tonight as
getting started obviously we had some technical issues, speaker Sheena Goss and I want to
talk to you about grading and feedback as well. In the course, we are using Canvass.
It is new to all of us so I think that's causing some people some grief as we get sort of situated
and accustomed to it. I'm going to go over the class schedule moving forward and talk
specifically about the final pieces to homework three, the presentation and the final project.
And, we'll go ahead and move here. So first off in the class, once you've completed an
assignment or a discussion, you can on the top right, you could go into that old one.
On the top right, you'll see a rubric. So that rubric is important. That tells you why
you got four out of five, or two out of five. And then, I have a text feedback, you can
see that right there in the middle, number two read my feedback. And if you want to,
you can add a response and I'll get that immediately. If you want to respond, you can even add a
video, or an audio response if you want to. You can also do this for all of the assignments
in one place. On the right hand side, you can see there, if you click on assignments,
in canvass, you'll get this list, and then you can click for the text, for the rubric
itself and the score. You get more information there on the right hand side. So there's actually,
canvass is very rich when it comes to the ability to give feedback on assignments and
deliver feedback to assignments too. So, any questions about getting feedback in canvass?
Okay, great. Thanks Eric. So, I'm going to move ahead one here. The rubrics that I have
created and the thing that delayed me was to create these rubrics, you can see I've
condensed the rubric that I've shown in the previous slides at a previous--in the beginning
of session that we have. I condensed it into a five point scale. So, are you--if you are
discussion for the week or for the two weeks, does it apply course concepts? Your message
is do they give feedback to classmates, and I've really been sort of emphasizing the personal,
personalizing your feedbacks giving a name to the student that you're talking to. If
you bring in your own personal experiences, do you bring in examples that go beyond just
the readings themselves, and do you do the two minimum per week number of posts? So if
you do those five things, you get full score in the discussion area. Do you have any questions
about the discussion area and the mechanics of the discussion? And the assignment rubrics,
it's another one, these are between five and ten points. I broke the first assignment up
into two smaller assignments. So, assignment three coming up is a ten point assignment.
But, those that drastic key concepts in the seminar, I think some people have some problems
here. If they don't, sort of define their terms and describe the readings that they've
done. So, if you launch into some sort of a description of what you've done or description
of a problem without designing, you know, where you got the information from, which
part of the readings you're talking about. All those things you need to put those right
in there and be very concrete in the assignment. That shows that you've done the reading it
allows you to communicate that later, maybe with somebody else, so practicing communicating
the lessons that you're getting in the class that's key and if you don't do that then I
don't have any proof. I don't have any really clear understanding of what you gained through
the readings of the materials. You've included all the required pieces as I tried to be very,
very concrete and clear about what is required for each one of this, number two, two pieces
of turned in, comments to three, you know classmates. I've been very, very specific
about those. And people are doing a really good job of cropping their screenshots, annotating
with images or circling things being really rich in the way that they illustrate their
submissions. And they are also replying to their peers with the substance it's not just
a sense or treated it says, good job, you know, it's a substantial description of what
the--which appears done and, yeah go ahead, go ahead Caroline just let me turn off the
mic. Yes, my good question just about cropping the images, sometimes I wanted to use images
with the wording is a little small so I feel like sometimes if I crop the image you will
not be able to fill the wording that I want to show. In that case it's not cropping the
images all right. That's fine the problem with cropping comes when maybe you loaded
a couple of web pages or you've got three or four levels of controls on the browser
and you could feel those, you know, not cropping the image always because the problem with
cropping in the image it always comes not being able to read the text well enough. So,
if you're doing something that makes the text more readable, whatever it is all the better.
Go ahead Eric. Yeah, what's works best for me is if tried to copying the image and the
looks that I go ahead and started at 400 pixels width and that doesn't work so well since
it's not very readable but if I crop it and then save it in a dairy large size let's say
twice that and then I resized it to 400 item. That's always worked best for both, for both,
for me for some reason in that it's--I'm not sure why that is but it's I just had a better
experience that way. Hey, Caroline, what tool are you using to do the cropping and the illustrations?
I've been using imager.com and it does allow you sometimes to upload the image as a thumbnail
and I find that, that crops the image behind image if I don't do that then it just, it
comes out too big. Okay, I guess when I say crop the image get rid of all the extraneous
stuff that doesn't have to do with the point you're trying to make. And I do realize that
the T-12 was a lot easier dealing with images and discussions. So, I don't know what to
do about canvass in that in that sense. Yeah, you have to upload outside of canvass and
then make a link to it and it's . So, there's definitely a minus for canvass in discussions,
go ahead Eric. At the beginning I used to be able to upload a big picture and then resize
it but lately I haven't been able to do that and I just want to know if we still can and
I'm not doing something right or that features not available anymore or. You know they update
the software every two weeks on the fly so they may have taken away that ability, that's
the first I've heard of that. Given the schedule moving forward now we've done seminar 2, design,
seminar 2 and I owe you grades on your homework 2, the sketches. So far this term I've not
really kept up with homework and grading as well as I have. So, I'm going to try and get
that done in the next day or two. Then you're watching lectures seven and eight and talking
about those and you're doing a paper prototype for homework three that is due on July the
25th. Then some paper prototype you are going to be doing presentations in the week of July
25th through August 1st. That's your paper prototyping with users, homework 3. Then the
final project due August 10th at noon, lets go over those in a little bit more details.
So, so far in the course you've done the seminars one and two from June through July 11th and
you've seen lectures, but the first two beginning lectures have been two and three, four, five
and six, you've done quizzes one and two and you've done homeworks one and now you've just
finished homework two. Tonight we had a guest speaker, a wonderful Sheena Goss from Wells
Fargo tonight. Coming up starting tonight lecture seven on user experiences live, lecture
8 there's two parts to lecture 8 on prototyping. Then homework three, paper prototypes, and
the presentations July 25th through the 1st, about the presentation time. So okay Eric
let me turn off my mic. What I remember that I had asked you before, but I just wondered
since we were talking now about if they have already been decided but I see now that they
haven't so, that's all I ever need to say about that. I think I have an error here,
paper prototypes are due on July the 25th, sorry about that. So mine is wrong. Okay,
all right so, when it comes to scheduling the presentations. So, I'm going to be opening
up a survey here. I'll publish the survey today or tomorrow actually I'll publish the
survey tomorrow. And this will give you--I'll give you 10 or 15 of these times and you'll
choose the ones that that work for you. I mean you'll choose the ones that actually,
you'll tell me the ones that don't work. I'm really looking for people to be as flexible
as possible. So, if you have a specific conflict in a time like Friday 10 to 11 a.m. if you
can't do that then mark it down as undoable. Some people in the past caused us a lot of
problems, if you say I only want to present on that Thursday at 2 p.m. and that's the
only time I'm available it causes a lot of problems, you try to avoid that, try to just
mark the times, the conflict times, times that you really can't make it and then that
will make it more flexible for everybody. So, anyway you'll have doodle survey up with
10 or 15 times, mark the times you can't do and then I'll give you--I'll start a forum
post so that you can go in and just select the post, select the date that works for you.
So, don't, don't I know this causes a lot of anxiety upfront but it really has not caused
any--anybody any problems because the two step process here makes it possible for us
to give you everybody gets a time that that works for them. So, I'll have the, initial
survey up a doodle survey app tomorrow. If you can go in and mark the times which just
won't work for you. And then, as soon as I get everyone's entries I'll give you the full
set of 4, 5 times that everybody could sign up for. So, that guarantee you there will
not be a conflict here. There will not be a conflict here, it will be not--will not
be, you know, no presentation times that work for you it doesn't work that way. So, Eric
you said you got the way that this works, right? I do I didn't realize that you were
about to answer the question that I'm going to have--just to ask them probably better
now. No it's a good question and everybody has--so this, you know, there obviously there's
no reason I feel bad here. So, homework 3 is up and ready for you to complete. That
is working with two testers. So you can find two testers and conduct two testing sessions
with the paper prototype that changes. So you're going to be making changes to this
prototype. You're going to add one forum post with your presentation files. And then attend
you were on Live, Illuminate sessions and you can deliver a 10 minute presentation.
So this is two parts you're going to upload a PowerPoint presentation and then the following
week you're going to enter illuminate or collaborate to your--and then you're going to give a presentation
based on your PowerPoint presentation. So this is a matter of finding people to do it
and finding a specific problem from this last round that you did the sketches that you did
and homework for two. So we're building our homework 2, and we're putting your solutions
in front of users and making changes. The internet features in your design work are
really important. So you're going to make a change, you're going to do something show
it to people they're going to give you some feedback which are you don't expect and then
you are going to make changes to your sketches and give them back to people. Don't necessarily
want them to be the same people so you should have at least two testers. The first round
you showed it to someone the second round you show it to someone else. So, I give you--wealth
of--no, no, no, no, no, no in fact, they're yeah, there are other classmates are a bad
idea probably. You want to go out and find folks who are not involved in the course.
And you really want to see people that are directly in front of you. So, you know, it's
okay to use a dad or a brother or a wife or a husband. Get those people and show them
your prototype. Don't ask them what they think but give them a set of tasks and then have
them work through those tasks. So you're going to create your step to reviewing the detail
description of the method there's some web page there. You're going to create the paper
prototype if the user is set tasks I'm not so concerned, you know, that it talks about
conducting a post session interview. I'm not so concerned about the interview I'm not so
concerned about the interview and what the people thought of the interface. So it's less
important as our guest speaker pointed out it's really the completing tasks and getting
something done with the interface. You are the expert, the people you're testing they're
not the experts. So don't give them an artifact and ask their opinion asked them to be subjective
about your interface. You know the patterns, they don't. So, you are the expert walking
on that situation be the expert. Give the user interface to a user and watch them fail
and make iterations of design changes based on those. All right, so, conduct at least
two sessions with separate users and changed your design at least once. They're going to
create a presentation no more than 10 minutes. Scan or take photos of two versions of prototype.
Be sure to describe your users so their demographics, how you found them, the tasks, you ask them
to complete it's specific design changes but you may base on specific user feedback and
be concrete that will be the key and core of this presentation in this homework 3. Problems
that you find in the interface and ways that you solved those problems and present your
work here in Illuminate. So, we're going to be in groups of four or five maybe three or
four depending. And so, each of you will be asking questions of the other presenters.
You only need to attend a single session, your own session. So, Eric go ahead, let me
turn off my mic. So, it sounds like we're going to be talking about the two sessions
in our presentation or am I not saying that correctly here. That's exactly right. You're
going to be talking about your testing sessions. You're going to be doing the testing in your
aluminate. No, you're going to be out there preparing the--your slide presentation a presentation
about this interface with live people, hopefully face to face. I think that's really the best
way to do it. You can use PowerPoint if you want to the some of these other tools as--Sheena
pointed out paper pencil often used often work really, really well. So, did that answer
question. Yes, thank you that's going to be interesting to talk about, you know, the process
in going through the testing again, as well as what we come up with so I'm looking forward
to that. Yeah, that's a great point. I'm not so much interested in a beautiful final product,
you know, your interface is beautiful, and then it looks great all that. More important
is your description of the process. I did this, you know, maybe it did looks little
sloppy at the start. I found these problems. This is the next version. The user seem to
react with it well this way and I seem to solve the problem that I found in my sketches
from my work too. So it's not about a beautiful final product. It's about the description
of your process, how you work through that those problems getting those designs and using
iterative design to create a better interface. Is that--good? It is thank you. Cohen did
you have any questions about homework three before we move on. And I'm really glad to
have you guys engaged. There's only a couple of you here. But you are really asking questions
that all the rest of the class is thinking about as they watch this later on. So Caroline,
do you have any questions, you could raise your hand if you do, I won't put you on the
spot. But, so for final project 20 point it's call it a capstone but basically for the entire
class in miniature so you're doing the evaluation piece, the design piece, the implementation
piece, all together on a problem you haven't worked with before. So, I mean you can stick
with watch or lots or as just your website if you want to, very few people do that they're
sick of it by then. You can go out find another interface anything that dulls a computer and
a person working together. This could be I've seen people go to safe way and do grocery
kiosk, people have done text only interfaces for through their ILS, so it's up to you.
Then you'll put that whole thing packaged together and--in about 5 pages of text, 10
pages with the images total. And then you're going to submit that, that actually doesn't
require any discussions, so you're not going to be talking about your classmates' work.
I really designed this to be something that could plug very easily into your e-portfolio
piece, as you finish the program. This is a 5 to 10 page report submit it through dropbox.
Any questions overall? Okay, great and I encourage anybody just listening to this or Caroline
or Eric if you feel lost, you want to talk with me about the class on myself. We could
do e-mail, I really want to be totally available for you. Moving forward, if you I have a little
bit more in the class we have 1 more round of discussion, 1 more round of homework the
presentation and the final project, and then we're wrapping it up. So, the classes, I think
it's been little rough with this user management system, but I think once I get the hang of
this and once I gets you rapid feedback. I think we're going to pick it right up. So,
thank you very much and we will finish the recording and talk you again for you presentations.