Tip:
Highlight text to annotate it
X
Dr: Jeff Higginbotham, University of Buffalo: Our overt focus is going to be on different
technologies for the most part,
and then there are some themes that I think will be brought out as we begin.
I, really, I want to thank our panel: Peggy Barker, Greg Lesher and Dave Hershberger
for agreeing to talk today.
And they've agreed that they are going to get into a pushing fight during the panel discussion
to keep things interesting after lunch.
And of course we want to give credit where credit is due, and thank NIDRR for supporting all of our work.
When we started out, Sue and I started out talking about what we're going to do,
we identified about six areas and started fleshing those out, and then had a conversation
with Janice and Melanie and they said, "Well, we've got three,
and we don't know how we're going to get done."
So, we're not going to get done.
I guarantee that we will ultimately miss a lot, disappoint, and it will be very biased in what we have to say.
We're really out to, in part, to inform and also to provoke discussion for today.
And I think we have, you know it's a challenging thing to be up here
talking to this widely diverse group.
And because of the diversity, everybody knows more collectively than we do,
and at the same time everyone has very particular areas of interest and areas of knowledge.
So here are the areas we are going to try to address over the course of the half hour.
I would like to start out though -- I was at the Museum of Modern Art a few weeks ago
with my wife and we were looking around, actually,
Graham, who's the person who did the leg splints?
[ audience member replies, "Charles and Ray Eames." ]
Yes, the Eames.
So there's a really nice design exhibition in which the Eames are shown, both for the art
for the splint as well as for all the furniture and things like that came out, a terrific show.
But we were there and I saw this artist that I had recognized before.
The artist's name is Tempt One, or Tempt.
And Tempt is sort of his tag.
And Tempt is one of the major graffiti artists along with Banksy and people world wide
that have been elevating that form of cultural art into a much higher art form
and doing really risky kinds of things, and here's an example of his art.
It's shown all over the world, and in 2003 Tempt acquired ALS.
And you can see what a physical art form it is.
And in graffiti art a lot of it has to do with how fast you're doing it,
and how physically involved, where you're doing it
because the more the precarious places you do it, the better.
This is some of his latest work and so basically he got together with an engineer,
or an engineer contacted him and wanted to help him do that.
And so this is, really, I think, embraces what Sue and I have tried to think about,
and that is some of the underlying themes, the obvious ones:
science, technology, clinical practice, but perhaps more important are issues of performance and identity
because that really embraces, I think, what we all do as communicators.
Sometimes it's hard for us social and behavioral scientists to really realize
some of those issues which are particularly...related to identity.
And I have to say that Graham is talking a lot about that.
But I think that Tempt really embraces all of those issues in incredible ways.
So, here's Tempt.
This is the first device that he had.
He later, I think, has a Tobii screen that he interacts with, but this was a $50 Play Station concoction
that his engineer worked with, and you can see this is a reproduction
of Tempt's eye movements, and here she doing some of his work.
And I have a quick video to show you.
So moving his eyes is part of Tempt's therapy, right there in his room
and he creates art we see out here in public
just last night we were out in downtown Los Angeles
Dark corners of downtown LA, and I stood here on a dark corner
fearing for my life ( laughter) and right there on the tall building
unbelieveable - Tempt was writing a tag, from a hospital room miles away
and it was being projected onto the wall of the building
So until now, Tempt -- at least up until now -- Tempt is still a productive artist
and, in fact, you can think of this marriage of technology and innovation and identity and performance
that Tempt now has expanded as a consequence of this in arenas that he's never been
able to do before. So that's one of the book ends of this.
Sue's going to talk about multimodal input and general computing and social networks,
and then I'll get back on and talk.
Dr Susan Fager, University of Nebraska, Lincoln: Can you guys hear me in the back?
I sat in the back all day and it's kind of hard to hear. So, just do that sign if I get low.
So, I'm going to start with multi-modal, multi-input access.
I really don't have a good term for this, but what I'm referring to is just the physical
access to the technology,
and traditionally we've focused on, not always but traditionally,
on a kind of single access method, and it's usually the best, most reliable physical access movement system
that we look at when we're doing clinical evaluations and access to AAC technology.
But, if you look at the way individuals communicate, and the AAC systems that they use,
they're typically not just a single access sort of system.
And usually the person themselves, in a low and a high tech kind of way,
incorporate a variety of modalities and movements and signals when they're communicating,
and so I want us think a bit about how we can get technology to start to implement these different
sorts of movements and signals for more effective communication.
I'm just going to touch briefly on some of the challenges and benefits
of both sorts of methods.
And obviously the benefit using a single access strategy is you targeted the most
reliable and consistent movement, but what might be challenging is that while at the
point of evaluation that was the most reliable and consistent movement, it may not be the
most reliable and consistent throughout the day so, it might limit the amount of time the
person can actually access the system of the device they're using or their computer.
It also sets up a person potentially for fatigue depending on what the medical condition
is and also something that we don't really know a lot about,
but kind of see clinically is the potential for overuse sorts of injuries.
When people used head sticks, when that was implemented, quite frequently we ended up seeing,
about 10 or 15 years later, a lot of head and neck pain associated with head stick use.
And recently, clinically, I've noticed a lot of individuals who use head switches
as they grow older they're starting to have similar issues and pain associated with that,
so we're starting to look at alternative access such as eye gaze for those individuals.
So, given a multi-input strategy, What are some of the potential benefits?
Maybe a more efficient method to your ... accessing your computer system
allowing for access throughout the day.
Another idea might be a way to promote additional movements and train additional movements.
So, for an individual who has like an acquired injury, maybe a brain stem stroke or
severe impairment where they're pretty locked in but they're starting to have some emerging movements,
physical movements, instead of zeroing in on the most consistent and
reliable movement which may be eye gaze, perhaps they have a little bit of hand movement
and you can start incorporating that, then maybe over time that becomes more efficient
and more effective for them so it's kind of, it might have a therapeutic benefit.
And some of the challenges associated with that again is just their technological
limitations that there are no computers or SGDs that really incorporate multiple
modalities, multiple methods, and also it's a really difficult task
because there are a lot of individual differences.
One person might have a certain set of communication capabilities
and another person may have a completely different set,
and how to anticipate that and how to design technology that can support that?
I'm just going to give you an example here and this is really more of a
multi-input strategy research that I got involved with, and it focused on using speech recognition
as part of this multi-input strategy.
There's been a considerable amount of engineering efforts behind
dysarthric speech recognition over the years.
It's primarily been successful for people who are mildly dysarthric
or for very small, limited vocabulary sets such as environmental control applications, and a set of ten commands.
But for people who are more severe -- moderate and severely and profoundly dysarthric -- speech
recognition typically isn't a functional writing tool, a functional communication tool.
So, this project to supplement speech recognition project that collaborators, Tom Jacobs
with InvoTek, Dave Beukelman and John Hosom looked at a system that used speech
recognitions based on the subject's speech, but also incorporating first letter
identification or alphabet supplementation.
So, as the person intended to write a word, they would type the first letter of the
word, that is the target word, and then also incorporating language modeling and word
prediction, so multiple inputs into the computer in addition to the dysarthric speech
signal help the computer understand more effectively what the individual was saying.
We've had individuals who are severe to profoundly dysarthric having key stroke savings
up to 75 and 80 percent using this sort of technology.
And I want to also recognize others, Dang and Patel and colleagues have also been looking
at this whole dysarthric speech recognition area and they couple that with surface EMG
signals so not just that signal itself but surface EMG to help increase the accuracy of recognitions.
Just a little bit of -- a few examples of using multiple inputs for computer applications.
In general computer navigation, mouse control and writing -- this is a bit of a quandary
because insurance in the United States pays for SGDs based on, really, medical and
face-to-face communication type needs.
It doesn't quite recognize writing, e-mailing, texting -- those sorts of forms of
communication -- as medically necessary.
So, obviously, we've focused our development efforts and our software development efforts
on software that supports face-to-face communication.
However, those clinicians in the audience and those people who know and work with individuals
with disabilities know that they really have a strong desire for mainstream computer
access options.
There's quite an emphasis on that, and, in fact, the majority of patients with ALS that I
work with at our facility use e-mail, facebook -- that's their primary mode of
communication, and when we evaluate, that's their determining factor.
That's how they make decisions on the kind of device that they want to use.
We had a follow-up study - I think that Dave Beukelman mentioned it earlier --
on early eye gaze use, early eye gaze system use,
that's when ERICA was the first fundable eye gaze system on the market, and we followed 15
individuals with ALS and documented the use of the system, and well over 70% of them used
it for web-type applications: e-mailing those types of things.
So, even then, they were used pretty significantly for that.
But with current SGDs there's a lot of limitations how the computers (�.powerful?) and
they often have to use some sort of interface that a person has to use in order to do
traditional web applications.
And just to show you an example, this is a grid software that's housed on
an Intelligaze eye gaze system and this is the interface you would use if you wanted to do web browsing.
It's different than you or I ... how we use the computer.
We do web browsing.
And the reason, there's a very good reason they set these up.
It's because eye-gaze systems have very poor resolution.
It's very difficult to fit small things in targets on a computer, so you really can't
access a computer desktop with an eye-gaze system in a traditional way that we're all
probably used to doing, so you have to have these larger targets, but it requires you to
learn something new.
It's visually a bit distracting.
Here's an example of the same sort of application.
You just have to learn a whole system.
There's also limitations within the system itself that you can't click on links in your e-mail.
It just can be really cumbersome especially if you're really proficient at using this
traditional mainstream method of doing e-mail and web access.
So, some of the benefits to being able to access your computer in a standard way is some
of the ... as Melanie had mentioned earlier, the working memory and the sort of ... the
cognitive aspects of that.
We really would like to tap in, to be able to tap into things that are over learned,
things that we're familiar with, especially if you have an acquired injury.
You might have some mild cognitive deficits associated with that, or you might be having
a declining condition where your cognition will be declining over time.
If you can tap into things that you have already learned and you're very used to using,
chances are you going to be more successful and you're going to be able
to use it for a longer period of time.
And this is also, basically, you know, how we now communicate through a computer.
We use these sort of standard interfaces, so it would make sense that we would strive to
make sure we have access to these sorts of standard interfaces.
And then, social networking is just my last point and it really, it kind of falls along
with the general computing piece but it really warrants a little special attention on the
side because this is really a common form of communication and it's daily, it's hourly,
it's minute by minute, it's second by second for some people.
And social networking is kind of this broad umbrella term and I'm using it very broadly
and I'm referring to social networking sites like Facebook, Myspace, what is it...
GooglePlus now? texting, chat, instant messaging, all of those types of social networking
applications, and this is very appealing for those who have limited access to the
face-to-face communication which is ... it's a pretty large population of individuals
who rely on AAC.
A lot of them are, they might be in nursing facilities, they might be hospitalized
for a period of time, many are not employed as they may be at home, they have
limited opportunities to get out into the community.
It's how they interface with their friends and interface with the community
and i t's a way to express their own personal identity, socially interact without the time
constraints. There are those with a face-to-face interaction that Jeff will probably talk a bit
about later. So, some of the challenges with interfacing with social network sites.
I think in our focus groups we kind of mentioned that earlier today that the software
doesn't very seamlessly work with some of these sites.
Usually you have to use the interface and get on the web in that way and access them that
way, that they're not specifically designed to nicely interact with those sites.
And then also the whole concept ... and not to just beat up the manufacturers and the
technology developers, this is really difficult task because it's a constantly moving
target and so, How to we adapt software and continue to make it accessible and work well
to something that's changing monthly and almost weekly, and daily in some cases?
Also, how do people actually use social networking with their systems?
There's a whole area of research there that we don't truly understand.
A lot of things I'm telling you today are based on my own personal experiences and my own
assumptions, but I can't speak for everyone in this room, and I certainly can't
speak for the whole population of individuals who rely on AAC.
We really don't understand how they're currently doing it and also what are some
potential opportunities. EasyView I mentioned that
earlier and then how do you seamlessly integrate these, sort of, what I consider more
mainstream functions into AAC technologies is definitely a challenge for us.
Dr. Jeff Higginbotham, University of Buffalo
:I'm going to pop back just a second to the social network to make this transition.
We've been working on a project in the RERC where we're interviewing individuals with
ALS and their families.
We were recently out in Detroit working with Lisa Bardach on this project and brought
home some videos and we were looking at a video of a young man, 26 years old,
with motor-neuron disease and he's in an interview situation and he's telling us that,
as you'd expect, that his social network has really shrunk and that he has
a few friends that come and visit him -- he's really lost a lot of friends.
Well, we also have two video cameras and one is on the screen, so what's the guy doing?
while he's interviewing, and he and his mom are talking to us while he's downloading free
movies during that time as he's talking to us, and he's also on Facebook.
And so, as we're looking at this, you know, I ask, well how many friends
does he have on Facebook?
He has the type of Facebok account that you can go and look at -- he has 530 friends.
So we're really looking at a game-changing technology, particularly for people within
whose culture and age aggregate the kinds of displays of identity and those kinds of
things, who are not afraid of, who are not afraid of hiding that ... and who, infact, transact their
friendships through Facebook, not like many of us, and, you know, go back and we all
looked at each other and ... How many friends do we have on Facebook? ... those
of us that use it. It's pretty humbling.
So, I'm going to switch to mobile computing, primarily, sort of, the iPad era.
The first picture up there was done in 1980. It's the first video I made.
Jean Mirrors and Cinda Bartorf -- some of you know Cinda, some of you might know Jean.
And here he is that long ago interacting.
His primary mode of interaction with the device is a language board.
This is from a project just over the last couple of years and that young man working with
one of my research assistants with a dedicated device, and this was taken last summer of
another young man in our clinic who is getting ready to go to camp and didn't want to
take his dedicated device, so he's on an iPad, and now he can go to camp and get in and out of
his wheelchair really easily, and he can play games with all his buddies at camp.
He can do all these kinds of things.
Here's the state of the tablets as of yesterday, the day before yesterday.
A Google tablet came out a while ago -- Microsoft Surface came out ... I have a hard time
remembering that name for some reason -- Kindle Fire and, of course, the ubiquitous iPad.
There's a plethora of these technologies we know will never go away.
It's always going to get more.
And with Google entering the scene, the android market, I think, is in a game-changing situation.
So, the state of mobile computing -- you guys gave this talk during your conversations
today, and in the meetings that we had at our table, so I'm going to be pretty brief.
Just to reflect what everybody said, it's an incredibly disruptive technology and I don't
think that we've witnessed anything like it before.
And one of the key characteristics is just the incredibly rapid change which is
isolating, or sort of disempowering the research, because any time that we grab something
to look at, the technology has shifted and shifted dramatically, and I think that that's a
big challenge for us as people who define ourselves as researchers.
I think it marks the rise of the individual, and, that is, it allows individuals to
purchase the over-the-counter devices and not rely necessarily on prescriptions, and it
is challenging to the existing institutions, institutions like clinicians and the stable
industry, and it's causing all kinds of problems that we've been talking about for the
last couple of years. Two quick stories.
I have an ex-student that is a director of a clinic in Halifax.
Last year she wrote me that they're doing a study, that they have two iPads and two
iPhones and they were working on figuring out how they were going to integrate them into
their clinical practice. Nine months later they wrote me and they had 44 devices.
That's one story.
The other story is, I was at the New York Speech and Hearing Association giving a talk on
iPads and there was an issue where someone wanted to know something about Proloquo and
how you could do something with it and I didn't have the answer, and my colleague didn't
have the answer and, I guess it was crowd-sourced because the whole group of 80 or 90
people that were there not only provided the answer but took the talk away from us and it became a
massive help session. And when I asked a friend of mine, What happened?
He said, "Well, it's because it doesn't rely on specialists anymore." That is, every body
has an iPad.
The interfaces are similar enough, and, certainly, there's some specific knowledge, but
it has shifted, so we're looking at very obviously a very big challenge to us
professionally.
What I do want to show you is a little piece of video because it's one of the persisting
problems, and I wanted us to share an issue about access and to look at it, and to look
at it from -- I'm not saying that when you look at this individual that the iPad is the
right device for them, but you certainly can see what some of the problems are in terms
of simple, physical access which alienates many of the people that we work with.
You can see he's flipping it unintentionally, he's flipping it back, he's selecting
something he didn't mean to select.
So, my challenge to developers here, and really to all of us, is Why haven't we resolved
these issues? Why isn't this at the top of the list?
So, we really need to be working on improving physical access, making use of gestures and
this incredible sophisticated gesture interface on these devices, and figuring out ways
that we can leverage that for people who need physical accommodations.
There are major inter-operability issues.
People need to have equal access across applications which we certainly can't have
on the iPad, coordinate across technologies, single source inputs such as here's
Google Glass that's up here that came off the web the other day with a new announcement about
Google Glass being available in 2013 -- and here's the original Google Glasses and I
have heard that Jordy may sue Google for patent infringement.
I'm going to shift since I now have only three minutes for two more topics.
Natural language processing and AAC are two examples that we're all familiar with.
Down below is an Inkado close to my heart, word prediction system which is a natural
language processing approach, and up here is Cliff Kushler's Swipe system, all using
natural language processing techniques in terms of trying to predict.
There are a number of emerging areas in natural language processing using the internet,
projects that we've worked on for topic specification, utterance-based communication,
geo-positioning -- down below you can see My Voice's solution to that, one of the only
commercial apps out doing that.
We've done work on Partner talk and trying to leverage Partner talk into prediction.
These are all what I would call emerging areas.
Text simplification which has a lot of application, I'll show you in a minute.
This is an example of text simplification.
You get a piece of text, lets say from the internet and through an NLP process you can
parse that text in a couple of different ways.
You can make simple sentences out of it, and I would say that the challenge to
developers, that one of things we should be doing is taking that and making it into utterance --
making it so that we can issue these texts as utterances so that you can download what
happened with the Yankees, and be able to talk about it easily.
The other is text summarization similar, which is a really important type of technology,
I think, for people with compromised reading skills so that you can read the newspaper
and be able to comprehend more.
State of the NLP: keystroke savings ... really, I mean, there's some nice research out
there, but it has not moved very much in the last 10 or 15 years.
And the technologies that I've talked about as emerging, really are, for the most part,
in the lab right now, and there's some, I think, some problems with making
those transitions.
Speech recognition still doesn't support dysarthric speech or conversational discourse.
This comes from one of our colleagues Albert Robelard who's a sociologist with a
motor neuron disease, and he says, "Stock replies don't have built-in recipient design."
That is, they are not made for the person that you're talking to.
"Pre-formulated turns at talk cannot adapt to the particular words used and
to the specific setting of the verbal exchange. These predesigned turns at talk feel like heavy
weights, encumbering natural conversation."
I bring this up because utterance-based systems are very close to my heart
and I feel like I always need to be reading the detractors, but what Albert
brings up is the first point, Does the innovation matter?
Does the five percent difference in word prediction matter?
Does the text simplification innovation matter to the person?
So, considering what we were talking about earlier, about divided attention, operability,
What are the operation requirements of these technologies? What's the payoff?
We do not have robust access to the internet, so how do we build these new technologies
which rely on the internet so that they work without the internet?
And how do we develop the underlying data structures and databases?
It's the type of thing that we're mostly oblivious to, but some of the real advancements
in this field, whether it's an utterance-based communication system, or whether it's
word prediction for the next generation are going to rely on much more sophisticated
approaches, and I'm not sure we have the industry to do that.
I'm wrapping up.
I would be glad to talk to you about social interaction which is what I really wanted to
talk about. I'll wrap up with two things.
I really think that as we look at these devices, we really need to look at what people do with them.
It's ... we have speech-generation devices, that's what they're labeled. And I would make a
challenge now to go and interact, and watch people interacting with these devices
and see how much of that is visually mediated.
OK, there's some pictures up here about people looking at display screens.
My colleague Cre Engelke and I have been looking at how people use visual displays,
how to leverage it, and it's pretty interesting to see how much do displays and
devices that are never intended to be used as code constructing instruments are used.
And second, I'll leave you with two quotes.
This is from Dan Keplinger who is an artist and was the focus of a movie "King Gimp".
It's about voice and identity and I really wanted to say this to follow up on
what Graham said. He said, "If people told me if I was going to make it in college
I'd have to master the computer voice. But I hated the damn thing.
Nobody knows the real man, not even my mom. I'm worried that people will not talk to me,
but to the computer. There is no way in hell a computer voice can express the
emotion I have inside of me."
And the second one, a doctoral student of mine, Jenn Seale, who I think will be a really
important person when she finally gets out.
This came out of a paper she just wrote and part of her PhD and she says -- part of the
introduction: "Advancements in the field can be credited, in part, for the reality that
some augmented communicators are employed, college graduates, living independently,
and engaged in various other community activities." I think we should feel
really good about that. And she finishes up by saying, "But increased opportunities
for augmented speakers to express themselves in a predominantly speaking culture
are perhaps the most important result of AAC developments." So, just keeping those kinds
of goals in mind. And I'll turn this over to the panel for further discussion.