Tip:
Highlight text to annotate it
X
I am going to tell you a little bit about the, um,
we are going to tell you about the saga of
FUSE, this is a three-part presentation
I'll talk first, and then Dennis McCarthy
will tell you about, how we actually did it
how we got from essentially the beginning
of CD to the end, to a very successful launch
and then, this is a lesson I think here
sailing was not smooth after launch.
Things worked well for a while, and we knew
that there were some problems in the wings,
and we were worried about them
and of course the problems you worry about are
the other ones that don't, will they happen too
but other things happen. So, if there is
a moral to my tale, it's going to be that stuff
happens.
Now, what I'm going to do in the sort of 10 or 15 minutes
that I have, is I'm going to first tell you about the mission, what it is.
It's quite different from a planetary mission
it was quite ambitious in the scope of the instrumentation
and then tell you a little bit about the history of the early part
of the mission, which as you can see went on for 15 years.
And then, Dennis will tell you
about phases C and D and Jeff will tell you about phase E.
and the extended mission that follows. The
this is a picture of the launch
in June '99, it's a NASA mission
and it basically what it has, is a large FUV high resolution
Spectrograph, and it complements the Hubble space telescope
if you were to equip the
Hubble to do this spectral region, it would require an enormous
investment in technology, and provide an awful lot
of risk for the Hubble. So the solution
in a way, you build another satellite, a dedicated
specialized satellite. We are in a low
earth orbit, we operated for about eight years
and we were very successful in acquiring data of interest
to the astronomical community. It is a PI class mission
it became a PI class mission of the type you think it was a
PI class mission, only really at the end of phase B.
And uh, it's about
operated as a, it was operated as a
observatory in the first three years, half the time went to the PI team
half the time went to the community, and then after that
in the extended mission, all of the time went to the community.
It was very popular, oversubscription rates
were a factor of three or more. This
were a factor of three or more. This not a little
instrument.
If you were to look at the instrument, it stands
a little shy of 20 feet high, you know, it's
3,000 pounds, it's a big thing
and, just shipping it is a big deal
and, so, uh
I say it's one and a half, MIDEX
maybe when the MIDEXes really got small
it was two MIDEXes I don't know but
we were, there were no MIDEXes at the time
that this mission went into CD, and in fact
we were sort of the pipe cleaner for how to do a
PI class mission.
Now why should you do this? Well basically
because of technical reasons, the Hubble cuts
off a little bit below 1,200 angstroms.
Fortunately, the spectroscopic region really extends to
912 angtroms, where
the photoionization of hydrogen makes the galaxy opaque.
So there's a gap
and FUSE basically covers this gap. And the reason why it covers this gap
is that's where the transitions are. You think about
your elementary physics, you know
where's the resinence in the atoms,
and the answer is it's above 10 volts
and that's where all the strong transitions
are, and so FUSE is in this region, and if you want to do things like
deuterium, you want to look at very hot plasmas at 300,000
to 1,000,000 degrees, and you've got work in the spectral region.
And it's very, very hard using the indirect evidence
that you get with Hubble, to always know what's going on.
We were a very
broad mission, we addressed a very large number of problems
ranging from deuterium, which is related to the abundance
of baryons in the, right in the, three minutes into the
Big ***.
There's a big problem by the way, one of the problems we have, is we have
models that predict how many baryons based
on the number of baryons that we know
were there, that we think we know
were there in the beginning. We can predict how many baryons there are
in the local universes, E equals zero, half of them are gone.
Where are they? Half are missing. The answer is
they are in very hot plasma, they're very difficult to
observe. Now FUSE can't observe that plasma either
but, using the 06 transaction, which
can measure slightly cooler plasma, we can sort of see the hair on the dog
and therefore decide whether or not there is a dog there
and therefore decide whether or not there is a dog there. And the answer is
there appears to be a dog there. That's where they're hidden.
Oops, let me go back, what did I do?
Thank you.
Another discovery
was that, we knew that the galaxy had a corona
but we were quite surprised to learn that beyond the corona
there's a huge halo, that goes way out and it's very hot
it's very thin, and you don't see it very easily
but FUSE could see it, because we could see the incoming high velocity
clouds, and lo and behold you could see, almost like a shockwave
on the front end.
And another question, this is something
we didn't think we would, we never thought of it, God was good
to us and gave us this problem.
I remember sitting in a room in one of my colleagues said,
"Let's look at an A star,"
and I said, "Alfred, why do you want to look at an A star? That's crazy,"
"Don't have any missions in the far ultraviolet, we know that,"
He says "Let's look."
We saw this very strong chromospheric line from the
central star, very broad, huge.
What's going on? Then you suddenly realize it's the interaction between the
disc and the star. Which is still not well
explained at this time. Well, and on and on
okay, all right. So, this was a very successful
mission, we have lots of programs, lots of observations
lots of data, I would say
about 1,000 scientists worked on the program over the eight years of
operation. Very successful in terms of data,
many many publications, we stopped counting in March
of '08. Now one of the things you have to remember, the data
is in the mass archive
at the Space Telescope Science Institute.
It's accessible over the Web. And among astronomers
archival research is a flourishing field
I'll give you an example, on the Hubble space telescope
half of the papers that are published that refer to
the Hubble space telescope
are in new data, the other half are in archival
data. The lesson there I think for the planetary
community. I say this as a person who was involved in Voyager
and my colleagues have tried to dig back in the Voyager data, it's very hard.
If you're in a PI, if you're on
on the team, and you have access to all the software tools it's
fine, but if you're an outsider you're dead. And, uh
that's not true in astronomy and astrophysics
and the data sets are generally available to any astronomer
and they are used.
The other interesting thing that came out of this mission, is that the
cosmic origin spectrograph on the Hubble space telescope is
directly based on the technology that was
developed in this mission.
And a lot of the learning pains that we went through with the
detector, and with the gratings were transferred, and were an enormous
savings in the construction of this instrument.
Okay, now I'm going to change gears here, and I'm going to tell you
a little bit of the saga. At the end of the Copernicus
mission, a group of scientists
one of them was Ed Weiler. Ed
actually worked on the Copernicus mission,
and so he and a couple of his
colleagues, were just talking about the fact that it would be
wonderful to have another follow-on mission. And in fact the field
report, the Cato report that, at 80
recommended that at its highest priority for
small to moderate sized missions, are far ultraviolet spectrograph.
Working group was set up,
we wrote of report, and we baseline a grazing
incidence telescope, not as something as Chandra, but we wanted
in order to get short wavelengths we wanted a grazing incidence.
Delta class missions of those days were much more
shall we say, ambitious than, uh
explorers are today. Now when I, the original
addition of this slide did not say extensive
discussions, it said talk talk talk.
We, you know, who was going to join in
a party, you never have enough money, so who's going to join in a party?
You talk to ESA, we talked to various national
entities, we talked Australia, we talked to the United
Kingdom. There were many different players.
And, eventually it didn't look like ESA was going to play.
We put in Explorer in 1986
which unfortunately followed the Challenger accident
and there was a delay going in to phase A.
Um,
the, uh, when I remember thinking
when I finished getting that proposal in, the scramble to get the
proposal in, and we, I took my wife and daughters out
for dinner, and she loves Chinese food, so
we went to a Chinese restaurant that we like
and of course at the end of the dinner, there's always a fortune cookie
so I open up the fortune cookie, and it says,
"Beware of what you desire..."
"...You may get it." [ audience laughter ]
I think that's probably a moral to the story.
In any case, we originally
baseline and ELV, a dedicated spacecraft, and ESA
was going to be part of that, we were going to be
high earth orbit, it was really a
very ambitious mission. Unfortunately,
in 1988
uh, ESA bailed out
okay, they decided they wanted to build the Huygens spacecraft
on Cassini, and they couldn't do both. Well,
at that point the only way out was the low earth orbit Explorer platform.
And, uh, that
basically a shuttle mission would replace
it would take a previous mission off the Explorer platform
and put us on it. I wasn't happy about that
because nobody wants to get hooked up to the shuttle, but
I, uh, that was the only way to survive.
We did in fact enter phase B in '89
and, as I mentioned we had now
had to go to low Earth orbit, and there was a
big concern in the community because of the cost of Hubble
it was clear that low Earth orbit could be very complex, and very
expensive. All right, they said you're going to need another
Institute, you're going to have two institutes on San Martin Drive.
So, so, more to come.
Okay, more to come.
We thought we are going into phase
B, all right, now
I have to say, by this point
I want to make a point, that we had
gone through a couple of what you would
call de-scopes. We have gone from
grazing incidence, to, uh
normal incidence, and there was a case where the technology
came to the point, that in fact it was
attracted to go to the normal incidence
so wasn't really a de-scope, in fact I think we certainly
reduced our risk, and we probably
kept the same performance or better. We also went to a
what's known as the delay line type detector
which allowed us to get very very many pixels, at the
risk of drift, in other words it wasn't geometrically as stable
and we paid for that. On the other hand if I kept my first choice
in detectors, I think I would've never gotten it, I mean
the history of that other detector was that
$20 million dollars for a very small one
on STIS, and uh, we had a very very
large one on FUSE, it just, just would've never happened.
But the problem was, having solved those problems
circumstances caught up to us.
If you remember, this was the era of faster better cheaper
or that was when it was going to happen, and
as the pressure was put on the Explorer program
to do something, it was clear that the large delta
classed Explorer missions, and in particular FUSE, was clogging the
way. So, the solution was
you know, a phone call.
And a phone call came the day after Labor Day
you know Labor Day I always am very relaxed
at the end of that nice long weekend,
I was chair of the department at that time, so I got a call.
I was up in my department office, I was told
to meet Ed Weiler and Weidman
in about 20 minutes, and I ran downstairs and I
grabbed Dennis, we went to my office
and this is the call we got, okay, it was cancelled.
Now, NASA really wanted to preserve the science.
I mean, I wasn't like you know, they say, alright cross this off the list
we're moving on. I think people were pretty upset, but they were really
caught in a bind with a budget crunch, and then began
very intense week, I can't tell you all of the people we saw.
I'm not allowed to, but, the point is that
the science community, and the people like John Bacall
played important roles, uh, we talked all the players
and I must say, one of the important things in our case, remember this is
astrophysical community is kind of wired, they,
big, there's a lot of herd things that go on, and
there were people who wanted to start e-mail
campaigns, they said, "I can get access
to the American astronomical Society e-mail list."
And I said, "No, please don't."
And, uh, so, there was an issue of keeping the community
damped down. Finally, if you
can believe this, in uh,
come on, whoops,
yeah, here we are, four days later
Friday afternoon, Dennis
Mark Perry, and myself, met with
Wes Huntress, and basically this is the outcome-
we created a PI class mission.
Okay, so that in fact was the first real PI class mission in,
in, [inaudible]. PIs have always been PIs of the instruments
but they never took complete control of the mission.
And, uh, we,
were given, just about four months to prepare
a CD proposal, and of course we were, we had something very
ambitious, we had a high Earth orbit, on and on and on
and uh, we had to restructure that mission.
And, when you talk about the interaction between scientists and engineers
being important, I can tell you it was very important, okay.
It's very important. And when you decide on your
orbit in one day, you've got to have
all the players in the room, it's the ultimate skunk works
okay. And FUSE by the way was not
we were distributed in many ways, but the core group
was, was in Baltimore
at Hopkins in one building.
And, I mentioned the, uh also
I talked about, high Earth orbit versus low Earth orbit
and in the end, we were to show, with a lot of help from
Space Telescope Science Institute, that if you had a simple
instrument without a lot of modes, you could also build an operation
system that was relatively simple.
Not cheap, just simpler, okay.
So phase D began,
a day after Thanksgiving in '95, the
story that Dennis will tell then is how we got
from there to June 24, 1999, when we
launched. Thank you.
[ Audience applause ]
Dennis McCarthy: So I went to this Chinese restaurant
[ audience laughter ] I mean I think the,
let's see here,
after the COBE mission which we talked about yesterday, I
I was the deputy for the Hubble repair mission, the first
repair mission. And after that, that was it, it's time to go.
So I went to Hopkins, I was going to learn
to be a teacher and build an instrument
and I was there for six weeks to the day
when Warren and I got this phone call from Ed Weiler
and looking out the window, and go, and I went
"I don't have tenure." [ audience laughter ],
"Warren's got tenure, I don't have tenure...
...I don't have tenure." So we went to, uh
as Warren said, we can't tell you all the places we went
because, we just did what we had to do, and I can tell you that Ed Weiler
since I knew him, and know him, at the end of this
telephone call, he said, "Dennis, do what you gotta
do." And on the way home I went, "I know who...
...I know who to call." So we made some phone calls, and we went to
Washington and met some people.
And it turned out, that, if you have full cost accounting
FUSE was going to be done in-house at Goddard, which is another
training program like COBE which was good.
But with full cost accounting it was over $300 million dollars
and the system had decided, faster, better and
cheaper, and $300 million wouldn't do it.
So we went downtown and we met some people
and they said, "You know if you could do this for a third of the price,"
"we'd consider it." And I said, "I'll be...
back." [ audience laughter ] I had to leave to make it look like we
gave it some thought. [ audience laughter ] But since I didn't have
tenure, the answer was going to be, "We'll do it for a third of the price."
And in the end, the bottom line was
we did it for $120 million versus $300 million
and there were a lot of questions like how could you possibly do
this for one third the cost. Well, first
of all, we made it a PI mission, we concentrated
the work at Hopkins, and I can tell you, and I'll speak to
this, the system engineering played a big role in this.
And the other thing is, we procured what was available
commercially, we bought a spacecraft that was fixed price
we developed a ground system that was fixed
price, and as I said yesterday, if you
can fix two of the three things, of the triangle
if you can fix the ground station, and the spacecraft, then
you can develop the instrument, and that's what we did.
whoops,
I got, so if you look at this, uh
Warren alluded to the fact that this program, had
has, three countries involved, the United States, Canada, and France.
It has four universities, Hopkins
Colorado, and Berkeley, and then we had the
University of Puerto Rico, which is
where the ground station went. We ended up with a commercial spacecraft
from orbital sciences, and we ended up with
on a fixed-price contract, and we ended up with a ground system
which was a fixed price contract from Allied Signal, and they also
developed the operations control center, which actually
ended up in Hopkins, in the Bloomberg building.
So if you look at this
complex organization, and you're gonna do it for one third the price
you're going to have to do it differently
now I'm just skip the next chart because
you've already heard some of this yesterday, and it's a lot of
motherhood, but, I think I'll speak to
this by the time we get done.
So to do it for $120 million, we designed it
to cost, as I said yesterday, I put a
little logo in my office, which was right next to Warren's office,
and it had one word in, that said the word was "changes," with a circle
and with an X through the word. We're not changing
anything. And I think, and I think
Warren speaks to it, and there's a chart further back
it probably took a year for everybody
to understand, we're not going to change
it. I mean we've got to get this done on time, we did it
in less than four years, and were not going to make any changes
and because of the fact that the spacecraft was a fixed price
contract, you could not change anything
at the interface between the spacecraft
and the instrument. The other thing was, the ground system
was a fixed price contract, and between the spacecraft
and ground system you can't change that either. It actually made it
easier for me, once Warren understood what I meant.
[ audience laughter ] I was going to tell you
something but, we also tailored the
documentation, I think what I found, and I know what
Jim Fanson believes about 7120, it was not there at the time.
We developed a lot of processes, and we
eliminated a lot of documentation, and I can tell
you what we did when it came to procedures, normally
on other programs, the procedures are
written, and then there's another document that
explains why we're going have to do this
and there's a plan, and then there's the procedure. What we did is we put
the plan as page one of the procedure, if we
can do it in one page, that eliminated a whole set of documents.
We also streamlined the
decision, and the approval process, and we actually did it online.
all of the change orders, and all of the documents, were
electronically in the system, and people could put changes into the system
and we did it all electronically, which made a big difference.
The other thing, and I'll show you a chart, the key to this whole thing, in my
mind, and that's the key to most programs, is systems engineering.
You could not do this program with people all over
the country, and other countries, if you did not have a good system
engineering team, and that to me was key to this program.
We integrated the team, I mean, in the end
Warren and I sat next to each other, and the scientists and engineers.
It was not a skunk works, but they were all at the Homewood campus
in Baltimore, and they all worked together. And I think Warren
since this program, since I left, it's
become obvious, it took months for us, for the engineers
and the scientists to understand what they were
saying to each other, it would be a lot of this
this wavelength, and this, and this, and after a while
and it was, it was just a process, the engineers
and scientists got to understand what each other meant.
This program became a PI mission.
The PI Warren, was responsible for this
program, and everyone understood he was responsible.
I as the project manager managed the cost and the schedule and the technical
but as far as the, reserves, as far as the schedule
and as far as the cost, the PI was responsible for that.
I think the key
was the system engineering, as I said, as you pointed out
we were not responsible for the launch vehicle, but we are responsible for interfaces
to the delta launch vehicle, we had a system engineer
at work directly with that launch vehicle
that worked directly with that launch vehicle, we had a system engineer
directly responsible for the ground system,
a system engineer for the spacecraft, and across various
instruments, systems engineers, and we had one
mission systems engineer, and this to me was the key for us to pull this entire
complex organization together. I think the key is systems
engineering.
We uh, as I mentioned, we had a web-based tool because there were people
throughout the country, at Berkeley, at Colorado, APL, at Hopkins
and in Canada, and I think because
of the fact that we could do a processes and procedures electronically,
it made a big difference and it was very efficient.
The engineering change orders, on
CCBs, Configuration Control Board, we did
that electronically, and people did not have to come to Baltimore, they could do it
and send their comments in, whether they approved the change or didn't approve
the change, and it was all done electronically, you can do it
with people spread out, if you do it that way.
I'm not going to read this chart to you, but they'll be
quiz when I get done. [ Audience laughter ]
I think the key to this program, there was a lot of keys to the program
but one of the keys was, that this spacecraft was fully redundant.
And the other thing was, it was a fixed-price contract
and all of the fee for that contract was based on
on-orbit performance, was not based on the delivery
of the spacecraft, it was based on the performance
of the spacecraft on orbit, that was evaluated every six months.
And because of that, you could rest assured, this
was orbital sciences, that that spacecraft was fully tested, because if they wanted their
fee, the spacecraft had to work. And I think the other thing was
because it was redundant, it gave us the ability to know that we did not
have many single point failures.
De-scopes, I mean Warren mentioned
the fact that, before I got there they were de-scoping, and
changing the design of the instrument. When we got involved with
this program, and we restructured the program and said we would do it for a third
of the price, we actually went off on a retreat
which is now called an advance I think, but
I think this is important, we took everybody on this team
to a hotel like this, and we spent two days
and it was done before it had to be,
before it was needed, and if you do it before it's needed
people will tell you what they can do, figuring not going to have to
do this, but, you try to, after a while you have this group
environment, and everyone tries to come up with something they could de-scope.
So we ended up with 2 1/2 pages of de-scopes.
Well we could skip that test, we could
skip this spare, we could not do this analysis, and it went
on and on and on. And I think the key was
to have it done before it's needed, because when it's needed
you bring them all back in the room and everyone's going to go, "It can't be me...
I cant the de-scope." So we had the list.
And I can tell you, as we went through the program, most of those
de-scopes came to fruition, but I think you need to have it in advance.
And the other thing was, this program had a minimum
performance flow, which is now a requirement
and because of the fact that we went off and discussed the de-scopes,
the de-scopes took us down to the minimum performance floor, if required
I think the other thing we did was,
we formed our own room review team
we brought in people, and I think you saw yesterday on
COBE that we brought three senior engineers in to evaluate
COBE, we did the same thing on FUSE, we didn't get caught up in
the normal, without being derogatory, the normal
system of reviews. We brought in senior people
who were retired, who reviewed this program
periodically before we would have a review, with the government
and these people were so helpful, and it was, it worked very
well. The other thing is we had peer reviews
and we had these over, and over, and over
again through the lifecycle of this project, and the peer reviews
were, not view graphs, they were detailed
reviews, we brought in a senior electrical
engineer, senior mechanical engineer, and we would go through the drawings
and go through the schematics, and those were the peer reviews, and I think that
was the most helpful thing we did.
The other thing is this review team,
the review team that Warren and I put together
I mean it worked out really well, they would actually present to the NASA
review team what their findings were, but they were done,
it was done independently, but it also worked well.
I think as Warren pointed out, I think this was the first PI mission
it was a big satellite, and it was
$120 million. I think the key was to procure
the spacecraft off the shelf,
and procure the ground system, we actually put the ground system at
the University of Puerto Rico, and it was an autonomous ground system
where at Hopkins, and Homewood, would send the ephemeris
data to the ground station, the antenna down at
Puerto Rico, and it would acquire the satellite and it was done autonomously.
I think this is the key to
all your programs, buy two and develop one
and I think that, to keep this
in 1996, that we bought
that spacecraft, fully redundant for $35 million,
that included the on-orbit performance fee.
The ground system was fixed price.
The international partnerships, I think
Jeff Kruk's gonna talk about that, that was difficult, and it was, we were
by the time I got there, to do this in less than four years that became
a hard thing to do, I mean ITAR was not as
formal as it is now, but it was difficult to deal with,
and I think Jeff is going to talk to them.
I think the science team, because they work directly with the engineering team,
after a while they realize we're not going to change it, I mean
this is academia where I worked at Hopkins, and we could sit there and study it
and, analyze it, and write a report, no we're not doing that.
I mean, they wouldn't come to my office because the answer was no.
I think the key was the interaction between the scientists and the engineers.
it took months for us to understand each other, but that to me was the key
and it was $120 million versus
$300 million.
So we
accelerated the program by two years, we reduced the costs by 60%,
the FUSE instrument basically remained the same, as what it
would've been with the Goddard larger program.
There are credible
spacecrafts that you can buy, and ground systems you can buy
on fixed-price contracts, which allows you to develop the instrument and control the cost.
and I think the key to this program is the fact that it
it, had, had inadequate phase B
program. So when we got there to do
phase C-D the phase B design was intact.
And I think the other key was, that we were emphatic that this PI
was responsible, and it was his program, and we
worked directly for the PI. As Warren would tell me
once a year, I serve at his pleasure
because I don't have tenure.
Any questions?
[ Audience applause ]
Jeff Kruk: I should clarify I only took over the
mission systems engineering, a few weeks before launch
so everything they were talking about was actually Larry Franks doing,
after that it was up to me.
I'm going to be talking a lot about
the phase E today, and I think I'm the first one who is really saying much about that
some of the lessons learned really are peculiar
to our circumstance, you're unlikely... anyway I'm going to try to
in each case at least distill lessons that I hope will be useful to you.
This is, um, the sort of dramatic
points in our on-orbit timeline, all the
items in pink, here, have to do with gyro failures.
All the items in purple are reaction wheel failures.
And a green are positive events, where we've
uploaded new flight software to deal with those problems.
One interesting point is that we first got our on-orbit
indication of problems with the ring laser gyros only six months after
launch. As Dennis mentioned gyros had been reworked
um, sort of in the middle of our thermal vacuum
testing, the vendor thought they had solved the problem, so we went in to
launch thinking that those were not an issue, and I'll
also indicate that the reaction wheel problems
uh, we had no hints of, prior to our in-orbit problem
with them. And then finally,
I just wanted to mention, we decommissioned the satellite in October of 2007.
Um, Warren and Dennis have already mentioned
just briefly, we did all of the mission operations and science
operations, and community support
from Hopkins, I'm not sure for most of the planetary
missions if you'll ever have all of those functions under one roof.
But in some ways it was a very nice, um, circumstance, because
it really made it a lot easier to deal with the on-orbit problems when everyone was
physically co-located. Warren made
the point that the operations were simple, compared to other
low earth orbit missions, but in a lot of respects when I look at the other proposals
I've been involved with for Explorers, our operations were more
complex than any of them. It was very much like Hubble apart
from the fact we only had one instrument, in fact we leveraged a lot of
our planning software and data processing software
from the Space Telescope Science Institute. We were sort of
the first mission where they exported a lot of those tools to other missions
and so, we benefited a lot from their efforts
so I'm going to start by talking about
a lot of the problems we encountered, in the first six months of the mission.
A lot of them were not very major problems in the grand scheme
of things, but a lot of them do illustrate what can go wrong when
you don't think of absolutely everything you might need to test, during
INT. So the first example I'm going to give
here is, the fact that our four co-aligned telescopes
didn't stay co-aligned once in orbit. We found that whenever we
would reorient the spacecraft to a significantly different
angle in respect to the Sun, but the four channels would all move by
say 10 arc seconds, when you have one arc second slit, that's much
too big. The lesson here really
is that, even though we had a very extensive thermal vacuum testing
program, this was not something we could test in a thermal vac tank.
There was no way to simulate moving the sun around inside the thermal vacuum chamber.
Um,
I don't know if any of you will have that particular problem,
but the point is you really need to always be thinking
about what it is you can't test, and what it is you should be
starting to worry about as a result. We worked around this primarily by
just, changes to our on-orbit procedures.
It did add somewhat to our overall workload
in phase E. So the real lesson here is
just simply if you can keep your operations of flexible, it may
stand you in good stead.
Another problem that shouldn't have happened
was that the embedded processor in our detectors was
susceptible to single event upsets
they were apparently, quite well rated in terms of
total dose, but down in the very center of the SAA when we would go through
perhaps it was a multiple event, uh, thing that would cause
the SEUs, but then the processor would have to be rebooted.
This was in itself, not necessarily that
big a problem, but during integration and test we discovered that
high voltage supplies in the detector had a potential failure mode that the software
wasn't watching for, so we had to update the
software in the detector, which again would not have been a problem except that the
detector stored all of the code in PROMs, as opposed to EEPROMs
so we would have to reload the software from the ground
every time we had an SEU. And because you had
only a few 10 minute contacts a day, that was a very painful process
and then you have to ramp the high voltage back up, so once a week we were losing
1 to 2 days of observing time, which is not acceptable.
And so the way we dealt with that, is that we have plenty of spare
EEPROMs in the instrument processor, and we had the ability to reprogram
the instrument processor, we had all the flight software
developers still available. So we stored
the detector code in the instrument computer, and added commands to the instrument
computer, to detect SEUs and to reload the code
autonomously. It was a few lines of flight software
but you have to be able to update it, and a couple of if statements and scripts that were
watching telemetry. But again the point was
we actually had real margins that we carried into launch
If you're finding yourself in phase A or B, mentally starting to use
up some of your margin, you should stop and
take stock about what you're doing.
This is another subtle problem,
the bulk memory manager
in our instrument computer, had a minor bug when you would finish
an observation script and delete it, there'd be one little, couple of bytes of the header
that wouldn't get deleted, and it would fragment the memory. You would
think, "This is a stupid problem," but it was only evident after you had
done a couple of weeks of continuous operation. Well there was never
any test we ever did on the ground, that ran the code
uninterrupted for weeks at a time. All the
development testing, you know, you would
always want to make sure you had a nice clean
well-defined configuration, every day you come in, you reboot everything, powered up
start clean, so this problem would never manifest itself.
And once you get the mission sequence testing with a real satellite, again
test time in INT is very expensive, you've got lots of people
clamoring to do different things with the satellite,
you never just going to take two weeks of
time out of the schedule, and run a mission sequence test.
It just simply wasn't a practical thing to do.
So again, this was a couple line update
to the software one we detected it.
So it was not a major problem, but the real lesson here is simply that there
are some things that you're not even going to think to test, and it can come back to bite
you. Um,
another very subtle problem, came out with the
processor and the fine error sensor.
There was a very subtle bug here that, where
an interrupt would get clobbered, under very unusual time
circumstances, it was sort of a one in a million type logic
raised condition, but that timing would depend on just where the guide stars
happened to be in the field of view, so during INT we would
maybe have a dozen different test configurations for the fine error sensor we would
execute, we thought those words stressed conditions, but it wasn't really
a stress issue. The processor was not being overloaded, it was just
a random event of when certain events were tied to the bus schedule
collided with things that were driven by the CCD macro
timing. So is the kind of bug that
you have to run a million different cases in INT, but
here is an example where,
this particular process was running maybe a dozen times a second.
It doesn't take very many days, running a couple dozen times a second
to hit your one in a million chance. So this is a
example where to really run a test like this, you need to have your, almost a
Monte Carlo for your test conditions and commanding of
your system. I can't even guarantee you would
catch it then, but it was the kind of thing that you just
want to put a bug in the back of your mind, it's not always easy
to determine what your worst-case
conditions are, to run your tests on.
There were lots of other,
minor problems, none of them would be worthy of note, the kind of thing
that anyone's going to run into, but the workload
just turned out to be, under estimated by say, about 10%
and it's probably impossible for anyone to really predict what they're
workload is going to be at the 10% level, far in advance.
So we round up, uh, I forget it was just a couple
months until operations, we decided to hire two more people. And suddenly everyone
was able to get some sleep. The other issue
there is you can't predict where your problems really are going to be
so just crosstraining of your staff is invaluable, and that way you can actually shift
resources, as you need them.
Preparation for the extended mission was
nothing really remarkable, but it was a very large effort.
The ground station was autonomous, but all the
procedures in the control center were not, and it was quite a bit of work to
automate all of that. I'm not sure if nowadays a lot of the commercial
ground station providers, or ground system providers
in mission ops centers, probably already have a lot of this stuff automated, and maybe that
wouldn't be such a big deal, but if that's not the case you,
it's important to actually plan a fair bit of resources to managing that effort.
Okay, gyro failures.
Um,
we did get some sign of this during INT, the vendor thought they identified the
problem. I've heard, probably about two years ago that
they think this time they really have identified the problem.
The difficulty is, it's a
the gyros have, even so, lasted several years, if you
identify a problem during INT, your vendor claims they fixed it, there's no way you can then
sit back and run two years or three years or five years of life testing
so you're going to have to decide your own inherent
appetite for that kind of risk. In our case we really
didn't have a choice, at that point we were a couple months
from launch, there was no way we were
go and buy completely different gyros, qualify them, modify all our
systems to use them.
Once we faced up to the fact we were going to
have to live without gyros, we did have a
extensive skunk works program, I should actually
extensive is not the right word, it was only a handful of us doing it.
The thing I want to talk about
briefly, is that once you're in that kind of situation,
performance is nowhere near as predictable
as it is for a normal flight program, and a flight
program, you're designing your attitude control system or whatever it is to
be in a linear regime. You've got lots of margin, you can define what the margin is
you can define stressed cases, and
you can then run tests to verify that you can handle
the stress cases. In a situation like this, you can't really define what the stress
cases are, you start propagating noise.
Once we had lost reaction wheels, you start getting
positive feedback between lack of controllability, and lack of being able to
propagate your equations of motion, so you just simply half to live
with, not necessarily being able to predict that your system is going to
work. When the alternative
is just turning off the mission, it's not a difficult choice to make but,
the point here is simply, there are times you can't necessarily define
what your stress cases are, or figure out how to test them.
The reaction wheel problems were quite a bit different from the
gyros, in the sense that, we really had no clue prior to
launch, when orbital was deciding which
reaction wheels to pick, they really had no reason to pick one vendor over the other.
The main reason they went with the wheels we did is that they had, the
greater resolution
of the wheel speed brought back in telemetry, and they thought that would be useful.
It did turn out to be useful for dealing with the gyro
problems, we had a much better way of measuring
exactly what torque had been applied, but
it was a very much second order decision prior to launch.
Once again, there was a lot of rework of the
flight software involved.
The only reason I'm bringing this up, is that since our
problems, quite a few other missions have had problems
with their reaction wheels and I've been
called in a number of CDR panels and providing other advice
and, one of the things that
people don't tend to think about, is that they focus entirely on the ACS
flight software, and in the case of dealing with the reaction wheels
changes to the flight software are pretty minor. Changes for dealing with the gyros
were major, but, we're dealing with reaction wheels
and changes to the flight software were small, but the changes to the ground system
were enormous. Was an order of magnitude of more increase in complexity
in our observation planning, and so
my word of advice for anyone who's ever faced with this situation, of having
do additional redesign work, is to think about
your ground system. The impacts there can be a lot greater than on the flight system.
Okay, some of these lessons learned are not necessarily
tied, I didn't want to give all the examples but I'm going to talk to a few of these in more
detail. You can't make your
you can't make your optical end test high enough fidelity.
You can do them early enough, really
you probably would not be able to do them really as well as you would like.
They're expensive, and finally as they say in the
commercials, they are priceless. We put a lot of effort into
our optical end-to-end tests. They were extremely valuable.
We found one problem,
right in the middle of many months of thermal vac
That was a very good thing indeed that we found, and
we were able to rework some of the hardware and go back into thermal vac and verify we'd
fixed it. But the lesson here, is that you really have to have funded
schedule reserve, all the way up to launch.
You can have a, you can differ a lot of tests into your observatory
level testing, some of them don't make sense to do before observatory level testing
but if you find a problem, you better have time and money to fix it.
You know, you look at an Explorer program, or some of these other programs,
your total development schedule is three years, you barely fit anything in
... that's three years. And then you also have to have time to fix things
that you, that you find. It's very easy to write a success oriented schedule,
it's another to actually have to live with it.
Another sort of motherhood and apple pie statement
that a lot of vendors will give lip service
to, is you design your instruments
with always an eye towards facilitating verification.
Don't give it lip service.
It's probably the most important thing you can do in designing your instrument
is trying to figure out from day one, how you're actually going to verify
that it works. We put a lot of effort into that, every
single requirement, that we flowed down, form our science requirements
down to our lower level requirements we had tests for.
And we've verified every one of those, there are a few things we didn't think to
write a lower level requirement on, they weren't necessarily crucial to our prime
science. They would've proved useful to some of the other science we did.
They weren't part of the flow down, we didn't do analysis
on them, we didn't test for them, and a few of the things just simply didn't work
as well as we would've liked.
I don't want to go into all the details, some of them are much too
specific to our FUSE instrument, but it's just a
sort of a warning that can't really think of everything no matter how hard you try
but if you don't try, you really won't get it.
A high fidelity flight ops test bed was not something we
had prior to launch, we had it after launch, when we
assembled the different software development test beds from the spacecraft
and the instrument.
This is a lesson learned, that really only struck me as I was
putting together these slides.
We did all of our control center procedure development
or tested them anyway, with the real satellite
during INT, it really wasn't, it was always a struggle.
The control center would want to do a test, the people on the ground
were doing something with the
satellite, were always fighting over some schedule, when to put a test in
you know, I'd really like to do this test next Tuesday
and guys at the Cape would say, well we've got something else going
on that's going to take three
days, how about next Wednesday,
and then you say you want two days, and they say no, you can only have one day.
So every major procedure, that um, that we had to do, we did
get tested, but, the example of
wanting to run an emissions sequence test for two weeks straight was something
you simply could not fit into the INT schedule, and there was no really
obvious compelling reason why we had to do it. But if we'd had a
test bed, a kind of thing where the people on the ground could've shaken out all the
procedures for how we're actually gonna string together weeks or a month of observations,
how we are gonna generate command loads, how we're gonna actually do this
on an ongoing basis, and actually see what the workload is, that's the
kind of thing you could've done, if we'd had a test bed.
And I've tried to make this argument and, uh,
some subsequent proposal efforts, people would say that's an enormously expensive thing to do
but what I've been thinking about it, I don't think that's
really the case, every piece in this kind of test bed, something you're already developing
for your instrument software developers, or spacecraft software developers
or whatever, and if you just decide on day one you're going to have this,
you'll find, I think that it's not so expensive, the hardware is typically just
PCs, for the case of the flight processors
you really want, a representative piece of hardware
but it doesn't have to be space-graded, it can be commercial grade parts
I think it's something, really, you ought to give serious consideration to.
The sustaining engineering,
contracts I think are self-evident, in our case we had
support from all the component providers.
And including from the Canadian space agency, we started having
problems with the gyros, we needed to rework the fine error sensor software
and so they stepped up, and basically provided that support for us
and redesigned the FES software.
For planetary mission, this bullet here on TDRSS it's probably
not so important. During our in-orbit
checkout, TDRSS scheduling was very painful.
We didn't use it very much then, later in the mission
they had developed some really wonderful tools, for being able to access
unscheduled TDRSS time, that was
essential for surviving some of our days, dealing with reaction wheel problems.
The one thing I haven't really talked about till now, but I thought I'd
just mention it, FUSE had very stringent contamination
control requirements, more stringent than Hubble. And a lot of effort
went in from day one, to figuring out how to manage that.
I just want to say here, I don't know how many of you will be interested in this
but you really can do it, at modest cost if you figure out from day one
what you need to do. Dennis
mentioned that I might talk about the finer,
[inaudible] development with the Canadians.
The lesson learned there I think, if you've got international
partners, is to, again from day one in your MOU
or whatever, establish the mechanism by which the project
is going to control what goes on in the international partners,
industrial partners.
with FUSE the industrial partner in Canada
was Condev, their contract was with the Canadian space agency
not with us, so um, whenever I wanted to
relax a requirement, I couldn't just tell them, I couldn't just write a little letter
and say you can relax this, I had to go to the Canadian space agency and have them tell
Condev to relax this.
it was a much more cumbersome procedure then one would've liked.
I don't know how to
do this, provide advice in a very general way but to think about
this issue when you have international partners, and try to establish from day one
what the actual mechanism is for making sure that ICDs are
have legal force across national boundaries
how to decide, how to establish which requirements documents have legal force
across international boundaries. If it all is done through the contract
with the foreign partner,
it can make life very difficult. And there was one other
bullet I was going to add, but escapes me at the moment, so I'll just stop here.
Thank you. [ Audience applause ]