Tip:
Highlight text to annotate it
X
>> Craig: So good morning, ladies and gentlemen.
It is my deep pleasure to call the IEEE Cluster 2013 Conference into full session and welcome you
to Indianapolis to IEEE Cluster 2013.
I want to thank the sponsors, Cray and DDN, who are gold sponsors, very,
very much deeply appreciate their sponsorship, IBM as a silver sponsor, and Matrix Integration
and Hewlett Packard as bronze sponsor as well.
Our corporate sponsors contributed mightily to this conference, and their sponsorship money went
in part to do things like put good food on the tables at meals and also
to support student participation in the conference.
I want to thank the not-for-profit sponsors as well.
Case Western University and then a number of universities are all members of the Coalition
for Advanced Scientific Computation.
University of Chicago Research Computing Center, Clemson University, Georgia Tech,
University of Miami, Mississippi State, University of Notre Dame, which is the best private school
in the state of Indiana, and the San Diego Super Computer Center, and UCSD.
I also want to thank and acknowledge the National Science Foundation.
So one of the things that we did that's new with this conference is submit a proposal
to the National Science Foundation, which supports the presence of many
of the students that are with us here today.
Just a very, very quick note, the Pervasive Technology Institute
at Indiana University's hosting this conference.
As part of our efforts to be engaged and participate in the National Scientific
and Computing community, we exist through the generous initial support of the Lily Endowment,
which is a private, charitable trust created by the children of Mr. Lily,
who founded Lily Drug Company, and now to introduce our keynote speaker,
David Keyes is Professor of Applied Mathematics and Computer Science at King Abdullah University
of Science and Technology in Saudi Arabia.
He is the founding dean of the Division of Computer and Electrical
and Mathematical Sciences and Engineering there.
How cool is it to go to create a brand new university
and essentially create a new school from the ground up?
This is one of Dr. Keyes' major organizational accomplishments.
His scientific accomplishments are focused in the area of algorithms,
and I made a list of bullet points because otherwise I would have forgotten one
of the major awards that he has been given in his career so far.
Fellow of SIAM and AMS, the Sidney Fernbach Award, ACM's Gordon Bell Prize, and the 2011 SIAM prize
for Distinguished Service to the Profession, which I think is really, really significant.
With no further ado, I am going to happily turn the podium over to Dr. Keyes.
>> David: Thank you, Craig.
Greetings, colleagues.
It's a wonderful honor and privilege to set a keynote for IEEE Cluster 2013.
After I scrolled through the abstracts of the conference in detail last week
and understood the affinity of some of you to my own technical interests,
I was tempted to scratch my original talk and go directly to some
of those shared technical interests.
However, that would be too narrow a use of the keynote that Craig and I had planned after getting
to know each other for about a year and a half charting directions of what was then the Office
of Cyber Infrastructure at the National Science Foundation.
So in keeping with the purpose of a keynote, I will try to be expansive and reflective
on the role of scalable computing in the university of the twenty-first century.
I will do this from the perspective of a university that has known no other century,
and indeed is still in its first half decade, and you could question whether a university
so young had anything by which to inform its sisters, and from track record alone, of course,
the answer would be no, but the advantage of a university that is very purposefully planned
as opposed to randomly evolved over decades or centuries, is that a lot of talented consultants,
very probably a few of you here, were involved in helping to set up this experiment,
and its philosophy should therefore be a very contemporary interest.
What would you do differently if you could start over is the question that the KAUST team asked
as it toured many university presidential offices in the US, UK, and Europe and Asia.
So KAUST's future also depends very much on the future progress of the community gathered here
because having flung itself on computing, in part for the enablement of a mission
in a fairly unprecedented way, it will succeed or fail in large part with the achievements
or disappointments of scalable computing in science and engineering
and indeed as a focus of education.
There are many tributaries into this talk and branches off of this talk that I have pruned
in order to attempt to fit it into the time allowed,
but this means that you can customize afterwards with questions, and I do hope to leave time
for that with interaction both on the podium and then I'll be around all day at a few
of the receptions and posters and so forth.
So I do look forward to meeting many of you.
The title I've chosen as sort of a runner here is "Compute to Breathe."
I think you will probably all agree with that in the sense that, well,
let's see, computing is essential.
It's natural and it's available in the sense of access and cost to any university just
like breathing is to any person, and I skipped the map.
The pinpricks on this map show where you have been over these past 14 years, and the white circle is
where I'm coming to you from on the coast of the Red Sea.
In fact, I live in a villa on an island in the Red Sea about the latitude of Maui.
It's a very attractive environment.
I notice that in all of Africa and the Middle East, you know, this conference hasn't
yet made an appearance and maybe it'll be appropriate to do so in less time
than you imagined before, you know, hearing this talk.
So even though I think we would all in this room agree about the potential contributions
of scalable computing to a research university, there's a whole crop of new research universities
that have been founded after KAUST, which to my surprise are not built around this foundation,
and this makes KAUST even more perhaps interesting experiment to many.
When we were launched in June of 2009, and I remember well the posting of that top 500 list
that year in Hamburg at the ISC, Saudi Arabia suddenly found itself third in the list
of nations among, you know, computing capability following the US and Germany,
and that was one of the earliest appearances on the list of the IBM Blue Gene/P,
and you see all the occurrences of that in red, and that's the system
that we chose to launch our university.
That machine, that fixed machine, is now third in the country.
So time evolves, and machines slip, but we are in the market for a new machine in 2014.
That should put us back in the top 20.
It would be in the top 10 today with about 5 pedaFLOPS, but of course even a year is...
you can slip quite a bit on that list.
Now what about computing at other startups?
Here are some notable startups that have occurred since KAUST in 2009.
The Masdar, the Masdar's the Arabic word for source.
It's true to science and technology.
It's in Abu Dhabi.
The Okinawa Institute of Science and Technology, the Nazarbayev, the university named
after the president of oil rich Kazakhstan, the Skolkovo Tech,
which many of you have probably heard about, started under [inaudible] in Moscow,
Shanghai Tech, started by the son of [inaudible], a PhD in Material Science from Drexel,
in a very well placed square kilometer in downtown Shanghai, the new Hamad Bin Khalifa University,
which is being organized by Thomas Zachariah, who many of you know from the number 2 position
at Oakridge National Labs, taking advantage of the interest of the Qatar Foundation in going big
on research university and a new medical school.
So these are all new graduate research universities with major international partners,
admirable missions, bold webpages, and seemingly very strong financial backing
and influential high-level government backing.
In many ways they're following KAUST's model, yet computing is not really
from their webpages a detectable curricular or facility priority.
So I think it's, you know, important for the credibility of our field
that these major starts pick up on computing as an essential part of the university.
So with that as an introduction, this presentation will give a little bit of background
to scalable computing at KAUST, and we'll sing the hymns of, you know,
the motivation of why scalable computing should be important to any twentieth century university,
and then I'll spend the last third of my talk indulging myself in some descriptions
of some research highlights in my group at KAUST today.
So the purpose is, first of all, to try to be encouraging but cautious about the hurdles.
We're in this together.
We have to work hard at the frontier and then to lead into a productive discussion period.
So part 1, KAUST and computing.
There's nothing like a curse of high expectations, and in 2009, a month after we launched,
Science Magazine ran a lengthy article on us about the expectations on KAUST as a university
in support of sustainable technology for the arid regions
of the world-energy, environment, food, and water.
There's a snapshot of most of the initial faculty, two-thirds of which was from US and Europe,
and of the student body, two-thirds from the Middle East and Asia.
Amusingly as you can see over in the bottom right, the top three countries
in the initial class were Saudi Arabia, China, and Mexico, another oil country actually,
and many of those students have actually stayed in Saudi Arabia since graduating.
We still have a ways to go, and Science Magazine was happy to remind us of that with a follow
up even larger article in December of 2012.
You can, if you're interested in starting a university,
this is recommended reading, and, you know, it's...
it will help you to fill in a little bit about our background and our mandate.
To specialize down to computer science and electrical engineering,
these are the two largest and, as it turns out, most selective majors at KAUST.
By the way, we're a graduate-only university.
We have had the largest number of graduates in those areas, mostly Masters,
just the PhD trickle is starting to develop into a flow as we hit the 5 year mark this year,
and these two areas draw overwhelmingly the largest number of applications both
from the region and our international student body.
We have students from 70 countries at KAUST.
Also KAUST may have the highest number of women students of any graduate school of engineering
that I know of anywhere including the US.
I came from Columbia where the women comprised 26% of the engineering school.
Here it's 36%, and it's also rich in regional women who find, you know,
less easy to travel abroad, and this is helped by computer science and electrical engineering.
Our totals are higher than the university average along with biology in part
because the work environment of a computer science professional,
you know, can be a desktop environment.
You don't have to be held on an oil rig or whatever,
and so it's considered a good career platform for talented young women.
The three programs of CS, EE, and applied math and computational science have been quick starts.
We weren't waiting for labs to be set up and various agreements with customs in terms
of importation of samples or different things that are new to a country in terms
of what are all these requirements for research.
So we got out of the, you know, box in a hurry.
We had our big computer set up before the students and faculty got there, and we've had, I think,
a very encouraging performance if you look at the kind of metrics that provosts
and presidents are interest in in terms of Thompson Reuters rankings.
But all four of the new initiatives since operations began have partly in response
to this student demographic been is these three areas of CS, EE, and applied math.
Another aspect of progress that you would be interested
in at this meeting is the facilities angle.
We did start with our flagship super computer, 222 teraFLOPS, IBM Blue Gene/P,
named Shahin after the Arabic word for falcon, which is the fastest animal.
It has approximately 200 users over 50 active projects,
a good percentage of which scale to the 64K cores on the machine.
Over a billion core hours have been utilized by ourselves and our international partners.
In terms of Linux clusters, we have several including the flagship, [inaudible],
which comes from the Arabic word for light.
It has approximately 650 users, 100 active projects in the distributed computing mode,
and over 250 applications maintained.
We have a couple of recently built experimental clusters of Zion, Phi, and Kepler,
early access donated by their companies, and we've had training workshops.
Even this week we had a workshop on the Kepler, which drew 150 students, faculty,
staff people from the region in terms of industry or other universities.
Nine vendors now serve the region.
When we started, IBM was the only one that could offer on site support.
Now all the major vendors are in the bidding process for our next machine.
Last year we hosted the Saudi HPC User Forum the week after supercomputing.
It had 315 registrants.
That grew from 25 two years before.
So HPC is gaining in visibility among Saudi government agencies and companies
as well as the other universities.
This is our university logo.
It has many interpretations.
One of the interpretations I like to give it is the four pillars of sustainability,
which we were chartered by the Royal Court of Saudi Arabia to work on,
and I always add in computational science as an enabler.
We're not one of the headliners, but we undergird all these different efforts, which are, you know,
a combination of experiment, theory, simulation, and data, and that's the other way that I
like to look at our logo is representing the four ways
of knowing-theory going back millennia to the Greeks.
It began to be corroborated with experiments just centuries ago,
and most of the world's great research universities were founded and evolved
in that mix of, you know, the scientific method of comparing theory
and experiment and reproducibility and so forth.
Then, you know, about 50 years ago simulation came along as a force,
starting really with [inaudible] Solomos in terms of trying to solve practical problems
and growing to, you know, the side act program, the ASCII program, genuine agency decisions made
on the basis of simulations alone in many cases,
and within the last decade we've seen big data emerge,
the volume of digitally stored information doubling about every year,
and KAUST was fortunate enough to be founded downstream of these other two revolutions
in the ways of doing science and was founded with a fairly good balance among those,
whereas many of the institutions with inertia and prestige are still firing
on just two of those four cylinders.
This is not data.
This is just a schema that I like to show in terms of the relative percentage
of scientific effort devoted to, let's say,
experiment and observation versus simulation and prediction.
So back in 1950 when digital computers first came into use for nonlinear problems
like weather prediction and neutron transport, they were more or less a curiosity to try
to explain things that defied theory, you know, nonlinear equations, no, you know,
use of the traditional analytical techniques that lead to closed forum solutions,
and things could be, you know, maybe explained with very low resolution computing.
By the year 2001 we have the bold declaration of the scientific discovery
through advanced computing program of the US Department of Energy saying that "Hey,
we're going to actually do scientific prediction and engineering design.
Resolution is high enough, speeds are high enough, and multiple decades
of natural phenomena can be resolved and integrated for long periods of time.
Computing is here to stay," and then, of course, we were founded downstream of that philosophy.
So at many typical research institutions, the emphasis is on the discovery through experiments
and then understanding by computing, but the future will certainly be
for cost effectiveness among many other reasons, prediction by computer, and then verification
by experiment, and notice the computing margin never goes to the upper right.
There's always a large gap that needs to be closed by very expensive experimentation,
and let's remind ourselves we're talking in the tens of billions of dollars for ITER
in France right now, the world's largest thermonuclear effusion reactor, and, you know,
4 billion for the NIF at Lawrence Livermore and so forth.
So these experiments are extremely expensive.
They have to be undertaken globally.
Simulation, although the facilities are not cheap, are relatively very inexpensive
and of course can be done in a globally distributed way
and should really be done before those expensive experiments are turned on.
So this is my own philosophy.
This is a side act philosophy.
This is what we're trying to reproduce.
KAUST was given three missions by the Royal Court-advanced science and technology,
catalyze economic diversification in a country that's primarily extractive moving
to an information service economy, and connecting Saudi Arabia
to the best practices in research and academic culture.
One could wait a century following the leaders to try to emerge at the other end,
but the impatient ambitious country said, "No, reinvent the university.
Do what you need to.
Do what others find less natural in order to try to find a niche."
And so we have a...
I'll go through a fourfold strategy very briefly here, the first one of which is obvious
to any academic administrator, multidisciplinary.
Everybody tries to do it, gives it a lot of lip service.
It's impeded at a lot of institutions by fiscal and faculty counting territoriality.
We managed to avoid that by not having undergraduate programs and by setting
up research centers that for the large part fall between the disciplines and infusing most
of the university's financial resources
through those topically oriented centers as opposed to through deans.
Another, you know, nice feature is that science and engineering are integrated.
They're not separate schools under separate deans, and the overall, you know, spirit of the campus is
to try to make progress on problems.
It's more really, you could say, of a DUE lab that has, you know, significant curricular add-on.
So these are the founding nine research centers, two of which are in the general area of math
and computer science, the Computational Science Research Center
and the Scientific Visualization Research Center.
The others are related to those four pillars of energy, environment, food,
and water that I mentioned, and it's through these centers that the projects originate
and are funded, and the divisions then staff them up with faculty and students.
Since founding we've created four new strategic initiatives, all of them in the IT
or EE area-solid-state lighting, extreme computing, uncertainty quantification,
and then numerical simulation of porous media.
Our second strategy is big sisters.
We had many founding partners to create the degree programs, to hire the faculty,
to host the students and faculty before our campus was ready and begin initial collaborations,
and you can see the partnering institutions at 9 of the 11 degree programs that they helped there,
and we also started offshore research collaborations
with a number of the world's research elite.
This is just a small sample of them, but these people and their schools got between 10
to 25 million dollars over a 5-year period to work on the topics that we wanted then to import
to campus, you know, desalination, batteries, you know, fault tolerant...
sorry, drought tolerant, salt tolerant agriculture, and so forth, and indeed this group
of initial investigators has been part of the faculty hiring pool
after they got in line with our vision.
A third strategy was industry and entrepreneurship, and you'll see here
about 40 companies that are part of the KAUST industrial collaboration's partnerships.
Some of them you don't recognize their names.
They're Saudi companies, but many are multinationals.
For instance, the second on over on the top, SABIC,
is now the world's second largest materials manufacturer after BASF.
It has recently purchased six other companies including General Electrics Plastics Divisions,
and therefor it inherited 6 research centers around the world,
but it's building its corporate research center on our campus, and this is the kind of thing
that we're trying to inculcate with our start.
Now, you'll notice an absence of IT companies, other than one, in that initial partner.
IT is not very big in terms of a commercial sense in the Middle East at the moment compared
to all these largely petroleum materials and other companies, but interestingly enough
since we started, two government agencies have been created
that build substantially upon the simulation capabilities that are coming to the country.
One of them is KAPSARC, which is a petroleum derivatives institute in the capital.
The Saudi's are used to having London and Zurich and New York set the price of oil.
They want to actually start understanding the strategies involved in global pricing of oil.
They, after all, are the largest marketer, and to a large extent do set the price,
but now want to have in house PhD's working in the financial area.
The founder of that institute is the same one who founded our campus, namely the Saudi Minister
of Petroleum and Minerals, Ali Al-Naimi.
You see him on the front page of the New York Times with some regularity.
The other agency there is the new Department of Energy, the King Abdullah City for Atomic
and Renewable Energy, also in the capital but also in liaison with our university.
These two, as I say, were started downstream of the vision
of having an in-kingdom research enterprise.
Along with the attraction of multinationals and large companies, we aim to start small ones.
We saturate the students with short courses on entrepreneurship.
We have industrial partners on the campus.
We have 45 square kilometers so they can lease and build their own labs.
They can use our core labs.
We have seed funding available to students and faculty up to a quarter of a million dollars
on a competitive basis and expert staffing for intellectual property, which is a challenge
in the Middle East, fund raising, and so forth.
It doesn't have the culture of, let's say, Silicon Valley.
Of the four companies that have actually been launched so far out of many startups
that are still in progress, of the five of them, I mean,
four of them are in the domain of information technology.
That's not a surprise.
It's an easier thing to get started.
You don't need to buy a lot of capital equipment.
They range from, you know, sort of social networking sites to underwater, you know,
data gathering and photography with digital electronics, a entertainment use
of scientific visualization, and a robotic cleaner for solar cells in the desert.
The fourth strategy is shock and awe with respect to facilities.
I list here our seven core labs.
The Marine Science Lab just got a boat, an ocean going vessel,
to help with its own sample collection.
There was about a billion dollars invested in scientific toys for the new faculty.
This is, of course, our best facility.
It was free.
It's the largest undiscovered body of water in the world really.
Very little science has been done on it, rather still pristine ecosystem of reefs and fisheries,
and before it is too drilled, like the Gulf on the other side of the kingdom,
we want to have a very good understanding of its potential contributions to, you know,
food to energy through algae, you know, all kinds of things that come from a rich,
diverse biological ecosystem that has adapted to rather extreme points
in salt and temperature and so forth.
These are our IT facilities, the 6-wall cave called CORNEA and the original supercomputer.
Of course, you can't just turn scientists loose on such sophisticated facilities.
You have to spend a lot of time training them and having in house research support for them.
So each of the two facilities got a core laboratory.
This one was trained by IBM at Yorktown and brought to the campus to help, you know,
the scientists and engineers who are expert at something else get up and running on Shahin.
That was a bit of an introduction to who we are and how we got started in IT, but, you know,
for all of us, what's the real basis, the strong motivation
for pushing a scalable computer agenda in any research university today?
The first is price and capability.
Quite clearly we've been through decades of thousand-fold improvement every decade.
This includes cost as you can see in the left column
of documented Gordon Bell Prizes for price performance.
One of those might have been Thomas Sterling's, who you'll hear from later in the week,
the adventure of the Beowulf cluster and one of the forces
to drive the cost per installed gigaFLOPS down.
The other Gordon Bell Prize, of course, is on total capability where cost is no object.
The government labs sort of end, and there also a factor of a thousand every decade,
not on the LINPACK benchmark, but on real applications.
I also like to do this thought experiment.
When I'm in the US, I usually use peanut butter, but since I'm coming
from Saudi Arabia, I'll use dates.
So it's a delicacy.
At today's prices you would just use it to eat, but what if you knew, you know,
you had the market insider knowledge that in 3 years the price
of dates would drop from $16 a kilogram to $4?
You would go around through the recipe books figuring out how to, you know,
manufacture foodstuffs replacing dates, replacing other sugars and oils with dates.
If your, you know, insider tips continued that in 3 more years you'd have another factor
of four reduction, you would use it as a feed stalk for biopolymers,
plastics, pharmaceuticals, and so forth.
If this trend could continue, you'd figure out how to rejigger furnaces to heat your homes,
pave your roads, etc. What's the point?
The point is that if any commodity is predictably getting cheaper and cheaper at a fast rate,
it displaces others, and that is, of course, the curve that computing has been on for decades,
and everybody knows this in banking and entertainment and commerce
and transportation and telecommunications.
Who doesn't know it is the scientists, really.
I mean we're the conservative ones, but in fact it's a deserved lag because, you know,
computing has lagged its peers in reproducibility, you know, our archiving and so forth,
and the real ability to, you know, take a paper and, you know, apply it in another area.
We are, of course, working very *** that.
That's one of the research frontiers of high end computing
and certainly quantification, reproducibility, and so forth.
Now if you were to take, you know, this progress and map it into other industries,
just think what a miracle it would be.
The 15-hour flight from JFK to Narita would take one-one twentieth of a second if computers...
if airplanes had kept up with computing.
Now we won't mention that the latency of the airport, like IO, would still add 3 hours
to this one twentieth of a second trip, but that's another research frontier, right?
If similar improvements in storage had been realized,
your offices could hold all the text material of the Library of Congress, and if similar reductions
in price had occurred, college at a private university
in the US would be 20 cents today per year.
So computing is amazing.
What should we do with this power?
Well, of course we have been using it for capability.
Nature is multi-scale.
It soaks up all those decades of performance taking their third root or their fourth root in,
you know, three or four spatial or time dimensions
in increasing resolution, and, you know, it's a real...
it's a real thirst to resolve multi-scale nature.
If we can resolve many such systems, we can combine them, for instance, ocean atmosphere,
terrestrial gas exchange, ice modeling, and build complex systems without making all the assumptions
of fluxes and body forcing by putting things together, and then if we really have a lot
of computing, we can wrap loops around these to do the real work of science to test hypotheses
to vary the inputs to find the sensitivities to optimize the systems.
This is the vision, and for many years that vision has been reflected in the upper oval
on this chart, raising the peak of capability, but I think today one
of the main challenges is lowering the threshold of access,
and if the upper ones are the grand challenge of computational engineering, computational physics,
computational chemistry, and so forth, the lower one is the grand challenge of computer science
to make all this power available at high level abstractions in flexible ways to a community
of people that are really expert in something else, and we're interested in both
of these aspects at KAUST, as, of course you...
and picking up on that last point, we could really characterize the modern aspect of research
and computational science and engineering as heavily focused
in scientific software and engineering.
We've been mathematizing nature for hundreds of years, making numerical methods, you know,
non-sensitive to errors, stable, accurate for decades through numerical analysis,
pushing performance again for decades in computer architecture sense and now spending a lot
of attention going back really to the founding of [inaudible] in 1992 with the standards for MPI
and PETSI and many of these other workforce libraries
to bring this power to the scientific masses.
This is the diagram that launched the SIDAC initiative back in 2000 by Tom Dunning,
the retiring Director of the NCSA.
Above this, you know, flowchart we could really put two loops, the validation
and verification loop, which is primarily the work of the scientists and the mathematicians
to perfect the representation of physical nature in the digital world,
and then the performance loop, the often neglected loop that takes just as many years executed
by the mathematicians and the computer scientists to turn the computer
into a bona fide scientific instrument with predictive power, and indeed,
you could think of the computer as a universal scientific instrument.
Our campus has 3 of these 4 facilities in abundance mass specs.
We have about 10 electron microscopes.
We have dozens of DNA sequencers both on land and at sea.
We don't have our own synchrotron.
We go to Cornell or Europe for that, but the supercomputer is used by people
in all of these disciplines, right?
It's a facility that boosts all boats, and this was, of course, recognized in the quotation
of James Langer the year before the SIDAC program first made it through Congress.
The computer is literally providing a new window through which we can observe the natural world
in exquisite detail, and this has been an inspiration to these past 13 or so years of SIDAC.
Well, I come from the Middle East.
How would such scalable power be applied at an oil company?
Well first of all is the most obvious case is better resolution of, say,
a reservoir in terms of length or time scales.
Accommodate physical effects with greater fidelity.
Oil is not some black, homogeneous mob.
It has many different constituents, different carbon length chains.
It has water.
It has gas, and to put in all the compositional effects, soaks up another dimension,
if you will, of representation space.
Then, of course, we want to occupy all of the physical dimensions,
not rely on axisymmetry or slab symmetry or steady state.
Better isolate artificial boundary conditions, you know, with many cycles of propagation, you know,
from unknown data in that doesn't affect the phenomena under study.
Combine multiple complex models.
Solve inverse problems.
Don't just run the forward model, but use the output of wells and data
to improve those models and predictions.
Of course, do optimization or control.
You have many ways of exploiting a reservoir, many ways of injecting water or CO2.
What is the one that will produce the most over time?
Quantify uncertainty with many runs to use up the parameter space of unknown inputs
and improve statistical estimates.
All of these are things that you can do with a model,
and we have to apply them to extreme situations.
These are web source, nothing confidential here, maps of the world's largest oil reservoir.
You can see it over there.
It's about the...
it's about 150 miles long, and it's one continuous pressure reservoir called Ghawar.
It has thousands of injection wells and thousands of production wells.
It's been in action since the 1950's producing 5 million barrels per day for the world.
That's about half of Saudi's output from that one reservoir.
This is a tiny corner of Ghawar as modeled by the former megapowers code
and the new gigapowers code, and what you can see from this real field data, these are production...
these are injection wells, these are production wells...
are that the simulator picked up that there would be 2 pockets of oil left behind, and, indeed,
horizontal wells were drilled and found those pockets,
and that pays for a lot of computers and staff.
But the exciting thing in computing today is not just simulation, the third paradigm we could say,
but the combination of the third and the fourth.
Instead of having inverse problems where you output some prediction,
if you actually have some data to put in, then you can take care of some of the uncertainties
in your model, some of the subterranean properties for example in the case of...
Now this is a non-trivial mathematical enterprise.
It's a good interface between math and CS because it's not a well posed problem,
but we have a lot of different areas of third and fourth paradigm combination going
in our research centers, particularly [inaudible], formerly of UC San Diego and scripts,
working in data assimilation and many aspects of geophysical flows.
This is a composite of 9 of our largest supercomputing projects at KAUST ranging
from molecular dynamics, quantum chemistry, ocean simulation, seismic conversion for discovery,
seismic conversion for earthquakes, sheer flows for efficient combustion, bioinformatics,
magneto hydrodynamics, and global climate.
The SIDAC philosophy is many applications drive.
They're built on a common base of meshes and intelligent agents and particles
that can be supported by a common base of distributed memory, math, and CS algorithms,
and one of our scientists who came over from Rutgers originally was a member
of the large 2007 Nobel Prize team on the international panel on climate change.
His specialty is aerosols, which is very important to the Middle East because we're going to invest
in all these solar collectors and they get covered with dust and airborne sand on a regular basis,
that not only effects their maintenance, but also their, you know, their daily productivity
in terms of diluting the atmospheric flux.
By the way, a quarter of Saudi Arabia's land area could supply all of Europe's electrical needs
if we had high temperature superconductivity or better storage.
So it's definitely an emerging area.
So what are our plans for extreme computing?
And I'll go a little bit fast here to save more time for your questions, but as Craig knows,
for many years, for about a decade, I've been working with NSF, with Department of Energy,
with NASA to basically try to establish some good directions
for computational science and engineering in the US.
These are all, you know, collaboratively edited reports in a variety of areas
to which we've contributed, and if you're interested in these, you know,
I would say mostly unimplemented reports, you know done with best intentions
on almost an annual basis for the agencies and then gathering dust, then of course you're welcome
to ask me where to find them, but what next?
I wanted really a platform to start implementing some of this.
So this is the vision for extreme computing as we call this new strategic initiative at KAUST.
Of course it has to...
it really should wave the flag in all three of these legs of simulation, data, and architecture,
although my own expertise is limited to, you know, small pieces of this, you know,
fork on the left hand side, and shown here are different sort of methods that one uses
and most notably assimilation and inversion are a combination of the third and fourth paradigms,
and then underneath are some of the applications to which we put these tools at KAUST,
and bottom row is some of the agencies and companies that are our prime customers.
So this is the target.
You know, it's all work in progress, but the twin deliverables are the enabling
and the actual discovery.
So the discovery on top, develop and demonstrate algorithms for scientific simulation
and data analytics and emerging scales.
We expect axiscale somewhere between 2019, 2022.
Today's software libraries, as Craig alluded to at the beginning,
are designed for something very different.
We can no longer ride Moore's Law.
We can no longer replicate billions of cores and assume they're going
to be synchronous or performance reliable.
We have to move to new programming models to support these apps,
and the second aspect is, of course, implementing them.
Even today at the petascale, most users don't take full advantage of their scalability or their
on ship performance, and, of course, this is the legitimate, you know, they have an excuse.
This is complimentary to their own training and requires this multidisciplinary collaboration
that your community is so good at.
Within the first deliverable there are really I think four thrusts
for architecture adaptive algorithms that at least we want to address,
the most important one being the new rule of, you know, no more weak scaling with extra memory.
You have to do strong scaling within a fixed memory with multi core or accelerators.
This is a challenge for [inaudible] algebra or PDE based codes because it looks much better
for certain things like fast multipole
and interval formulations in terms of the ability to scale.
With respect to arithmetic intensity as well,
these algorithms allow many more FLOPS per byte moved,
which bytes moving is the energy premium today.
We need to build fault tolerance into the algorithms themselves.
We can't rely on energy expensive alternatives of the hardware or system software doing it for us.
We have to partition today's codes into small, safe regions that we expect to be executed
as reliably as all computing today and then some fast, large unsafe regions
where we can take all kinds of shortcuts and know that errors will be caught.
Many of our favorite iterative methods in applied mathematics actually can fall very well
into this category because, you know, they're computer residual
and the residual derives the new correction, and if some errors are made
in that expensive correction, they will be caught automatically, and yet we don't exploit
with this ability to do the vast majority
of our coding less energy expensively and reliably than now.
We also, of course, have to reduce reliance on synchronization.
All the good algorithms in terms of multi grid and [inaudible] and so forth rely heavily
on frequent synchronization, which will simply not be realistic
with performance unreliable nodes and billions of cores.
So there are many implications of emerging architecture on algorithms.
Weak scaling could keep going another thousand.
Strong scaling is the real challenge.
So you can do the real hard work of exascale with your local desktop and your accelerator.
That work, when multiplied by a million, is the exascale system,
and I think I'll skip my little 4A into why sparse grids...
sparse matrices and [inaudible] grids are hard.
I'll just note that it's my life's work gone down the drain.
I mean, I've been working on Newton methods and [inaudible] methods
and domain decomposed conditioner methods for 20 years, and I see that they are not going
to make it to the exascale in the current manner.
So we're looking where we can make incremental improvements
and even revolutionary changes in programming models.
Here's an incremental improvement.
The irregular curve here is algebraic multi grid in terms or preconditioning [inaudible],
and the very steady blue curve, which begins to overtake it, is fast multipole method for solving
that exact same [inaudible] kernel.
It can be a preconditioner for something that's variable coefficient,
but it's something that has extremely good arithmetic intensity.
It's much more amenable to asynchronous implementation.
It's very arithmetically intensive.
It's the ideal, you know, looking kernel for the new architectures,
and these are some of the hardware improvements that can be made to the algorithm.
This is a comparison of 5 publicly available fast multipole codes and then one
of my collaborators, Rio Yokota, is actually the fastest one here at the bottom.
Another area of work is to move from regular loops to data flow and to direct an acyclic evaluation,
sort of a retreat, you know, for 3 decades to the way, you know,
to the frontiers of computer science, but it's coming back because of the need
to be more concurrent and less synchronous.
So these for a tiny little matrix doing a generalized [inaudible] problem,
you can see these 3 chains of 3 of the 4 arithmetically expensive steps.
Right now we would do them in sequence between different subroutine calls,
and the width is the concurrency, and the height is the time, but they can be combined
as in this figure on the left if we could get all the data dependencies expressed
in a proper data flow way and then when mapped onto GPU's using plasma and magma as shown
on the right, literally orders of magnitude of improvement are available, and these are...
the top one shows some work on dense BLAS, and the bottom one on some new sparse iterative methods.
On the top you see the original CUBLAS.
That's here, the NVIDIA product, and then you see the cable as the KAUST BLAS on top as we continue
to evolve linear algebra for GPU's.
Here it's more or less even draw between the sparse BLAS
of NVIDIA and our own research projects.
There's still a lot of work to be done there on the sparse case.
Here's an example of work stealing within NUMA.
So you don't want to steal globally.
That's too energy expensive, but you want to be able to fill up starved threads
with extra work within the local memory.
Here's some work by one of my Saudi women grad students in association
with the Barcelona Supercomputing Center.
We also are very interested in multi physics apps where the PDE
and the molecular dynamics aren't cohosted in a very natural way,
and you can see here the PDE solution scales very well strongly
on Shahin whereas the [inaudible] solution is tailing off,
and these provide all kinds of challenges.
So I think the programming models will at first be message passing because of legacy,
but must be stretched to MPI3 into less synchrony.
Load balance blocks, which are scheduled today with loops, should be separated into critical
and noncritical parts, the critical with...
scheduled with DAGS and the noncritical made available for work stealing.
I have some ideas of how that applies to my own methods
that I'll skip here and just end with some challenge.
This is a list, maybe not complete, of the sort of software that's needed to do scalable computing.
Some of it's related to building the model, the code, the mesh, partitioning, you know,
quantifying uncertainty, visualizing, and so on.
Some of it is related to developing the code, you know,
the performance monitoring, the debugging and so forth.
Some of it is related to scheduling production runs and controlling the workflow
and archiving the data and understanding.
You know, all of it needs to be ported to the exascale, most of it non-trivially.
I have this tiny little niche that I'm interested in.
I hope I can go around the room and point the laser and the rest of you are working
on all those other bits of the environment.
So we are at a point of a baton pass, as Craig mentioned in his introduction, from the generation
that rode Moore's Law and could conveniently work in the bulk synchronous programming environment,
the SPMD environment that we know and love, to an energy-aware generation that will really have
to put up with asynchrony, non-reliability of components,
and just the huge cost of moving data relative to FLOPS.
This is the team that I have working with me now.
The top two rows are students.
Not all of these Saudi ladies like to have their pictures videoed
in Indiana, but this is a great team.
It comes from several countries-Egypt, Lebanon, Mexico, China, Jordan, Saudi Arabia itself,
Palestine, Pakistan, and this is my team of research scientists and affiliates
in the KAUST supercomputing lab and some of our alumni.
Maybe you can join us in this quest.
We certainly will be in this quest globally together, but this quest I like to characterize
with the same words with which Kennedy launched the Apollo space program.
Seven years before humans walked on the moon or about seven years maybe before exascale,
facing a similar, you know, amount of uncertainty, Kennedy said, "We choose to do these things not
because they are easy but because they are hard..."
that's why to do them because they are hard...
"Because that goal will serve to organize and measure the best of our energies and skills
because that challenge is one that we are willing to accept,
we are unwilling to postpone, and that we intend to win."
So let's go at it with this conference.
Thank you very much.
[ Applause ]
And I get to put in a pitch.
Your office here...
we're still hiring faculty, recruiting students, recruiting post docs, and so forth.
So, be happy to talk to you about that today.
>> Hi. I'm wondering what does it take to create a university from scratch and make it number 3
in the world in terms of competition and power?
What does it take to do that?
>> David: Okay, so first of all, it was for a brief time number 3 among the world's countries,
but it was number 14, but you can always buy a facility, right?
That only takes money.
We all know that that's hard to come up with, but that's the easy part.
The hard part is getting the facility to perform at scale on real science,
and that's a challenge that we all share.
There's nothing unique to doing that in the Middle East.
In fact there's nothing more portable in the world than a Linux cycle,
and that's one of our challenges actually because you can attract top scientists
with an electron microscope, but if you build a supercomputer, they say, "Well,
you know, I can log into Stampede.
I can log into NCSA.
Why do I need to come to the Middle East to use, you know, supercomputing cycles?"
The challenge of starting a university, while I could give another talk
on that, I, of course, didn't start it.
I was just, you know, adopted into it.
I went originally on a sabbatical from Columbia.
I spent a second year, and then I decided A, I couldn't leave because the opportunities in terms
of the effort I had to make to raise resources versus use them was small compared to US, NSF/DOE,
and so forth and B, I felt that the institution was still fragile in the sense that if too many
of the founders left, you know, it would look like the experiment wasn't succeeding,
and I firmly believed that it was succeeding and will succeed and decided,
well, you know, you only live once.
This is a chance to create history.
What if the door were to close?
What if the door were to close because the university didn't succeed as it should?
So there's, you know, a real challenge, as you say, to create a university from scratch,
but there's a lot of, you know, good models, good big sisters, you could say,
to land both credibility and initial support.
We were born with age.
You know, we weren't born of 80.
We were born, you know, with many collaborators, many research projects already going.
Faculty could, of course, bring some of their students
that they had just started at the schools they came from.
The hardest part about starting a university from scratch is the culture of the graduate students.
At first they think they're fifth year undergrads.
They think they can go scuba diving on weekends and, you know, stuff like that,
whereas they should be in the lab, as we know, 7 days a week, and, you know,
they don't have older grad students to tell them to do that.
So it actually takes a few years before the graduate student culture becomes
like the ones at, you know, at Cal Tech and MIT.
There are many things that are surprising on the organizational side.
For instance, I thought it would be very natural to have a US type system of qualifying exams.
My European colleagues said, "What's qualifying exams?
Get the students into research as soon as possible.
What are all these extra courses, you know, that you think they should take?"
So there are, I mean, and even though I've worked with many Europeans, you know,
throughout my career, I've always done it in the US, you know, so I, you know,
everybody assumed their own system would be the one that we reproduced.
So it was a big salad bowl as opposed to a soup when we got started,
and there are many such surprising issues.
One was, of course, we were created by an oil company, the Saudi Aramco Company.
We at first had their procurement, their HR, all their processes we inherited.
They were not used to ordering math books.
They were used to ordering valves, you know, so there were many things that were challenging in,
you know, establishing our own, you know, academic culture.
The country of Saudi Arabia didn't have the concept of a post doc
in its visa or immigration framework.
So originally we called them faculty, and then we taught the country, hey,
this is going to be something you're going to be working with for a long time.
Let's get these, you know,
these university-ranking distinctions clear in the paperwork.
You know, there's all kinds of issues that are...
you don't think about, but, you know, the ability to answer a lot of questions by default
like if you don't know what should be in the library, just buy everything.
You know, that's an easy one.
[Laughter]
>> It's impressive that your graduate program has one-third women.
In Saudi Arabia are there any issues about coed classes or something like that?
>> David: Certainly we're the only coed university in Saudi Arabia.
That was part of the founding document.
That was part of the king's intention.
Incidentally, may I remind you that in 1880, Stanford was founded with a very similar mandate.
At that time all the universities that had tried to emulate Harvard, Yale, Princeton, Cornell,
they were all male, and Stanford said, "By golly, we're going to create a non-elite, you know,
coeducational university that's purposefully built to, you know, provide on the west coast all
of those great things that they have on the east coast, but do it our way."
And KAUST is very much, you know, the king's own dream.
It's not part of the Ministry of Higher Education, which has its own curricular and funding rules,
which would interfere with the, you know, practices that we wanted to emulate in terms
of non-discrimination, merit-based promotion, intellectual freedom.
All these sort of real charter pillars of western education,
they are part of our founding documents.
We live on a 45 square kilometer campus, which is big enough to create a mini-international city.
We can go out very freely, of course.
We love to mix with, you know, the local society, but we have,
you know, very strict entry requirements.
For obvious reasons we might be offensive to some
of the native cultures, and we might even be a target.
So it's a bit of a bubble, but it's one way.
I mean, we can go out as much as we...
as much as we feel.
Now in terms of what those women will do with their degrees, of course,
many of them teach in women's-only universities, and certainly as the number of universities
in Saudi Arabia goes from about 25 a few years ago to 300,
there's a real need to soak up a professoriate.
Unfortunately the people we are training are not ideally suited for those jobs
because those jobs are heavily teaching load intensive.
They don't really support the research culture.
I mean we actually believe and certainly plan to train students at the highest international level,
and they should really go to other research universities, and so until Saudi Arabia, you know,
evolves more of those, probably a good percentage of the graduates will go abroad.
Two-thirds of them come from abroad anyway.
So, you know, they will, you know, in high numbers return.
What's surprising to me is how many decide, "Hey this country actually has better opportunities
than mine right now," and many of them,
especially the Latin Americans have just, you know, stayed almost 100%.
So, but in terms of opportunities for women, they are definitely limited in today's Saudi society.
More and more are being created.
It's amazing to watch cultural change over a 4-year window but no one would pretend
that it's a country where women have equal access to opportunity.
My feeling is you have to change that from the inside by creating, you know,
the obvious talent pool that then can be of benefit to the country.
So it's encouraging that, you know, so many of them can stay within the country and pursue a PhD.
I might point out that right now in the US, there are over 100,000 Saudi students coming on...
almost all of the on full scholarships, probably supporting tuition at Indiana as much
as any other place, and they can get exactly the same fellowship to come
to Indiana as they can come to get to KAUST.
So why go to KAUST, you know, if you can come work in, you know, this kind of an environment,
but we do manage to keep, for that reason, a larger number of the women Saudi students
and provide them a Stanford-like opportunity without, you know, having to uproot.
>> Thank you.
>> David: Thank you.
>> So let's thank our speaker one more time.
[Applause]