Tip:
Highlight text to annotate it
X
Okay, this new set of slides discuss monitoring
on FutureGrid, and we've already stressed that
if you have a testbed you'd better have great
monitoring and great metrics, cuz there're gonna
be a lot of users whose goal is to do monitoring,
cuz they're doing testing, they're reporting performance
and possibly reporting robustness and reliability
and things like that. So here's our composite
of the four major monitoring environments,
Inca, Ganglia, SNAPP and perfSONAR
doing software functionality, cluster monitoring,
network and coupla types of measurements,
lperf and SNMP, and those are all done on
FutureGrid and reported on the FutureGrid websites.
[pause]
So we think... we've already stressed that a
feature of FutureGrid is you know what you're doing,
and given you should know what you're doing
you'd better be able to monitor it
and report what you're doing.
[pause]
Here we have Nimbus and Ganglia providing
information, which is... you can use to see
what happens to the load on your nodes
and things like that. And what is the status of
the actual images that you have when you're...
in your environment, because you can have
an image and it may not actually be running,
so these are distinct monitoring capabilities.
[pause]
We have developed at Texas a messaging
environment which allows us to accumulate and
put into a database... all this performance
information and then we have a dashboard,
a user interface, that allows you to access it.
It happens to use CouchDB, which is a well-known
NoSQL object store. The [unknown] uses a very
well-known excellent messaging system,
RabbitMQ, which uses the standard AMQP...
format to... do the messaging.
[pause]
So here's some notes about some of the work at...
Tennessee on PAPI, performance measurement.
And one of the troubles is these virtual
machines do not come with the same...
although you've virtualized most things,
you haven't always virtualized the performance counters.
And so it's not so trivial to take standard
performance tools, which are what PAPI is...
supports and use them in virtual machines.
And this is stressed here, the VMs...
may not expose to hardware counters.
That's gradually getting better. All of this is
partly the immaturity of the environments.
[pause]
Another important issue we see, of course,
we've already mentioned with network cards...
[pause]
virtualization is done using software to emulate
the hardware. That software can have significant
overheads, so you need to carefully...
take that into account in your...
[pause]
on your environment. And these overheads
are particularly important in things like
cache events and TLB events.
[pause]
So there're various ways of doing timekeeping.
Tickless timekeeping with no interrupt is the best.
And PAPI has two types of timing routines,
wallclock time and virtual time. And the virtual time
is affected by the time when a virtual machine
is descheduled on a busy environment.
And PAPI has already implemented the ability
to correct for this descheduling on KVM...
which is important for loaded virtual machines.
That's illustrated on this slide here, the effect of
the steal time. Here we have non-virtualized
environment. We have the real wallclock time,
and here we get the time... these are matrix multipliers,
so it should be a fixed time, so this is
the time for the matrix multiply.
As a function of the number of other things
running on your system, and here we do not...
here we have a stealing event and that
stealing effect here where the... what you naively
report is actually just the... time to run a matrix multiplier,
not the virtual time, which should run along here.
That is because the ceiling is not properly
corrected for. So that needs to be put into
environments like PAPI with the current
virtual machine systems we have.
[pause]
Here we have the last section of this...
MOOC on FutureGrid appliances.
Appliances are very, very important cuz you
could say they are the heart of cloud technology.
Well, as far as the user's concerned, cuz appliances
mean an image, with everything you possibly want.
Your OS, the user code, your virtual network,
multiple images, that's what appliances are, they're everything.
[pause]
They're particularly important for education and
training, because appliances can be used to
[pause]
to capture a piece of software, and that same
appliance can be run on your Virtual Box
on your client, or on FutureGrid.
And we have developed and may develop,
all for people to test, Condor, MPI, Hadoop,
and iterative MapReduce appliances.
[pause]
The general idea is a system like FutureGrid,
is a natural platform for programming labs.
Let's say you want to have these appliances
so people can just use them. If you have a lab
involving Hadoop then the Hadoop appliance
is what you want to use to execute.
And we also want to accumulate appliances
from lots of different people.
[pause]
And as we discussed, these appliances are
actually deployable not just on FutureGrid, but in...
as we sort of discussed with CloudMesh,
they can be deployed on other platforms.
Currently we actually make the appliances
separately for each platform. In the future
we can probably use CloudMesh to make that...
deployment of appliances across different
IaaS approaches in a more seamless fashion.
We have grid appliances on FutureGrid.
It captures MPI, Hadoop or Condor.
We have the same image for each node,
we have the ViNE virtual network to form a cluster,
and you can deploy this any way you like.
You can deploy it on a distributed environment,
on a localized environment and so on.
Put it on your client. Cuz this is... everything is
virtualized. You get your virtual grid you can
put where you want, which is probably not
terribly useful for production work, but it is
highly useful for education. That's why we stressed
education here, with these integrated appliances,
which have everything you need to build an environment.
Here's the last slide of this particular MOOC
on grid appliances, and it just illustrates with
Hadoop and a virtual network how this works.
You have your Hadoop workers which you
instantiate. These are your virtual machines
which you just copy your appliance,
appliance is just downloaded from the library.
Then you need to get some credentials where
there's some technology, virtual private network
technology, to... describe a group of,
say, of students who can use this cluster.
[pause]
Then you get these virtual IP addresses... and that's it.
[pause]
So that's... that shows the components,
the virtual network, the virtual... the credentials,
and the... customized pieces of software
which give you your... in this case, your Hadoop appliance.
Thank you very much, this is the end of my
detailed discussion of FutureGrid... which...
is available as an expansion on the Overview,
which I... is also recorded and available.
Thank you very much, this is Geoffrey Fox
signing off for the FutureGrid team. Thank you.