Tip:
Highlight text to annotate it
X
Welcome to the 8th screencast on Grid computing brought to you by the Direct User Support
team. During this screencast we will discuss the gfal API. GFAL is a C API for performing
input and output operations on grid files directly.
GFAL functions have similar names with their corresponding I/O functions used on local
file systems except for being prepended with the prefix "gfal_".
GFAL accepts Logical File Names, GUIDs and SURLs as filename arguments on top of local
filenames which may also be used.
Before moving on to the examples we should note that several environment variables need
to be set in the first place. These are: the LCG_GFAL_INFOSYS variable which should point
to a BDII server the LCG_CATALOG_TYPE variable which in our case is LFC the LCG_GFAL_VO variable
which should be the name of our Virtual Organization (in our case this is the hgdemo VO) and optionally
the LFC_HOST variable which should point to the Logical File Catalog service.
We now turn to a few simple examples and login the User Interface using putty. We then export
the appropriate environmental variables LCG_GFAL_INFOSYS, LCG_CATALOG_TYPE, LCG_GFAL_VO and LFC_HOST
s
discussed earlier. On a per user basis these should be set accordingly.
Once we are done setting the necessary environment variables we enter the directory write_to_file
which contains our first gfal source code example. This is written in C++ and as you
can see in the top section of the source code the gfal header file is included.
As can be seen the first argument to our executable will be the filename that will be opened and
to which our data (in this case a line buffer) will be written to.
We then open our Grid file with gfal_open and set its permissions. Notice the usage
of the O_CREAT flag that instructs our executable to create the file if it does not yet exist.
In the next section we define our line buffer which is simply Hello from gfal API and then
write that buffer to our Grid file with gfal_write.
Finally we close the file with gfal_close before exiting.
To compile our source code we will use the g++ compiler. Notice however that as the header
files and the gfal library are not in the systems INCLUDE PATH and LIBRARY PATH I have
to explicitly define the include and library directories for gfal.
Once the executable has been built I may execute it with any argument pointing to the local
file system. Thus for instance I use the argument some_file and as you can see a new file has
been created on the local file system with the name some_file. This file contains the
buffer string that is defined within the source code write.cpp.
I now erase the file and use a logical file name as an argument for my executable. As
you can see creating and writing the file now takes a little bit longer as the new file
is not local but rather resides on the Grid.
With the lfc-ls command I see that the file has indeed been created. If I would like to
view its contents one alternative would be to download it with the lcg-cp command locally.
As a first argument to the lcg-cp command I provide the logical file name of my Grid
file and as a second argument I define the filename under which my file will be stored
on the local file system. This is yet_another_file. Once the transfer has been completed I may
view the files contents.
Another alternative however is using once again gfal this time in order to view its
contents.
To demonstrate this I enter the read_from_file directory and open our second source code
example read.cpp. Notice that we now open the file with the READ ONLY flag in the gfal_open
function call. We next write its contents to a character buffer using the gfal_read
function. Once this has been performed we close the file with gfal_close and exit.
Once again to compile the source code we have to explicitly provide the Include and Library
directories for the header and library files. Once compilation has been performed we may
directly fire up our executable and provide as an argument the logical file name we previously
wrote to using gfal. As you can see the contents of the Grid file are written in the last line
of the output.
As a final example and in order to demonstrate that similar operations may be performed directly
from Grid Worker Nodes we have prepared a small DAG job consisted of two normal jobs
that are to be executed consequently. In the first normal job we use the gfal_write executable
to write a new file on the Grid and in the second job we use the gfal_read executable
to read that file from the Grid.
Notice that in the two executable shell scripts (one per job) we also take care of exporting
the necessary environment variables prior to executing the two executables.
To submit the job we use the glite-wms-job-submit command as usual.
Once the status has reached the Done Success state we may retrieve the output using the
glite-wms-job-output command.
As you can see the ids_nodes.map has been saved together with my output from the WMS.
I now browse the output files of the write job and as you can see the error file is empty
and the output file contains the output of my executable. In the read directory which
contains the output of the second job once again the error file is empty and the output
file contains in the last line the string that was read directly from the Grid file
created by the first job originally. I may also check directly that the file indeed exists
simply with an lfc-ls command.
This concludes our screencast of the gfal API. Join us in our next screencast where
we will be discussing The job perusal mechanism used in order to get real time output files
directly from a Grid job. Until then on behalf of the Direct user Support team thank you
for watching.