Tip:
Highlight text to annotate it
X
Hello everyone! In the previous videos we made sure you had IRAF installed and ready to go,
so now we can start with the actual spectra reduction
First of all I want to thank professors Elena and Roberto Terlevich, as well as, Professor Angeles Díaz,
for sponsoring these videos by allowing me to use their observational data
The spectra I am going to use are high quality recordings from the WHT
To complement the things I am going to show you in these videos I advice you to take a good look at the links within this video description
Indeed, the task configurations configurations I am going to use come from the third link
Finally, for those of you who unlike me do actually speak english, you will still find subtitles for these videos...
but now they must be manually activated within the Closed Captions menu of the YouTube video player
I have done it this way so in the future I can add additional languages
Ok that's all you needed to see me for...
Thank you very much for all the comments, subscriptions and likes and let's get started!
So what are we doing today:
In this diagram, which by the way you can find within this video description,
we have a simplified diagram of the observational log from the data I am going to use in these tutorials...
Each of these boxes represents the different sets of fits produced by the telescope recordings.
We have a Flat fields, sky flats, bias and moreover, for each scientific recording we have measured the flux from a calibration star
and "arc" files to apply the corresponding wavelength calibration
Finally, in this observation each object has its own Flatfield recording.
On the right, we have a summary of the steps necessary to perform the Bias, biassec, trimsec, flat field, and illumination correction...
Let's go over them: First First of all, We need to combine all the bias files in order to generate a global bias fits file
With this file we apply the bias correction to all our files. Actually we shall apply the biassec and trimsec correction at the same time with the
"ccdproc" task
The third step consists in generating a global average flat fits file from all the flatfield sets.
This global flat we need to normalize it
The following step consists in producing a global average sky flat fits file.
This file needs a flat field correction which we can do with our global normalized flat, we produced before
We can continue then by generating an illlumination flat with the illumination task
Now since in this particular observation we have recorded flat fields before each scientific data set, we are going to take advantage of this
opportunity, to produce a local flatfield for each scientific object
We use flatcombine to produce these local flats
And then we normalize them with the response task and the global flat field.
The final step will consist in using the local normalized flatfield and the global illumination frame to correct the scientific files.
Ok, before taking each of these steps, however, since I am assuming this is the first time you use these tasks,
we are going to check their configuration
Indeed, before reducing a set of data the first thing you should do is make sure your tasks are well properly configured for your work
For these initial tasks I am using the set up described in the manual of Philip Massey:"A User's Guite to CCD reductions with IRAF"
This is a good start for any data type, but in any case you should stick to your corresponding lab experience or preferences
The tasks we are going to use in this tutorial are: zerombine, ccdproc, imcombine, flatcombine, response and Illumination.
Now, here I have an IRAF terminal and a Ubuntu terminal to run DS9
This is the folder where I have my HII galaxies spectra.
In the form of fit files. Moreover, you can see several text files. These files are lists of my spectra organized by the frame type
For example, this file file over here is the list of bias frames for this observation...
At the end of this video I shall give you some tips on how to generate these lists. But for the time being you may wonder
why I have included this 1 between squared brackets
This has to do with my data format, it may not apply to so I shall be brief.
Fits files consist in a series of images encapsulated with in a single file with one header.
In my case, every single file has only one image and one header.
However, IRAF, does not know that, so I need, for the time being, to specify which frame I want it to use
Since I only have one frame and counter starts at "1" I need to type 1 between squared brackets.
You will find that lists like, this one, are effective to apply a single task to many files. For example.
Before going into Step 1, lets check all these frames, using this list, are uniform enough...
For that we are going to use the image statistics task:
In this command there is only one parameter
The "@" symbol tells Iraf that the input is in the form of a txt file with the corresponding image names
These images must be in the same folder where we are executing the command...
This is going very slow for me, but do not worry for you it will be much faster
Here we can see for every image the number pixels, the mean, the standard deviation in the pixel value,
the minimum and maximun values.
From this table you can decide which bias deviate too much from the rest...
For example, the first file has higher mean that the rest...
Even worse than that, it has a lower number of pixels which means the image has a different size altogether...
On the contrary, the third image has a much smaller mean..
On the contrary, the third image has a much smaller mean..
for some reason...
since I have many frames I can exclude the first three images from analysis continue working...
let's do that!
We open again our list of bias frames and... and I delete the first three images from it... and we save the it
Now I can proceed with the first step: We are going to combine all the bias frames to produce a global bias file.
Now I can proceed with the first step: We are going to combine all the bias frames to produce a global bias file.
You might remember that for changing a task's flags, I can use the epar command with the name of given task.
in this case zerocombine
The Zerocombine task is used to average you bias files. In case you are wondering what were these bias frames I have been talking about...
these images are used to account for the telescopes' camera "zero" level.
In this analysis, we do not need dark frames since in these professional telescopes the cooling reduces any dark current
In this analysis, we do not need dark frames since in these professional telescopes the cooling reduces any dark current
Here, we only need to modify some values :
The input and output names... better to set those with the command line
Here we have the combine flag with sets the combining operation... You can opt between average and median... In my book it says median
Reject stands for the rejection operation, in my guide it recommends crreject:
This method rejects only positive pixels according noise data from the CCD.
Ahh the ccdtype flag... let me try to explain it... If you put valid type, such as bias, or flat or dark... IRAF tries to read this ccdtype on the header
and only applies the current operations to the images belonging to these types.
Myself however, I do not have a valid ccdtype description on my headers so I need to leave it blank.
That should be all, to leave we type.... anywhere ":q"
We are now ready to apply this correction by running this command:
In this line, we see several flags, the first one we already know it, it represents input list of bias frames.
Here we have the "out" flag telling IRAF the name of the output file while "combine" decides the averaging operation.
We didn't really need to include these last two flags in our command line since we already defined them as default values
Indeed, you will notice I like really long command lines in IRAF
However, I think it is easer to use this approach to play with the tasks configuration...
but you should try the most natural way for you.
We run it...
and we wait
Let's take a look at this image with DS9
We go to our ubuntu terminal and we type :
It looks good enough...
As we did before... we use the imstat task on this file... to confirm we get similar values as in the individual frames...
Ok, it seems fine
Now Step 2: we need to remove from all our frames, scientific, flat, sky... all of them, this base-line signal from the CCD
which is accounted by the bias. This can be done with the ccdproc task.
Let's take a look at it...
As you can see this task has really many parameters. This is because it can be used to apply all these initial corrections.
Again it is safer to run it using command lines...
First, we set ccdtype to none. In case I did not say how before we can use the "space" key for that...
If I did not say it before, you can do that by typing space and enter
And now let's set "no" in all corrections as default
And now let's set "no" in all corrections as default
This is important because there are many tasks in IRAF which make use of the ccdproc task in the background
So they will try to use this default configuration and ask you for files not needed
This is important as well... here we need to set the orientation of the read out axis in my case as column.
This is because my spectra as you have seen, are oriented vertically
Finally, we need to decide a fitting function... In my guide it is advised to set it to legendre... but you can change it for the most suitable for you.
We leave...
Now we are ready to apply the bias correction to our files.
However, in order to decrease the number the of steps, I want to define the bias section and image section, or trimsec, simultaneously
These two corrections can be applied at the same time we remove the bias.
By the way, you might be asking what are these biassec and trimesec corrections ?
Well, in a few words:
Most current CCDs have one or even more regions not exposed to the incoming light.
These regions, in the CCD, measure a base signal,which should approach zero
These regions are known as overscan. This is somehow equivalent to a bias measurement performed with each observation.
On the other hand the trimsec correction, is used to defined where the scientific data is located in our CCD.
By default the trimsec is the CCD size minus the overscan region
By default most telescopes write down the biassec and trimsec regions on the fits headers.
Let's check this!
For that we are going to use the image header task
You can use this command line:
where the longheader flag has been activated to show the complete header
let's run it...
You see? I forgot to type the frame number and I got an error... let's try again...
Now it works... let's scroll up a bit...
Here you have them...
These are the default biassec and trimsec values for my images.
I should explain that in IRAF square sections are defined this way:
The forme rcouple of values represent the first and last column pixel coordinate
while the latter pair of values define the first and last row pixel coordinate.
In most cases these sections provided by the telescope should be fine
However, if you don't have them or you don't like them, you can check these coordinates from the images...
First, let's start by opening this image with DS9
A flat frame is necessary to check this...
...here we have a flat frame...
It seems nice enough...at the edges...except maybe for these hot pixels...
which start around... you can see there the coordinates... at 4098
As, we cannot appreciate the overscan from here, lets open this image with IRAF
and now we can are going to do that with the image plot task
Here we have it...
In this window, we have a plot of the pixel intensity value along the line 2100... actually it means row.. .
Let's check the edge values...
We zoom over region using the "e" key twice. The first one on the left down corner "e" and the second one on the right top corner...
The pixels start around column 1, but we could take 5 on the safer side...
to see gain the whole we type twice the "a" key in the same location...
The same on the other edge...
The last measurement occurs at pixel 966, we could use 960 safely...
and now "a" "a" to go back to the whole plot
To see the intensity along a pixel column we press "c"
At the edge, of this pixel column we can see the overscan region... let's zoom over it
Here we have the overscan region... which starts around 4097 and ends at 4200
now we zoom over the other column pixels edge and the data measurements start at pixel 1
and to quit this window remember we press "q"
I hope you wrote down these values for you need to compare them with the header biassec and trimsec definitions
Now that we have this information, we can actually apply the corrections This is the command line I shall be using...
let's check its flags...
As in the previous case the first parameter corresponds to the list of input files.
The list includes all the image types except the bias frames.
The second parameter is the list of output files
IRAF will give you an error if the number of files is different in both lists, so no need to worry about that.
Again, we don't have a cddtype value in our headers so no need to look for it...
in general to set a none value in IRAF we just type quotation marks ("")
Ok this we have not seen yet: All this parameters are single value flags which tell ccdproc which corrections to apply
You might have notice how within the "epar" menu these flags could have a "yes" or "no" value
In the command line in addition to these values we can use the plus symbol for "yes" and the minus symbol for "no"
In this case we are going to activate the overscan flag, this is equivalent to the biassec correction.
and we are going to trim our image. This is the trimsec correction
Finally, we are going to apply a bias correction by activating the zero correction flag
Now let's say we were happy with the overscan definition from the telescope.
in that case we type biassec equals image.
This way IRAF will take the biassec coordinates from each image header. On the other hand, if we were not happy with data region
we need to type trimsec equals our new coordinates... As I have done here
For the bias correction, we need to give the name of the image to use as the zero level for the CCD
This image must be in the same folder where we are running this command
Finally, since we trust IRAF with this task and we don't need to see every step: Interact, no thanks
Lets run the command...
It will take a while since I have many files....
In the mean time, let me show you the input and output lists I have used in this task...
This is the input list..
and this is the output list...
On the first one we have included the frame number so IRAF can recognize the Images
While on the second one, we have the same file names but with a "fits" extension
Taking these two steps IRAF will recognize these files as single frames and in the future it will open them directly
That's all I wanted to show you...
ccdproc has finished!
Now we can continue by averaging the flat files
As you know, flat images are evenly illuminated CCD recordings which are used to account the difference in pixel sensibilities.
Easy enough we are going to use the flatcombine task, let's take a look at it...
Here we have it... This time we leave average as the combine operation... but we set again the rejection type as crreject...
Again no ccdtype..
At the end of this list you will find two parameters related to the CCD:
the CCD Readnoise and the CCD gain... Here we have good news and bad news
The good news is that IRAF can read this information from each frame header... the bad news is that you need to tell it
the keyword it is using for these stored data
For example in my frames from the WHT
the CCD readout noise appears in the headers as "readnois"
without the final "e"
While the gain is described as "gain"
To run this task, we are going to use this command line.
I think I have explained all these parameters already... so let's run it...
by the way... you know it has finished when the command line symbol (vocl>) reappears...
Let's open it with DS9
It seems uniform enough
Now we are going to normalize it... for that we need the response task
This task along with the illumination task have been design to correct both the small and large scale variations
caused by illumination and transmission effects
Let's start by taking a look at this response task
... Sorry! you must go to the longslit package to use this task
There you have it...
A few things I need to change here... the fitting function...
which it is also advised to set it as legendre
and in the low and high rejection in sigma my guide advices 3... and 3
Now we are going to use this command line
In this line we have three parameters:
the first one is the input frame, the third one is the output frame, while the second one is the image which will be use for the response.
In most cases it is the very input image
The fourth flag activates the user input which will launch the "icfit" window
Let me run this command to show you what this means.
Fit the normalization spectrum for flat interactively? Yes!
Here we have our "icfit" window
In here we need our fitting function to reproduce as much as possible the shape of the lamp continuum
In here we need our fitting function to reproduce as much as possible the shape of the lamp continuum
This must be done while avoiding negative effects such as dead pixels, hot pixels or fringes
This must be done while avoiding negative effects such as dead pixels, hot pixels or fringes
In our case the spectrum seems quite clean so we are going to assume all the accidents are natural
In our case the spectrum seems quite clean so we are going to assume all the accidents are natural
In our case the spectrum seems quite clean so we are going to assume all the accidents are natural
and we can give them a high order
to change the order of the fitting function we type ":o 50"
in this case for example 50
We press "f" to "fit"
Once we are happy with the results
Once we are happy with the results
Once we are happy with the results
We press "q"... and we have produced our normalized global flat
We press "q"... and we have produced our normalized global flat
Let's minimize this window and we let's take a look at this image
And it seems quite clean...
We are now at step 5: Combine all the sky flat frames. This can be done with the combine task. We can do this with the combine task
I can use most of the default values of the combine task for this...
In this command line The firs parameter is my sky flat images list while the second is the output name...
the rejection... this time we choose the average sigma clipping while we set the scale and the weight as mode
Finally, the default output value when there are no pixels, that is the blank, we set it as one
We run the command...
Ok, let's check the image...
It seems a sky flat uniform enough...
Ok... now we need to apply the flat correction to this image... To do that we can use again the ccdproc task
This is what you need:
This time only the flatcor flag has been activated and we just need to specify the frame to be used in the global normalized flat
The final thing to do with the skyflat images is using the illumination task to produce the illumination Flat.
Let's take a look at its parameters
Let me try to explain this task:
Let me try to explain this task:
Let me try to explain this task:
One source of error in our spectra is having a different amount of light; not along the slit but parallel to it, in the spatial axis
One source of error in our spectra is having a difference in the amount of light not along the slit but paralell to it in the spatial axis
One source of error in our spectra is having a difference in the amount of light not along the slit but paralell to it in the spatial axis
We use sky flats for this correction, since in most cases, they are the most evenly illuminated recordings we can have
We use sky flats for this correction, since in most cases, they are the most evenly illuminated recordings we can have
To do this, the illumination task needs you to divide you spectrum in several regions
These are the bins, which we are going to leave with the default value of 5
then as we did in response task, you need to make sure manually that objects, such as bad pixles
then as we did in response task, you need to make sure manually that objects, such as bad pixles
then as we did in response task, you need to make sure manually that objects, such as bad pixles
or in the particular case of sky flats, stars, are not affecting the illumination
or in the particular case of sky flats, stars, are not affecting the illumination
And finally, we leave, as we did before, the high and low rejection in Sigma as 3 and 3
And finally, we leave, as we did before, the high and low rejection in Sigma as 3 and 3
And finally, we leave, as we did before, the high and low rejection in Sigma as 3 and 3
Ok, that should be it...
Now we can run this command...
Now it is going to ask us if we want to determine the illumination interactively for skyflat...
We say yes...
Here we are again in the icfit window...
We can set the location of the bins...
The only advise I can give you is to make sure that none of the bins edges is place close to a emission or absorption feature...
The only advise I can give you here, is to make sure that none of the bins edges is place close to an emission or absorption feature...
Let's delete the default bins using the "i" key
And now we can set the new bins using the s key...
And now we can set the new bins using the s key...
And now we can set the new bins using the "s" key...
Fourth...
and fifth
Now that we have done this we can press "q"
We return to our IRAF terminal ...
And it will ask us if we want to determine the illumination iteractively for skyFlat at bin 1
We say yes...
At this point we need to give a higher order to this fitting function...
You remember how to do that...(:o order) and let's try a lower number this time... 20
You remember how to do that...(:o order) and let's try a lower number this time... 20
Very good.... now we are going to delete these points
We can delete them with the key "d"
"f" to fit
Let's say we are happy there... we press "q"
And now it will ask us to try at the second bin.... we say yes
Now the fitting has improved...
We delete again some points...
Ok we are happy with these points... let's try again...
bin number 3...
The situation has improved... we delete more points..
We can increase the order a bit ...
We can increase the order a bit...
30
Ok... good enough...
Ok... good enough...
Bin number 4...again
and the final bin...ok
We should increase the order?
It should be fine... "q"
It should be fine... "q"
We are done... we can minimize this window...
And now let's take a look at this illumination flat...
There you have it
As you can see along the slit location the pixel intensity variation remains almost to the third decimal
This is what you should have: very small variations in pixels intensity. Rather these are large scale variations.
Rather these are large scale variations...
ok... we are happy with this illumination flat...
Now we have all the pieces to complete our data treatment
Normally, you would use this illumination flat and the global normalized flat to treat all your data with ccdproc task
Normally, you would use this illumination flat and the global normalized flat to treat all your data with ccdproc task
Myself, since I have flat frames recorded during all the night, I am going to try something, not harder... just longer...
Myself, since I have flat frames recorded during all the night, I am going to try something, not harder... just longer...
I am going to repeat the previous steps to produce local flat frames for each object...
I am going to repeat the previous steps to produce local flat frames for each object...
You already know how to do that...
For example... let's say these images are the flats corresponding to object 1
I can use the flatcombine task to produce a local flat file
I can use the flatcombine task to produce a local flat file
Press enter...and it is done...
And I can do the same for object 2
And now I have de flat for object 2
Now I need to normalize them...
Now I need to normalize these flat files...
For that I am going to use the local flat... and normalized it with the global flat...
And it will give me the normalized object 1 flat...
And I press enter...
This is the same set up as in the previous global flat and we were ok with it...
It is done
For the second object...
minimizing...
Now in the final step, we are going to use the ccdproc task to apply the flat and illumination correction to our object
For that we are going to use this command line...
In this case, I have activated the flat correction and the illumination correction
For that I have given the object 1 flat and global illumination flat frame names
and we run it
and we run it
That's it!
With these ten steps you have completed the fundamental pixel intensity corrections in your data
Now you just need to repeat these steps, this final step in particular...
For all your objects, as well as, your calibrations stars
In the next video we shall be looking at technical issues in the CCD which might have affected our frames...
And on the following ones we shall be taking care of the physical corrections
But before saying good bye, I promised you a few trics on how to generate these lists
But before saying good bye, I promised you a few tricks on how to generate these lists
On many occasions, you will be working with data from someone else or from a telescope which has free access to the data you need.
In that case, you most likely will have a log file such as the one, I have here... from the WHT
The approach I am going to show you will consist in opening this file with a spreadsheet software
The one which is freely available and it can be used, I believe, in all operative systems is it is Libre Office, so let's use it!
Grab and pull the file to the spread sheet window
And now we can open it...
Now make sure that when you set "separated by" all the options are deactivated
We are going to do that manually... press ok
I need you to notice how all this information is written within the same column.
I need you to notice how all this information is written within the same column.
I need you to notice how all this information is written within the same column.
That's not usefull for us... we want each piece of data in a single column... How are we going to do that?
So how are we going to do that?
Well first we are going to delete these rows... we don't need them
Well first we are going to delete these rows... we don't need them
(right mouse button delete rows)
Let me check everything is ok...
We need to select this column... And we go to "Data" -> "Text to columns"
I move this down a bit...
I know this separated by can be very tempting... but in most cases it does not work...you should work with "fixed width"
And now to create these separators we click on the ruler. Move them to define columns...
For some reason we cannot extend this window...
In my case, Libre Office remembered my separators from the previous test. You'll need to create them and check the columns fit properly
Everything seems fine so we press "ok"
There you have it...no there is a problem here...
I delete this cell...
We press twice at each column edge to adjust the column width to the text width
Object fine, coordinates fine, air mass fine, instrument ok...
Now, I need to sort these data
What you need to do is to select the columns
What you need to do is to select the columns
and again go to "Data" -> "Sort"
The first thing I want is to sort by "Instrument", secondly by "Object" and thirdly by "Run"
Now all my rows are organized, according to these parameters
For example let's say we wanted a list of the blue bias frames
We select this column with the images... and we generate a new sheet
Copy and paste
Now this is still useless to us...Since we don' t have an extension...
So in another column we type the extension...
and you might remember how my files started by "r"... so "r" in another column
And now you can use the "ampersand" (&) symbol to combine these strings
And the same for all the columns
Now we copy... new sheet and paste special
Here we just need the text...no formula nor anything else...
Ok this is my list of bias frames... now we save it
Ok this will be my "BiasFiles_FirstFrame1.csv"
Now... I am not fully aware of IRAF limitations on text files... I just now that comma separated values (.csv) work with IRAF. So we chose them
Now... I am not fully aware of IRAF limitations on text files... I just now that comma separated values (.csv) work with IRAF. So we chose them
Now... I am not fully aware of IRAF limitations on text files... I just now that comma separated values (.csv) work with IRAF. So we chose them
Now... I am not fully aware of IRAF limitations on text files... I just now that comma separated values (.csv) work with IRAF. So we chose them
Now we save it...
Happily enough, it will save just the actual sheet... which is the one we want
The second trick, for working with lists, I am going to show you today makes use of a text editor...
The second trick, for working with lists, I am going to show you today makes use of a text editor...
The second trick, for working with lists, I am going to show you today makes use of a text editor...
I am going to use emacs because is the only one I know the shortcuts but I am sure you can do this with any text editor you like... so no worries
let's open this list
There you have it... let's say we wanted to add the frame number this list...
For that you can use these short cuts...
to open the "Query replace" we type dot fit and enter...
"Query replace ... with" .fit[1] and we press enter
And now we need
And it has changed all the fit extensions...
That's all what I wanted to show you
If you have any doubt or you did not understand some of the steps I took please post a coment
See you very soon