Primer on data import¶
Data import - Specifying custom camera information¶
In order to use all features of pyplis, certain specifications related to camera
and image acquisition need to be defined. Basic information about the camera
(e.g. detector specifics) and the corresponding file convention (image type,
which data can be extracted from file names) are specified within
Camera
objects. This tutorial introduces the Camera
class and
how to set up your custom camera type based on your data format, including
definitions of your file naming convention.
Specifying your file naming convention¶
At the very beginning of the analysis, the images need to be imported and
separated by image type (e.g. on-band plume, off-band plume, dark, offset, on /
off-band cell calibration). In order to use the automated separation for a given
dataset (e.g. a single folder IMG_DIR
containing images of all types) it is
required that the image type can be identified from the file names.
The relevant information for identifying different image types (e.g. plume on-band, dark, offset) can be specified using either of the following two classes:
Filter
: specifies file access information for all image types that
- are NOT dark or offset images (e.g. on / off images plume / background)
DarkOffsetInfo
: specifies different types of dark images and
offset images.
Such a collection of Filter
and DarkOffsetInfo
objects is then
stored within a Camera
object.
These information is used to separate the individual image types when creating a
Dataset
object. The latter searches all valid image files in a given
folder IMG_DIR
and creates ImgList
objects for each Filter
and DarkImgList
objects for each DarkOffsetInfo
object defined
in the Camera
. Each of these lists is then filled with the file paths
of the corresponding image type located in IMG_DIR
. The Camera
object furthermore includes relevant specs of the camera (e.g. pixel geometry,
lens).
The following list provides an overview of relevant parameters for filename access information using exemplary filenames of the ECII camera type.
The ECII camera standard¶
In the following, an exemplary Camera
class is specified based on the
ECII-camera standard and file naming convention (cf. Example 0.2 - The camera class).
To start with, an empty Camera
instance is created:
cam = pyplis.Camera()
# prints the string representation which gives a nice overview over the
# relevant parameters
print cam
If you wish to store the camera as default you need to specify a unique camera ID (string) which is not yet used for any of the pyplis default cameras stored in the file cam_info.txt (package data). You can check all existing IDs using:
print pyplis.inout.get_all_valid_cam_ids()
Let’s call our new camera “ecII_test”:
cam.cam_id = "ecII_test"
Now specify some relevant attributes of the camera, starting with the image file type:
cam.file_type = "fts"
You can also provide information about detector and camera lens:
cam.focal_length = 25e-3 #m
# Detector geometry
cam.pix_height = 4.65e-6 # pixel height in m
cam.pix_width = 4.65e-6 # pixel width in m
cam.pixnum_x = 1344
cam.pixnum_y = 1024
In the following, the camera file naming convention is specified. This enables to extract certain information from the image file names (e.g. image acq. time, image type, exposure time).
Start with setting the file name delimiter of your file naming convention:
cam.delim = "_"
Based on that, specify the position of acquisition time (and date) in the image file names after splitting with delimiter:
cam.time_info_pos = 3
The acq. time strings in the file names need to be converted into datetime
objects thus, specify the string for internal conversion (is done using
datetime.strptime()
):
cam.time_info_str = "%Y%m%d%H%M%S%f"
If the file name also includes the image exposure time, this can also be specified:
cam.texp_pos = "" #the ECII does not...
as well as the unit (choose from “s” or “ms” if applicable):
cam.texp_unit = ""
Furthermore, the image type identification can (and should) be specified in the
camera, in order to make life easier. This ensures, that images of different
types (e.g. on / off-band, dark, offset) can be identified and separated
directly from the filename. The relevant information is specified in a
collection of Filter
and DarkOffsetInfo
objects.
Let’s start off with defining the different image access types for on and
off-band images (these are stored in Filter
objects, while dark /
offset image access information is stored in DarkOffsetInfo
objects,
follows below):
# On-band images
on = pyplis.Filter(id="on", type="on", acronym="F01",
meas_type_acro="F01", center_wavelength=310)
# Off-band images
off = pyplis.Filter(type="off", acronym="F02",
meas_type_acro="F02", center_wavelength=330)
Now add the two filters to the camera (i.e. put them into a list and assign it to the camera):
filters = [on, off]
cam.default_filters = filters
# Checks and sets filters in cam
cam.prepare_filter_setup()
Tell the camera, which of the filters is the “central” filter for the emission rate analysis (usually “on”):
cam.main_filter_id = "on"
The latter information is used for internal linking of image lists within a
Dataset
object, for instance, if the camera contains multiple
type="on"
filters (i.e. on-band SO2).
Note
This parameter main_filter_id
is irrelevant for standard setups like here
(i.e. one on and one off-band filter)
Similar to the filter setup (which specifies access to the actual images to be
analysed), the filename access information for dark (type=dark
) and offset
(type=offset
) image identification needs to be specified using
DarkOffsetInfo
instances:
offset_low_gain = pyplis.DarkOffsetInfo(id="offset0",type="offset",
acronym="D0L",
meas_type_acro="D0L",
read_gain=0)
offset_high_gain = pyplis.DarkOffsetInfo(id="offset1",type="offset",
acronym="D0H", read_gain=1)
dark_low_gain = pyplis.DarkOffsetInfo(id="dark0",type="dark",
acronym="D1L", read_gain=0)
dark_high_gain = pyplis.DarkOffsetInfo(id="dark1",type="dark",
acronym="D1H", read_gain=1)
# put the 4 dark info objects into a list and assign to the camera
dark_info = [offset_low_gain, offset_high_gain,
dark_low_gain, dark_high_gain]
cam.dark_info = dark_info
Note
You might have recognised, that in the last 3 DarkOffsetInfo`
objects, the meas_type_acro was not specified. This is because it is actually irrelevant for the ECII camera which does not include a sub string specifying different measurement modi like, for instance, the HD-Custom camera (i.e. K, M, D).
Now that all different image types are specified, the camera needs to know where
to find the actual information in the file names (after splitting using
delim
).
The position of the strings specified in the attribute acronym
(see
definitions of the Filter
and DarkOffsetInfo
objects above) can be set
using:
cam.acronym_pos = 4
and the position of the strings specified in attribute meas_type_acro
:
cam.meas_type_acro_pos = 4
Note
If meas_type_acro
is irrelevant (like for this camera) it is required to
be set equal acronym_pos
Furthermore, the dark correction type needs to be specified, pyplis includes two options for that, the ECII uses option 1:
cam.DARK_CORR_OPT = 1
Todo
Include information about the two different dark correction modes
That’s it! You might want to check if everything is in place:
print cam
If you are happy, you might want to check if the data access from the file names
works. You can do a fast check using a file path IMG_PATH
to one of your
images and run:
acq_time, filter_id, meas_type, texp, warnings =\
cam.get_img_meta_from_filename(IMG_PATH)
You might also test it for a whole dataset of images located in a directory
IMG_DIR
and check if pyplis can identify the different image types. You can
do this, for instance, by creating a Dataset
object. First, create a
measurement setup with minimum information:
meas_setup = pyplis.MeasSetup(base_dir=IMG_DIR, camera=cam)
and create a Dataset from that:
ds = pyplis.Dataset(meas_setup)
The Dataset
object should now detect all individual image types and
puts them into separate lists, which can be accessed using the IDs of the
corresponding Filter
objects, e.g.:
lst = ds.get_list("on")
print "Number of images in list: %d" %lst.nof
These lists are of type ImgList
. Similarly, dark and offset image lists
(DarkImgList
classes) were created using the information stored in the
DarkOffsetInfo
objects specified in our camera:
dark_list_low_gain = ds.get_list("dark0")
offset_list_low_gain = ds.get_list("offset0")
You can also easily access all lists, that actually contain images (i.e. for
which image matches could be found in IMG_DIR
), e.g. all lists that contain
images and correspond to one of the Filter
objects:
all_imglists = ds.img_lists_with_data #this is a dictionary
print all_imglists.keys() #prints the list / Filter IDs
and similar, all DarkImgList
objects that contain data:
all_darklists = ds.dark_lists_with_data #this is a dictionary
print all_darklists.keys() #prints the list IDs
If everything works out nicely, you can add the camera as new default using:
cam.save_as_default()
After saving the camera as new default, you can load it using:
import pyplis
cam = pyplis.Camera(cam_id="ecII_test")
print cam
Done!