Changelog¶
Automatically generated (and slightly modified) using:
git log --pretty=format:"- %ad, %aN%x09%s" --date=short v1.3.0..HEAD > changelog/CHANGELOG_v130_v143.rst
Release 1.3.0 -> 1.4.3¶
- 2019-05-22, jgliss Updated README
- 2019-05-22, jgliss Updated README
- 2019-05-22, jgliss Updated README
- 2019-05-22, jgliss Updated README
- 2019-05-22, jgliss Updated README
- 2019-05-22, jgliss Removed release note for version 1.3.0
- 2019-05-21, jgliss Fixed some formatting errors in README
- 2019-05-21, jgliss Updated version to 1.4.3, cleaned up and reformatted README so that it passes twine check and included Python 3 support into classifiers in setup.py
- 2019-05-21, jgliss Updated version to 1.4.2
- 2019-05-21, jgliss Removed / reformatted problematic parts from README.rst, so that it will be (hopefully) rendered properly in PyPi
- 2019-05-21, jgliss Removed pyplis logo from README (since it could not be rendered on PyPi) and updated version to 1.4.1
- 2019-05-21, jgliss Changed version to 1.4.0 (ready for release)
- 2019-05-21, jgliss Added LICENSE file to MANIFEST.in
- 2019-05-21, jgliss Refined tests performed in script ex06_doas_calib.py and accounted for minor differences in results between python 2.7 and 3.7
- 2019-05-21, jgliss Minor changes in output of radius search in doascalib.py
- 2019-05-21, jgliss Updated requirements file and added explicit conda environment file for python 2.7
- 2019-05-20, jgliss Updated README
- 2019-05-20, jgliss Updated README
- 2019-05-20, jgliss Added numpy and six to environment file
- 2019-05-20, jgliss Updated version
- 2019-05-20, jgliss Added pydoas to conda-forge dependency in pyplis_env.yml (no pip dependencies anymore)
- 2019-05-20, jgliss Added import of mpl_toolkits.mplot3d in 3d plotting routine of MeasGeometry
- 2019-05-20, jgliss Updated, simplified and merged two conda environment files for python 2 and 3 into new one that works for both: pyplis_env.yml
- 2019-05-20, jgliss Updated version and fixed one test in ex02_meas_geometry.py
- 2019-05-20, jgliss Merge branch ‘py3’
- 2019-05-12, jgliss Fixed some minor Deprecationwarnings
- 2019-05-12, jgliss Fixed failing test related to viewing direction correction of camera
- 2019-05-12, jgliss Fixed import error in geometry.py of geonum class TopoAccessError, due to recent API changes in geonum
- 2019-05-12, Jonas Merge pull request #23 from johannjacobsohn/fix-typos
- 2019-05-12, jgliss Fixed relative imports of geonum in example scripts; fixed failing test in ex06_doas_calib.py
- 2019-05-12, jgliss Added conda environment file for python 3 installation (not yet tested)
- 2019-04-23, Jonas Gliss Updated pyplis_env_py27.yml
- 2019-04-23, Jonas Gliss Updated README
- 2019-04-23, Jonas Gliss Updated README
- 2019-04-23, Jonas Gliss Updated README
- 2019-04-23, Jonas Gliss Updated VERSION to 1.4.0.dev1
- 2019-04-23, Jonas Gliss Updated installation instructions in README
- 2019-04-23, Jonas Gliss Updated script ex02_meas_geometry.py due to failing tests
- 2019-04-23, Jonas Gliss Added python 2.7 conda environment file pyplis_env_py27.yml
- 2019-04-22, jgliss Merged and updated README from py3 branch
- 2019-04-22, jgliss Updated geometry.py due to recent API updates in new geonum release v1.4.0
- 2019-04-22, jgliss Updated example scripts due to recent API changes in latest version 1.4.0 of geonum
- 2019-03-01, Jonas Update README.rst
- 2019-02-25, Johann Jacobsohn cleanup
- 2019-02-22, Johann Jacobsohn fixup use of depreciated methods and properties
- 2019-02-19, Johann Jacobsohn fixup typos
- 2019-02-19, Solvejg Dinger Fix broken link in Readme
- 2019-02-19, Solvejg Dinger Merge pull request #10 from heliotropium72/patch-1
- 2019-02-19, Solvejg Dinger Adapt suggestion for one-liner
- 2019-02-11, Jonas Gliss Refers to #13: Added new default camera usgs_9mm; the 12mm focal length version can now be accessed via the former ID usgs but also via usgs_12mm
- 2019-01-25, Solvejg Dinger Fixed rendering problem of README
- 2019-01-25, Solvejg Dinger Update README.rst
- 2019-01-22, Solvejg Dinger Merge pull request #16 from johannjacobsohn/py3
- 2019-01-17, Johann Jacobsohn WIP: fixup to make tests pass, needs to be investigated
- 2019-01-17, Johann Jacobsohn py3.7 doesn’t like __dir__ to be overwritten…
- 2019-01-17, Johann Jacobsohn add python 3.5 and 3.7 to tox, update pytest dependency and remove sitepackages from tox to make tests more consistent
- 2019-01-15, heliotropium72 Fix pytest fixtures to correct usage
- 2019-01-14, heliotropium72 Revert previous WIP
- 2019-01-02, jgliss Partly fixed failing tests
- 2018-12-27, jgliss Fixed bug arising from division returning float rather than int, due to recent Python 3 port
- 2018-11-29, Johann Jacobsohn Revert “skip failing test until fixed”
- 2018-11-29, Johann Jacobsohn WIP to pass tox scripts - needs to be reverted and fixed!
- 2018-11-29, Johann Jacobsohn add RUN_EXAMPLE_SCRIPTS.py to tox, use sitepackages for Basemap
- 2018-11-29, Johann Jacobsohn fix floordiv issue from python3 work, and cleanup
- 2018-11-28, Johann Jacobsohn protect possible None value against < operator, which fails in python3. Remove try..except block that only obscures exceptions
- 2018-11-28, Johann Jacobsohn remove __deepcopy__, underlying issue needs to be fixed in geonum
- 2018-11-28, Johann Jacobsohn make sure to use int for array indices
- 2018-11-28, Johann Jacobsohn numpy.sum does not handle .values() as you would expect in python3…
- 2018-11-28, Johann Jacobsohn truediv -> __truediv__, fix typo
- 2018-11-27, Johann Jacobsohn fix reading of cam_info.txt in python3
- 2018-11-27, Johann Jacobsohn try to load test data from envvar PYPLIS_DATADIR if not found in default location (helpful for testing)
- 2018-11-27, Johann Jacobsohn fixup: CalibData.senscorr_mask does actually not accecpt ndarray
- 2018-11-23, Johann Jacobsohn add RUN_INTRO_SCRIPTS.py to tox
- 2018-11-16, Johann Jacobsohn fix pytest to version 3
- 2018-11-16, Johann Jacobsohn fix difference in exception handling between python 2 and 3, restrict exceptions to KeyError
- 2018-11-14, Johann Jacobsohn skip failing test until fixed
- 2018-11-14, Johann Jacobsohn fixup requirements and tox environment
- 2018-11-14, Johann Jacobsohn fixup import syntax for py3
- 2018-11-14, Johann Jacobsohn fixup deepcopy of MeasGeometry in py3
- 2018-11-14, Johann Jacobsohn fix test_io in py3
- 2018-11-14, Johann Jacobsohn fix urlib import for python3
- 2018-11-14, Johann Jacobsohn small pydocstyle fixes
- 2018-11-14, Johann Jacobsohn __div__ -> truediv
- 2018-11-14, Johann Jacobsohn return value of .keys() and .values() have changed between python 2 and 3, this should make it consistent between the two
- 2018-11-14, Johann Jacobsohn add __future__ to help make python 2 and 3 behave consistent
- 2018-11-09, Johann Jacobsohn add pylint and disable every offending check
- 2018-11-05, Johann Jacobsohn scripts/ex07_doas_cell_calib.py: improve python3 compability, fix bugbear lint
- 2018-11-05, Johann Jacobsohn pyplis/utils.py: improve python3 compability, fix bugbear lint
- 2018-11-05, Johann Jacobsohn pyplis/setupclasses.py: improve python3 compability, fix bugbear lint
- 2018-11-05, Johann Jacobsohn pyplis/processing.py: improve python3 compability, fix bugbear lint
- 2018-11-05, Johann Jacobsohn pyplis/plumespeed.py: improve python3 compability, fix bugbear lint
- 2018-11-05, Johann Jacobsohn pyplis/plumebackground.py: improve python3 compability, fix bugbear lint
- 2018-11-05, Johann Jacobsohn pyplis/model_functions.py: improve python3 compability, fix bugbear lint
- 2018-11-05, Johann Jacobsohn pyplis/inout.py: improve python3 compability, fix bugbear lint
- 2018-11-05, Johann Jacobsohn pyplis/image.py: improve python3 compability, fix bugbear lint
- 2018-11-05, Johann Jacobsohn pyplis/imagelists.py: improve python3 compability, fix bugbear lint
- 2018-11-05, Johann Jacobsohn pyplis/geometry.py: improve python3 compability, fix bugbear lint
- 2018-11-05, Johann Jacobsohn pyplis/forms.py: improve python3 compability, fix bugbear lint
- 2018-11-05, Johann Jacobsohn pyplis/fluxcalc.py: improve python3 compability, fix bugbear lint
- 2018-11-05, Johann Jacobsohn pyplis/doascalib.py: improve python3 compability, fix bugbear lint
- 2018-11-05, Johann Jacobsohn pyplis/dilutioncorr.py: improve python3 compability, fix bugbear lint
- 2018-11-05, Johann Jacobsohn pyplis/dataset.py: improve python3 compability, fix bugbear lint
- 2018-11-05, Johann Jacobsohn pyplis/cellcalib.py: improve python3 compability, fix bugbear lint
- 2018-11-05, Johann Jacobsohn pyplis/calib_base.py: improve python3 compability, fix bugbear lint
- 2018-11-05, Johann Jacobsohn add bugbear lint to improve python 3 compability
- 2018-11-05, Johann Jacobsohn Light phrasing changes to conform to PEP 257 and pass pydocstyle validation
- 2018-06-01, Johann Jacobsohn lint pyplis/test/test_dataset_module.py
- 2018-05-25, Johann Jacobsohn lint scripts/more_scripts/ex002_plume_background_combi.py
- 2018-05-25, Johann Jacobsohn lint scripts/more_scripts/ex001_save_all_calib_imgs.py
- 2018-05-25, Johann Jacobsohn lint scripts/ex12_emission_rate.py
- 2018-05-25, Johann Jacobsohn lint scripts/ex11_signal_dilution.py
- 2018-05-25, Johann Jacobsohn lint scripts/ex10_bg_imglists.py
- 2018-05-25, Johann Jacobsohn lint scripts/ex0_7_cellcalib_manual.py
- 2018-05-25, Johann Jacobsohn lint scripts/ex0_6_pcs_lines.py
- 2018-05-25, Johann Jacobsohn lint scripts/ex0_5_optflow_livecam.py
- 2018-05-25, Johann Jacobsohn lint scripts/ex0_4_imglists_auto.py
- 2018-05-25, Johann Jacobsohn lint scripts/ex0_3_imglists_manually.py
- 2018-05-25, Johann Jacobsohn lint scripts/ex0_2_camera_setup.py
- 2018-05-25, Johann Jacobsohn lint scripts/ex0_1_img_handling.py
- 2018-05-25, Johann Jacobsohn lint scripts/ex09_velo_optflow.py
- 2018-05-25, Johann Jacobsohn lint scripts/ex08_velo_crosscorr.py
- 2018-05-25, Johann Jacobsohn lint scripts/ex07_doas_cell_calib.py
- 2018-05-25, Johann Jacobsohn lint scripts/ex06_doas_calib.py
- 2018-05-25, Johann Jacobsohn lint scripts/ex05_cell_calib_auto.py
- 2018-05-25, Johann Jacobsohn lint scripts/ex04_prep_aa_imglist.py
- 2018-05-25, Johann Jacobsohn lint scripts/ex03_plume_background.py
- 2018-05-25, Johann Jacobsohn lint scripts/ex02_meas_geometry.py
- 2018-05-25, Johann Jacobsohn lint scripts/ex01_analysis_setup.py
- 2018-05-25, Johann Jacobsohn lint scripts/SETTINGS.py
- 2018-05-25, Johann Jacobsohn lint scripts/RUN_INTRO_SCRIPTS.py
- 2018-05-25, Johann Jacobsohn lint scripts/RUN_EXAMPLE_SCRIPTS.py
- 2018-05-25, Johann Jacobsohn lint pyplis/utils.py
- 2018-05-25, Johann Jacobsohn lint pyplis/test/test_io.py
- 2018-05-25, Johann Jacobsohn lint pyplis/test/test_image_module.py
- 2018-05-25, Johann Jacobsohn lint pyplis/test/test_highlevel_examples.py
- 2018-05-25, Johann Jacobsohn lint pyplis/test/test_dataset_module.py
- 2018-05-25, Johann Jacobsohn lint pyplis/test/__init__.py
- 2018-05-25, Johann Jacobsohn lint pyplis/setupclasses.py
- 2018-05-25, Johann Jacobsohn lint pyplis/processing.py
- 2018-05-25, Johann Jacobsohn lint pyplis/plumespeed.py
- 2018-05-25, Johann Jacobsohn lint pyplis/plumebackground.py
- 2018-05-25, Johann Jacobsohn lint pyplis/optimisation.py
- 2018-05-25, Johann Jacobsohn lint pyplis/model_functions.py
- 2018-05-25, Johann Jacobsohn lint pyplis/inout.py
- 2018-05-25, Johann Jacobsohn lint pyplis/imagelists.py
- 2018-05-25, Johann Jacobsohn lint pyplis/image.py
- 2018-05-25, Johann Jacobsohn lint pyplis/helpers.py
- 2018-05-25, Johann Jacobsohn lint pyplis/glob.py
- 2018-05-25, Johann Jacobsohn lint pyplis/geometry.py
- 2018-05-25, Johann Jacobsohn lint pyplis/forms.py
- 2018-05-25, Johann Jacobsohn lint pyplis/fluxcalc.py
- 2018-05-25, Johann Jacobsohn lint pyplis/exceptions.py
- 2018-05-25, Johann Jacobsohn lint pyplis/doascalib.py
- 2018-05-25, Johann Jacobsohn lint pyplis/dilutioncorr.py
- 2018-05-25, Johann Jacobsohn lint pyplis/dataset.py
- 2018-05-25, Johann Jacobsohn lint pyplis/custom_image_import.py
- 2018-05-25, Johann Jacobsohn lint pyplis/cellcalib.py
- 2018-05-25, Johann Jacobsohn lint pyplis/calib_base.py
- 2018-11-01, Solvejg Dinger Bug Fix: Setting of calibration coefficients
- 2018-05-25, Johann Jacobsohn lint pyplis/__init__.py
- 2018-05-25, Johann Jacobsohn lint scripts/priv_01_plot_raw_vs_calib.py
- 2018-05-24, Johann Jacobsohn add python 3 to tox
- 2018-05-17, Johann Jacobsohn add tox to run tests
- 2018-09-09, jgliss Merge branch ‘heliotropium72-multi_layer_fits’
- 2018-09-09, jgliss fixed minor bug (download location of test data)#
- 2018-09-09, jgliss Fixed minor bug
- 2018-09-09, jgliss Merge branch ‘multi_layer_fits’ of https://github.com/heliotropium72/pyplis into heliotropium72-multi_layer_fits
- 2018-06-22, heliotropium72 Fix binary_mask fixture
- 2018-06-21, heliotropium72 Add sky_mask to image lists
- 2018-06-21, heliotropium72 Add class ImgListLayered
- 2018-06-21, heliotropium72 Make OptflowFarneback.calc_flow compatible with OpenCV 2.x and 3.x
- 2018-06-20, Jonas Merge pull request #8 from heliotropium72/patch-3
- 2018-06-20, Jonas Merge pull request #7 from heliotropium72/patch-2
- 2018-06-19, heliotropium72 Add get_masked_img method and corresponding tests
- 2018-06-18, Solvejg Dinger Merge pull request #11 from heliotropium72/patch-3
- 2018-06-18, Solvejg Dinger Fix indentation level
- 2018-06-18, Solvejg Dinger Merge pull request #10 from heliotropium72/patch-3
- 2018-06-18, Solvejg Dinger Merge pull request #9 from heliotropium72/patch-2
- 2018-06-07, Solvejg Dinger Add custom import method for Comtessa image data
- 2018-06-07, Solvejg Dinger Extended image.meta dictionary
- 2018-05-27, jgliss Fixed minor bug; added custom image read method template for QSI camera LMV group, France
- 2018-05-26, jgliss Updated formatting
- 2018-05-26, jgliss Updated formatting
- 2018-05-26, jgliss Deleted history of old binary files (repo size down to 30MB from ~500MB); Added content of scripts_out folder to .gitignore; updated README
- 2018-05-23, jgliss Removed strict PIL dependency for image read (now using opencv method imread)
Release 1.0.0 -> 1.0.1¶
This release includes only minor changes compared to the last one. These are mainly related to the access and handling / modelling of the plume background intensities in ImgList
objects.
25/11/2017 - 12/01/2018 (v1.0.0 -> v1.0.1)¶
- Fixed some bugs
- Added more tests.
- Improved access of plume background images in
ImgList
objects
Release 1.0.1 -> 1.3.0¶
Note
This release includes major API changes, performance improvements and bug fixes compared to version 1.0.1. Please update your installation as soon as possible.
Summary¶
Measurement geometry (MeasGeometry
):
- more accurate plume distance retrievals (i.e. now also in dependency of vertical distance).
- redesigned API -> improved user-friendliness.
Image analysis: Image registration shift can now be applied to images.
shift()
in class Img.- Comes with new mode (
shift_mode
) inImgList
objects. - Default on / off shift for camera can be set in
Camera
using attributereg_shift_off
(and correspondingly, in file cam_info.txt).
Camera calibration. Major improvements and API changes:
- new abstraction layer (
calib_base
) including new calibration base class.CalibData
: bothDoasCalibData
andCellCalibData
are now inherited from new base classCalibData
. Advantages and new features:- arbitrary definition of calibration fit function.
- fitting of calibration curve, I/O (read / write FITS) and visualisation of DOAS and cell calibration data are now unified in
CalibData
.
Further changes
ImgStack
more intuitive and flexible (e.g. dynamically expandable).- Improved index handling and performance of image list objects (
imagelists
). PlumeBackgroundModel
: revision, clean up and performance improvements.- Improved user-friendliness and performance of plume background retrieval in
ImgList
objects. - Correction for signal dilution (
DilutionCorr
): increased flexibility and user-friendliness. - Improved flexibility for image import using
Dataset
class (e.g. on / off images can be stored in the same file). - Reviewed and largely improved performance of general workflow (i.e. iteration over instances of
ImgList
incalib_mode
,dilcorr_mode
andoptflow_mode
).
Major bug fixes
- Fixed conceptual error in cross-correlation algorithm for velocity retrieval (
find_signal_correlation()
in moduleplumespeed
). - Fixed:
ImgList
in AA mode used current off-band image (at indexidx_off
) both for the current and next on-band image (and notidx_off+1
).
1.0.1 -> 1.1.0¶
Img
object
- Included read / write of meta info dictionary for FITS load / save
- New method
is_darkcorr()
- DOAS calibration (
doascalib
)
- More flexible retrieval of DOAS calibration curves
ImgStack
- new method
ImgStack.sum()
- Can now be dynamically extended (i.e. dynamic update of 3D array size). Corresponding API changes:
- REMOVED: method
append_img()
- NEW methods:
insert_img()
,add_img()
,init_stack_array()
- Measurement geometry (
MeasGeometry
)
- More accurate plume distance retrieval (now also in vertical direction, cf. Fig. 2 from example script 2)
- Other changes
- Moved Etna test data to new URL
- Fixed bugs
1.1.0 -> 1.2.1¶
Img
object
- new method
Img.sum()
- Image list classes (
imagelists
, MAJOR API CHANGES)
- Improved flexibility and clarity in index management
- New attribute
skip_files
(i.e. load only every nth image from the filelist)- New method
iter_indices()
- Renamed method
update_index_linked_lists()
tochange_index_linked_lists()
- Removed method
change_index()
- Plume background retrieval (
plumebackground
)
- Getter / setter for attr.
surface_fit_mask
(ensure it is typeImg
)
- Changes related to I/O
- Moved option
`LINK_OFF_TO_ON
fromDataset
toBaseSetup
(no API changes inDataset
)- New I/O option
ON_OFF_SAME_FILE
inBaseSetup
that can be set if on and off images are stored in one (e.g. FITS) file (like for the new USGS CVO camera type)- I/O options for data import can now be specified in file cam_info.txt for each camera individually using keyword
io_opts
and is stored as dict inCameraBaseInfo
(base class ofCamera
)- Included I/O info for camera of USGS CVO (uses previous point)
- Source info can now be saved automatically to file my_sources.txt
- Other changes
- New method
matlab_datenum_to_datetime()
inhelpers
- Fixed bugs
1.2.1 -> 1.3.0¶
Note
This version includes major refactoring and changes in API, aiming for more transparency and intuitive design. For instance, both the DoasCalibData
and CellCalibData
now inherit from a new base class CalibData
(in new module calib_base
).
Note
Changes related to camera calibration API (e.g. renaming, refactoring or removing of methods) are not resolved in full detail below (following point 1.).
- Camera calibration
- NEW MODULE
calib_base
containing new calibration base classCalibData
(bothDoasCalibData
andCellCalibData
inherit from this base object)- MAIN CHANGES associated with with refactoring into general base class
CalibData
- NEW FEATURE: Fit function for calibration data (both cell and DOAS) can now be defined arbitrarily (before, only polynomials were possible). See also module
model_functions
, in particular new classCalibFuns
- I/O (.e.g to / from FITS, or csv) are now unified for cell and DOAS calibration
- Visualisation (e.g. plot of calibration curve and data) now unified for cell and DOAS calibration
- New default fit function based on Kern et al. 2015
- UNCERTAINTY treatment: Error in calibrated CDs is now computed based on the standard deviation of fit residual (if more than 10 datapoints are available for retrieval of calibration curve).
Img
object
- Renamed attribute “alt_offset” -> “altitude_offset”
- Moved custom import for ECII camera into new custom method
load_ecII_fits()
in modulecustom_image_import
- New attributes:
is_cropped
is_resized
is_shifted
- New methods:
shift()
(applies x/y pixel shift of image)convolve_with_mask()
, for instance, when applied to an AA image, the input mask may be, e.g. a parameterised DOAS FOV (e.g. fitted 2D super-Gauss). The function then returns the weighted average AA within the FOV.get_thresh_mask()
- Image list classes (
imagelists
)
- New list mode
shift_mode
(only for offband lists, i.e. lists with attributetype="off"
): activate / deactivate shift (dx, dy) of images on image load (cf. other list modes, such astau_mode
,calib_mode
,optflow_mode
). If activated, the default shiftreg_shift_off
of the assignedCamera
instance is used (is set (0, 0) if not explicitly defined (either in file cam_info.txt for a camera type (cf. cam “usgs” therein) or in instance ofCamera
directly).- Reviewed and optimised:
correct_dilution()
reviewed, largely rewritten and optimised- New attributes:
update_cam_geodata
(default isFalse
). If True, the measurement geometry (i.e. plume distance) is automatically updated if image files contain camera geodata (e.g. lat, lon, viewing direction).- New methods:
calc_plumepix_mask()
(for dilution correction)timestamp_to_index()
(returns list index corresponding to a datetime object)_iter_num()
(number of iterations to loop through the whole list, resulting from the total number of filesnof
andskip_files
)pop()
now raises NotImplementedError- Introduced @property methods (getter / setter) for the attributes
skip_files
(newly introduced in v1.2.1, see above) andedit_active
to ensure index update and reload (on change)- Further changes, deprecated, renamed
- Introduced new input parameter reload_here in
goto_img()
(if True,load()
is called even if the new index is the same as the current index, defaults toFalse
)- Deprecated:
- Removed attribute
which_bg
(now handled automatically by @property attributebg_img
)- Renamed
aa_corr_mask
->senscorr_mask
DARK_CORR_OPT
->darkcorr_opt
- Bug fixes:
- Fixed: on-band list AA mode used current off-band image (at index
idx_off
) both for the current and next on-band image (and notidx_off+1
).
- Measurement geometry (
MeasGeometry
, MAJOR API CHANGES)
- Improved user-friendliness and performance: getter / setter methods for all attributes
- Intended access / modification of attributes is via new getter / setter methods (e.g.
`geom.cam["lon"]
->geom.cam_lon
)- Comes with better handling of recomputation requirements of geometry in case individual parameters (e.g. camera viewing direction, position, wind direction) are updated (in this context, note new attribute
update_cam_geodata
inImgList
objects). Specifically:- Method
update_geosetup()
is called whenever a relevant attribute is updated via the corresponding setter method. This ensures, that derived values such as plume distance are always up-to-date with the current attributes.- Attribute dictionaries now private (e.g.
.cam
->._cam
,.source
->._source
).- New methods:
get_topo_distance_pix()
(determines distance to local topography in viewing direction of individual image pixel)
PlumeBackgroundModel
(Review and clean-up)
- New attribute
last_tau_image
- New method
_init_bgsurf_mask()
: initiate mask for 2D background polynomial surface fit (only relevant for correction modemode=0
)- Removed
- dictionary
_current_imgs
: kept copies of input images (private dictionary)- Methods:
get_current()
,pyrlevel()
,current_plume_background()
,subtract_tau_offset()
,_prep_img_type()
,set_current_images()
,plot_tau_result_old()
DilutionCorr
- Retrieval of extinction coefficients for dilution correction based on dark terrain features can now also be performed for individual pixel coordinates in the images, in addition to the distance retrieval based on lines in the images (see example script 11)
- New methods:
add_retrieval_point()
add_retrieval_line()
- Module
model_functions
:
- New calibration fit function(s) based on Kern et al., 2015
- New class
CalibFuns
for access of calibration fit functions
- Plume velocity retrievals (
plumespeed.py
)
- Cross correlation method (
find_signal_correlation()
)
- Improved retrieval robustness: introduced percentage max shift that describes the maximum shift in percent of the second relative to the first time-series based on the total length of both series.
- Fixed systematic retrieval error: Before, the second signal was rolled over the first, meaning, that the “end” of the 2. signal was attached to it’s beginning and thus, correlated with the beginning of the first signal. That behaviour has been resolved.
- Optical flow (
OptflowFarneback
andFarnebackSettings
)
i_min
(lower end of contrast range for optical flow calculation) can now also be smaller than 0.
- I/O and setup classes (modules
inout
andsetupclasses
)
- my_pyplis folder is now created on installation (in user home directory)
- includes copies of cam_info.txt file and my_sources.txt
- New method
save_default_source()
ininout
(is saved in file my_sources.txt)- New method
save_to_database()
inSource
(wrapper method forsave_default_source()
)- New I/O option
REG_SHIFT_OFF
in classesBaseSetup
andMeasSetup
: if True (and if image lists are created usingDataset
and correspondingMeasSetup
object), then, the off-band images (in off-bandImgList
) are automatically shifted to on-band images (in on-bandImgList
) using the registration shift that is specified inCamera.reg_shift_off
(can be set in file cam_info.txt)
- Other changes
- New method
integrate_profile()
in classLineOnImage
- New method
make_circular_mask()
in modulehelpers.py
- In
fluxcalc
(and all included classes): renamed attrcd_err_rel
tocd_err
(note changes in uncertainty treatment of calibration data!)EmissionRateSettings
: new option / attributemin_cd_flow
(in addition to already existingmin_cd
) that may be used to explicitly define the minimum column-density of an image pixel for it to be considered valid with respect to optical flow histogram analysis (before, the thresholdmin_cd
was used). Is set equalmin_cd
if not explicitly specified- Moved class
LineOnImage
into moduleutils
- Moved method
model_dark_image()
fromprocessing
toimage
as well as classProfileTimeSeriesImg
- Changed input parameter of
model_dark_image()
inprocessing
- Changed default colormap for optical density (and calibrated) images from bwr to viridis (perceptually uniform)
- Major performance improvements: reviewed typical workflow chain and removed irrelevant duplications of image arrays in certain objects ()
- Fixed bugs
- Included new tests (test suite still very low coverage…)
Release 0.11.2 -> 1.0.0¶
30-31/03/2017¶
- Renamed
roi_rad
in Farneback classes toroi_rad_abs
(makes it clearer in which coordinates it is supposed to be defined). The old name also still works but a warning is given if used. - Renamed
hist_dir_sigma
in Farneback classes tohist_sigma_tol
since it is applied both to the fit result of the main peak of the orientation histogram but also to determine the uncertainty in the length histogram from the moments analysis. The old name also still works but a warning is given if used. - More features in
LineOnImage
- Global velocity estimates (and uncertainties) can now be assigned (e.g. for emission rate analysis)
LocalPlumeProperties
can now be assigned (e.g. for emission rate analysis and if a time series of local displacement vectors for velocity retrieval was calculated beforehand and the results are accessible in a text file)).
01-07/04/2017¶
- Added features in
LocalPlumeProperties
- Interpolation can now also be performed onto index array of other time series (e.g. image list time stamps)
- New method
apply_significance_thresh()
inLocalPlumeProperties
: can be used to remove data points with significance lower than provided thresh (and combined e.g. with interpolation).- New method
to_pyrlevel()
: converts displacement lengths (and errors) to a given Gauss pyramid level.
- Expanded functionality in classes
EmissionRateAnalysis
andEmissionRateSettings
:
- global velocities can now be assigned for each PCS line individually (need to be assigned in the
LineOnImage
objects directly).LocalPlumeProperties
assigned to retrieval lines (LineOnImage
objects) inEmissionRateAnalysis
are now considered for analysis when usingvelo_mode_farneback_histo
(e.g. they might be calculated and post processed beforehand).- New velocity method
farneback_hybrid
for velocity retrievals: uses optical flow output along line modified such that vectors that are not in expectation range (retrieved from histo analysis) are replaced by the average flow vector from the histo analysis.
- New attribute
residual
inDoasCalibData
- Fixed some bugs related to scale space conversion in
ImgList
objects (e.g. related to activation oftau_mode
, dilution correction) - Corrected bug related to SO2-uncertainty based on slope error of calibration curve from covariance matrix of poly fit. Previously: used value of slope error as measure of uncertainty (wrong), now: use relative error, e.g. calibration curve zero y-axis offset and with slope, slope err:
m=1e19, m_err=1e17
then the mapped SO2 error (for a given tau valuetau0
) is determined as :so2 = tau0 * m
andso2_err = so2 * m_err / m
- Added mathematical operators to
EmissionRateResults
class
- __add__: use “+” operator to add results (e.g. retrieved at two different lines from two crater emissions)
- __sub__: use “-” operator to subtract results (e.g. retrieved at two different positions downwind of the crater emissions)
10/04/2017¶
- Added option in
make_stack()
inImgList
objects: the method includes now the option to specify a reference ROI in the image (e.g. sky reference area) and a corresponding min / max range for the expectation value in that range: if the input is specified, then only images are added to the stack that are within the specified range within the ROI. - New features in
EmissionRateAnalysis
andEmissionRateSettings
- Added same feature (as described in 1.) to emission rate retrieval classes, relevant attributes in
EmissionRateSettings
class are:
ref_check_mode
: activate / deactivate the new modebg_roi_abs
(ROI used for check)ref_check_lower_lim
: lower intensity limitref_check_upper_lim
: upper intensity limit- Moved attr.
bg_roi
from analysis class to settings class and renamed tobg_roi_abs
.
11/04/2017¶
- Added check of date information in
get_img_meta_all_filenames()
ofImgList
which is, for instance, used for accessing datetime information of acq. times of all images in the list: a problem may occur if the file names only include information of acq. times of the images but not dates. Then, the retrieved timestamps (numpy array of datetime objects) will only include acq. times of all images and the default date: 1/1/1900. If this is the case, then the method replaces these default dates in the array using the date stored in the meta header of the currently loaded image in the the list. This is, for instance relevant for the HD default camera which includes date information in the tiff header (will be loaded and stored in meta header ofImg
class on load, but not in the file names).
12/04 - 04/05/2017¶
- Minor changes in plot style for standard outputs
- Worked on docs
04/05 - 21/05/2017 (v0.11.4 -> v0.12.0)¶
Note
Not downwards compatible change in fluxcalc.py
: changed name of velocity retrieval modes and functions related to optical flow from e.g. farneback_hybrid
to flow_hybrid
.
- Minor improvements in documentation of example scripts
- Changes in docs
- Minor changes in plot style for standard outputs
- DOAS calibration polynomial is now fitted only using mantissa of the CDs (to avoid large number warning in polyfit)
- Changes in optimisation strategy for optical flow histogram analysis and correction (modules:
plumespeed.py
,fluxcalc.py
)
- Minimum required length (per line and image is set at lower end of 1sigma of expectation interval of histo analysis
- More sophisticated uncertainty analysis for effective velocities
- Changed all names in
fluxcalc.py
related to optical flow based velocity retrievals which includedfarneback
toflow
(not downward compatible) - New class
EmissionRateRatio
influxcalc
22/05 - 29/08/2017 (v0.11.4 -> v0.12.0)¶
- Minor bug fixes
- Added functionality to
Img
objects - DOAS calibration data can now be fitted using weighted regression based on DOAS fit errors. Note, that new default is weighted fitting, if applicable (i.e. if uncertainties are available).
- New class
VeloCrossCorrEngine
inplumespeed.py
for high level computing of cross correlation based velocity retrievals. Note that this includes changes in example script 8, which now uses the new class. Thus, running the current version of example script 8 will not work with older versions of Pyplis. - Started with implementation of test suite using pytest
30/08 - 05/10/2017 (v0.12.0 -> v0.13.4)¶
- Minor bug fixes
- Improved convenience functionality of classes in
doascalib
by adding some @property decorators. - New high-level default method
run_fov_fine_search()
inDoasFOVEngine
- Renamed key vor wind velocity (and error) in
MeasGeometry
from “vel” to “velo” - New method
find_movement()
inplumespeed
. The method performs an iterative computation of the optical flow between two images under variation of the considered input brightness ranges. - Improved functionality for automated retrieval of sky-background pixels in an plume image (now uses new method
find_movement()
to identify and exclude pixels showing motion.
5/10/2017 - 25/11/2017 (v0.13.4 -> v1.0.0)¶
- Fixed some bugs
- Started with setting up a test-suite (available in the GitHub repo but not yet included in standard installation of the code)
- Added test-dataset of size reduced images from the Etna testdat (mainly for tests. This dataset is not yet included in the standard installation
# Automatic SRTM access can now be deactivated in MeasGeometry
objects
- Made MultiGaussFit optional for histogram post analysis of optical flow
- Removed requirement for
progressbar
- Changed colour and plot styles in some of the standard plotting methods (e.g. cross-correlation velocity)
- Improvements and new methods in
CellCalibData
objects (e.g. fitting of calibration curve, access to covariance matrix, slope error, calculation of uncertainties). - Renamed some methods
- Improvements in efficiency and new methods in
MeasGeometry
objects.
# New methods in helpers.py
- Minor changes to example scripts
- Major changes to
ImgList
objects
- New list mode:
dilution_corr
: images are loaded as dilution corrected images using the method from Campion et al., 2015. Can be activated and deactivated like all other modes (e.g.tau_mode
).- Updated all list methods related to signal dilution correction.
- @property decorators (and setters) for plume distance and integration step length, i.e.
plume_dists
andintegration_step_length
- Renamed
next_img()
andprev_img()
togoto_next()
andgoto_prev()
respectively (old names still work as well)
- Changes to
DoasFOV
:fov_mask
is now calledfov_mask_rel
. Renamedtransform_fov_mask_abs_coords()
tofov_mask_abs()
. EmissionRateAnalysis
can now also be run with settingdilcorr
using the newdilcorr_mode
ofImgList
objects (see above and example script 12).- Some new features in class
Img
(e.g.avg_in_roi()
, orerode()
).
Release 0.9.2 -> 0.11.2¶
28/02/2017 - 01/03/2017¶
- Allowing for defining custom image import method (file custom_image_import.py)
- Fixed bug regarding assignment of dark / offset lists for HD-Custom camera: if multiple type dark (and / or offset) image lists exist (based on camera file naming convention after file separation in Dataset), then the list with the shortest difference in the exposure time is set (using the first image of the respective image lists for intercomparison)
- Expanded handling of start / stop time specifications in Setup classes and Dataset classes (for initialisation of working environment) -> now, the user can also provide time stamps (datetime.time objects) or dates (datetime.date objects) as input and they will converted automatically to datetime. Also, if the default date is set (i.e. 1/1/1900) in a Setup, it will be disregarded and only the time stamps HHMMSS are considered to identify image files belonging to specified start /stop interval.
- Minor convenience changes in Dataeset enabling to set attributes and options of MeasSetup class directly from Dataset class using @property decorators (e.g. for start, stop, base_dir, etc.)
- Updated specs for HD-Custom camera such that cell calibration data can be imported
- Expanded functionality for dark and offset list assignments in Dataset and ImgList objects.
- Master dark / offset images are now searched for each dark / offset image list individually
- Customised assignment of dark / offset lists in image lists for cameras where meas type is specified in own filename substring (e.g. HD cam).
02/03/2017¶
- Included new default camera type “hd_new” (2. camera from group in Heidelberg, Germany). Currently missing detector and optics specs
- Expanded flexibility for meta information access via filename for acquisition time, meas_type and filter_id in Camera class: now, the conversion / identification strings can also include the actual delimiter (e.g. delim=”_”, time_info_pos - 0, time_info_str=”%Y%m%d_%H%M%S_%f” or filter_id_pos=3 and filter.acronym=”A_dark”). This is for instance required for file naming convention of new default SO2 camera type “hd_new”.
- Improved functionality for dark and offset image access in ImgList classes
- Improved data import speed in Dataset -> search of master_dark image is only applied to lists that actually include image data
03/03/2017¶
- Included image check for negative numbers or zeros after dark image correction and before tau / AA image calculation: correction is directly applied to images (no warning), i.e. pixels <- 0 are set to smallest positive float of system.
- Removed bugs regarding image time stamps in MeasSetup and image match search in Dataset (when specifying start / stop time stamps are provided as time object and not as datetime object). These two bugs resulted from changes applied in 0.9.3.dev1 (1/3/2017) and are irrelevant for previous versions.
05/03/2017¶
Note
Detected and fixed bug related to signal cross correlation based plume velocity retrievals after pandas updgrade from 0.16.2 -> 0.19.2.
06/03/2017¶
- Removed bug in
ImgStack
methodmerge_with_time_series
: generalised catch of first exception (applies ifdoas_series
is pandasSeries
object and not pydoasDoasResults
).
07/03/2017¶
- Improved performance in long “for” loops (e.g.
make_stack()
,get_mean_value()
inBaseImgList
or file searching methods inDataset
by removingself.
operations within the loops)EmissionRateResults
can now be saved as text file and has some new methods, the most important ones are:
__str__()
: informative string representationto_dict()
: converts results to dictionaryto_pandas_dataframe()
: converts object into pandasDataFrame
classfrom_pandas_dataframe()
: imports data from pandasDataFrame
classsave_txt()
: save results as text file
- Updated options for xlabel formatting when plotting time series
- Improved optical flow histogram analysis
- Renamed settings param
sigma_tol_mean_dir
tohist_dir_sigma
- New: choose from two options for retrieval of average displacement length from length histogram (in
get_main_flow_field_params()
ofOpticalFlowFarneback
):
- “argmax”: uses bin with largest count as mean displacement estimate (new)
- “multigauss”: tries to perform
MultiGaussFit
to data and if this fails, uses method “argmax”- new global settings parameters for maximum number of fitted gaussians in both orientation and length histogram, can now be set via
OpticalFlowFarnebackSettings
08/03/2017¶
- New functions in
Img
class:
to_binary
and corresponding entryis_bin
inedit_log`
dict.dilate
: apply morphological transform dilation to image using methodcv2.dilate
invert
: inverts an image object (added entry in edit_log)
- New method
get_mean_img
in image list classes: determines average image based on start / stop index (or for all images in list)- Removed bug in
Img
methodto_pyrlevel
for going up in pyramid
09/03/2017¶
- Class
Dataset
objects can now be initiated with differntImgList
types (e.g.CellCalibEngine
is now initiated withCellImgList
objects)- implementation of method to apply dilution correction to an plume image
correct_img
moved as global method indilutioncorr.py
and is now a wrapper inDilutionCorr
class.- New methods in
DilutionCorr
class:
get_ext_coeffs_imglist
: retrieve extinction coefficients for all images in anImgList
object.
13/03/2017¶
New functions in
ImgList
:
get_thresh_mask()
: get mask based on intensity threshold (e.g. tau thresh)prepare_bg_fit_mask()
: (BETA) for background modelling mode 0 (PolySurfaceFit). Determines a mask specifying background pixels based on intensities in reference rectangles and optical flow analysis (slow).correct_dilution()
: correct current image for signal dilutionset_bg_img_from_polyfit()
: determines an initial background image in list usingPolySurfaceFit`. The result is set as new ``bg_img
attribute, i.e. is used for backrgound modelling in modes 1-6. This can be done if no measured sky radiance image is available.correct_dilution()
: applies Dilution correction to current image if all requirements are fulfilled for that- start / stop indices can now be set in
make_stack()
Removed automatic load of previous image in
ImgList
objectsIncluded AA image calculation for CORR_MODE – 0 in
PlumeBackgroundModel
.Removed dark corr check between plume and BG image in
PlumeBackgroundModel
when modelling tau images.
14/03/2017¶
- Image dilution correction method (
correct_img()
,dilutioncorr
) can now also be called with a single floating point number specifying a constant plume distance (less accurate, but may be required for a fast check or if measurement geometry is not available)- New methods in
ImgList
:
prep_data_dilutioncorr()
: prepares relevant data for dilution correction (used e.g. incorrect_dilution()
)correct_dilution_all()
: corrects and saves all images in list for signal dilution (optionally also attached off band list)
15/03/2017¶
- NEW background correction mode (VALUE - 99) in
PlumeBackgroundModel
. In this mode, plume and background image are used as they are without any modifications. This mode may be used in case a plume background imagePlumeBackgroundModel
raisesAttributeError
in case plume and background image have different vignetting correction states.- Changed scaling of plume background (mode=0) to plume image such that it is done for an initial tau image fulfilling tau=0 in
scale_rect
(little faster)- Additional features in
ImgList
objects:
- Background modelling mode can now be set directly using
BG_MODEL_MODE
which takes care of changing the mode and directly reloads the images in the list.- New mode for background correction in lists: now, also a background list can be linked and assigned using
bg_list
. The background image access mode (from list vs. global BG image) can be set viawhich_bg
- Renamed method
apply_current_edit
to_apply_edit
inImgList
classes
16/03/2017 - 20/03/2017¶
Note
major changes related to optical flow histogram analysis. Retrieval of main flow field parameters using MultiGaussFit does not work at the moment. Will be fixed tomorrow in
- Improved performance and stability of optical flow histogram analysis by better controlling the number of bins and settings allowing for a threshold specifying the minimum number of significant vectors for retrieval of mean flow field parameters (see new parameter
min_count_frac
in optical flow settings class). LineOnImage
object can now also created rotated ROIs, i.e. rectangles aligned with the line orientation- Remove
prepare_intensity_condition_mask()
fromOpticalFlowFarneback
(it was causing more confustion than help). - Removed multigauss analysis of length histogram due to instability of the fit. The mean displacement length is now determined using mean and std of all pixels longer than
min_length
and which are pointing into the right direction. The latter is retrieved from multi Gauss fit applied to orientation histogram in specified ROI.
- removed
hist_len_how
inOpticalFlowFarnebackSettings
- removed
hist_len_gnum_max
inOpticalFlowFarnebackSettings
- removed
estimate_mean_len_argmax()
fromOpticalFlowFarneback
- New parameter
hist_dir_binres
in optical flow settings class: can be used to set the bin width in degrees for the fit of the flow orientation histogram (defaults to 10) - New parameter
roi_rad
in optical flow settings class: can be used to set the ROI used for setting min / max intensity range before calculating flow field (only relevant ifauto_update
is True) - Finalised and tested new retrieval of local optical flow parameters including rotated ROIs of multiple lines.
- Applied appropriate changes to all relevant example scripts.
- Improved plotting methods for optical flow histogram analysis, including new methods:
plot_orientation_histo()
plot_length_histo()
- Added features to
LocalPlumeProperties
, most important ones:
displacement_vectors
: array containing all displacement vectors of the time seriessignificance
: array containing information about the fraction of significant optical flow vectors within ROI used to retrieve the mean flow field parameters (can be used, e.g. for quality check and is included inplot()
function).plot_magnitudes()
: plots time series (+/- error) of displacement magnitudes.plot()
(Beta): plot 3 subplots showing detailed information of retrieval resultsto_dict()
: dictionary representation of objectfrom_dict()
: import data from dictionaryto_pandas_dataframe()
: convert object into pandas DataFrame objectfrom_pandas_dataframe()
: self explanatorysave_txt()
: save data as text fileload_txt()
: load data from text fileinterpolate()
: interpolate missing data points (uses pandas DataFrame interpolation method)dropna()
: remove missing data points (uses pandas DataFrame dropna method)apply_median_filter()
: applies median filter to dataapply_gaussian_filter()
: applies median filter to data
22/03/2017¶
- Added older versions of example scripts (for older pyplis versions)
- Renamed some output files of example scripts
- Added version check in all example scripts (raises error if version conflict occurs)
23/03/2017¶
- Added functionality to
ImgList
objects:
optflow_histo_analysis()
: performs optical flow histo analysis for all images in list and arbitrary number of PCS lines
- Renamed method ``get_main_flow_field_params` in
OpticalFlowFarneback
to ``local_flow_params` (old name still also works) - A colour can now be assigned on init (using kwargs) to
LocalPlumeProperties
- Renamed classes:
OpticalFlowFarnebackSettings
->FarnebackSettings
(old name still works but gives a warning on class initiation)OpticalFlowFarneback
->OptflowFarneback
(old name still works but gives a warning on class initiation)
24/03/2017¶
- Improved tau-preview plot in
PlumeBackgroundModel
- Added option
save_tau_prev
tocorrect_dilution_all()
inImgList
class: if True, then tau preview images are stored as well.
28-29/03/2017¶
- Reviewed and optimised fitting routine in
MultiGaussFit
- Changed retrieval of main peak params and analysis of magnitude histogram using moments of distributions