Changelog

Automatically generated (and slightly modified) using:

git log --pretty=format:"- %ad, %aN%x09%s" --date=short v1.3.0..HEAD > changelog/CHANGELOG_v130_v143.rst

Release 1.3.0 -> 1.4.3

2019-05-22, jgliss Updated README
2019-05-22, jgliss Updated README
2019-05-22, jgliss Updated README
2019-05-22, jgliss Updated README
2019-05-22, jgliss Updated README
2019-05-22, jgliss Removed release note for version 1.3.0
2019-05-21, jgliss Fixed some formatting errors in README
2019-05-21, jgliss Updated version to 1.4.3, cleaned up and reformatted README so that it passes twine check and included Python 3 support into classifiers in setup.py
2019-05-21, jgliss Updated version to 1.4.2
2019-05-21, jgliss Removed / reformatted problematic parts from README.rst, so that it will be (hopefully) rendered properly in PyPi
2019-05-21, jgliss Removed pyplis logo from README (since it could not be rendered on PyPi) and updated version to 1.4.1
2019-05-21, jgliss Changed version to 1.4.0 (ready for release)
2019-05-21, jgliss Added LICENSE file to MANIFEST.in
2019-05-21, jgliss Refined tests performed in script ex06_doas_calib.py and accounted for minor differences in results between python 2.7 and 3.7
2019-05-21, jgliss Minor changes in output of radius search in doascalib.py
2019-05-21, jgliss Updated requirements file and added explicit conda environment file for python 2.7
2019-05-20, jgliss Updated README
2019-05-20, jgliss Updated README
2019-05-20, jgliss Added numpy and six to environment file
2019-05-20, jgliss Updated version
2019-05-20, jgliss Added pydoas to conda-forge dependency in pyplis_env.yml (no pip dependencies anymore)
2019-05-20, jgliss Added import of mpl_toolkits.mplot3d in 3d plotting routine of MeasGeometry
2019-05-20, jgliss Updated, simplified and merged two conda environment files for python 2 and 3 into new one that works for both: pyplis_env.yml
2019-05-20, jgliss Updated version and fixed one test in ex02_meas_geometry.py
2019-05-20, jgliss Merge branch ‘py3’
2019-05-12, jgliss Fixed some minor Deprecationwarnings
2019-05-12, jgliss Fixed failing test related to viewing direction correction of camera
2019-05-12, jgliss Fixed import error in geometry.py of geonum class TopoAccessError, due to recent API changes in geonum
2019-05-12, Jonas Merge pull request #23 from johannjacobsohn/fix-typos
2019-05-12, jgliss Fixed relative imports of geonum in example scripts; fixed failing test in ex06_doas_calib.py
2019-05-12, jgliss Added conda environment file for python 3 installation (not yet tested)
2019-04-23, Jonas Gliss Updated pyplis_env_py27.yml
2019-04-23, Jonas Gliss Updated README
2019-04-23, Jonas Gliss Updated README
2019-04-23, Jonas Gliss Updated README
2019-04-23, Jonas Gliss Updated VERSION to 1.4.0.dev1
2019-04-23, Jonas Gliss Updated installation instructions in README
2019-04-23, Jonas Gliss Updated script ex02_meas_geometry.py due to failing tests
2019-04-23, Jonas Gliss Added python 2.7 conda environment file pyplis_env_py27.yml
2019-04-22, jgliss Merged and updated README from py3 branch
2019-04-22, jgliss Updated geometry.py due to recent API updates in new geonum release v1.4.0
2019-04-22, jgliss Updated example scripts due to recent API changes in latest version 1.4.0 of geonum
2019-03-01, Jonas Update README.rst
2019-02-25, Johann Jacobsohn cleanup
2019-02-22, Johann Jacobsohn fixup use of depreciated methods and properties
2019-02-19, Johann Jacobsohn fixup typos
2019-02-19, Solvejg Dinger Fix broken link in Readme
2019-02-19, Solvejg Dinger Merge pull request #10 from heliotropium72/patch-1
2019-02-19, Solvejg Dinger Adapt suggestion for one-liner
2019-02-11, Jonas Gliss Refers to #13: Added new default camera usgs_9mm; the 12mm focal length version can now be accessed via the former ID usgs but also via usgs_12mm
2019-01-25, Solvejg Dinger Fixed rendering problem of README
2019-01-25, Solvejg Dinger Update README.rst
2019-01-22, Solvejg Dinger Merge pull request #16 from johannjacobsohn/py3
2019-01-17, Johann Jacobsohn WIP: fixup to make tests pass, needs to be investigated
2019-01-17, Johann Jacobsohn py3.7 doesn’t like __dir__ to be overwritten…
2019-01-17, Johann Jacobsohn add python 3.5 and 3.7 to tox, update pytest dependency and remove sitepackages from tox to make tests more consistent
2019-01-15, heliotropium72 Fix pytest fixtures to correct usage
2019-01-14, heliotropium72 Revert previous WIP
2019-01-02, jgliss Partly fixed failing tests
2018-12-27, jgliss Fixed bug arising from division returning float rather than int, due to recent Python 3 port
2018-11-29, Johann Jacobsohn Revert “skip failing test until fixed”
2018-11-29, Johann Jacobsohn WIP to pass tox scripts - needs to be reverted and fixed!
2018-11-29, Johann Jacobsohn add RUN_EXAMPLE_SCRIPTS.py to tox, use sitepackages for Basemap
2018-11-29, Johann Jacobsohn fix floordiv issue from python3 work, and cleanup
2018-11-28, Johann Jacobsohn protect possible None value against < operator, which fails in python3. Remove try..except block that only obscures exceptions
2018-11-28, Johann Jacobsohn remove __deepcopy__, underlying issue needs to be fixed in geonum
2018-11-28, Johann Jacobsohn make sure to use int for array indices
2018-11-28, Johann Jacobsohn numpy.sum does not handle .values() as you would expect in python3…
2018-11-28, Johann Jacobsohn truediv -> __truediv__, fix typo
2018-11-27, Johann Jacobsohn fix reading of cam_info.txt in python3
2018-11-27, Johann Jacobsohn try to load test data from envvar PYPLIS_DATADIR if not found in default location (helpful for testing)
2018-11-27, Johann Jacobsohn fixup: CalibData.senscorr_mask does actually not accecpt ndarray
2018-11-23, Johann Jacobsohn add RUN_INTRO_SCRIPTS.py to tox
2018-11-16, Johann Jacobsohn fix pytest to version 3
2018-11-16, Johann Jacobsohn fix difference in exception handling between python 2 and 3, restrict exceptions to KeyError
2018-11-14, Johann Jacobsohn skip failing test until fixed
2018-11-14, Johann Jacobsohn fixup requirements and tox environment
2018-11-14, Johann Jacobsohn fixup import syntax for py3
2018-11-14, Johann Jacobsohn fixup deepcopy of MeasGeometry in py3
2018-11-14, Johann Jacobsohn fix test_io in py3
2018-11-14, Johann Jacobsohn fix urlib import for python3
2018-11-14, Johann Jacobsohn small pydocstyle fixes
2018-11-14, Johann Jacobsohn __div__ -> truediv
2018-11-14, Johann Jacobsohn return value of .keys() and .values() have changed between python 2 and 3, this should make it consistent between the two
2018-11-14, Johann Jacobsohn add __future__ to help make python 2 and 3 behave consistent
2018-11-09, Johann Jacobsohn add pylint and disable every offending check
2018-11-05, Johann Jacobsohn scripts/ex07_doas_cell_calib.py: improve python3 compability, fix bugbear lint
2018-11-05, Johann Jacobsohn pyplis/utils.py: improve python3 compability, fix bugbear lint
2018-11-05, Johann Jacobsohn pyplis/setupclasses.py: improve python3 compability, fix bugbear lint
2018-11-05, Johann Jacobsohn pyplis/processing.py: improve python3 compability, fix bugbear lint
2018-11-05, Johann Jacobsohn pyplis/plumespeed.py: improve python3 compability, fix bugbear lint
2018-11-05, Johann Jacobsohn pyplis/plumebackground.py: improve python3 compability, fix bugbear lint
2018-11-05, Johann Jacobsohn pyplis/model_functions.py: improve python3 compability, fix bugbear lint
2018-11-05, Johann Jacobsohn pyplis/inout.py: improve python3 compability, fix bugbear lint
2018-11-05, Johann Jacobsohn pyplis/image.py: improve python3 compability, fix bugbear lint
2018-11-05, Johann Jacobsohn pyplis/imagelists.py: improve python3 compability, fix bugbear lint
2018-11-05, Johann Jacobsohn pyplis/geometry.py: improve python3 compability, fix bugbear lint
2018-11-05, Johann Jacobsohn pyplis/forms.py: improve python3 compability, fix bugbear lint
2018-11-05, Johann Jacobsohn pyplis/fluxcalc.py: improve python3 compability, fix bugbear lint
2018-11-05, Johann Jacobsohn pyplis/doascalib.py: improve python3 compability, fix bugbear lint
2018-11-05, Johann Jacobsohn pyplis/dilutioncorr.py: improve python3 compability, fix bugbear lint
2018-11-05, Johann Jacobsohn pyplis/dataset.py: improve python3 compability, fix bugbear lint
2018-11-05, Johann Jacobsohn pyplis/cellcalib.py: improve python3 compability, fix bugbear lint
2018-11-05, Johann Jacobsohn pyplis/calib_base.py: improve python3 compability, fix bugbear lint
2018-11-05, Johann Jacobsohn add bugbear lint to improve python 3 compability
2018-11-05, Johann Jacobsohn Light phrasing changes to conform to PEP 257 and pass pydocstyle validation
2018-06-01, Johann Jacobsohn lint pyplis/test/test_dataset_module.py
2018-05-25, Johann Jacobsohn lint scripts/more_scripts/ex002_plume_background_combi.py
2018-05-25, Johann Jacobsohn lint scripts/more_scripts/ex001_save_all_calib_imgs.py
2018-05-25, Johann Jacobsohn lint scripts/ex12_emission_rate.py
2018-05-25, Johann Jacobsohn lint scripts/ex11_signal_dilution.py
2018-05-25, Johann Jacobsohn lint scripts/ex10_bg_imglists.py
2018-05-25, Johann Jacobsohn lint scripts/ex0_7_cellcalib_manual.py
2018-05-25, Johann Jacobsohn lint scripts/ex0_6_pcs_lines.py
2018-05-25, Johann Jacobsohn lint scripts/ex0_5_optflow_livecam.py
2018-05-25, Johann Jacobsohn lint scripts/ex0_4_imglists_auto.py
2018-05-25, Johann Jacobsohn lint scripts/ex0_3_imglists_manually.py
2018-05-25, Johann Jacobsohn lint scripts/ex0_2_camera_setup.py
2018-05-25, Johann Jacobsohn lint scripts/ex0_1_img_handling.py
2018-05-25, Johann Jacobsohn lint scripts/ex09_velo_optflow.py
2018-05-25, Johann Jacobsohn lint scripts/ex08_velo_crosscorr.py
2018-05-25, Johann Jacobsohn lint scripts/ex07_doas_cell_calib.py
2018-05-25, Johann Jacobsohn lint scripts/ex06_doas_calib.py
2018-05-25, Johann Jacobsohn lint scripts/ex05_cell_calib_auto.py
2018-05-25, Johann Jacobsohn lint scripts/ex04_prep_aa_imglist.py
2018-05-25, Johann Jacobsohn lint scripts/ex03_plume_background.py
2018-05-25, Johann Jacobsohn lint scripts/ex02_meas_geometry.py
2018-05-25, Johann Jacobsohn lint scripts/ex01_analysis_setup.py
2018-05-25, Johann Jacobsohn lint scripts/SETTINGS.py
2018-05-25, Johann Jacobsohn lint scripts/RUN_INTRO_SCRIPTS.py
2018-05-25, Johann Jacobsohn lint scripts/RUN_EXAMPLE_SCRIPTS.py
2018-05-25, Johann Jacobsohn lint pyplis/utils.py
2018-05-25, Johann Jacobsohn lint pyplis/test/test_io.py
2018-05-25, Johann Jacobsohn lint pyplis/test/test_image_module.py
2018-05-25, Johann Jacobsohn lint pyplis/test/test_highlevel_examples.py
2018-05-25, Johann Jacobsohn lint pyplis/test/test_dataset_module.py
2018-05-25, Johann Jacobsohn lint pyplis/test/__init__.py
2018-05-25, Johann Jacobsohn lint pyplis/setupclasses.py
2018-05-25, Johann Jacobsohn lint pyplis/processing.py
2018-05-25, Johann Jacobsohn lint pyplis/plumespeed.py
2018-05-25, Johann Jacobsohn lint pyplis/plumebackground.py
2018-05-25, Johann Jacobsohn lint pyplis/optimisation.py
2018-05-25, Johann Jacobsohn lint pyplis/model_functions.py
2018-05-25, Johann Jacobsohn lint pyplis/inout.py
2018-05-25, Johann Jacobsohn lint pyplis/imagelists.py
2018-05-25, Johann Jacobsohn lint pyplis/image.py
2018-05-25, Johann Jacobsohn lint pyplis/helpers.py
2018-05-25, Johann Jacobsohn lint pyplis/glob.py
2018-05-25, Johann Jacobsohn lint pyplis/geometry.py
2018-05-25, Johann Jacobsohn lint pyplis/forms.py
2018-05-25, Johann Jacobsohn lint pyplis/fluxcalc.py
2018-05-25, Johann Jacobsohn lint pyplis/exceptions.py
2018-05-25, Johann Jacobsohn lint pyplis/doascalib.py
2018-05-25, Johann Jacobsohn lint pyplis/dilutioncorr.py
2018-05-25, Johann Jacobsohn lint pyplis/dataset.py
2018-05-25, Johann Jacobsohn lint pyplis/custom_image_import.py
2018-05-25, Johann Jacobsohn lint pyplis/cellcalib.py
2018-05-25, Johann Jacobsohn lint pyplis/calib_base.py
2018-11-01, Solvejg Dinger Bug Fix: Setting of calibration coefficients
2018-05-25, Johann Jacobsohn lint pyplis/__init__.py
2018-05-25, Johann Jacobsohn lint scripts/priv_01_plot_raw_vs_calib.py
2018-05-24, Johann Jacobsohn add python 3 to tox
2018-05-17, Johann Jacobsohn add tox to run tests
2018-09-09, jgliss Merge branch ‘heliotropium72-multi_layer_fits’
2018-09-09, jgliss fixed minor bug (download location of test data)#
2018-09-09, jgliss Fixed minor bug
2018-09-09, jgliss Merge branch ‘multi_layer_fits’ of https://github.com/heliotropium72/pyplis into heliotropium72-multi_layer_fits
2018-06-22, heliotropium72 Fix binary_mask fixture
2018-06-21, heliotropium72 Add sky_mask to image lists
2018-06-21, heliotropium72 Add class ImgListLayered
2018-06-21, heliotropium72 Make OptflowFarneback.calc_flow compatible with OpenCV 2.x and 3.x
2018-06-20, Jonas Merge pull request #8 from heliotropium72/patch-3
2018-06-20, Jonas Merge pull request #7 from heliotropium72/patch-2
2018-06-19, heliotropium72 Add get_masked_img method and corresponding tests
2018-06-18, Solvejg Dinger Merge pull request #11 from heliotropium72/patch-3
2018-06-18, Solvejg Dinger Fix indentation level
2018-06-18, Solvejg Dinger Merge pull request #10 from heliotropium72/patch-3
2018-06-18, Solvejg Dinger Merge pull request #9 from heliotropium72/patch-2
2018-06-07, Solvejg Dinger Add custom import method for Comtessa image data
2018-06-07, Solvejg Dinger Extended image.meta dictionary
2018-05-27, jgliss Fixed minor bug; added custom image read method template for QSI camera LMV group, France
2018-05-26, jgliss Updated formatting
2018-05-26, jgliss Updated formatting
2018-05-26, jgliss Deleted history of old binary files (repo size down to 30MB from ~500MB); Added content of scripts_out folder to .gitignore; updated README
2018-05-23, jgliss Removed strict PIL dependency for image read (now using opencv method imread)

Release 1.0.0 -> 1.0.1

This release includes only minor changes compared to the last one. These are mainly related to the access and handling / modelling of the plume background intensities in ImgList objects.

25/11/2017 - 12/01/2018 (v1.0.0 -> v1.0.1)

Fixed some bugs
Added more tests.
Improved access of plume background images in ImgList objects

Release 1.0.1 -> 1.3.0

Note

This release includes major API changes, performance improvements and bug fixes compared to version 1.0.1. Please update your installation as soon as possible.

Summary

Measurement geometry (MeasGeometry):

more accurate plume distance retrievals (i.e. now also in dependency of vertical distance).
redesigned API -> improved user-friendliness.

Image analysis: Image registration shift can now be applied to images.

shift() in class Img.
Comes with new mode (shift_mode) in ImgList objects.
Default on / off shift for camera can be set in Camera using attribute reg_shift_off (and correspondingly, in file cam_info.txt).

Camera calibration. Major improvements and API changes:

new abstraction layer (calib_base) including new calibration base class. CalibData: both DoasCalibData and CellCalibData are now inherited from new base class CalibData. Advantages and new features:
- arbitrary definition of calibration fit function.
- fitting of calibration curve, I/O (read / write FITS) and visualisation of DOAS and cell calibration data are now unified in CalibData.

Further changes

ImgStack more intuitive and flexible (e.g. dynamically expandable).
Improved index handling and performance of image list objects (imagelists).
PlumeBackgroundModel: revision, clean up and performance improvements.
Improved user-friendliness and performance of plume background retrieval in ImgList objects.
Correction for signal dilution (DilutionCorr): increased flexibility and user-friendliness.
Improved flexibility for image import using Dataset class (e.g. on / off images can be stored in the same file).
Reviewed and largely improved performance of general workflow (i.e. iteration over instances of ImgList in calib_mode, dilcorr_mode and optflow_mode).

Major bug fixes

Fixed conceptual error in cross-correlation algorithm for velocity retrieval (find_signal_correlation() in module plumespeed).
Fixed: ImgList in AA mode used current off-band image (at index idx_off) both for the current and next on-band image (and not idx_off+1).

1.0.1 -> 1.1.0

Img object

Included read / write of meta info dictionary for FITS load / save

New method is_darkcorr()

DOAS calibration (doascalib)

More flexible retrieval of DOAS calibration curves

ImgStack

new method ImgStack.sum()

Can now be dynamically extended (i.e. dynamic update of 3D array size). Corresponding API changes:

REMOVED: method append_img()

NEW methods: insert_img(), add_img(), init_stack_array()

Measurement geometry (MeasGeometry)

More accurate plume distance retrieval (now also in vertical direction, cf. Fig. 2 from example script 2)

Other changes

Moved Etna test data to new URL

Fixed bugs

1.1.0 -> 1.2.1

Img object

new method Img.sum()

Image list classes (imagelists, MAJOR API CHANGES)

Improved flexibility and clarity in index management

New attribute skip_files (i.e. load only every nth image from the filelist)

New method iter_indices()

Renamed method update_index_linked_lists() to change_index_linked_lists()

Removed method change_index()

Plume background retrieval (plumebackground)

Getter / setter for attr. surface_fit_mask (ensure it is type Img)

Changes related to I/O

Moved option `LINK_OFF_TO_ON from Dataset to BaseSetup (no API changes in Dataset)

New I/O option ON_OFF_SAME_FILE in BaseSetup that can be set if on and off images are stored in one (e.g. FITS) file (like for the new USGS CVO camera type)

I/O options for data import can now be specified in file cam_info.txt for each camera individually using keyword io_opts and is stored as dict in CameraBaseInfo (base class of Camera)

Included I/O info for camera of USGS CVO (uses previous point)

Source info can now be saved automatically to file my_sources.txt

Other changes

New method matlab_datenum_to_datetime() in helpers

Fixed bugs

1.2.1 -> 1.3.0

Note

This version includes major refactoring and changes in API, aiming for more transparency and intuitive design. For instance, both the DoasCalibData and CellCalibData now inherit from a new base class CalibData (in new module calib_base).

Note

Changes related to camera calibration API (e.g. renaming, refactoring or removing of methods) are not resolved in full detail below (following point 1.).

Camera calibration

NEW MODULE calib_base containing new calibration base class CalibData (both DoasCalibData and CellCalibData inherit from this base object)

MAIN CHANGES associated with with refactoring into general base class CalibData

NEW FEATURE: Fit function for calibration data (both cell and DOAS) can now be defined arbitrarily (before, only polynomials were possible). See also module model_functions, in particular new class CalibFuns

I/O (.e.g to / from FITS, or csv) are now unified for cell and DOAS calibration

Visualisation (e.g. plot of calibration curve and data) now unified for cell and DOAS calibration

New default fit function based on Kern et al. 2015

UNCERTAINTY treatment: Error in calibrated CDs is now computed based on the standard deviation of fit residual (if more than 10 datapoints are available for retrieval of calibration curve).

Img object

Renamed attribute “alt_offset” -> “altitude_offset”

Moved custom import for ECII camera into new custom method load_ecII_fits() in module custom_image_import

New attributes:

is_cropped

is_resized

is_shifted

New methods:

shift() (applies x/y pixel shift of image)

convolve_with_mask(), for instance, when applied to an AA image, the input mask may be, e.g. a parameterised DOAS FOV (e.g. fitted 2D super-Gauss). The function then returns the weighted average AA within the FOV.

get_thresh_mask()

Image list classes (imagelists)

New list mode shift_mode (only for offband lists, i.e. lists with attribute type="off"): activate / deactivate shift (dx, dy) of images on image load (cf. other list modes, such as tau_mode, calib_mode, optflow_mode). If activated, the default shift reg_shift_off of the assigned Camera instance is used (is set (0, 0) if not explicitly defined (either in file cam_info.txt for a camera type (cf. cam “usgs” therein) or in instance of Camera directly).

Reviewed and optimised:

correct_dilution() reviewed, largely rewritten and optimised

New attributes:

update_cam_geodata (default is False). If True, the measurement geometry (i.e. plume distance) is automatically updated if image files contain camera geodata (e.g. lat, lon, viewing direction).

New methods:

calc_plumepix_mask() (for dilution correction)

timestamp_to_index() (returns list index corresponding to a datetime object)

_iter_num() (number of iterations to loop through the whole list, resulting from the total number of files nof and skip_files)

pop() now raises NotImplementedError

Introduced @property methods (getter / setter) for the attributes skip_files (newly introduced in v1.2.1, see above) and edit_active to ensure index update and reload (on change)

Further changes, deprecated, renamed

Introduced new input parameter reload_here in goto_img() (if True, load() is called even if the new index is the same as the current index, defaults to False)

Deprecated:

Removed attribute which_bg (now handled automatically by @property attribute bg_img)

Renamed

aa_corr_mask -> senscorr_mask

DARK_CORR_OPT -> darkcorr_opt

Bug fixes:

Fixed: on-band list AA mode used current off-band image (at index idx_off) both for the current and next on-band image (and not idx_off+1).

Measurement geometry (MeasGeometry, MAJOR API CHANGES)

Improved user-friendliness and performance: getter / setter methods for all attributes

Intended access / modification of attributes is via new getter / setter methods (e.g. `geom.cam["lon"] -> geom.cam_lon)

Comes with better handling of recomputation requirements of geometry in case individual parameters (e.g. camera viewing direction, position, wind direction) are updated (in this context, note new attribute update_cam_geodata in ImgList objects). Specifically:

Method update_geosetup() is called whenever a relevant attribute is updated via the corresponding setter method. This ensures, that derived values such as plume distance are always up-to-date with the current attributes.

Attribute dictionaries now private (e.g. .cam -> ._cam, .source -> ._source).

New methods:

get_topo_distance_pix() (determines distance to local topography in viewing direction of individual image pixel)

PlumeBackgroundModel (Review and clean-up)

New attribute last_tau_image

New method _init_bgsurf_mask(): initiate mask for 2D background polynomial surface fit (only relevant for correction mode mode=0)

Removed

dictionary _current_imgs: kept copies of input images (private dictionary)

Methods: get_current(), pyrlevel(), current_plume_background(), subtract_tau_offset(), _prep_img_type(), set_current_images(), plot_tau_result_old()

DilutionCorr

Retrieval of extinction coefficients for dilution correction based on dark terrain features can now also be performed for individual pixel coordinates in the images, in addition to the distance retrieval based on lines in the images (see example script 11)

New methods:

add_retrieval_point()

add_retrieval_line()

Module model_functions:

New calibration fit function(s) based on Kern et al., 2015

New class CalibFuns for access of calibration fit functions

Plume velocity retrievals (plumespeed.py)

Cross correlation method (find_signal_correlation())

Improved retrieval robustness: introduced percentage max shift that describes the maximum shift in percent of the second relative to the first time-series based on the total length of both series.

Fixed systematic retrieval error: Before, the second signal was rolled over the first, meaning, that the “end” of the 2. signal was attached to it’s beginning and thus, correlated with the beginning of the first signal. That behaviour has been resolved.

Optical flow (OptflowFarneback and FarnebackSettings)

i_min (lower end of contrast range for optical flow calculation) can now also be smaller than 0.

I/O and setup classes (modules inout and setupclasses)

my_pyplis folder is now created on installation (in user home directory)

includes copies of cam_info.txt file and my_sources.txt

New method save_default_source() in inout (is saved in file my_sources.txt)

New method save_to_database() in Source (wrapper method for save_default_source())

New I/O option REG_SHIFT_OFF in classes BaseSetup and MeasSetup: if True (and if image lists are created using Dataset and corresponding MeasSetup object), then, the off-band images (in off-band ImgList) are automatically shifted to on-band images (in on-band ImgList) using the registration shift that is specified in Camera.reg_shift_off (can be set in file cam_info.txt)

Other changes

New method integrate_profile() in class LineOnImage

New method make_circular_mask() in module helpers.py

In fluxcalc (and all included classes): renamed attr cd_err_rel to cd_err (note changes in uncertainty treatment of calibration data!)

EmissionRateSettings: new option / attribute min_cd_flow (in addition to already existing min_cd) that may be used to explicitly define the minimum column-density of an image pixel for it to be considered valid with respect to optical flow histogram analysis (before, the threshold min_cd was used). Is set equal min_cd if not explicitly specified

Moved class LineOnImage into module utils

Moved method model_dark_image() from processing to image as well as class ProfileTimeSeriesImg

Changed input parameter of model_dark_image() in processing

Changed default colormap for optical density (and calibrated) images from bwr to viridis (perceptually uniform)

Major performance improvements: reviewed typical workflow chain and removed irrelevant duplications of image arrays in certain objects ()

Fixed bugs

Included new tests (test suite still very low coverage…)

Release 0.11.2 -> 1.0.0

30-31/03/2017

Renamed roi_rad in Farneback classes to roi_rad_abs (makes it clearer in which coordinates it is supposed to be defined). The old name also still works but a warning is given if used.
Renamed hist_dir_sigma in Farneback classes to hist_sigma_tol since it is applied both to the fit result of the main peak of the orientation histogram but also to determine the uncertainty in the length histogram from the moments analysis. The old name also still works but a warning is given if used.
More features in LineOnImage

Global velocity estimates (and uncertainties) can now be assigned (e.g. for emission rate analysis)

LocalPlumeProperties can now be assigned (e.g. for emission rate analysis and if a time series of local displacement vectors for velocity retrieval was calculated beforehand and the results are accessible in a text file)).

01-07/04/2017

Added features in LocalPlumeProperties

Interpolation can now also be performed onto index array of other time series (e.g. image list time stamps)

New method apply_significance_thresh() in LocalPlumeProperties: can be used to remove data points with significance lower than provided thresh (and combined e.g. with interpolation).

New method to_pyrlevel(): converts displacement lengths (and errors) to a given Gauss pyramid level.

Expanded functionality in classes EmissionRateAnalysis and EmissionRateSettings:

global velocities can now be assigned for each PCS line individually (need to be assigned in the LineOnImage objects directly).

LocalPlumeProperties assigned to retrieval lines (LineOnImage objects) in EmissionRateAnalysis are now considered for analysis when using velo_mode_farneback_histo (e.g. they might be calculated and post processed beforehand).

New velocity method farneback_hybrid for velocity retrievals: uses optical flow output along line modified such that vectors that are not in expectation range (retrieved from histo analysis) are replaced by the average flow vector from the histo analysis.

New attribute residual in DoasCalibData
Fixed some bugs related to scale space conversion in ImgList objects (e.g. related to activation of tau_mode, dilution correction)
Corrected bug related to SO2-uncertainty based on slope error of calibration curve from covariance matrix of poly fit. Previously: used value of slope error as measure of uncertainty (wrong), now: use relative error, e.g. calibration curve zero y-axis offset and with slope, slope err: m=1e19, m_err=1e17 then the mapped SO2 error (for a given tau value tau0) is determined as :so2 = tau0 * m and so2_err = so2 * m_err / m
Added mathematical operators to EmissionRateResults class

__add__: use “+” operator to add results (e.g. retrieved at two different lines from two crater emissions)

__sub__: use “-” operator to subtract results (e.g. retrieved at two different positions downwind of the crater emissions)

10/04/2017

Added option in make_stack() in ImgList objects: the method includes now the option to specify a reference ROI in the image (e.g. sky reference area) and a corresponding min / max range for the expectation value in that range: if the input is specified, then only images are added to the stack that are within the specified range within the ROI.
New features in EmissionRateAnalysis and EmissionRateSettings

Added same feature (as described in 1.) to emission rate retrieval classes, relevant attributes in EmissionRateSettings class are:

ref_check_mode: activate / deactivate the new mode

bg_roi_abs (ROI used for check)

ref_check_lower_lim: lower intensity limit

ref_check_upper_lim: upper intensity limit

Moved attr. bg_roi from analysis class to settings class and renamed to bg_roi_abs.

11/04/2017

Added check of date information in get_img_meta_all_filenames() of ImgList which is, for instance, used for accessing datetime information of acq. times of all images in the list: a problem may occur if the file names only include information of acq. times of the images but not dates. Then, the retrieved timestamps (numpy array of datetime objects) will only include acq. times of all images and the default date: 1/1/1900. If this is the case, then the method replaces these default dates in the array using the date stored in the meta header of the currently loaded image in the the list. This is, for instance relevant for the HD default camera which includes date information in the tiff header (will be loaded and stored in meta header of Img class on load, but not in the file names).

12/04 - 04/05/2017

Minor changes in plot style for standard outputs
Worked on docs

04/05 - 21/05/2017 (v0.11.4 -> v0.12.0)

Note

Not downwards compatible change in fluxcalc.py: changed name of velocity retrieval modes and functions related to optical flow from e.g. farneback_hybrid to flow_hybrid.

Minor improvements in documentation of example scripts
Changes in docs
Minor changes in plot style for standard outputs
DOAS calibration polynomial is now fitted only using mantissa of the CDs (to avoid large number warning in polyfit)
Changes in optimisation strategy for optical flow histogram analysis and correction (modules: plumespeed.py, fluxcalc.py)

Minimum required length (per line and image is set at lower end of 1sigma of expectation interval of histo analysis

More sophisticated uncertainty analysis for effective velocities

Changed all names in fluxcalc.py related to optical flow based velocity retrievals which included farneback to flow (not downward compatible)
New class EmissionRateRatio in fluxcalc

22/05 - 29/08/2017 (v0.11.4 -> v0.12.0)

Minor bug fixes
Added functionality to Img objects
DOAS calibration data can now be fitted using weighted regression based on DOAS fit errors. Note, that new default is weighted fitting, if applicable (i.e. if uncertainties are available).
New class VeloCrossCorrEngine in plumespeed.py for high level computing of cross correlation based velocity retrievals. Note that this includes changes in example script 8, which now uses the new class. Thus, running the current version of example script 8 will not work with older versions of pyplis.
Started with implementation of test suite using pytest

30/08 - 05/10/2017 (v0.12.0 -> v0.13.4)

Minor bug fixes
Improved convenience functionality of classes in doascalib by adding some @property decorators.
New high-level default method run_fov_fine_search() in DoasFOVEngine
Renamed key vor wind velocity (and error) in MeasGeometry from “vel” to “velo”
New method find_movement() in plumespeed. The method performs an iterative computation of the optical flow between two images under variation of the considered input brightness ranges.
Improved functionality for automated retrieval of sky-background pixels in an plume image (now uses new method find_movement() to identify and exclude pixels showing motion.

5/10/2017 - 25/11/2017 (v0.13.4 -> v1.0.0)

Fixed some bugs
Started with setting up a test-suite (available in the GitHub repo but not yet included in standard installation of the code)
Added test-dataset of size reduced images from the Etna testdat (mainly for tests. This dataset is not yet included in the standard installation

# Automatic SRTM access can now be deactivated in MeasGeometry objects

Made MultiGaussFit optional for histogram post analysis of optical flow
Removed requirement for progressbar
Changed colour and plot styles in some of the standard plotting methods (e.g. cross-correlation velocity)
Improvements and new methods in CellCalibData objects (e.g. fitting of calibration curve, access to covariance matrix, slope error, calculation of uncertainties).
Renamed some methods
Improvements in efficiency and new methods in MeasGeometry objects.

# New methods in helpers.py

Minor changes to example scripts
Major changes to ImgList objects

New list mode: dilution_corr: images are loaded as dilution corrected images using the method from Campion et al., 2015. Can be activated and deactivated like all other modes (e.g. tau_mode).

Updated all list methods related to signal dilution correction.

@property decorators (and setters) for plume distance and integration step length, i.e. plume_dists and integration_step_length

Renamed next_img() and prev_img() to goto_next() and goto_prev() respectively (old names still work as well)

Changes to DoasFOV: fov_mask is now called fov_mask_rel. Renamed transform_fov_mask_abs_coords() to fov_mask_abs().
EmissionRateAnalysis can now also be run with setting dilcorr using the new dilcorr_mode of ImgList objects (see above and example script 12).
Some new features in class Img (e.g. avg_in_roi(), or erode()).

Release 0.9.2 -> 0.11.2

28/02/2017 - 01/03/2017

Allowing for defining custom image import method (file custom_image_import.py)

Fixed bug regarding assignment of dark / offset lists for HD-Custom camera: if multiple type dark (and / or offset) image lists exist (based on camera file naming convention after file separation in Dataset), then the list with the shortest difference in the exposure time is set (using the first image of the respective image lists for intercomparison)

Expanded handling of start / stop time specifications in Setup classes and Dataset classes (for initialisation of working environment) -> now, the user can also provide time stamps (datetime.time objects) or dates (datetime.date objects) as input and they will converted automatically to datetime. Also, if the default date is set (i.e. 1/1/1900) in a Setup, it will be disregarded and only the time stamps HHMMSS are considered to identify image files belonging to specified start /stop interval.

Minor convenience changes in Dataeset enabling to set attributes and options of MeasSetup class directly from Dataset class using @property decorators (e.g. for start, stop, base_dir, etc.)

Updated specs for HD-Custom camera such that cell calibration data can be imported

Expanded functionality for dark and offset list assignments in Dataset and ImgList objects.

Master dark / offset images are now searched for each dark / offset image list individually

Customised assignment of dark / offset lists in image lists for cameras where meas type is specified in own filename substring (e.g. HD cam).

02/03/2017

Included new default camera type “hd_new” (2. camera from group in Heidelberg, Germany). Currently missing detector and optics specs

Expanded flexibility for meta information access via filename for acquisition time, meas_type and filter_id in Camera class: now, the conversion / identification strings can also include the actual delimiter (e.g. delim=”_”, time_info_pos - 0, time_info_str=”%Y%m%d_%H%M%S_%f” or filter_id_pos=3 and filter.acronym=”A_dark”). This is for instance required for file naming convention of new default SO2 camera type “hd_new”.

Improved functionality for dark and offset image access in ImgList classes

Improved data import speed in Dataset -> search of master_dark image is only applied to lists that actually include image data

03/03/2017

Included image check for negative numbers or zeros after dark image correction and before tau / AA image calculation: correction is directly applied to images (no warning), i.e. pixels <- 0 are set to smallest positive float of system.

Removed bugs regarding image time stamps in MeasSetup and image match search in Dataset (when specifying start / stop time stamps are provided as time object and not as datetime object). These two bugs resulted from changes applied in 0.9.3.dev1 (1/3/2017) and are irrelevant for previous versions.

05/03/2017

Note

Detected and fixed bug related to signal cross correlation based plume velocity retrievals after pandas updgrade from 0.16.2 -> 0.19.2.

06/03/2017

Removed bug in ImgStack method merge_with_time_series: generalised catch of first exception (applies if doas_series is pandas Series object and not pydoas DoasResults).

07/03/2017

Improved performance in long “for” loops (e.g. make_stack(), get_mean_value() in BaseImgList or file searching methods in Dataset by removing self. operations within the loops)

EmissionRateResults can now be saved as text file and has some new methods, the most important ones are:

__str__(): informative string representation

to_dict(): converts results to dictionary

to_pandas_dataframe(): converts object into pandas DataFrame class

from_pandas_dataframe(): imports data from pandas DataFrame class

save_txt(): save results as text file

Updated options for xlabel formatting when plotting time series

Improved optical flow histogram analysis

Renamed settings param sigma_tol_mean_dir to hist_dir_sigma

New: choose from two options for retrieval of average displacement length from length histogram (in get_main_flow_field_params() of OpticalFlowFarneback):

“argmax”: uses bin with largest count as mean displacement estimate (new)

“multigauss”: tries to perform MultiGaussFit to data and if this fails, uses method “argmax”

new global settings parameters for maximum number of fitted gaussians in both orientation and length histogram, can now be set via OpticalFlowFarnebackSettings

08/03/2017

New functions in Img class:

to_binary and corresponding entry is_bin in edit_log` dict.

dilate: apply morphological transform dilation to image using method cv2.dilate

invert: inverts an image object (added entry in edit_log)

New method get_mean_img in image list classes: determines average image based on start / stop index (or for all images in list)

Removed bug in Img method to_pyrlevel for going up in pyramid

09/03/2017

Class Dataset objects can now be initiated with differnt ImgList types (e.g. CellCalibEngine is now initiated with CellImgList objects)

implementation of method to apply dilution correction to an plume image correct_img moved as global method in dilutioncorr.py and is now a wrapper in DilutionCorr class.

New methods in DilutionCorr class:

get_ext_coeffs_imglist: retrieve extinction coefficients for all images in an ImgList object.

13/03/2017

New functions in ImgList:

get_thresh_mask(): get mask based on intensity threshold (e.g. tau thresh)

prepare_bg_fit_mask(): (BETA) for background modelling mode 0 (PolySurfaceFit). Determines a mask specifying background pixels based on intensities in reference rectangles and optical flow analysis (slow).

correct_dilution(): correct current image for signal dilution

set_bg_img_from_polyfit(): determines an initial background image in list using PolySurfaceFit`. The result is set as new ``bg_img attribute, i.e. is used for backrgound modelling in modes 1-6. This can be done if no measured sky radiance image is available.

correct_dilution(): applies Dilution correction to current image if all requirements are fulfilled for that

start / stop indices can now be set in make_stack()

Removed automatic load of previous image in ImgList objects

Included AA image calculation for CORR_MODE – 0 in PlumeBackgroundModel.

Removed dark corr check between plume and BG image in PlumeBackgroundModel when modelling tau images.

14/03/2017

Image dilution correction method (correct_img(), dilutioncorr) can now also be called with a single floating point number specifying a constant plume distance (less accurate, but may be required for a fast check or if measurement geometry is not available)

New methods in ImgList:

prep_data_dilutioncorr(): prepares relevant data for dilution correction (used e.g. in correct_dilution())

correct_dilution_all(): corrects and saves all images in list for signal dilution (optionally also attached off band list)

15/03/2017

NEW background correction mode (VALUE - 99) in PlumeBackgroundModel. In this mode, plume and background image are used as they are without any modifications. This mode may be used in case a plume background image

PlumeBackgroundModel raises AttributeError in case plume and background image have different vignetting correction states.

Changed scaling of plume background (mode=0) to plume image such that it is done for an initial tau image fulfilling tau=0 in scale_rect (little faster)

Additional features in ImgList objects:

Background modelling mode can now be set directly using BG_MODEL_MODE which takes care of changing the mode and directly reloads the images in the list.

New mode for background correction in lists: now, also a background list can be linked and assigned using bg_list. The background image access mode (from list vs. global BG image) can be set via which_bg

Renamed method apply_current_edit to _apply_edit in ImgList classes

16/03/2017 - 20/03/2017

Note

major changes related to optical flow histogram analysis. Retrieval of main flow field parameters using MultiGaussFit does not work at the moment. Will be fixed tomorrow in

Improved performance and stability of optical flow histogram analysis by better controlling the number of bins and settings allowing for a threshold specifying the minimum number of significant vectors for retrieval of mean flow field parameters (see new parameter min_count_frac in optical flow settings class).
LineOnImage object can now also created rotated ROIs, i.e. rectangles aligned with the line orientation
Remove prepare_intensity_condition_mask() from OpticalFlowFarneback (it was causing more confustion than help).
Removed multigauss analysis of length histogram due to instability of the fit. The mean displacement length is now determined using mean and std of all pixels longer than min_length and which are pointing into the right direction. The latter is retrieved from multi Gauss fit applied to orientation histogram in specified ROI.

removed hist_len_how in OpticalFlowFarnebackSettings

removed hist_len_gnum_max in OpticalFlowFarnebackSettings

removed estimate_mean_len_argmax() from OpticalFlowFarneback

New parameter hist_dir_binres in optical flow settings class: can be used to set the bin width in degrees for the fit of the flow orientation histogram (defaults to 10)
New parameter roi_rad in optical flow settings class: can be used to set the ROI used for setting min / max intensity range before calculating flow field (only relevant if auto_update is True)
Finalised and tested new retrieval of local optical flow parameters including rotated ROIs of multiple lines.
Applied appropriate changes to all relevant example scripts.
Improved plotting methods for optical flow histogram analysis, including new methods:

plot_orientation_histo()

plot_length_histo()

Added features to LocalPlumeProperties, most important ones:

displacement_vectors: array containing all displacement vectors of the time series

significance: array containing information about the fraction of significant optical flow vectors within ROI used to retrieve the mean flow field parameters (can be used, e.g. for quality check and is included in plot() function).

plot_magnitudes(): plots time series (+/- error) of displacement magnitudes.

plot() (Beta): plot 3 subplots showing detailed information of retrieval results

to_dict(): dictionary representation of object

from_dict(): import data from dictionary

to_pandas_dataframe(): convert object into pandas DataFrame object

from_pandas_dataframe(): self explanatory

save_txt(): save data as text file

load_txt(): load data from text file

interpolate(): interpolate missing data points (uses pandas DataFrame interpolation method)

dropna(): remove missing data points (uses pandas DataFrame dropna method)

apply_median_filter(): applies median filter to data

apply_gaussian_filter(): applies median filter to data

22/03/2017

Added older versions of example scripts (for older pyplis versions)
Renamed some output files of example scripts
Added version check in all example scripts (raises error if version conflict occurs)

23/03/2017

Added functionality to ImgList objects:

optflow_histo_analysis(): performs optical flow histo analysis for all images in list and arbitrary number of PCS lines

Renamed method ``get_main_flow_field_params` in OpticalFlowFarneback to ``local_flow_params` (old name still also works)
A colour can now be assigned on init (using kwargs) to LocalPlumeProperties
Renamed classes:

OpticalFlowFarnebackSettings -> FarnebackSettings (old name still works but gives a warning on class initiation)

OpticalFlowFarneback -> OptflowFarneback (old name still works but gives a warning on class initiation)

24/03/2017

Improved tau-preview plot in PlumeBackgroundModel
Added option save_tau_prev to correct_dilution_all() in ImgList class: if True, then tau preview images are stored as well.

28-29/03/2017

Reviewed and optimised fitting routine in MultiGaussFit
Changed retrieval of main peak params and analysis of magnitude histogram using moments of distributions