improver.calibration package

Submodules

• improver.calibration.dataframe_utilities module
  • Ingestion of DataFrames into iris cubes
    • Forecast DataFrame
    • Truth DataFrame
  • _dataframe_column_check()
  • _define_height_coord()
  • _define_time_coord()
  • _drop_duplicates()
  • _ensure_consistent_static_cols()
  • _fill_missing_entries()
  • _prepare_dataframes()
  • _preprocess_temporal_columns()
  • _quantile_check()
  • _training_dates_for_calibration()
  • _unique_check()
  • forecast_and_truth_dataframes_to_cubes()
  • forecast_dataframe_to_cube()
  • get_forecast_representation()
  • truth_dataframe_to_cube()
• improver.calibration.dz_rescaling module
  • ApplyDzRescaling
    • ApplyDzRescaling.__init__()
    • ApplyDzRescaling._abc_impl
    • ApplyDzRescaling._check_mismatched_sites()
    • ApplyDzRescaling._create_forecast_period_constraint()
    • ApplyDzRescaling._create_forecast_reference_time_constraint()
    • ApplyDzRescaling.process()
  • EstimateDzRescaling
    • EstimateDzRescaling.__init__()
    • EstimateDzRescaling._abc_impl
    • EstimateDzRescaling._compute_scaled_dz()
    • EstimateDzRescaling._compute_scaled_dz_cube()
    • EstimateDzRescaling._create_hour_coord()
    • EstimateDzRescaling._fit_polynomial()
    • EstimateDzRescaling.process()
• improver.calibration.ensemble_calibration module
  • Ensemble Model Output Statistics (EMOS)
    • Estimating EMOS coefficients
      • Distribution description
      • What is the location parameter?
      • What is the scale parameter?
    • Implementation details
    • Estimating EMOS coefficients using the ensemble mean
    • Estimating EMOS coefficients using the ensemble realizations
    • Applying EMOS coefficients
  • ApplyEMOS
    • ApplyEMOS.__init__()
    • ApplyEMOS._abc_impl
    • ApplyEMOS._check_additional_field_sites()
    • ApplyEMOS._convert_to_realizations()
    • ApplyEMOS._format_forecast()
    • ApplyEMOS._get_attribute()
    • ApplyEMOS._get_input_forecast_type()
    • ApplyEMOS.process()
  • CalibratedForecastDistributionParameters
    • CalibratedForecastDistributionParameters.__init__()
    • CalibratedForecastDistributionParameters._abc_impl
    • CalibratedForecastDistributionParameters._calculate_location_parameter_from_mean()
    • CalibratedForecastDistributionParameters._calculate_location_parameter_from_realizations()
    • CalibratedForecastDistributionParameters._calculate_scale_parameter()
    • CalibratedForecastDistributionParameters._create_output_cubes()
    • CalibratedForecastDistributionParameters._diagnostic_match()
    • CalibratedForecastDistributionParameters._spatial_domain_match()
    • CalibratedForecastDistributionParameters.process()
  • ContinuousRankedProbabilityScoreMinimisers
    • ContinuousRankedProbabilityScoreMinimisers.BAD_VALUE
    • ContinuousRankedProbabilityScoreMinimisers.TOLERATED_PERCENTAGE_CHANGE
    • ContinuousRankedProbabilityScoreMinimisers.__init__()
    • ContinuousRankedProbabilityScoreMinimisers._abc_impl
    • ContinuousRankedProbabilityScoreMinimisers._calculate_percentage_change_in_last_iteration()
    • ContinuousRankedProbabilityScoreMinimisers._minimise_caller()
    • ContinuousRankedProbabilityScoreMinimisers._normal_crps_preparation()
    • ContinuousRankedProbabilityScoreMinimisers._prepare_forecasts()
    • ContinuousRankedProbabilityScoreMinimisers._process_points_independently()
    • ContinuousRankedProbabilityScoreMinimisers._process_points_together()
    • ContinuousRankedProbabilityScoreMinimisers.calculate_normal_crps()
    • ContinuousRankedProbabilityScoreMinimisers.calculate_truncated_normal_crps()
    • ContinuousRankedProbabilityScoreMinimisers.process()
  • EstimateCoefficientsForEnsembleCalibration
    • EstimateCoefficientsForEnsembleCalibration.__init__()
    • EstimateCoefficientsForEnsembleCalibration._abc_impl
    • EstimateCoefficientsForEnsembleCalibration._add_predictor_coords()
    • EstimateCoefficientsForEnsembleCalibration._create_cubelist()
    • EstimateCoefficientsForEnsembleCalibration._create_temporal_coordinates()
    • EstimateCoefficientsForEnsembleCalibration._get_spatial_associated_coordinates()
    • EstimateCoefficientsForEnsembleCalibration._set_attributes()
    • EstimateCoefficientsForEnsembleCalibration._validate_distribution()
    • EstimateCoefficientsForEnsembleCalibration.compute_initial_guess()
    • EstimateCoefficientsForEnsembleCalibration.create_coefficients_cubelist()
    • EstimateCoefficientsForEnsembleCalibration.guess_and_minimise()
    • EstimateCoefficientsForEnsembleCalibration.mask_cube()
    • EstimateCoefficientsForEnsembleCalibration.process()
• improver.calibration.rainforest_calibration module
  • RainForests calibration
    • Sub-grid variability as a means of calibration
    • The RainForests method
      • GBDT vs manually constructed DT
    • Implementation details
      • Model training
      • Forecast calibration
  • ApplyRainForestsCalibration
    • ApplyRainForestsCalibration._abc_impl
    • ApplyRainForestsCalibration._check_num_features()
    • ApplyRainForestsCalibration._get_feature_splits()
    • ApplyRainForestsCalibration._get_num_features()
    • ApplyRainForestsCalibration._parse_model_config()
    • ApplyRainForestsCalibration.check_filenames()
    • ApplyRainForestsCalibration.process()
  • ApplyRainForestsCalibrationLightGBM
    • ApplyRainForestsCalibrationLightGBM.__init__()
    • ApplyRainForestsCalibrationLightGBM._abc_impl
    • ApplyRainForestsCalibrationLightGBM._align_feature_variables()
    • ApplyRainForestsCalibrationLightGBM._calculate_threshold_probabilities()
    • ApplyRainForestsCalibrationLightGBM._evaluate_probabilities()
    • ApplyRainForestsCalibrationLightGBM._get_ensemble_distributions()
    • ApplyRainForestsCalibrationLightGBM._get_num_features()
    • ApplyRainForestsCalibrationLightGBM._make_decreasing()
    • ApplyRainForestsCalibrationLightGBM._prepare_features_array()
    • ApplyRainForestsCalibrationLightGBM._prepare_threshold_probability_cube()
    • ApplyRainForestsCalibrationLightGBM.process()
  • ApplyRainForestsCalibrationTreelite
    • ApplyRainForestsCalibrationTreelite.__init__()
    • ApplyRainForestsCalibrationTreelite._abc_impl
    • ApplyRainForestsCalibrationTreelite._get_num_features()
• improver.calibration.reliability_calibration module
  • AggregateReliabilityCalibrationTables
    • AggregateReliabilityCalibrationTables._abc_impl
    • AggregateReliabilityCalibrationTables._check_frt_coord()
    • AggregateReliabilityCalibrationTables.process()
  • ApplyReliabilityCalibration
    • ApplyReliabilityCalibration.__init__()
    • ApplyReliabilityCalibration._abc_impl
    • ApplyReliabilityCalibration._apply_calibration()
    • ApplyReliabilityCalibration._apply_point_by_point_calibration()
    • ApplyReliabilityCalibration._calculate_reliability_probabilities()
    • ApplyReliabilityCalibration._ensure_monotonicity_across_thresholds()
    • ApplyReliabilityCalibration._extract_matching_reliability_table()
    • ApplyReliabilityCalibration._interpolate()
    • ApplyReliabilityCalibration.process()
  • ConstructReliabilityCalibrationTables
    • ConstructReliabilityCalibrationTables.__init__()
    • ConstructReliabilityCalibrationTables._abc_impl
    • ConstructReliabilityCalibrationTables._add_reliability_tables()
    • ConstructReliabilityCalibrationTables._create_probability_bins_coord()
    • ConstructReliabilityCalibrationTables._create_reliability_table_coords()
    • ConstructReliabilityCalibrationTables._create_reliability_table_cube()
    • ConstructReliabilityCalibrationTables._define_metadata()
    • ConstructReliabilityCalibrationTables._define_probability_bins()
    • ConstructReliabilityCalibrationTables._populate_masked_reliability_bins()
    • ConstructReliabilityCalibrationTables._populate_reliability_bins()
    • ConstructReliabilityCalibrationTables.process()
  • ManipulateReliabilityTable
    • ManipulateReliabilityTable.__init__()
    • ManipulateReliabilityTable._abc_impl
    • ManipulateReliabilityTable._assume_constant_observation_frequency()
    • ManipulateReliabilityTable._combine_bin_pair()
    • ManipulateReliabilityTable._combine_undersampled_bins()
    • ManipulateReliabilityTable._create_new_bin_coord()
    • ManipulateReliabilityTable._enforce_min_count_and_montonicity()
    • ManipulateReliabilityTable._extract_reliability_table_components()
    • ManipulateReliabilityTable._sum_pairs()
    • ManipulateReliabilityTable._update_reliability_table()
    • ManipulateReliabilityTable.process()
• improver.calibration.simple_bias_correction module
  • ApplyBiasCorrection
    • ApplyBiasCorrection.__init__()
    • ApplyBiasCorrection._abc_impl
    • ApplyBiasCorrection._check_forecast_bias_consistent()
    • ApplyBiasCorrection._get_mean_bias()
    • ApplyBiasCorrection.process()
  • CalculateForecastBias
    • CalculateForecastBias.__init__()
    • CalculateForecastBias._abc_impl
    • CalculateForecastBias._create_bias_cube()
    • CalculateForecastBias._define_metadata()
    • CalculateForecastBias._ensure_single_valued_forecast()
    • CalculateForecastBias.process()
  • apply_additive_correction()
  • evaluate_additive_error()
• improver.calibration.utilities module
  • _ceiling_fp()
  • broadcast_data_to_time_coord()
  • check_data_sufficiency()
  • check_forecast_consistency()
  • check_predictor()
  • convert_cube_data_to_2d()
  • create_unified_frt_coord()
  • filter_non_matching_cubes()
  • flatten_ignoring_masked_data()
  • forecast_coords_match()
  • get_frt_hours()
  • merge_land_and_sea()

Module contents

init for calibration that contains functionality to split forecast, truth and coefficient inputs.

add_warning_comment(forecast)

Add a comment to warn that calibration has not been applied.

Parameters:

forecast (Cube) – The forecast to which a comment will be added.

Return type:

Cube

Returns:

Forecast with an additional comment.

split_forecasts_and_coeffs(cubes, land_sea_mask_name=None)

Split the input forecast, coefficients, static additional predictors, land sea-mask and probability template, if provided. The coefficients cubes and land-sea mask are identified based on their name. The static additional predictors are identified as not have a time coordinate. The current forecast and probability template are then split.

Parameters:

• cubes (CubeList) – A list of input cubes which will be split into relevant groups. This includes the forecast, coefficients, static additional predictors, land-sea mask and probability template.

• land_sea_mask_name (Optional[str]) – Name of the land-sea mask cube to help identification.

Returns:

• A cube containing the current forecast.

• If found, a cubelist containing the coefficients else None.

• If found, a cubelist containing the static additional predictor else None.

• If found, a land-sea mask will be returned, else None.

• If found, a probability template will be returned, else None.

Raises:

• ValueError – If multiple items provided, when only one is expected.

• ValueError – If no forecast is found.

split_forecasts_and_truth(cubes, truth_attribute)

A common utility for splitting the various inputs cubes required for calibration CLIs. These are generally the forecast cubes, historic truths, and in some instances a land-sea mask is also required.

Parameters:

• cubes (List[Cube]) – A list of input cubes which will be split into relevant groups. These include the historical forecasts, in the format supported by the calibration CLIs, and the truth cubes.

• truth_attribute (str) – An attribute and its value in the format of “attribute=value”, which must be present on truth cubes.

Return type:

Tuple[Cube, Cube, Optional[Cube]]

Returns:

• A cube containing all the historic forecasts.

• A cube containing all the truth data.

• If found within the input cubes list a land-sea mask will be returned, else None is returned.

Raises:

• ValueError – An unexpected number of distinct cube names were passed in.

• IOError – More than one cube was identified as a land-sea mask.

• IOError – Missing truth or historical forecast in input cubes.

validity_time_check(forecast, validity_times)

Check the validity time of the forecast matches the accepted validity times within the validity times list.

Parameters:

• forecast (Cube) – Cube containing the forecast to be calibrated.

• validity_times (List[str]) – Times at which the forecast must be valid. This must be provided as a four digit string (HHMM) where the first two digits represent the hour and the last two digits represent the minutes e.g. 0300 or 0315. If the forecast provided is at a different validity time then no coefficients will be applied.

Return type:

bool

Returns:

If the validity time within the cube matches a validity time within the validity time list, then True is returned. Otherwise, False is returned.