improver.calibration.stochastic_noise module

Plugin for adding stochastic noise to a cube using Short-Space Fourier Transform (SSFT).

class StochasticNoise(ssft_init_params=None, ssft_generate_params=None, db_threshold=0.03, db_threshold_units='mm/hr', scale_non_positive_noise=False, allow_seeded_parallel_processing=False, arbitrary_offset=5.0, wet_noise_floor=None, dry_fallback_range=None)

Bases: BasePlugin

Class to apply spatially-structured stochastic noise (randomly generated noise with specific statistical properties) to non-positive regions of a field, building on the Short-Space Fourier Transform (SSFT) approach from Nerini et al. (2017), as implemented in the pySTEPS library.

This plugin is intended for use with positive zero-bounded diagnostics only, and is a particularly useful tool for Ensemble Copula Coupling-Quantile (ECC-Q) realization generation. While ECC-Q is used to improve the accuracy of forecasts by calibrating ensemble members to better represent the true distribution of the forecast variable, the rank-based reordering (sorting) of ensemble members at each grid point can lead to unrealistic individual members (e.g. single-pixel precipitation artifacts) when multiple raw ensemble members have identical values (‘ties’) of zero (very common in precipitation forecasts) and the post-processed calibrated probabilities indicate a non-zero value should occur. By adding spatially-structured stochastic noise to break ties in these non-positive regions, more realistic spatial structures can be generated in the final ECC-Q realizations, while still respecting the calibrated probabilities.

__init__(ssft_init_params=None, ssft_generate_params=None, db_threshold=0.03, db_threshold_units='mm/hr', scale_non_positive_noise=False, allow_seeded_parallel_processing=False, arbitrary_offset=5.0, wet_noise_floor=None, dry_fallback_range=None)

Initialise the plugin. For a typical input field e.g. a precipitation field with some positive values for precipitation spread across the domain and some zero values, the plugin will add stochastic noise to the zero values using the SSFT approach, while leaving the positive values unchanged. For fields that contain insufficient spatial variability to derive meaningful SSFT perturbations (for example completely dry, nearly dry, or otherwise near-constant fields), referred to here as degenerate fields, the plugin will generate fallback stochastic noise (“dry fallback noise”) in linear space. This noise uses the wet_noise_floor and dry_fallback_range arguments to ensure that the fallback noise is strictly non-positive and does not exceed the noise added to wet regions.

If ssft_init_params or ssft_generate_params are not provided, default values from the Pysteps documentation will be used.

Parameters:

• ssft_init_params (Optional[dict]) – Keyword arguments for initializing SSFT filter using pysteps.noise.fftgenerators.initialize_nonparam_2d_ssft_filter. Default is an empty dict, which will use the pysteps defaults.

• ssft_generate_params (Optional[dict]) – Keyword arguments for generating stochastic noise using pysteps.noise.fftgenerators.generate_noise_2d_ssft_filter. Default is an empty dict, which will use the pysteps defaults.

• db_threshold (float) – Threshold value below which data will be set to a constant in dB scale to avoid issues with log(0). Value provided in units of db_threshold_units. Default is 0.03 mm/hr.

• db_threshold_units (str) – Units of the db_threshold value. Default is “mm/hr”.

• scale_non_positive_noise (bool) – If True, noise in non-positive regions (where template.data <= 0) will be scaled such that the maximum noise value in those regions is zero and all other noise values are negative. This prevents the addition of positive noise to non-positive regions, which could artificially increase values where the input cube indicates no signal should occur. If this is true, wet_noise_floor must be set, so that totally dry fields do not receive noise that exceeds noise given to fields that are wet. Default is False.

• allow_seeded_parallel_processing (bool) – If True, allows multiple workers to be used even when a seed is provided in ssft_generate_params. This may improve computation speed, but can introduce run-to-run variation because pySTEPS uses global RNG seeding. If False, seeded runs are forced to a single worker for reproducibility. Default is False.

• arbitrary_offset (float) – An arbitrary offset value to add to the dB values of sub-threshold pixels. This is used to ensure that all sub-threshold pixels have a distinct value in dB space, which allows them to be handled appropriately in the _from_dB method. The default value of 5 was chosen to provide a clear separation from the threshold value in dB space, but can be adjusted if needed.

• wet_noise_floor (Optional[float]) – Optional lower bound for noise in non-positive regions after scaling, in linear units of db_threshold_units. Must be negative if set. This can be used to limit the magnitude of negative SSFT-derived wet-member noise. This value must be less than the SSFT-derived noise in wet regions. Any generated noise below the floor value will be set to the floor value, potentially resulting in more ties when used in conjunction with Ensemble Copula Coupling. Default is None (no floor).

• dry_fallback_range (Optional[tuple]) – Optional range (min_value, max_value) for dry fallback noise in linear units of db_threshold_units. Provide as a Python tuple string, e.g. “(-10.0, -5.0)”. Both values must be <= 0 and (min_value < max_value). If wet_noise_floor is set and this is not provided, this defaults to (2 * wet_noise_floor, wet_noise_floor) to keep dry fallback below the wet floor. If wet_noise_floor is set and dry_fallback_range is provided, the max_value of dry_fallback_range must be <= wet_noise_floor to ensure separation between dry-fallback and wet noise ranges.

Raises:

• ValueError – If db_threshold is not a positive value.

• ValueError – If wet_noise_floor is provided and is non-negative.

• ValueError – If wet_noise_floor is provided while scale_non_positive_noise is False.

• ValueError – If dry_fallback_range does not contain exactly two values.

• ValueError – If dry_fallback_range does not satisfy min_value < max_value <= 0.

• ValueError – If both wet_noise_floor and dry_fallback_range are provided and dry_fallback_range max exceeds wet_noise_floor.

Warning

If a seed is provided in ssft_generate_params and allow_seeded_parallel_processing is True, a warning is raised to indicate that using multiple workers with a fixed seed may introduce run-to-run variation because pySTEPS uses global RNG seeding.

Example dictionaries for initializing and generating SSFT filter:

ssft_init_params = {"win_size": (100, 100), "overlap": 0.3, "war_thr": 0.1}
ssft_generate_params = {"overlap": 0.3, "seed": 0}

See Pysteps documentation for further keyword arguments.

_abc_impl = <_abc._abc_data object>

_fallback_noise_linear(shape)

Generate strictly non-positive fallback noise in linear space.

If a seed is configured in ssft_generate_params, this returns reproducible noise. The resulting field has a maximum value slightly below zero so dry fields remain dry while still receiving tie-break noise.

Parameters:

shape (tuple) – Target 2-D output shape.

Return type:

ndarray

Returns:

Fallback 2-D noise field in linear units.

_from_dB(data)

Convert cube data from dB scale with thresholding.

Function based on dB_transform function (with arg inverse=True) from pySTEPS/pysteps.

Parameters:

data (ndarray) – data in dB scale.

Return type:

ndarray

Returns:

np.ndarray with data converted from dB scale to original scale. Note: After conversion to original scale, values below the threshold are set to zero.

static _is_degenerate_field(data)

Return True if field has no dynamic range for SSFT initialisation.

Return type:

bool

_process_single_realization(input_cube)

Add stochastic noise to a cube containing a single realization (or no realization coord). For non-degenerate fields e.g. precipitation fields with some positive values, the plugin will add stochastic noise to the non-positive regions using the SSFT approach, while leaving the positive values unchanged. For degenerate fields (for example completely dry, nearly dry, or otherwise near-constant fields), fallback noise is generated in linear space.

Parameters:

input_cube (Cube) – Cube to which stochastic noise will be added.

Return type:

Cube

Returns:

Cube with added stochastic noise.

Raises:

ValueError – If a degenerate field is detected for SSFT initialisation and wet_noise_floor has not been configured (which means no default dry_fallback_range is available).

Warns:

UserWarning – If a degenerate field is detected for SSFT initialisation, or if SSFT initialisation fails for any reason, a warning is raised to indicate that linear fallback stochastic noise generation will be used instead.

_to_dB(cube)

Convert cube data to dB scale and apply thresholding using db_threshold specified in the plugin initialization.

Function based on dB_transform function (with arg inverse=False) from pySTEPS/pysteps.

Parameters:

cube (Cube) – Cube containing data to be converted to dB scale.

Return type:

Cube

Returns:

Cube with data converted from linear scale to dB scale.

do_fft(data)

Generate stochastic noise using SSFT for a 2-D array slice (one realization).

This may raise ValueError if individual windows within the field are degenerate (constant-valued), even if the overall field has variation. In such cases, the caller should fall back to linear noise generation.

Parameters:

data (ndarray) – 2D array for which stochastic noise is to be added.

Returns:

2D array of generated stochastic noise.

Return type:

np.ndarray

process(input_cube)

Add locally-conditioned stochastic noise to a cube object using Short-Space Fourier Transform (SSFT).

While this plugin accepts any cube with “x” and “y” dimensions, it is recommended to first slice the cube over the realization dimension and parallelize the processing of individual realizations using the plugin on each slice, to improve performance. This extraction and later merging of realization slices can be easily achieved using the improver CLI extract and merge functionality, respectively.

Parameters:

input_cube (Cube) – Cube to which stochastic noise will be added. Must contain “x” and “y” dimensions, and may optionally contain a “realization” dimension.

Return type:

Cube

Returns:

Cube with added stochastic noise.

Warning

If the input cube contains a “realization” dimension, a warning is raised to indicate that processing will be slower than necessary, and that it is recommended to process each realization separately.