improver.utilities.compare module

improver.utilities.compare module#

Compare netcdf files using python-netCDF4 library.

This comparison is done using netCDF4 rather than iris so that it is not coupled to changes in netCDF representation across iris versions. It is also able to compare non-CF conventions compliant netCDF files that iris has difficulty loading.

Many functions in this module take an argument called ‘reporter’ which is a function to be called to report comparison differences. This provides flexibility regarding what action should be taken when differences are found. For example, the action to take could be to print the message, log the message, or raise an appropriate exception.

compare_attributes(name, actual_ds, desired_ds, ignored_attributes, reporter)[source]#

Compare attributes in a netCDF dataset/group.

Parameters:

  • name (str) – group name

  • actual_ds (Dataset) – dataset produced by test run

  • desired_ds (Dataset) – dataset considered good

  • ignored_attributes (Optional[List[str]]) – list of attributes to exclude from comparison.

  • reporter (Callable[[str], None]) – callback function for reporting differences

Return type:

None

compare_data(name, actual_var, desired_var, rtol, atol, reporter)[source]#

Compare attributes in a netCDF variable.

Parameters:

  • name (str) – variable name

  • actual_var (Variable) – variable produced by test run

  • desired_var (Variable) – variable considered good

  • rtol (float) – relative tolerance

  • atol (float) – absolute tolerance

  • reporter (Callable[[str], None]) – callback function for reporting differences

Return type:

None

compare_datasets(name, actual_ds, desired_ds, rtol, atol, exclude_vars, ignored_attributes, reporter)[source]#

Compare netCDF datasets. This function can call itself recursively to handle nested groups in netCDF4 files which are represented using the same Dataset class by python-netCDF4.

Parameters:

  • name (str) – group name

  • actual_ds (Dataset) – dataset produced by test run

  • desired_ds (Dataset) – dataset considered good

  • rtol (float) – relative tolerance

  • atol (float) – absolute tolerance

  • exclude_vars (List[str]) – variable names to exclude from comparison

  • ignored_attributes (Optional[List[str]]) – list of attributes to exclude from comparison.

  • reporter (Callable[[str], None]) – callback function for reporting differences

Return type:

None

compare_dims(name, actual_ds, desired_ds, exclude_vars, reporter)[source]#

Compare dimensions in a netCDF dataset/group.

Parameters:

  • name (str) – group name

  • actual_ds (Dataset) – dataset produced by test run

  • desired_ds (Dataset) – dataset considered good

  • exclude_vars (List[str]) – variable names to exclude from comparison

  • reporter (Callable[[str], None]) – callback function for reporting differences

Return type:

None

compare_netcdfs(actual_path, desired_path, rtol=0.0001, atol=0.0001, exclude_vars=None, ignored_attributes=None, reporter=None)[source]#

Compare two netCDF files.

Parameters:

  • actual_path (PathLike) – data file produced by test run

  • desired_path (PathLike) – data file considered good eg. KGO

  • rtol (float) – relative tolerance

  • atol (float) – absolute tolerance

  • exclude_vars (Optional[List[str]]) – variable names to exclude from comparison

  • ignored_attributes (Optional[List[str]]) – list of attributes to exclude from comparison.

  • reporter (Callable[[str], None]) – callback function for reporting differences

Return type:

None

compare_objects(actual_var, desired_var, reporter)[source]#

Compare two pickled objects. This is not a complete comparison as two objects may have the same string representation but be different objects.

Parameters:

  • actual_var (PathLike) – Path to the pickled object produced by test run.

  • desired_var (PathLike) – Path to the pickled object considered good.

  • reporter (Callable[[str], None]) – Callback function for reporting differences.

Return type:

None

compare_pickled_forest(output_path, kgo_path, reporter=None)[source]#

Load a pickled forest (e.g. a Random Forest) and compare its contents. :type output_path: PathLike :param output_path: data file produced by test run :type kgo_path: PathLike :param kgo_path: data file considered good e.g. KGO :type reporter: Optional[Callable[[str], None]] :param reporter: callback function for reporting differences

compare_vars(name, actual_ds, desired_ds, rtol, atol, exclude_vars, ignored_attributes, reporter)[source]#

Compare variables in a netCDF dataset/group.

Parameters:

  • name (str) – group name

  • actual_ds (Dataset) – dataset produced by test run

  • desired_ds (Dataset) – dataset considered good

  • rtol (float) – relative tolerance

  • atol (float) – absolute tolerance

  • exclude_vars (List[str]) – variable names to exclude from comparison

  • ignored_attributes (Optional[List[str]]) – list of attributes to exclude from comparison.

  • reporter (Callable[[str], None]) – callback function for reporting differences

Return type:

None