How to implement a command line utility

What is an IMPROVER CLI?

An IMPROVER command line interface (CLI) is a Python module located in improver/cli that uses clize to provide both a command line interface and a Python function interface. Clize automatically handles command line argument parsing, errors and help, including generated --help text based on the Python docstring.

When called as a Python function, input and output are via Iris cubes. When run from the command line, input and output are via netCDF files which are loaded and saved by Iris. These netCDF files contain a single cube with an ensemble-CDF related dimension as the first dimension - one of realization, percentiles, or probability over thresholds.

IMPROVER CLI utilities invoke one or more plugin classes and produce a result. For example, improver/cli/nbhood.py applies a neighbourhood processing plugin using code from improver/nbhood.py.

IMPROVER plugins

Plugins are Python classes with unit tests. See the Code style guide section on plugins for more information.

Command line testing

IMPROVER CLIs should have corresponding acceptance tests located in improver_tests/acceptance. These tests are written using the pytest test framework.

The point of acceptance tests is to provide reference test data to explain and test normal use of the CLI. These acceptance tests consist of reference NetCDF input files and a ‘known good output’ (KGO) NetCDF output file.

See the Running at your site page for information on how to run the acceptance tests.

Recreating acceptance test data

When a new test has been written and the input data has been put in place it is possible to generate the expected output:

export RECREATE_KGO=/path/to/new/kgos
pytest -m acc -k combine
unset RECREATE_KGO

The RECREATE_KGO path must be distinct from the input path (IMPROVER_ACC_TEST_DIR, see Running at your site). It is also possible to use this method to recreate all KGOs following changes that may affect them, for example modifications to metadata.

export RECREATE_KGO=/path/to/new/kgos
bin/improver-tests cli

In the singular or bulk case, it is important that the resulting KGO are sanity checked to ensure the test data remains sensible.

Acceptance test checksums

The input and output files for acceptance tests are identified by checksums listed in improver_tests/acceptance/SHA256SUMS. Changes to the checksum file in the code repository identify when changes are made to the code requiring corresponding changes to the data files. The acceptance test data files are maintained internally in a Met Office repository, with the directory structure being based on the CLI plugin names.

Minimising the usage of input and output data files is desirable in the future, as described in issue #1218.

Use of checksums

As stated above, the checksums are used to ensure the acceptance test data being used to test the code are the correct versions for the code revision being tested. To run all the acceptance tests use: bin/improver-tests cli

The first test to be run is improver_tests/acceptance/test_checksums.py which will fail if the test data and checksums do not correspond and stop any further tests. Running an individual test, e.g. pytest -m acc -k threshold will also compare the input data against the expected checksums and report if there is a mismatch.

It is possible to bypass the comparison of data against checksums using: export IMPROVER_IGNORE_CHECKSUMS=true

This may be useful when in the midst of making changes.

Updating acceptance test data checksums

Updating acceptance test data checksums is achieved using the command: bin/improver-tests recreate_checksums

The updated improver_tests/acceptance/SHA256SUMS file will need to be committed to the IMPROVER repository along with the code that has required / led to the modified test inputs / outputs.