deode.tasks package

Tasks submodule.

Submodules

deode.tasks.archive module

ARCHIVEHOUR // ARCHIVESTATIC.

class ArchiveFDB(config)

Bases: ArchiveTask

Archving task for time dependent data dedicated for FDB.

class ArchiveHour(config)

Bases: ArchiveTask

Archving task for time dependent data.

class ArchiveMergedSQLites(config)

Bases: ArchiveTask

Archving task for time dependent data.

class ArchiveStatic(config)

Bases: ArchiveTask

Archving task for static data.

class ArchiveStaticMember(config)

Bases: ArchiveTask

Archving task for static data.

class ArchiveTask(config, datatype=None, include=None, exclude=None)

Bases: Task

Archving data.

execute()

Loops over archive choices.

deode.tasks.base module

Base site class.

class Task(config, name)

Bases: object

Base Task class.

archive_logs(files, target=None)

Archive files in a log directory.

Parameters:

• files (str,list) – File(s) to be archived

• target (str) – Target directory for archiving

change_to_wdir()

Change to task working dir.

create_wdir()

Create task working directory and check for unix group and set permissions.

create_wrkdir()

Create a cycle working directory.

execute()

Do nothing for base execute task.

get_binary(binary_name)

Determine binary path from task or system config section.

Parameters:

binary_name (str) – Name of binary

Returns:

full path to binary

Return type:

bindir (str)

get_task_setting(setting)

Get task setting.

Parameters:

setting (str) – Setting to find in task.{self.name}

Returns:

Found setting

Return type:

value

post(source=None, target=None)

Do default postfix.

E.g. clean

prep()

Do default preparation before execution.

E.g. clean

remove_wdir(wdir=None)

Remove working directory.

rename_wdir(prefix='Failed_task_', source=None, target=None)

Rename failed/completed working directory.

run()

Run task.

Define run sequence.

class UnitTest(config)

Bases: Task

Base Task class.

deode.tasks.batch module

Batch process.

class BatchJob(rte, wrapper='')

Bases: object

Batch job.

run(cmd)

Run command.

Parameters:

cmd (str) – Command to run.

Raises:

• TypeError – If the provided command is not a string

• CalledProcessError – Execution error

deode.tasks.c903 module

C903.

class C903(config)

Bases: Task

C903, preform interpolation from IFS to IAL LAM.

execute()

Run task.

Define run sequence.

deode.tasks.canari module

Canari.

class Canari(config)

Bases: Task

Canari task.

execute()

Execute task.

deode.tasks.clean_old_data module

Clean old data.

class CleanEhypeData(config)

Bases: CleanOldData

Clean old data from scratch.

execute()

Run clean data from scratch.

class CleanIFSData(config)

Bases: CleanOldData

Clean IFS data.

execute()

Run clean IFS data.

class CleanOldData(config)

Bases: Task

Interface class to the cleaning of the old data.

cutoff(delay)

Calculates the time to determine the old data.

Parameters:

delay (int) – delay in days

Returns:

time in seconds

Return type:

cutoff (int)

get_old(path, format_, cutoff_time, ignore=None)

Get directories which are older than cutoff.

Parameters:

• path (str) – Path to start of searching old directories

• format (str) – regex for path

• cutoff_time (int) – Directories or files older then cutoff

• ignore (list) – List of directories to ignore

Returns:

List of directories for remove

Return type:

list_to_remove (list)

remove_list(dir_list, files=False)

Remove directories/files from the list.

Parameters:

• dir_list (list) – Dictionary of directories/files

• files (boolean) – If in list are files, Default False

class CleanScratchData(config)

Bases: CleanOldData

Clean old data from scratch.

execute()

Run clean data from scratch.

class CleanSuites(config)

Bases: CleanOldData

Clean old suites.

execute()

Run clean suites.

deode.tasks.cleaning_tasks module

Clean deode file systems.

class Cleaning(config)

Bases: Task

Interface class to the cleaning.

execute()

Run the cleaning.

prep_clean_task(cleaning_type)

Setup clean task.

Parameters:

cleaning_type (str) – Cleaning config section identifier

class CycleCleaning(config)

Bases: Cleaning

Cycle cleaning task.

class PostMortem(config)

Bases: Cleaning

Final cleaning task.

deode.tasks.collectlogs module

CollectLogs.

class CollectLogs(config, config_label='staticlogs')

Bases: Task

Collect task and scheduler logfiles and store them as tarfiles.

execute()

Execute collect logs .

scan_logs(tarlog, parent, target, pattern='', exclude='')

Search for files matching a pattern and add them to a tar file.

Parameters:

• tarlog (tarfile object) – Tar file used

• parent (str) – Top search directory

• target (str) – Main search directory

• pattern (str) – Optional search pattern

• exclude (str) – Optional string for files to be excluded

Raises:

FileNotFoundError – If parent directory does not exist

class CollectLogsHour(config)

Bases: CollectLogs

Collectlog task for static data.

class CollectLogsStatic(config)

Bases: CollectLogs

Collectlog task for static data.

deode.tasks.creategrib module

CreateGrib.

class CreateGrib(config)

Bases: GlGrib

Convert time dependent files.

class CreateGribStatic(config)

Bases: GlGrib

Convert static files.

class GlGrib(config, name='')

Bases: Task

Create grib files.

convert2grib(infile, outfile, filetype)

Convert FA to grib.

Namelist arguments are given in the creategrib.NAME config part per filetype :param infile: Input file :type infile: str :param outfile: Output file :type outfile: str :param filetype: filetype to look for in rules :type filetype: str

Raises:

FileNotFoundError – Could not find file

create_list(input_template, output_settings)

Create list of files to process.

execute()

Execute creategrib.

deode.tasks.discover_task module

Discover tasks.

available_tasks(reg)

Create a list of available tasks.

Parameters:

reg (DeodePluginRegistry) – Deode plugin registry

Returns:

Task objects

Return type:

known_types (list)

discover(package, base)

Discover task classes.

Plugin classes are discovered in a given namespace package, deriving from a given base class. The base class itself is ignored, as are classes imported from another module (based on cls.__module__). Each discovered class is identified by the class name by changing it to lowercase and stripping the name of the base class, if it appears as a suffix.

Parameters:

• package (types.ModuleType) – Namespace package containing the plugins

• base (type) – Base class for the plugins

Returns:

type): Discovered plugin classes

Return type:

(dict of str

discover_modules(package, what='plugin')

Discover plugin modules.

Parameters:

• package (types.ModuleType) – Namespace package containing the plugins

• what (str, optional) – String describing what is supposed to be discovered. Defaults to “plugin”.

Yields:

tuple – str: Name of the imported module types.ModuleType: The imported module

get_task(name, config) → Task

Create a deode.tasks.Task object from configuration.

Parameters:

• name (_type_) – _description_

• config (_type_) – _description_

Raises:

NotImplementedError – If task name is not amongst the known task names.

Returns:

The task object with name name. The task object has to be a subclass of Task to be retrievable.

Return type:

Task

deode.tasks.e923 module

E923.

class E923(config)

Bases: Task

Methods for the e923 work.

constant_part(constant_file)

Run the constant part of e923.

Parameters:

constant_file – filename of the resulting file

monthly_part(constant_file)

Run the monthly part of e923.

Parameters:

constant_file – filename of the input constant file

myexec(cmd, i)

Execute binary task.

Parameters:

• cmd (str) – Command to run

• i (int) – sequence number

print_part(part, month=None)

Run the constant part of e923.

Parameters:

• part – which step

• month – month

remove_links(link)

Remove link.

Parameters:

link (list) – List of links to remove

class E923Constant(config)

Bases: E923

E923Constant task.

execute()

Run task.

Define run sequence.

class E923Monthly(config)

Bases: E923

E923Monthly task.

execute()

Run task.

Define run sequence.

class PgdUpdate(config)

Bases: Task

PgdUpdate.

Ensure consistency in orography between the atmosphere and surfex Read gridpoint orography from the atmospheric file and insert it in the PGD file Const.Clim.sfx.

execute()

Run task.

deode.tasks.e923_update module

E923Update.

class E923Update(config)

Bases: Task

Methods for updating climfile for ALARO.

For mor information see doc/markdown/e923_update.md.

execute()

Run task.

Define run sequence.

deode.tasks.e927 module

E927.

class E927(config)

Bases: Task

E927 task.

execute()

Run task.

Define run sequence.

deode.tasks.forecast module

Forecast.

class FirstGuess(config)

Bases: Task

FirstGuess.

execute()

Find initial file.

class Forecast(config)

Bases: Task

Forecast task.

accelerator_device_input()

Copy the input files for GPU execution.

• parallel_method: input file with parallelisation technique for each algorithm

• synchost: input file defining optional device-to-host memory transfers

• select_gpu: wrapper file for sbatch, binding GPU to MPI rank

archive_output(filetype, periods)

Archive forecast model output.

Parameters:

• filetype (str) – Filename template

• periods (str) – Output list

execute()

Execute forecast.

merge_output(filetype, periods)

Merge distributed forecast model output.

Parameters:

• filetype (str) – File type (history, surfex, fullpos)

• periods (str) – Output list

Final result is the expected output file name in the working directory (as if there was no IO server). NOTE: This function has been replaced by io_merge tasks. It is only called if n_io_merge=0.

post()

Do special post for Foreast.

sst_sic_input(intp_bddir)

Link SST & SIC input file to forecast directory.

wfp_input()

Add wind turbine files to forecast directory.

class PrepareCycle(config)

Bases: Task

Task.

deode.tasks.gmtedsoil module

GMTED and SOILGRID.

class Gmted(config)

Bases: Task

GMTED.

define_gmted_input(domain_properties: dict) → tuple

Define GMTED input files.

Parameters:

domain_properties (dict) – Domain properties

Returns:

GMTED input files

Return type:

tuple

execute()

Run task.

Define run sequence.

get_domain_properties(config) → dict

Get domain properties.

Parameters:

config (Config) – Config object

Returns:

Domain properties

Return type:

dict

gmted_header_coordinates(east: float, west: float, south: float, north: float) → tuple

Get GMTED header coordinates.

Parameters:

• east (float) – East

• west (float) – West

• south (float) – South

• north (float) – North

Returns:

Header coordinates

Return type:

tuple

static tif2bin(gd, bin_file) → None

Convert tif file to binary file used by surfex.

Parameters:

• gd – gdal dataset

• bin_file (str) – Binary file

static write_gmted_header_file(header_file, hdr_north, hdr_south, hdr_west, hdr_east, hdr_rows, hdr_cols) → None

Write header file.

Parameters:

• header_file (str) – Header file

• hdr_north (float) – North

• hdr_south (float) – South

• hdr_west (float) – West

• hdr_east (float) – East

• hdr_rows (int) – Number of rows

• hdr_cols (int) – Number of columns

class Soil(config)

Bases: Task

Prepare soil data task for PGD.

static check_domain_validity(domain_properties: dict) → None

Check if domain is valid.

Parameters:

domain_properties (dict) – Dict with domain properties

Raises:

ValueError – If domain is outside soilgrid data area

static coordinates_for_cutting_dataset(domain_properties: dict, halo: float = 5.0) → tuple

Get coordinates for cutting dataset.

Parameters:

• domain_properties (dict) – Dict with domain properties

• halo (float) – Halo. Defaults to 5.0.

Returns:

Coordinates for cutting dataset

Return type:

tuple

execute()

Run task.

Define run sequence.

Raises:

FileNotFoundError – If no tif files are found.

get_domain_properties(config) → dict

Get domain properties.

Parameters:

config (deode.ParsedConfig) – Configuration

Returns:

Domain properties

Return type:

dict

static write_soil_header_file(header_file, soiltype, hdr_north, hdr_south, hdr_west, hdr_east, hdr_rows, hdr_cols, nodata=0, bits=8, write_fact=False, fact=10) → None

Write header file.

Parameters:

• header_file (str) – Header file

• soiltype (str) – Soil type

• hdr_north (float) – North

• hdr_south (float) – South

• hdr_west (float) – West

• hdr_east (float) – East

• hdr_rows (int) – Number of rows

• hdr_cols (int) – Number of columns

• nodata (int) – No data value. Defaults to 0.

• bits (int) – Number of bits. Defaults to 8.

• write_fact (bool) – Write factor. Defaults to False.

• fact (int) – Factor. Defaults to 10

deode.tasks.gribmodify module

AddCalculatedFields.

class AddCalculatedFields(config)

Bases: Task

Create grib files.

add_field_to_grib(fnames, params, operation, output_params, layer_weights, physical_range)

Add fields to the same grib following specified modification.

Parameters:

• fnames (list) – list of grib files

• params (list) – list of parameter dictionaries of input parameters

• operation (str) – operation to compute new field

• output_params (dict) – list of parameter dictionaries of output parameters

• layer_weights (list) – list of weights for each layer

• physical_range (list) – list of physical range for each parameter

Raises:

NotImplementedError – Operation not implemented yet”

calc_patch_averaging(params, layer_weights, values_list, bitmap_list, physical_range, nature_weighting, surface=False)

Calculate patch averaging.

Bitmaps are used to mask out invalid values. This is necessary, as the values for a parameter may be missing in some tiles despite the fact that the tile fraction is not 0. This is solved by updating the layer weights to always sum to 1 while excluding missing data.

execute()

Execute gribmodify.

expand_input_grib_id(rules)

Expand input_grib_id entries into separate instances.

find_additional_files(additional_files, validtime)

Find multiple additional files.

find_in_files(param, fname, additional_files)

Check if field with specified parameters exists.

Checks the main file and all additional files.

Parameters:

• param – parameter dictionary,

• fname – main grib file

• additional_files – list of additional grib files

Returns:

True if field exists in any file

Return type:

bool

find_par(param, fname)

Check if field with specified parameters already exists in grib.

Parameters:

• param – parameter dictionary,

• fname – grib file

Returns:

True if field exists

Return type:

bool

get_unique_values(dicts, key, exclude=None)

Get unique values of a key in a list of dictionaries.

get_value_for_params(params, match_dict, values_list, bitmap_list, apply_bitmap=False, return_bitmap=False)

Get the index of the dict in the list that matches all key-value pairs.

safe_codes_get(gid, key)

Safely get the value for a key from a GRIB message.

Returning None if the key is not found.

set_while_retaining_time_unit(gid_res, key, value)

Set a key in a GRIB message while retaining the time unit.

deode.tasks.impacts module

Impact model classes.

class Ehype(name: str, *_args, **_kwargs)

Bases: ImpactModel

EHYPE specific methods.

name: str = 'ehype'

run()

Starts the EHYPE suite.

class ImpactModel(name: str, *_args, **_kwargs)

Bases: ABC

Abstract class for impact models.

config: BasicConfig

dump(to_dump)

Write config to selected format.

Parameters:

to_dump (dict) – config dict to write

Raises:

TypeError – Unknown output file type

execute()

Prepares and runs the impact model commands.

filename: str | None = ''

name: str

platform: Platform

abstract run()

Abstract method.

class ImpactModels(config, taskname=None)

Bases: Task

Create info to and start impact models.

execute()

Start the impact model(s).

class StartImpactModels(config)

Bases: ImpactModels

Starts the impact models.

get_impact(config, taskname)

Gather impact settings.

Parameters:

• config (deode.ParsedConfig) – Configuration

• taskname (str) – Indicating taskname

Returns:

Impact model settings

Return type:

impact (dict)

deode.tasks.interpolsstsic module

InterpolSstSic.

class InterpolSstSic(config)

Bases: Task

Interpolate SST/SIC from the host model to the model geometry.

Supported host models:

• IFS

execute()

Run task.

Run sequence: set input files, create gl namelist and run gl.

Raises:

NotImplementedError – If an unsupported SST model is used

deode.tasks.iomerge module

IO merge task.

class IOmerge(config)

Bases: Task

IO merge task.

check_fc_path()

Check if fc_path exists.

Raises:

FileNotFoundError – If directory is not present

execute()

Execute IO merge.

merge_output(lt, filetype)

Merge distributed forecast model output.

Parameters:

• lt (timeDelta) – lead time

• filetype (str) – Type of file

static wait_for_file(filename, age_limit=15)

Wait until a file is at least N seconds old.

Parameters:

• filename (str) – full path to file

• age_limit (int) – minimum age of last modification

wait_for_io(filetype, lt, maxtries=2)

Wait for all io_server output to be stable.

Parameters:

• filetype (str) – kind of output file

• lt (timeDelta) – lead time

• maxtries (int) – maximum number of file search trials

Returns:

List of files expected

Return type:

file_list (list)

Raises:

RuntimeError – In case of erroneous number of files

deode.tasks.marsprep module

Marsprep.

class Marsprep(config)

Bases: Task

Marsprep task.

execute()

Run task.

Define run sequence.

Raises:

RuntimeError – If there is an issue with the work folder.

get_bdmember_fetch_list(bddir: str, bdfile: str)

Builds the list of missing files to fetch from MARS.

get_geopotential_latlon(first_member, bdmember_list)

Get geopotential in lat/lon.

Parameters:

• first_member (int) – First bdmember in ensemble

• bdmember_list (List[int]) – Members to fetch data for

get_gg_data(tag: str, steps: List[int], members_dict: Dict[str, List[int]])

Get gridpoint surface data.

get_gg_snow_data(tag: str, steps: List[int], members_dict: Dict[str, List[int]], param)

Get soil gridpoint data.

get_gg_soil_data(tag: str, params: dict)

Get soil gridpoint data.

get_grid_point_surface_data()

Get grid point surface data.

get_grid_point_upper_air_data()

Get gridpoint upper air data.

get_lat_lon_data(bdmember_list: List[int])

Get Lat/Lon data.

Parameters:

bdmember_list (List[int]) – Members to fetch data for

get_lat_lon_sst_data(missing_steps_per_member: Dict[int, List[int]], members_dict: Dict[str, List[int]])

Get lat/lon data for sst.

get_sfx_data()

Get SFX data.

get_sh_data(tag: str, steps: List[int], members_dict: Dict[str, List[int]])

Get SH data.

get_shz_data(tag: str)

Get geopotential in spherical harmonics.

get_spectral_harmonic_data()

Get spectral harmonic data.

get_sst_data()

Get SST data.

get_ua_data(tag: str, steps: List[int], members_dict: Dict[str, List[int]])

Get upper air data.

property mars: dict

Get mars selection.

static mars_selection(selection: str, config: ParsedConfig) → dict

Copy default settings if requested.

Parameters:

• selection (str) – The selection to use.

• config (deode.ParsedConfig) – Configuration object

Returns:

mars config section

Return type:

mars (dict)

update_data_request(request: BaseRequest, prefetch: bool, specify_domain: bool, bdmembers: List[int] | None = None, grid: str | None = None, source: str | None = None, fieldset: str | None = None)

Create ECMWF MARS system request.

Parameters:

• request – BaseRequest object to update

• prefetch – Retrieve or stage

• specify_domain – Use lat/lon and rotation or use global (default)

• bdmembers – Boundary members to retrieve in case of eps.

• grid – Specific grid for some request. Default None.

• source – Sorce for retrieve data from disk. Defaults None.

• fieldset – Name of fieldset. Defaults None.

deode.tasks.prep_run module

Preparatory task for a deode suite.

class PrepRun(config)

Bases: Cleaning

Preparatory run/cleaning task.

deode.tasks.serial module

Serial task.

class Serial(config)

Bases: Task

Serial task.

execute()

Execute task.

deode.tasks.sfx module

Surfex tasks.

class InputDataFromNamelist(nml, input_data, program, platform, basetime=None, validtime=None, one_decade=False)

Bases: object

Binary input data for offline executables.

extend_macro(key, val, macros, sep='#')

Extend entries from macro.

Parameters:

• key (_type_) – _description_

• val (_type_) – _description_

• macros (dict) – Macros

• sep (str, optional) – Namelist key separator. Defaults to “#”.

Raises:

• NotImplementedError – _description_

• NotImplementedError – _description_

Returns:

Key, value dictionary

Return type:

dict

get()

Return data.

static get_nml_value(nml, block, key, indices=None)

Get namelist value.

Parameters:

• nml (nmlf90.Namelist) – Namelist

• block (str) – Namelist block

• key (str) – Namelist key

• indices (list, optional) – Indices to read. Defaults to None.

Returns:

Namelist setting

Return type:

setting (any)

static get_nml_value_from_string(nml, string, sep='#', indices=None)

Get namelist value from a string.

Parameters:

• nml (nmlf90.Namelist) – Namelist

• string (str) – Namelist identifier

• sep (str, optional) – _description_. Defaults to “#”.

• indices (list, optional) – Indices to read. Defaults to None.

Returns:

Namelist setting

Return type:

setting (any)

matching_value(data, val, sep='#', indices=None)

Match the value. Possibly also read namelist value.

Parameters:

• data (dict) – Data to check keys for

• val (str) – Key to find

• sep (str, optional) – Namelist separator. Defaults to “#”.

• indices (list, optional) – Indices in namelist

Raises:

RuntimeError – “Malformed input data”

Returns:

Matching entry in data.

Return type:

dict

process_data(sep='#')

Process input definitions on files to map.

Parameters:

sep (str, optional) – Namelist separator. Defaults to “#”.

Returns:

A dict with mapped local names and target files.

Return type:

mapped_data (dict)

process_macro(key, val, macros, sep='#', indices=None)

Process macro.

Parameters:

• key (str) – Key

• val (str) – Value

• macros (dict) – Macros

• sep (str, optional) – Namelist key separator. Defaults to “#”.

• indices (list, optional) – Process macro from namelist indices.

Raises:

NotImplementedError – Only 2 dimensions are implemented

Returns:

Key, value dictionary

Return type:

dict

read_macro_setting(macro_defs, key, default=None, sep='#')

Read a macro setting.

Parameters:

• macro_defs (dict) – Macro definition

• key (str) – Macro setting to get.

• default (str, optional) – Default value. Defaults to None.

• sep (str, optional) – Namelist key separator. Defaults to “#”.

Returns:

setting (any)

substitute(key, val, macros=None, micro='@')

Substitute patterns.

Parameters:

• key (str) – _description_

• val (str) – _description_

• macros (dict, optional) – Macros. Defaults to None.

• micro (str, optional) – Micro character. Defaults to “@”.

Returns:

Substituted key=value

Return type:

dict

class Pgd(config)

Bases: Task

Task.

execute()

Execute.

class PgdFilterTownFrac(config)

Bases: Task

PgdFilterTownFrac.

Reduce Town frac in Pgd file made with eccoclimapII.

execute()

Run task.

class Prep(config)

Bases: Task

Prep.

execute()

Execute.

deode.tasks.sqlite module

ExtractSQLite.

class ExtractSQLite(config)

Bases: Task

Extract sqlite point files.

execute()

Execute ExtractSQLite on all output files.

class MergeSQLites(config)

Bases: Task

Extract sqlite point files.

execute()

Execute MergeSQLite on all member FC files.