deode.tasks package

Tasks submodule.

Submodules

deode.tasks.archive module

ARCHIVEHOUR // ARCHIVESTATIC.

class Archive(config, datatype=None)

Bases: Task

Archving data.

archive(pattern, inpath, outpath, archive_type=None, newname=None)

Send files to the file manager.

Parameters:

• pattern (str,list) – string of list of patterns to search for

• inpath (str) – Full path on the input archive

• outpath (str) – relative path on the output archive

• archive_type (str, optional) – Archive type. Defaults to None.

• newname (str, optional) – Forces a rename of an identified file. Defaults to None.

Raises:

FileNotFoundError – If file not found

execute()

Loops over archive choices.

trigger(trigger)

Return trigger.

class ArchiveHour(config)

Bases: Archive

Archving task for time dependent data.

class ArchiveStatic(config)

Bases: Archive

Archving task for static data.

deode.tasks.base module

Base site class.

class Task(config, name)

Bases: object

Base Task class.

archive_logs(files, target=None)

Archive files in a log directory.

Parameters:

• files (str,list) – File(s) to be archived

• target (str) – Target directory for archiving

change_to_wdir()

Change to task working dir.

create_wdir()

Create task working directory and check for unix group and set permissions.

create_wrkdir()

Create a cycle working directory.

execute()

Do nothing for base execute task.

get_binary(binary_name)

Determine binary path from task or system config section.

Parameters:

binary_name (str) – Name of binary

Returns:

full path to binary

Return type:

bindir (str)

get_task_setting(setting)

Get task setting.

Parameters:

setting (str) – Setting to find in task.{self.name}

Returns:

Found setting

Return type:

value

post()

Do default postfix.

E.g. clean

prep()

Do default preparation before execution.

E.g. clean

remove_wdir()

Remove working directory.

rename_wdir(prefix='Failed_task_')

Rename failed working directory.

run()

Run task.

Define run sequence.

class UnitTest(config)

Bases: Task

Base Task class.

deode.tasks.batch module

Batch process.

class BatchJob(rte, wrapper='')

Bases: object

Batch job.

run(cmd)

Run command.

Parameters:

cmd (str) – Command to run.

Raises:

• TypeError – If the provided command is not a string

• CalledProcessError – Execution error

deode.tasks.c903 module

C903.

class C903(config)

Bases: Task

C903, preform interpolation from IFS to IAL LAM.

execute()

Run task.

Define run sequence.

deode.tasks.canari module

Canari.

class Canari(config)

Bases: Task

Canari task.

execute()

Execute task.

deode.tasks.clean_old_data module

Clean old data.

class CleanIFSData(config)

Bases: CleanOldData

Clean IFS data.

execute()

Run clean IFS data.

class CleanOldData(config)

Bases: Task

Interface class to the cleaning of the old data.

cutoff(delay)

Calculates the time to determine the old data.

Parameters:

delay (int) – delay in days

Returns:

time in seconds

Return type:

cutoff (int)

get_old(path, form, cutoff_time, ignore=None)

Get directories which are older cutoff.

Parameters:

• path (str) – Path to start of searching old directories

• form (str) – regex for path

• cutoff_time (int) – Directories or files older then cutoff

• ignore (list) – List of directories to ignore

Returns:

List of directories for remove

Return type:

list_to_remove (list)

remove_list(dir_list, files=False)

Remove directories/files from the list.

Parameters:

• dir_list (list) – Dictionary of directories/files

• files (boolean) – If in list are files, Default False

class CleanScratchData(config)

Bases: CleanOldData

Clean old data from scratch.

execute()

Run clean data from scratch.

class CleanSuites(config)

Bases: CleanOldData

Clean old suites.

execute()

Run clean suites.

deode.tasks.cleaning_tasks module

Clean deode file systems.

class Cleaning(config)

Bases: Task

Interface class to the cleaning.

execute()

Run the cleaning.

prep_clean_task(cleaning_type)

Setup clean task.

Parameters:

cleaning_type (str) – Cleaning config section identifier

class CycleCleaning(config)

Bases: Cleaning

Cycle cleaning task.

class PostMortem(config)

Bases: Cleaning

Final cleaning task.

class PreCleaning(config)

Bases: Cleaning

Preparatory cleaning task.

deode.tasks.collectlogs module

CollectLogs.

class CollectLogs(config)

Bases: Task

Collect task and scheduler logfiles and store them as tarfiles.

execute()

Execute collect logs .

scan_logs(tarlog, parent, target, pattern='', exclude='')

Search for files matching a pattern and add them to a tar file.

Parameters:

• tarlog (tarfile object) – Tar file used

• parent (str) – Top search directory

• target (str) – Main search directory

• pattern (str) – Optional search pattern

• exclude (str) – Optional string for files to be excluded

Raises:

FileNotFoundError – If parent directory does not exist

deode.tasks.creategrib module

CreateGrib.

class CreateGrib(config)

Bases: Task

Create grib files.

convert2grib(infile, outfile, filetype)

Convert FA to grib.

Namelist arguments are given in the task.creategrib config part per filetype :param infile: Input file :type infile: str :param outfile: Output file :type outfile: str :param filetype: filetype to look for in rules :type filetype: str

Raises:

FileNotFoundError – Could not find file

create_list(input_template, output_settings)

Create list of files to process.

execute()

Execute creategrib.

deode.tasks.discover_task module

Discover tasks.

available_tasks(reg)

Create a list of available tasks.

Parameters:

reg (DeodePluginRegistry) – Deode plugin registry

Returns:

Task objects

Return type:

known_types (list)

discover(package, base)

Discover task classes.

Plugin classes are discovered in a given namespace package, deriving from a given base class. The base class itself is ignored, as are classes imported from another module (based on cls.__module__). Each discovered class is identified by the class name by changing it to lowercase and stripping the name of the base class, if it appears as a suffix.

Parameters:

• package (types.ModuleType) – Namespace package containing the plugins

• base (type) – Base class for the plugins

Returns:

type): Discovered plugin classes

Return type:

(dict of str

discover_modules(package, what='plugin')

Discover plugin modules.

Parameters:

• package (types.ModuleType) – Namespace package containing the plugins

• what (str, optional) – String describing what is supposed to be discovered. Defaults to “plugin”.

Yields:

tuple – str: Name of the imported module types.ModuleType: The imported module

get_task(name, config)

Create a deode.tasks.Task object from configuration.

Parameters:

• name (_type_) – _description_

• config (_type_) – _description_

Raises:

NotImplementedError – If task name is not amongst the known task names.

Returns:

_description_

Return type:

_type_

deode.tasks.e923 module

E923.

class E923(config)

Bases: Task

Methods for the e923 work.

constant_part(constant_file)

Run the constant part of e923.

Parameters:

constant_file – filename of the resulting file

monthly_part(constant_file)

Run the monthly part of e923.

Parameters:

constant_file – filename of the input constant file

myexec(cmd, i)

Execute binary task.

Parameters:

• cmd (str) – Command to run

• i (int) – sequence number

print_part(part, month=None)

Run the constant part of e923.

Parameters:

• part – which step

• month – month

remove_links(link)

Remove link.

Parameters:

link (list) – List of links to remove

class E923Constant(config)

Bases: E923

E923Constant task.

execute()

Run task.

Define run sequence.

class E923Monthly(config)

Bases: E923

E923Monthly task.

execute()

Run task.

Define run sequence.

class PgdUpdate(config)

Bases: Task

PgdUpdate.

Ensure consistency in orography between the atmosphere and surfex Read gridpoint orography from the atmospheric file and insert it in the PGD file Const.Clim.sfx.

execute()

Run task.

deode.tasks.e923_update module

E923Update.

class E923Update(config)

Bases: Task

Methods for updating climfile for ALARO.

For mor information see doc/markdown/e923_update.md.

execute()

Run task.

Define run sequence.

deode.tasks.e927 module

E927.

class E927(config)

Bases: Task

E927 task.

execute()

Run task.

Define run sequence.

deode.tasks.extractsqlite module

ExtractSQLite.

class ExtractSQLite(config)

Bases: Task

Extract sqlite point files.

execute()

Execute ExtractSQLite on all output files.

deode.tasks.forecast module

Forecast.

class FirstGuess(config)

Bases: Task

FirstGuess.

execute()

Find initial file.

class Forecast(config)

Bases: Task

Forecast task.

accelerator_device_input()

Copy the input files for GPU execution.

• parallel_method: input file with parallelisation technique for each algorithm

• synchost: input file defining optional device-to-host memory transfers

• select_gpu: wrapper file for sbatch, binding GPU to MPI rank

archive_output(filetype, periods)

Archive forecast model output.

Parameters:

• filetype (str) – Filename template

• periods (str) – Output list

execute()

Execute forecast.

merge_output(filetype, periods)

Merge distributed forecast model output.

Parameters:

• filetype (str) – File type (history, surfex, fullpos)

• periods (str) – Output list

Final result is the expected output file name in the working directory (as if there was no IO server). NOTE: This function has been replaced by io_merge tasks. It is only called if n_io_merge=0.

wfp_input()

Add wind turbine files to forecast directory.

class PrepareCycle(config)

Bases: Task

Task.

deode.tasks.gmtedsoil module

GMTED and SOILGRID.

class Gmted(config)

Bases: Task

GMTED.

define_gmted_input(domain_properties: dict) → tuple

Define GMTED input files.

Parameters:

domain_properties (dict) – Domain properties

Returns:

GMTED input files

Return type:

tuple

execute()

Run task.

Define run sequence.

get_domain_properties(config) → dict

Get domain properties.

Parameters:

config (Config) – Config object

Returns:

Domain properties

Return type:

dict

gmted_header_coordinates(east: float, west: float, south: float, north: float) → tuple

Get GMTED header coordinates.

Parameters:

• east (float) – East

• west (float) – West

• south (float) – South

• north (float) – North

Returns:

Header coordinates

Return type:

tuple

static tif2bin(gd, bin_file) → None

Convert tif file to binary file used by surfex.

Parameters:

• gd – gdal dataset

• bin_file (str) – Binary file

static write_gmted_header_file(header_file, hdr_north, hdr_south, hdr_west, hdr_east, hdr_rows, hdr_cols) → None

Write header file.

Parameters:

• header_file (str) – Header file

• hdr_north (float) – North

• hdr_south (float) – South

• hdr_west (float) – West

• hdr_east (float) – East

• hdr_rows (int) – Number of rows

• hdr_cols (int) – Number of columns

class Soil(config)

Bases: Task

Prepare soil data task for PGD.

static check_domain_validity(domain_properties: dict) → None

Check if domain is valid.

Parameters:

domain_properties (dict) – Dict with domain properties

Raises:

ValueError – If domain is outside soilgrid data area

static coordinates_for_cutting_dataset(domain_properties: dict, halo: float = 5.0) → tuple

Get coordinates for cutting dataset.

Parameters:

• domain_properties (dict) – Dict with domain properties

• halo (float) – Halo. Defaults to 5.0.

Returns:

Coordinates for cutting dataset

Return type:

tuple

execute()

Run task.

Define run sequence.

Raises:

FileNotFoundError – If no tif files are found.

get_domain_properties(config) → dict

Get domain properties.

Parameters:

config (deode.ParsedConfig) – Configuration

Returns:

Domain properties

Return type:

dict

static write_soil_header_file(header_file, soiltype, hdr_north, hdr_south, hdr_west, hdr_east, hdr_rows, hdr_cols, nodata=0, bits=8, write_fact=False, fact=10) → None

Write header file.

Parameters:

• header_file (str) – Header file

• soiltype (str) – Soil type

• hdr_north (float) – North

• hdr_south (float) – South

• hdr_west (float) – West

• hdr_east (float) – East

• hdr_rows (int) – Number of rows

• hdr_cols (int) – Number of columns

• nodata (int) – No data value. Defaults to 0.

• bits (int) – Number of bits. Defaults to 8.

• write_fact (bool) – Write factor. Defaults to False.

• fact (int) – Factor. Defaults to 10

deode.tasks.gribmodify module

AddCalculatedFields.

class AddCalculatedFields(config)

Bases: Task

Create grib files.

add_field_to_grib(fname, params, operation, short_name)

Add sum of two fields to the same grib.

Parameters:

• fname (str) – gribe file

• params (list) – list of paramaterNumbers of input parameters

• operation – operarion to compute new field

• short_name (str) – short_name of added field

Raises:

NotImplementedError – Operation not implemented yet”

execute()

Execute add_total_precip.

find_par(short_name, fname)

Check if field with shortName already exists in grib.

Parameters:

• short_name – shortName,

• fname – grib file

Returns:

True if field exists

Return type:

bool

deode.tasks.impacts module

Impact model classes.

class Ehype(name: str, *_args, **_kwargs)

Bases: ImpactModel

EHYPE specific methods.

name: str = 'ehype'

run()

Starts the EHYPE suite.

class ImpactModel(name: str, *_args, **_kwargs)

Bases: ABC

Abstract class for impact models.

config: BasicConfig

dump(to_dump)

Write config to selected format.

Parameters:

to_dump (dict) – config dict to write

Raises:

TypeError – Unknown output file type

execute()

Prepares and runs the impact model commands.

filename: str | None = ''

name: str

platform: Platform

abstract run()

Abstract method.

class ImpactModels(config, taskname=None)

Bases: Task

Create info to and start impact models.

execute()

Start the impact model(s).

class StartImpactModels(config)

Bases: ImpactModels

Starts the impact models.

deode.tasks.iomerge module

IO merge task.

class IOmerge(config)

Bases: Task

IO merge task.

execute()

Execute IO merge.

merge_output(lt, fc_path='../Forecast')

Merge distributed forecast model output.

Parameters:

• lt (timeDelta) – lead time

• fc_path (str) – path to forecast directory

static wait_for_file(filename, age_limit=15)

Wait until a file is at least N seconds old.

Parameters:

• filename (str) – full path to file

• age_limit (int) – minimum age of last modification

wait_for_io(filetype, lt, fc_path='../Forecast', maxtries=2)

Wait for all io_server output to be stable.

Parameters:

• filetype (str) – kind of output file

• lt (timeDelta) – lead time

• fc_path (str) – path to forecast directory

• maxtries (int) – maximum number of file search trials

Returns:

List of files expected

Return type:

file_list (list)

Raises:

RuntimeError – In case of erroneous number of files

deode.tasks.marsprep module

Marsprep.

class Marsprep(config)

Bases: Task

Marsprep task.

check_file_exists(steps, path, file_name)

Check if which mars file already exist.

check_value(value, key)

Check value according to key.

• If a string returnts the value itself

• If key is a date search for the most suitable match in value

• Else return the value matching the key.

Parameters:

• value (str, BaseConfig object) – Values to select

• key (str) – key for value checking

Returns:

Found value

Return type:

value (str)

Raises:

ValueError – Exception

create_executable(marsfile)

Create task for binary.

Parameters:

marsfile – intermediate request file

Returns:

None

execute()

Run task.

Define run sequence.

Raises:

RuntimeError – If there is an issue with the work folder.

fetch_info(steps, tag)

Print what we are actually fetching.

Parameters:

• steps (str) – list of steps to retrieve

• tag (str) – file type identifier

static get_domain_data(config)

Read and return domain data.

Parameters:

config – config method

Returns:

String containing the domain info for MARS

list_files_join(marsfolder, f_pattern)

Read and return file names based on given pattern.

Parameters:

• marsfolder – path with MARS file location

• f_pattern – glob pattern

Returns:

list of files that should be joined

mars_selection(selection=None)

Copy default settings if requested.

Parameters:

selection – default self.config[“boundaries.ifs.selection”]

Returns:

updated mars config section

Return type:

mars (dict)

split_date(date, length, interval)

Manipulate the dates for Mars request.

Parameters:

• date – full date provided for spliting into parts

• length – forecast cycle length in hours

• interval – boundary interval in hours “1H”,”3H”,”6H”

Returns:

Pandas dataframe object

update_data_request(param, data_type, date, time, steps, prefetch, levtype, specify_domain, target, grid=None)

Create ECMWF MARS system request.

Parameters:

• param – String of parameters to extract sep=/

• data_type – Type of data to extract, ‘analysis’ or ‘forecast’

• date – Date to extract

• time – Time to extract

• steps – Forecast steps to be extracted

• prefetch – Retrieve or stage

• levtype – SFC or ML

• specify_domain – Use lat/lon and rotation or use global (default)

• target – Filename to write

• grid – Specific grid for some request. Default None.

Returns:

Pandas dataframe object

Raises:

ValueError – Wrong call of read.

static write_mars_req(nam, name, method)

Write a request for MARS.

Parameters:

• nam – namelist object to write

• name – request file name

• method – selected method, retrieve or read

deode.tasks.serial module

Serial task.

class Serial(config)

Bases: Task

Serial task.

execute()

Execute task.

deode.tasks.sfx module

Surfex tasks.

class InputDataFromNamelist(nml, input_data, program, platform, basetime=None, validtime=None, one_decade=False)

Bases: object

Binary input data for offline executables.

extend_macro(key, val, macros, sep='#')

Extend entries from macro.

Parameters:

• key (_type_) – _description_

• val (_type_) – _description_

• macros (dict) – Macros

• sep (str, optional) – Namelist key separator. Defaults to “#”.

Raises:

• NotImplementedError – _description_

• NotImplementedError – _description_

Returns:

Key, value dictionary

Return type:

dict

get()

Return data.

static get_nml_value(nml, block, key, indices=None)

Get namelist value.

Parameters:

• nml (nmlf90.Namelist) – Namelist

• block (str) – Namelist block

• key (str) – Namelist key

• indices (list, optional) – Indices to read. Defaults to None.

Returns:

Namelist setting

Return type:

setting (any)

static get_nml_value_from_string(nml, string, sep='#', indices=None)

Get namelist value from a string.

Parameters:

• nml (nmlf90.Namelist) – Namelist

• string (str) – Namelist identifier

• sep (str, optional) – _description_. Defaults to “#”.

• indices (list, optional) – Indices to read. Defaults to None.

Returns:

Namelist setting

Return type:

setting (any)

matching_value(data, val, sep='#', indices=None)

Match the value. Possibly also read namelist value.

Parameters:

• data (dict) – Data to check keys for

• val (str) – Key to find

• sep (str, optional) – Namelist separator. Defaults to “#”.

• indices (list, optional) – Indices in namelist

Raises:

RuntimeError – “Malformed input data”

Returns:

Matching entry in data.

Return type:

dict

process_data(sep='#')

Process input definitions on files to map.

Parameters:

sep (str, optional) – Namelist separator. Defaults to “#”.

Returns:

A dict with mapped local names and target files.

Return type:

mapped_data (dict)

process_macro(key, val, macros, sep='#', indices=None)

Process macro.

Parameters:

• key (str) – Key

• val (str) – Value

• macros (dict) – Macros

• sep (str, optional) – Namelist key separator. Defaults to “#”.

• indices (list, optional) – Process macro from namelist indices.

Raises:

NotImplementedError – Only 2 dimensions are implemented

Returns:

Key, value dictionary

Return type:

dict

read_macro_setting(macro_defs, key, default=None, sep='#')

Read a macro setting.

Parameters:

• macro_defs (dict) – Macro definition

• key (str) – Macro setting to get.

• default (str, optional) – Default value. Defaults to None.

• sep (str, optional) – Namelist key separator. Defaults to “#”.

Returns:

setting (any)

substitute(key, val, macros=None, micro='@')

Substitute patterns.

Parameters:

• key (str) – _description_

• val (str) – _description_

• macros (dict, optional) – Macros. Defaults to None.

• micro (str, optional) – Micro character. Defaults to “@”.

Returns:

Substituted key=value

Return type:

dict

class Pgd(config)

Bases: Task

Task.

execute()

Execute.

class Prep(config)

Bases: Task

Prep.

execute()

Execute.