deode package

Package to run the Destination Earth on Demand Extremes system.

class GeneralConstants(*_args, **_kwargs)[source]

Bases: QuasiConstant

General package-related constants.

PACKAGE_DIRECTORY = PosixPath('/home/runner/work/Deode-Workflow/Deode-Workflow/deode')

PACKAGE_NAME = 'deode'

VERSION = '0.17.0'

Subpackages

Submodules

deode.archive module

Archive.

class Archive(config, datatype=None, include=None, exclude=None)[source]

Bases: object

Handle archiving.

archive(pattern, inpath, outpath, archive_type=None, newname=None)[source]

Send files to the file manager.

Parameters:

pattern (str,list) – string of list of patterns to search for
inpath (str) – Full path on the input archive
outpath (str) – relative path on the output archive
archive_type (str, optional) – Archive type. Defaults to None.
newname (str, optional) – Forces a rename of an identified file. Defaults to None.

Raises:

FileNotFoundError – If file not found

execute()[source]: Loops over archive choices.

trigger(trigger)[source]: Return trigger.

deode.argparse_wrapper module

Wrappers for argparse functionality.

add_keep_def_file(parser_object, help_message='Keep suite definition file after submission')[source]

Add object args.

Parameters:

parser_object (args oject) – args object to update
help_message (str) – Help text

add_namelist_args(parser_object)[source]

Add namelist args.

Parameters:: parser_object (args oject) – args object to update
Returns:: updated args object
Return type:: parser_object (args oject)

get_parsed_args(program_name='deode', argv=None)[source]

Get parsed command line arguments.

Parameters:

program_name (str) – The name of the program.
argv (list) – A list of passed command line args.

Returns:

Parsed command line arguments.

Return type:

argparse.Namespace

deode.aux_types module

Aux types used in the package.

class BaseMapping(*args, **kwargs)[source]

Bases: Mapping

Immutable mapping that will serve as basis for all config-related classes.

copy(update: Mapping | Callable[[Mapping], Any] | None = None)[source]: Return a copy of the instance, optionally updated according to update.

property data: Return the underlying data stored by the instance.

dict()[source]: Return a dict representation, converting also nested Mapping-type items.

dumps(section='', style: Literal['toml', 'json', 'yaml'] = 'toml', toml_formatting_function: Callable | None = None)[source]: Get a nicely printed version of the container’s contents.

class QuasiConstant(*_args, **_kwargs)[source]

Bases: object

Inheriting from this will make the class’ attributes (almost) immutable.

class QuasiConstantMetaclass(*args, **kwargs)[source]

Bases: type

Metaclass to help making a class behave as if it had quasi-constant attributes.

dict()[source]: Return a dict form of the instance, with nested instances also converted.

property type_conversions: Type conversions to be performed on the attributes.

deode.cleaning module

Clean deode file systems.

class CleanDeode(config, defaults=None, basetime=None)[source]

Bases: object

Clean data.

clean()[source]: Perform the cleaning.

prep_cleaning(choices, basetime=None)[source]

Prepare tasks for cleaning.

Parameters:

choices (dict) – Dict with cleaning settings
basetime (dateTime object) – Reference time

deode.commands_functions module

Implement the package’s commands.

class RunTaskNamespace(**kwargs)[source]

Bases: Namespace

Namespace for the ‘run’ command.

deode_home: str

members: List[int] | None = None

output: Path

task: str

task_job: Path

template_job: Path

troika: str

create_exp(args, config)[source]

Implement the ‘case’ command.

Parameters:

args (argparse.Namespace) – Parsed command line arguments.
config (.config_parser.ParsedConfig) – Parsed config file contents.

doc_config(args, config: ParsedConfig)[source]

Implement the ‘doc_config’ command.

Parameters:

args (argparse.Namespace) – Parsed command line arguments.
config (ParsedConfig) – Parsed config file contents.

namelist_convert(args, config: ParsedConfig)[source]

Implement the ‘namelist convert’ command.

Parameters:

args (argparse.Namespace) – Parsed command line arguments.
config (.config_parser.ParsedConfig) – Parsed config file contents.

namelist_format(args, config: ParsedConfig)[source]

Implement the ‘namelist format’ command.

Parameters:

args (argparse.Namespace) – Parsed command line arguments.
config (.config_parser.ParsedConfig) – Parsed config file contents.

namelist_integrate(args, config)[source]

Implement the ‘namelist integrate’ command.

Parameters:

args (argparse.Namespace) – Parsed command line arguments.
config (.config_parser.ParsedConfig) – Parsed config file contents.

Raises:

SystemExit # noqa – DAR401

run_task(args: RunTaskNamespace, config: ParsedConfig)[source]

Implement the ‘run’ command.

Parameters:

args (argparse.Namespace) – Parsed command line arguments.
config (.config_parser.ParsedConfig) – Parsed config file contents.

show_config(args, config)[source]

Implement the ‘show_config’ command.

Parameters:

args (argparse.Namespace) – Parsed command line arguments.
config (.config_parser.ParsedConfig) – Parsed config file contents.

show_config_schema(args, config)[source]

Implement the show config-schema command.

Parameters:

args (argparse.Namespace) – Parsed command line arguments.
config (.config_parser.ParsedConfig) – Parsed config file contents.

show_host(args, config)[source]

Implement the show host command.

Parameters:

args (argparse.Namespace) – Parsed command line arguments.
config (.config_parser.ParsedConfig) – Parsed config file contents.

show_namelist(args, config)[source]

Implement the ‘show_namelist’ command.

Parameters:

args (argparse.Namespace) – Parsed command line arguments.
config (.config_parser.ParsedConfig) – Parsed config file contents.

show_paths(args, config)[source]: Implement the ‘show_paths’ command.

ssh_cmd(host, user, cmd)[source]

SSH to remote server and execute basic commands.

Parameters:

host – Host name or server IP address
user – Username (string) to login to server with
cmd – Command to be executed

Returns:

Message to notify if failed and why or successfully completed.

start_suite(args, config)[source]

Implement the ‘start suite’ command.

Parameters:

args (argparse.Namespace) – Parsed command line arguments.
config (.config_parser.ParsedConfig) – Parsed config file contents.

Raises:

SystemExit – If error occurs while transferring files.

deode.config_parser module

Registration and validation of options passed in the config file.

class BasicConfig(*args, _metadata=None, **kwargs)[source]

Bases: BaseMapping

Base class for configs. Arbitrary entries allowed: no validation is performed.

property data: Return the underlying data stored by the instance.

classmethod from_file(path, **kwargs)[source]

Retrieve configs from a file in miscellaneous formats.

Parameters:

path (Union[pathlib.Path, str]) – Path to the config file.
**kwargs – Arguments passed to the class constructor.

Returns:

Configs retrieved from the specified path.

Return type:

cls

property metadata: Get the metadata associated with the instance.

save_as(config_file)[source]

Save config file.

Parameters:: config_file (str) – Path to config file

exception ConfigFileValidationError[source]

Bases: Exception

Error to be raised when parsing the input config file fails.

class ConfigParserDefaults(*_args, **_kwargs)[source]

Bases: QuasiConstant

Defaults related to the parsing of config files.

CONFIG_DIRECTORY = PosixPath('/home/runner/work/Deode-Workflow/Deode-Workflow/deode/data/config_files')

CONFIG_PATH = PosixPath('/home/runner/work/Deode-Workflow/Deode-Workflow/deode/data/config_files/config.toml')

DATA_DIRECTORY = PosixPath('/home/runner/work/Deode-Workflow/Deode-Workflow/deode/data')

MAIN_CONFIG_JSON_SCHEMA = mappingproxy({'$schema': 'http://json-schema.org/draft-07/schema#', 'title': 'Config', 'description': "Model for validation of `deode`'s main config file.", 'type': 'object', 'additionalProperties': True, 'properties': mappingproxy({'general': mappingproxy({'$ref': '#/definitions/general'}), 'suite_control': mappingproxy({'$ref': '#/definitions/suite_control'}), 'clean_old_data': mappingproxy({'$ref': '#/definitions/clean_old_data'}), 'system': mappingproxy({'$ref': '#/definitions/system'}), 'boundaries': mappingproxy({'$ref': '#/definitions/boundaries'}), 'collectlogs': mappingproxy({'$ref': '#/definitions/collectlogs'}), 'pgd': mappingproxy({'$ref': '#/definitions/pgd'}), 'fullpos': mappingproxy({'$ref': '#/definitions/fullpos'}), 'troika': mappingproxy({'$ref': '#/definitions/troika'}), 'include': mappingproxy({'$ref': '#/definitions/include'}), 'gribmodify': mappingproxy({'$ref': '#/definitions/gribmodify'})}), 'required': ['general'], 'definitions': mappingproxy({'include': mappingproxy({'title': 'Include Section', 'description': "Sections to be included from other config files. Specified as 'key = val' pairs, where the keys are the section names and the vals are the paths to the files containing them.", 'type': 'object', 'properties': mappingproxy({})}), 'troika': mappingproxy({'title': 'Troika', 'description': 'DEODE system specific path settings', 'type': 'object', 'properties': mappingproxy({'config_file': mappingproxy({'type': 'string', 'description': 'Troika config file', 'default': '@DEODE_HOME@/data/config_files/troika.yml'}), 'troika': mappingproxy({'type': 'string', 'description': 'Troika command, allows to specify troika with full path', 'default': 'troika'})})}), 'plugin_registry': mappingproxy({'title': 'Plug-in registry', 'type': 'object', 'properties': mappingproxy({'plugins': mappingproxy({'title': 'Loaded plugins', 'type': 'object'})})}), 'boundaries': mappingproxy({'title': 'Boundary Section', 'description': 'Settings for control of boundary treatment.', 'type': 'object', 'properties': mappingproxy({'bdint': mappingproxy({'description': 'Boundary interval', 'type': 'string', 'format': 'duration'}), 'bdshift': mappingproxy({'description': "Shift of boundary initial time. E.g. 'PT3H' would mean use 3H old boundaries", 'type': 'string', 'format': 'duration'}), 'max_interpolation_tasks': mappingproxy({'description': 'Number of interpolate tasks run in parallel', 'type': 'number'}), 'bdmodel': mappingproxy({'description': 'Coupling model', 'default': 'ifs', 'enum': ['lam', 'ifs'], 'type': 'string'}), 'humi_gp': mappingproxy({'type': 'boolean', 'description': 'Value of TFP_Q%LLGP in domain namelist for c903. True for gridpoint humidity.'}), 'lam': mappingproxy({'title': 'LAM related boundary settings', 'description': 'LAM related boundary settings', 'type': 'object', 'properties': mappingproxy({'bdcycle_start': mappingproxy({'description': 'lam boundary model cycle start hour', 'type': 'string', 'default': 'PT0H', 'pattern': '^P(?!$)(\\d+Y)?(\\d+M)?(\\d+W)?(\\d+D)?(T(?=\\d+[HMS])(\\d+H)?(\\d+M)?(\\d+S)?)?$'}), 'bdcycle': mappingproxy({'description': 'lam boundary model cycle interval', 'type': 'string', 'default': 'PT24H', 'pattern': '^P(?!$)(\\d+Y)?(\\d+M)?(\\d+W)?(\\d+D)?(T(?=\\d+[HMS])(\\d+H)?(\\d+M)?(\\d+S)?)?$'})})}), 'ifs': mappingproxy({'title': 'IFS related settings', 'description': "Model for the 'general.times' section.", 'type': 'object', 'properties': mappingproxy({'selection': mappingproxy({'type': 'string', 'description': 'Selection', 'enum': ['atos_bologna_DT', 'lumi_DT', 'ATOS_DT', 'HRES', '@HOST@_DT', 'DT12', 'i5qp', 'i7u4', 'i7ye', 'i8hy', 'IFSENS', 'iit7', 'ilv7']}), 'bdmembers': mappingproxy({'anyOf': [{'type': 'array', 'items': {'type': 'string'}}, {'type': 'array', 'items': {'type': 'integer'}}], 'description': 'List of members in ensemble if selection is IFS.', 'default': []}), 'bdmember': mappingproxy({'anyOf': [{'type': 'integer'}, {'type': 'string'}], 'description': 'Boundary member number a given ensemble member shall use if selection is IFS', 'default': 0})})}), 'bd_has_surfex': mappingproxy({'description': 'Set to true if the host model runs with surfex', 'type': 'boolean'})})}), 'collectlogs': mappingproxy({'title': 'CollectLogs Section', 'description': 'Settings for control of log collection.', 'type': 'object', 'properties': mappingproxy({'staticlogs': mappingproxy({'description': 'Collection settings for static tasks', 'type': 'object', 'properties': mappingproxy({'joboutdir': mappingproxy({'description': 'Top output dir to collect logs from ', 'type': 'string'}), 'tarname': mappingproxy({'description': 'Name of the resulting tar file', 'type': 'string'}), 'task_logs': mappingproxy({'description': 'Search path for task specific logs', 'type': 'string'})})}), 'hourlogs': mappingproxy({'description': 'Collection settings for time dependent tasks', 'type': 'object', 'properties': mappingproxy({'joboutdir': mappingproxy({'description': 'Top output dir to collect logs from ', 'type': 'string'}), 'tarname': mappingproxy({'description': 'Name of the resulting tar file', 'type': 'string'}), 'task_logs': mappingproxy({'description': 'Search path for task specific logs', 'type': 'string'})})})})}), 'clean_old_data': mappingproxy({'title': 'Cleaning old data', 'decription': 'Option for cleaning old data', 'type': 'object', 'properties': mappingproxy({'ehype_data_period': mappingproxy({'description': 'Age for old ehype data in days', 'type': 'string', 'pattern': '^P(?!$)(\\d+Y)?(\\d+M)?(\\d+W)?(\\d+D)?$', 'default': 'P14D'}), 'scratch_data_period': mappingproxy({'description': 'Age for old data in days', 'type': 'string', 'pattern': '^P(?!$)(\\d+Y)?(\\d+M)?(\\d+W)?(\\d+D)?$', 'default': 'P14D'}), 'suites_period': mappingproxy({'description': 'Age for old data in days', 'type': 'string', 'pattern': '^P(?!$)(\\d+Y)?(\\d+M)?(\\d+W)?(\\d+D)?$', 'default': 'P14D'}), 'IFS_period': mappingproxy({'description': 'Age for old data in days', 'type': 'string', 'pattern': '^P(?!$)(\\d+Y)?(\\d+M)?(\\d+W)?(\\d+D)?$', 'default': 'P7D'}), 'ignore': mappingproxy({'decription': 'File not to remove', 'type': 'array', 'default': []}), 'suite_format': mappingproxy({'decription': 'Format of path to ecf files', 'type': 'string', 'default': '/'}), 'scratch_format': mappingproxy({'decription': 'Format of path to scratch', 'type': 'string', 'default': '/'}), 'ifs_format': mappingproxy({'decription': 'Format of path to IFS', 'type': 'string', 'default': '/IFS/(\\d{4})/(0[1-9]|1[0-2])/(0[1-9]|[12]\\d|3[01])/([01]\\d|2[0-3])'}), 'scratch_ext': mappingproxy({'decription': 'Extention of cratche for operational user.', 'type': 'string', 'default': 'deode'})})}), 'pgd': mappingproxy({'title': 'Pgd Section', 'description': 'Settings for control of Pgd selection.', 'type': 'object', 'properties': mappingproxy({'npatch': mappingproxy({'title': 'Number of patches for surfex variables', 'type': 'integer', 'default': 3}), 'one_decade': mappingproxy({'title': 'Option for using one decade', 'type': 'boolean', 'default': True}), 'use_osm': mappingproxy({'title': 'Option to use Open Street map input data', 'type': 'boolean', 'default': False})})}), 'fullpos': mappingproxy({'title': 'Fullpos Section', 'description': 'Settings for control of fullpos selection.', 'type': 'object', 'properties': mappingproxy({'config_path': mappingproxy({'title': 'Path to fullpos config files', 'type': 'string', 'default': '@DEODE_HOME@/data/namelist_generation_input/@CYCLE@/fullpos'}), 'main': mappingproxy({'description': 'List of mandatory rules', 'type': 'array', 'items': mappingproxy({'type': 'string'}), 'default': ['rules', 'namfpc_header']}), 'selection': mappingproxy({'description': 'List of selections to include', 'type': 'object', 'additionalProperties': mappingproxy({'type': 'array', 'items': mappingproxy({'type': 'string'})})}), 'domain_name': mappingproxy({'type': 'string', 'description': 'Name of your domain as used by fullpos', 'maxLength': 20, 'default': 'DOMAIN'})})}), 'suite_control': mappingproxy({'title': 'Suite control', 'description': 'Suite control options', 'type': 'object', 'properties': mappingproxy({'suite_definition': mappingproxy({'description': 'Suite definition to play in EcFlow', 'type': 'string', 'default': 'DeodeSuiteDefinition'}), 'do_creategrib_static': mappingproxy({'description': 'Activate grib conversion for static files', 'type': 'boolean', 'default': False}), 'do_soil': mappingproxy({'description': 'Activate preparation of SOILGRID data as input for PGD', 'type': 'boolean'}), 'do_pgd': mappingproxy({'description': 'Activate generation of a PGD file', 'type': 'boolean'}), 'mode': mappingproxy({'description': 'Choose a strategy to achieve initial/initial_sfx files (start: start from interpolated files and then propagate the FG between cycles, restart: start from a previous run and then propagate the FG between cycles, cold_start: always start from interpolated files).', 'type': 'string', 'enum': ['start', 'restart', 'cold_start']}), 'interpolate_boundaries': mappingproxy({'description': 'Activates the boundary interpolation', 'type': 'boolean'}), 'do_marsprep': mappingproxy({'description': 'Run marsprep or skip extraction', 'type': 'boolean'}), 'do_extractsqlite': mappingproxy({'description': 'Activates extraction of selected points to SQLite files for verification.', 'type': 'boolean'}), 'do_mergesqlite': mappingproxy({'description': 'Activates merging of extracted sqlite files. Requires do_extractsqlite to be true, and number of ensemble members to be greater than 1, to have an effect.', 'type': 'boolean'}), 'create_static_data': mappingproxy({'description': 'Activate the generation of PGD and monthly climate files in the suite', 'type': 'boolean'}), 'create_time_dependent_suite': mappingproxy({'description': 'Activate the time dependent part of the suite', 'type': 'boolean'}), 'split_mars': mappingproxy({'description': 'Do marsprep only for one step before c903', 'type': 'boolean', 'default': False}), 'do_cleaning': mappingproxy({'description': 'Do file cleaning', 'type': 'boolean', 'default': False}), 'n_io_merge': mappingproxy({'description': 'Number of IOmerge tasks', 'type': 'integer', 'default': 1}), 'n_creategrib': mappingproxy({'description': 'Number of CreateGrib tasks', 'type': 'integer', 'default': 1}), 'n_addcalculatedfields': mappingproxy({'description': 'Number of AddCalculatedFields tasks', 'type': 'integer', 'default': 1}), 'member_specific_static_data': mappingproxy({'description': 'Whether to create member specific static data (true) or not', 'type': 'boolean', 'default': False}), 'member_specific_mars_prep': mappingproxy({'description': 'Whether to do member specific marsprep (true) or common marsprep', 'type': 'boolean', 'default': False}), 'max_static_data_tasks': mappingproxy({'description': 'Limit for max number of monthly tasks, run at same time', 'type': 'integer', 'default': 10}), 'DeodeCleaningSuiteDefinition': mappingproxy({'decription': 'Settings for Cleaning Suite', 'type': 'object', 'properties': mappingproxy({'do_clean_scratch_data': mappingproxy({'description': 'Activate cleaning the scratch data', 'type': 'boolean', 'default': True}), 'do_clean_suites': mappingproxy({'description': 'Activate cleaning the suites', 'type': 'boolean', 'default': True}), 'do_clean_IFS': mappingproxy({'description': 'Activate cleaning IFS data', 'type': 'boolean', 'default': True}), 'days_in_week': mappingproxy({'description': 'List of days when cleaning suite will run, 0-6 Sunday-Saturday', 'type': 'array', 'items': mappingproxy({'type': 'integer'})}), 'time_to_run': mappingproxy({'description': 'Time to run', 'type': 'string', 'pattern': '^([01]\\d|2[0-3]):[0-5]\\d:[0-5]\\dZ$'})})})})}), 'system': mappingproxy({'title': 'System', 'description': 'DEODE system specific path settings', 'type': 'object', 'properties': mappingproxy({'forecast_dir_link': mappingproxy({'type': 'string', 'description': 'Name of the softlink pointing to the Forecast task work directory', 'default': '../Forecast'}), 'bdfile_sfx_template': mappingproxy({'type': 'string', 'description': 'Name template for input initial surfex file for prep'}), 'bdfile_template': mappingproxy({'type': 'string', 'description': 'Name template for input boundary files'}), 'bddir_sfx': mappingproxy({'type': 'string', 'description': 'Location of input intial file for surfex'}), 'bddir': mappingproxy({'type': 'string', 'description': 'Location of input boundaries'}), 'intp_bddir': mappingproxy({'type': 'string', 'description': 'Location of boundaries on the current domain, i.e. the output directory of e927', 'default': '@WRK@'}), 'intp_bddir_sfx': mappingproxy({'type': 'string', 'description': 'Location of the inital surfex file for the current domain, i.e. the output directory of Prep', 'default': '@ARCHIVE@'}), 'climdir': mappingproxy({'type': 'string', 'description': 'Location of generated/used climate files (PGD/monthly files)'}), 'bindir': mappingproxy({'type': 'string', 'description': 'Location of binaries'}), 'bdclimdir': mappingproxy({'type': 'string', 'description': 'Location of host model climate files used for e927/c903'}), 'wrk': mappingproxy({'type': 'string', 'description': 'Location of date specific working directory', 'default': '@CASEDIR@/@YYYY@@MM@@DD@_@HH@@mm@'}), 'fullpos_config_file': mappingproxy({'type': 'string', 'description': 'Path to the fullpos yml config file'}), 'archive_timestamp': mappingproxy({'type': 'string', 'description': 'Relative path of date specific archive', 'default': '@YYYY@/@MM@/@DD@/@HH@/'}), 'archive': mappingproxy({'type': 'string', 'description': 'Absolute path of date specific archive', 'default': '@ARCHIVE_ROOT@/@ARCHIVE_TIMESTAMP@/@MEMBER_STR@'}), 'logs': mappingproxy({'type': 'string', 'description': 'Location of collected logfiles'}), 'namelists': mappingproxy({'type': 'string', 'description': 'Path to static namelists', 'default': '@DEODE_HOME@/data/namelists/@CYCLE@'}), 'marsdir': mappingproxy({'type': 'string', 'description': 'Location input files extracted from MARS by Marsprep'}), 'global_sfcdir': mappingproxy({'type': 'string', 'description': 'Path to SFC files needed for c903, which are not in MARS'}), 'prev_case': mappingproxy({'description': 'Name of the previous experiment', 'type': 'string', 'default': '@CASE@'}), 'sfx_input_definition': mappingproxy({'description': 'Path to the surfex json config file', 'type': 'string', 'default': '@CYCLE@/sfx.json'}), 'forecast_input_definition': mappingproxy({'type': 'string', 'description': 'Path to the forecast json config file', 'default': '@CYCLE@/forecast.json'}), 'c903_input_definition': mappingproxy({'type': 'string', 'description': 'Path to the c903 json config file', 'default': '@CYCLE@/c903.json'})})}), 'gribmodify': mappingproxy({'title': 'Section for modify grib', 'description': 'Rules for modifications to be made to the grib output', 'type': 'object', 'properties': mappingproxy({'conversions': mappingproxy({'description': 'List of file types to be converted', 'type': 'array', 'default': []}), 'gribmodify_rules_file': mappingproxy({'description': 'File specifying the gribmodify rules for the specified conversions', 'type': 'string'}), 'fullpos.1': mappingproxy({'title': 'Total rain', 'description': 'Calculates total rain from components', 'type': 'object', 'properties': mappingproxy({'$ref': '#/$gribmodify_rules'})}), 'fullpos.2': mappingproxy({'title': 'Total snow', 'description': 'Calculates total snow from components', 'type': 'object', 'properties': mappingproxy({'$ref': '#/$gribmodify_rules'})}), 'fullpos.3': mappingproxy({'title': 'Total precpipation', 'description': 'Calculates total precipitation from components', 'type': 'object', 'properties': mappingproxy({'$ref': '#/$gribmodify_rules'})}), 'fullpos.4': mappingproxy({'title': '10m wind speed', 'description': 'Calculates 10m wind speed', 'type': 'object', 'properties': mappingproxy({'$ref': '#/$gribmodify_rules'})}), 'fullpos.5': mappingproxy({'title': '100m wind speed', 'description': 'Calculates 100m wind speed', 'type': 'object', 'properties': mappingproxy({'$ref': '#/$gribmodify_rules'})}), 'fullpos.6': mappingproxy({'title': '10m gust wind speed', 'description': 'Calculates 10m gust wind speed', 'type': 'object', 'properties': mappingproxy({'$ref': '#/$gribmodify_rules'})}), 'surfex.1': mappingproxy({'title': 'Leaf area index', 'description': 'Calculates nature-weighted leaf area index', 'type': 'object', 'properties': mappingproxy({'$ref': '#/$gribmodify_rules'})}), 'surfex.2': mappingproxy({'title': 'Albedo', 'description': 'Calculates albedo', 'type': 'object', 'properties': mappingproxy({'$ref': '#/$gribmodify_rules'})}), 'surfex.3': mappingproxy({'title': 'Top 20 cm average soil moisture', 'description': 'Calculates the nature-weighted patch-averaged soil moisture in the top 20 cm', 'type': 'object', 'properties': mappingproxy({'$ref': '#/$gribmodify_rules'})}), 'surfex.4': mappingproxy({'title': 'Top 100 cm average soil moisture', 'description': 'Calculates the nature-weighted patch-averaged soil moisture in the top 100 cm', 'type': 'object', 'properties': mappingproxy({'$ref': '#/$gribmodify_rules'})}), 'surfex.5': mappingproxy({'title': 'Top 20 cm average soil temperature', 'description': 'Calculates the patch-averaged soil temperature in the top 20 cm', 'type': 'object', 'properties': mappingproxy({'$ref': '#/$gribmodify_rules'})}), 'surfex.6': mappingproxy({'title': 'Top 100 cm soil temperature', 'description': 'Calculates the patch-averaged soil temperature in the top 100 cm', 'type': 'object', 'properties': mappingproxy({'$ref': '#/$gribmodify_rules'})}), 'surfex.7': mappingproxy({'title': 'Root depth soil moisture', 'description': 'Calculates root depth soil moisture', 'type': 'object', 'properties': mappingproxy({'$ref': '#/$gribmodify_rules'})}), 'surfex.8': mappingproxy({'title': 'Canopy water', 'description': 'Calculates the patch-averaged canopy water', 'type': 'object', 'properties': mappingproxy({'$ref': '#/$gribmodify_rules'})})})}), 'general': mappingproxy({'title': 'General Section', 'description': "Model for the 'general' section.", 'type': 'object', 'properties': mappingproxy({'times': mappingproxy({'title': 'Time-Related Entries', 'description': "Model for the 'general.times' section.", 'type': 'object', 'properties': mappingproxy({'forecast_range': mappingproxy({'description': 'Forecast range', 'type': 'string', 'format': 'duration'}), 'start': mappingproxy({'description': "Start time, given as a date-time string 'YYYY-MM-DDTHH:mm:ssZ' or duration, e.g. '-P1D' for yesterday, which is relative to the current date at 00Z.", 'type': 'string', 'oneOf': [{'format': 'date-time'}, {'format': 'duration'}]}), 'end': mappingproxy({'description': 'End time if the run covers several cycles. Same format as start', 'type': 'string', 'oneOf': [{'format': 'date-time'}, {'format': 'duration'}]}), 'list': mappingproxy({'description': 'List of dates, placeholder for future needs.', 'type': 'array', 'items': mappingproxy({'type': 'string', 'format': 'date-time'})}), 'cycle_length': mappingproxy({'description': 'Cycle length defined as duration', 'default': 'PT3H', 'type': 'string', 'format': 'duration'}), 'basetime': mappingproxy({'description': 'Initial date and time for the current task.', 'type': 'string', 'format': 'date-time'}), 'validtime': mappingproxy({'description': 'Final date and time for the current task.', 'type': 'string', 'format': 'date-time'})}), 'oneOf': [{'anyOf': [{'required': ['start'], 'not': {'required': ['list']}}, {'required': ['basetime'], 'not': {'anyOf': [{'required': ['end']}, {'required': ['list']}, {'required': ['start']}]}}]}, {'required': ['list'], 'not': {'anyOf': [{'required': ['start']}, {'required': ['end']}]}}]}), 'accept_static_namelists': mappingproxy({'description': 'Allow usage of static namelists as input for the tasks. The namelist should be located in the directory specified by platform.namelists and named with the convention namelist_{kind}_{task} where kind could be surfex or master and task forecast/pgd or similar. For the extraction of namelist see [namelists.md](../docs/namelists.md).', 'type': 'boolean'}), 'case': mappingproxy({'description': 'Experiment name', 'default': 'deode_case', 'type': 'string'}), 'member_str': mappingproxy({'description': 'String identifier of a member in the ensemble', 'default': 'mbr000', 'type': 'string', 'pattern': '^mbr\\d{3}$'}), 'member': mappingproxy({'description': 'Member number of a member in the ensemble', 'default': 0, 'type': 'integer'}), 'cnmexp': mappingproxy({'description': 'Experiment short name', 'default': 'DEOD', 'type': 'string', 'minLength': 4, 'maxLength': 4}), 'loglevel': mappingproxy({'description': 'Logger output level', 'type': 'string', 'enum': ['CRITICAL', 'ERROR', 'DEBUG', 'INFO', 'WARNING']}), 'surfex': mappingproxy({'description': 'SURFEX switch', 'type': 'boolean'}), 'keep_workdirs': mappingproxy({'description': 'Do not remove working directories', 'type': 'boolean'}), 'csc': mappingproxy({'description': 'CSC', 'default': 'AROME', 'type': 'string', 'enum': ['AROME', 'ALARO', 'HARMONIE_AROME']}), 'cycle': mappingproxy({'description': 'IAL cycle', 'type': 'string', 'enum': ['CY49t2', 'CY48t3', 'CY46h1']}), 'event_type': mappingproxy({'description': 'Type of extreme event', 'type': 'string', 'default': 'nwp', 'enum': ['airquality', 'convection', 'flooding', 'heatwave', 'nwp', 'storm']}), 'initfile': mappingproxy({'description': 'Initial file for the forecast object. Normally no need to provide unless there are special requirements or the forecast object is used in stand alone mode.', 'type': 'string'}), 'initfile_sfx': mappingproxy({'description': 'Initial surfex file for the forecast object. Normally no need to provide unless there are special requirements or the forecast object is used in stand alone mode.', 'type': 'string'}), 'namelist': mappingproxy({'title': 'Namelist', 'description': 'Namelist settings. Various settings for the forecast namelist that have no other place to go.', 'type': 'object', 'properties': mappingproxy({'nrazts': mappingproxy({'description': 'Reset times for instantaneous fluxes ( e.g. min/max temperature )', 'type': 'string'}), 'radiation_frequency': mappingproxy({'description': 'Time interval for radiation compitations in the model.', 'type': 'string'})})}), 'output_settings': mappingproxy({'title': 'Output Settings', 'description': 'Output settings. Specified as duration for any of interval, endtime:interval, starttime:endtime:interval or a list of these options', 'type': 'object', 'properties': mappingproxy({'history': mappingproxy({'description': 'Output list for history files', '$ref': '#/$output_setting_rules'}), 'surfex': mappingproxy({'description': 'Output list for surfex files', '$ref': '#/$output_setting_rules'}), 'fullpos': mappingproxy({'description': 'Output list for fullpos files', '$ref': '#/$output_setting_rules'})})}), 'remove_sections': mappingproxy({'description': 'Sections to remove prior to config merge', 'type': 'array', 'default': []}), 'windfarm': mappingproxy({'description': 'Switch on wind farm parameterisation', 'type': 'boolean', 'default': False})}), 'required': ['times']})}), '$output_setting_rules': mappingproxy({'anyOf': [{'type': 'string', 'maxLength': 0}, {'type': 'string', 'format': 'duration'}, {'type': 'array', 'items': {'anyOf': [{'type': 'string', 'format': 'duration_slice'}]}}]}), '$gribmodify_rules': mappingproxy({'csc_specific': mappingproxy({'title': 'CSC specific selection', 'description': 'The CSCs for which the specified conversion should happen.', 'type': 'array', 'items': mappingproxy({'type': 'string'})}), 'output_name': mappingproxy({'title': 'Output parameter name', 'description': 'Chosen name of output parameter.', 'type': 'string'}), 'minumum_frequency': mappingproxy({'title': 'Minimum frequency', 'description': 'Minimum output frequency expected', 'format': 'duration'})})})

MAIN_CONFIG_JSON_SCHEMA_PATH = PosixPath('/home/runner/work/Deode-Workflow/Deode-Workflow/deode/data/config_files/config_file_schemas/main_config_schema.json')

PACKAGE_CONFIG_PATH = PosixPath('/home/runner/work/Deode-Workflow/Deode-Workflow/deode/data/config_files/config.toml')

PACKAGE_INCLUDE_DIR = PosixPath('/home/runner/work/Deode-Workflow/Deode-Workflow/deode/data/config_files/include')

SCHEMAS_DIRECTORY = PosixPath('/home/runner/work/Deode-Workflow/Deode-Workflow/deode/data/config_files/config_file_schemas')

class ConfigPaths[source]

Bases: object

Support multiple path search.

CONFIG_DATA_SEARCHPATHS = [PosixPath('/home/runner/work/Deode-Workflow/Deode-Workflow/deode/data')]

erroneous_paths = []

static path_from_subpath(subpath) → Path[source]

Interface to find full path given any subpath, by searching ‘searchpaths’.

Parameters:: subpath (str) – Subpath to search for
Returns:: Full path to target
Return type:: (Path)
Raises:: RuntimeRerror – Various errors

static print(config_file=None, host=None)[source]

Prints the available config directories.

Displays the main config search paths as defined by list_paths in addition to the actual search paths in the config file used.

exception ConflictingValidationSchemasError[source]

Bases: Exception

Error to be raised when more than one schema is defined for a config section.

class JsonSchema(*args, **kwargs)[source]

Bases: BaseMapping

Class to use for JSON schemas. Provides a validate method to validate data.

get_markdown_doc()[source]: Return human-readable doc for the schema in markdown format.

validate(data)[source]: Return a copy of data validated against the stored JSON schema.

class ParsedConfig(*args, json_schema, include_dir=PosixPath('/home/runner/work/Deode-Workflow/Deode-Workflow/deode/data/config_files'), host=None, **kwargs)[source]

Bases: BasicConfig

Object that holds parsed configs validated against a json_schema.

property data: Return the underlying data stored by the instance.

expand_macros(expand_all=False)[source]

Expand macros in config recursively.

Parameters:: expand_all (boolean) – Flag to expand all macros
Returns:: Parsed configuration
Return type:: config (ParsedConfig)

classmethod from_file(path, include_dir=None, **kwargs)[source]: Do as in BasicConfig. If None, include_dir will become path.parent.

property include_dir: Return the search dir used sections in the raw config’s include section.

property json_schema: Return the instance’s JSON schema.

evaluate_dynamic_dates(config_data)[source]: Replace dynamic dates like ‘yesterday’ with actual timestamps.

deode.custom_validators module

Define custom validators for Pydantic models.

Exports:: validate_durations: Validate the format of a duration string against ISO8601. import_from_string: Validate if a string can be imported and return the imported object.

import_from_string(value: str) → ModuleType | Type[source]

Validate if a string can be imported and return the imported object.

Only supports importing objects specified with full module path, e.g. module.submodule.object.

Parameters:

value – The string to import.

Raises:

ImportError – If the module or object cannot be imported.
TypeError – If the input is not a string.

Returns:

The imported object.

deode.datetime_utils module

Implement helper routines to deal with dates and times.

class DatetimeConstants(*_args, **_kwargs)[source]

Bases: QuasiConstant

Datetime-related constants.

DEFAULT_SHIFT = Timedelta('0 days 00:00:00')

ISO_8601_TIME_DURATION_REGEX = '^P(?!$)(\\d+Y)?(\\d+M)?(\\d+W)?(\\d+D)?(T(?=\\d+[HMS])(\\d+H)?(\\d+M)?(\\d+S)?)?$'

as_datetime(obj)[source]: Convert obj to string, parse into datetime and add UTC timezone iff naive.

as_timedelta(obj)[source]: Convert obj to string and parse into pd.Timedelta.

check_syntax(output_settings: Tuple[str] | List[str], length: int)[source]

Check syntax of output_settings.

Parameters:

output_settings (Union[Tuple[str], List[str]]) – Specifies the output steps
length (integer) – length to check on

Raises:

SystemExit – General system handler

cycle_offset(basetime, bdcycle, bdcycle_start=Timedelta('0 days 00:00:00'), bdshift=Timedelta('0 days 00:00:00'))[source]

Calculcate offset from a reference time.

Parameters:

basetime (datetime) – Reference time
bdcycle (timedelta) – Interval between cycles
bdcycle_start (timedelta) – Time of day when bdcycle starts
bdshift (timedelta) – shift of boundary usage

Returns:

a timdelta object of the offset

Return type:

timedelta

dt2str(dt)[source]

Convert timdelta object to file name suitable string.

Parameters:

dt (timedelta object) – duration

Returns:

string representation of duration: suitable for FA files

Return type:

duration (str)

evaluate_date(date: str) → str[source]

Parses an ISO 8601 datetime and/or duration and returns the computed datetime.

If the input is a datetime (e.g., “2025-03-19T00:00:00Z”), it returns it as-is.
If the input includes a duration (e.g., “2025-03-19T00:00:00Z/-P1D”), it applies the duration to the datetime.
If the input is only a duration (e.g., “-P1D”), it applies it to today’s midnight

(UTC).

Parameters:: date (str) – An ISO 8601 datetime, duration, or both.
Returns:: The computed datetime in ISO format.
Return type:: str

expand_output_settings(output_settings: str | Tuple[str] | List[str], forecast_range: str) → List[List[Timedelta]][source]

Expand the output_settings coming from config.

Parameters:

output_settings (Union[str, Tuple[str], List[str]]) – Specifies the output steps
forecast_range (str) – Forecast range in duration syntax

Raises:

RuntimeError – Handle erroneous time increment

Returns:

List of output subsections.: Can be empty in case of empty output_settings

Return type:

sections (List[List[pd.Timedelta]])

get_decadal_list(dt_start, dt_end) → list[source]: Return a list of dates for which decadal pgd files have to be created.

get_decade(dt) → str[source]: Return the decade given a datetime object.

get_month_list(start, end) → list[source]: Get list of months between to given dates (input as string).

oi2dt_list(output_settings: str | Tuple[str] | List[str], forecast_range: str) → List[Timedelta][source]

Build list of output occurences.

Parameters:

output_settings (Union[str, Tuple[str], List[str]]) – Specifies the output steps
forecast_range (str) – Forecast range in duration syntax

Returns:

Sorted list of output occurences

Return type:

dt (List[pd.Timedelta])

deode.derived_variables module

Derive runtime variables.

check_fullpos_namelist(config, nlgen)[source]

Find existing fullpos select files or generate them.

Parameters:

config (deode.ParsedConfig) – Configuration
nlgen (dict) – master forecast namelist

Returns:

Possibly updated forecast namelist

Return type:

nlgen (dict)

derived_variables(config, processor_layout=None)[source]

Derive some variables required in the namelists.

Parameters:

config (deode.ParsedConfig) – Configuration
processor_layout (ProcessorLayout, optional) – Processor layout object

Returns:

Derived config update

Return type:

update (dict)

Raises:

NotImplementedError – For configurations checking

set_times(config)[source]

Set basetime/validtime if not present.

Parameters:: config (.config_parser.ParsedConfig) – Parsed config file contents.
Raises:: ValueError – If start > end
Returns:: Dict of corrected basetime/validtime
Return type:: update (dict)

deode.experiment module

Experiment tools.

class EPSExp(config: ParsedConfig)[source]

Bases: Exp

Setup EPS experiment from config file.

setup_exp(host=None)[source]

Generate EPS member settings and update config file.

Only deviations of member settings from general EPS settings are added to the config.

class Exp(config: ParsedConfig, merged_config: dict | None)[source]

Bases: object

Experiment class.

class ExpFromFiles(config: ParsedConfig, exp_dependencies, mod_files: List[Path], host=None, merged_config=None)[source]

Bases: Exp

Generate Exp object from existing files. Use config files from a setup.

static deep_update(source, overrides)[source]

Update a nested dictionary or similar mapping.

Modify source in place.

Parameters:

source (dict) – Source
overrides (dict) – Updates

Returns:

Updated dictionary

Return type:

dict

static setup_files(output_file, case=None)[source]

Set up the files for an experiment.

Parameters:

output_file (str) – Output file
case (str, optional) – Experiment name. Defaults to None.

Returns:

Experiment dependencies from setup.

Return type:

exp_dependencies(dict)

static toml_dump(to_dump, fname)[source]

Dump toml to file.

Using tomlkit to preserve stucture

Parameters:

to_dump (dict) – Data to save
fname (str) – Filename

case_setup(config: ParsedConfig, output_file, mod_files: List[Path], case=None, host=None, expand_config=False)[source]

Do experiment setup.

Parameters:

config (.config_parser.ParsedConfig) – Parsed config file contents.
output_file (str) – Output config file.
mod_files (list) – Modifications. Defaults to None.
case (str, optional) – Case identifier. Defaults to None.
host (str, optional) – host name. Defaults to None.
expand_config (boolean, optional) – Flag for expanding macros in config

Returns:

Output config file.

Return type:

output_file (str)

get_git_info()[source]: Get git information.

deode.formatters module

Define format validators used in the JSON schema validation.

duration_format_validator(duration: str) → bool[source]

Validate the format of a duration string against ISO8601.

Parameters:: duration – The duration string to validate.
Returns:: True if the duration string is valid, False otherwise.

duration_slice_format_validator(duration_slice: str) → bool[source]

Validate the format of a duration slice string against ISO8601.

Each part of the slice is validated using duration_format_validator.

Parameters:: duration_slice – The duration slice string to validate.
Returns:: True if the duration slice string is valid, False otherwise.

deode.fullpos module

Fullpos namelist generation.

class Fullpos(domain, fpdir='.', fpfiles=None, fpdict=None, rules=None)[source]

Bases: object

Fullpos namelist generator based on (yaml) dicts.

check_non_instant_fields(selection, time_selection)[source]

Search for non instant fields.

Parameters:

selection (dict) – Dict with fullpos settings to be examined
time_selection (str) – Which selection time rules to check

Raises:

RuntimeError – Non instant fields found

construct()[source]

Construct the fullpos namelists.

Returns:: namfpc part selection (dict): xxtddddhhmm part
Return type:: namfpc_out (dict)
Raises:: InvalidSelectionCombination – Invalid combination in selection

expand(v, levtype, levels, domain, i)[source]

Expand fullpos namelists to levels and domains.

Parameters:

v (str) – parameter list
levtype (str) – type of vertical level in fullpos syntax
levels (list) – list of levels
domain (str) – domain name
i (int) – parameter conuter

Returns:

expaned names i (int): parameter counter

Return type:

d (dict)

load(fpdir, fpfiles)[source]

Load fullpos yaml file.

Parameters:

fpdir (str) – Path do fullpos settings
fpfiles (list) – List of yml files to read

Returns:

fullpos settings

Return type:

nldict (dict)

replace_rules(vin)[source]

Replace patterns from the rules dict.

Parameters:: vin (str|list) – Input values
Raises:: RuntimeError – Invalid type
Returns:: possibly replaced strings
Return type:: v (str|list)

update_selection(additions_list=None, additions_dict=None)[source]

Add choices to the selection section.

Parameters:

additions_list (list) – Additional selection to be read from files
additions_dict (dict) – Additional selection as a dictionary

exception InvalidSelectionCombinationError[source]

Bases: ValueError

Custom exception.

flatten_list(li)[source]: Recursively flatten a list of lists (of lists).

deode.general_utils module

General utils for use throughout the package.

expand_dict_key_slice(dict_: Dict[int | str, Any], indices: List[int]) → Dict[int, Any][source]

Expand key slices of a Dict.

Handles slices in the form of “start:stop:step”, expands them to individual keys, and assigns the original value to all individual keys. Keys are converted to integers.

Any of the start, stop and step can be ommited. If start is ommited, it is set to the minimum value of indices. If stop is ommited, it is set to the maximum value of indices. If step is ommited, it is set to 1.

Parameters:

dict – The dict, which keys shall be expanded.
indices – The indices to respect when expanding, i.e. if expanded index is not in indices, it will not be added to the new dict.

Returns:

New dict with expanded keys.

Return type:

dict

get_empty_nested_defaultdict()[source]: Return an empty nested (recursive) defaultdict object.

merge_dicts(dict1: dict, dict2: dict, overwrite: bool = False) → dict[source]

Merge two dictionaries with values from dict2 taking precedence.

If values are lists, they are concatenated.

Parameters:

dict1 (dict) – Reference dict
dict2 (dict) – Update dict
overwrite (bool) – Whether to overwrite values in dict1 with values from dict2 if the keys are the same, but the types of the values are not lists or dicts.

Returns:

Merged dict

Return type:

(dict)

Raises:

RuntimeError – Invalid type

modify_mappings(obj: Mapping, operator: Mapping | Callable[[Mapping], Any])[source]: Descend recursively into obj and modify encountered mappings using operator.

recursive_delete_keys(mapping: Dict[str, Any], keys_dict: Dict[str, bool])[source]

Recursively delete keys from a mapping based on a dictionary of keys to delete.

Keys are deleted in-place on the mapping object.

Parameters:

mapping – The mapping to delete keys from.
keys_dict – The dictionary of keys to delete.

recursive_dict_deviation(base_dict: dict, deviating_dict: dict) → dict[source]

Calculate the (recursive) difference between two dicts.

Parameters:

base_dict – The base dictionary to calculate deviation from.
deviating_dict – The dict to calculate the deviation of w.r.t to the base_dict

Returns:

The deviation as a dictionary.

value_from_any_generator(any_: Any | Sequence[Any] | Mapping[int, Any], indices: List[int], default_value: str | None = None) → Generator[str, None, None][source]

Yield values from any type.

Parameters:

any – The input object to yield values from.
indices – The indices to retrieve from the value.
default_value – The default value to use if an index is not found in Mapping.

Yields:

The value from the input object.

value_from_mapping_generator(mappable: Mapping[int, Any], keys: List[int], default_value: Any) → Generator[Any, None, None][source]

Yield values from a dictionary according to keys.

Parameters:

mappable – The mappable to yield values from
keys – The keys for which to retrieve corresponding values from the dictionary.
default_value – The default value to use if a key is not found.

Yields:

The value corresponding to the key.

value_from_sequence_generator(sequence: Sequence[Any]) → Generator[Any, None, None][source]

Yield alternately one of the values from a sequence of values.

The order of the yielded values is determined by the order of the sequence.

Parameters:: sequence – The sequence to yield values from.
Yields:: One of the values from sequence in alternate order.

deode.geo_utils module

Utilities for simple geographic tasks.

class Projection(proj4str)[source]

Bases: object

Projection class.

check_key(key: str, config: dict) → bool[source]

Check if key is in config.

Parameters:

key (str) – Key to check
config (dict) – Configuration

Returns:

True if key is in config

Return type:

bool

Raises:

ValueError – If key is not in config

get_domain_properties(domain_spec: dict) → dict[source]

Get domain properties.

Parameters:: domain_spec (dict) – Domain specification
Returns:: Domain properties
Return type:: dict

class Projstring[source]

Bases: object

Proj4 string class.

get_projstring(lon0=0.0, lat0=-90.0) → str[source]

Get proj4 string.

Parameters:

lon0 (float, optional) – Central longitude. Defaults to 0.0.
lat0 (float, optional) – Central latitude. Defaults to -90.0.

Returns:

Proj4 string

Return type:

str

deode.host_actions module

Handle host detection.

exception AmbigiousHostError[source]

Bases: ValueError

Custom exception.

class DeodeHost(known_hosts=None, known_hosts_file=None)[source]

Bases: object

DeodeHost object.

detect_deode_host(use_default=True)[source]

Detect deode host by matching various properties.

First check self.deode_host as set by os.getenv(“DEODE_HOST”), second use the defined hosts in known_hosts.yml. If no matches are found return the first host defined in known_hosts.yml

Parameters:: use_default (boolean, optional) – Flag to return default host if host not found
Raises:: RuntimeError – Ambiguous matches
Returns:: mapped hostname
Return type:: deode_host (str)

exception HostNotFoundError[source]

Bases: ValueError

Custom exception.

class SelectHost[source]

Bases: object

Class for the host selection.

set_deode_home(config, deode_home=None)[source]

Set deode_home in various ways.

Parameters:

config (.config_parser.ParsedConfig) – Parsed config file contents.
deode_home (str) – Externally set deode_home

Returns:

deode_home

deode.initial_conditions module

Initial_conditions.

class InitialConditions(config)[source]

Bases: object

FirstGuess.

check_if_found()[source]: Check if files are present.

find_initial_files()[source]: Find initial file.

success()[source]: Report of success.

deode.logs module

Logging-related classes, functions and definitions.

class InterceptHandler(level=0)[source]

Bases: Handler

Add logging handler to augment python stdlib logging.

Logs which would otherwise go to stdlib logging are redirected through loguru.

emit(record: LogRecord) → None[source]

Emit a record.

Parameters:: record (builtin_logging.LogRecord) – Output record

class LogDefaults(*_args, **_kwargs)[source]

Bases: QuasiConstant

Defaults used for the logging system.

DIRECTORY = PosixPath('/home/runner/.logs/deode')

LEVEL = 'INFO'

RETENTION_TIME = '1 week'

SINKS = mappingproxy({'console': <_io.TextIOWrapper name='<stderr>' mode='w' encoding='utf-8'>})

class LogFormatter(datetime: str = '<green>{time:YYYY-MM-DD HH:mm:ss}</green>', level: str = '<level>{level: <8}</level>', code_location: str = '<cyan>@{name}</cyan>:<cyan>{function}</cyan> <cyan><{file.path}</cyan>:<cyan>{line}>:</cyan>', message: str = '<level>{message}</level>')[source]

Bases: object

Helper class to setup logging without poluting the module’s main scope.

code_location: str = '<cyan>@{name}</cyan>:<cyan>{function}</cyan> <cyan><{file.path}</cyan>:<cyan>{line}>:</cyan>'

datetime: str = '<green>{time:YYYY-MM-DD HH:mm:ss}</green>'

format_string(loglevel: str)[source]: Return the appropriate fmt string according to log level and fmt opts.

level: str = '<level>{level: <8}</level>'

message: str = '<level>{message}</level>'

class LoggerHandlers(default_level: str = 'INFO', **sinks)[source]

Bases: Sequence

Helper class to configure logger handlers when using loguru.logger.configure.

add(name, sink, **configs)[source]: Add handler to instance.

log_elapsed_time(**kwargs)[source]: Return a decorator that logs beginning, exit and elapsed time of function.

deode.mars_utils module

Utility for marsprep.

class BaseRequest(*, class_: str, data_type: str, expver: str, levtype: str, date: str, time: str, steps: str, param: str, target: str)[source]

Bases: object

Represents a structured data request with configurable parameters.

This class facilitates the creation of data requests by encapsulating request attributes and providing methods for modifying and enhancing the request structure. The request data is stored in a dictionary accessible via the request attribute.

Parameters:

class (str) – The class of the request, e.g., “D1”.
data_type (str) – The type of data requested (e.g., analysis, forecast).
expver (str) – Experiment version identifier.
levtype (str) – Level type (e.g., surface, pressure levels).
date (str) – The date range for the request in YYYYMMDD format.
time (str) – The time range for the request in HHMM format.
steps (str) – Forecast steps or time intervals.
param (str) – Parameter codes for the requested data.
target (str) – Target file or identifier for the output.
request (dict) – A dictionary representation of the request, initialized automatically upon object creation.

add_database_options()[source]: Add database options.

add_eps_members(members: List[int], prefetch: bool)[source]

Fix parameters in case of eps members.

Parameters:

members (list) – members to retrieve
prefetch (string) – type of retrieve

add_levelist(levelist: str)[source]

Set multilevel if needed.

Parameters:: levelist (string) – list of levels
Returns:: None

class_: str

data_type: str

date: str

expver: str

levtype: str

param: str

replace(**kwargs)[source]: Return new instance with updated values.

request: dict

steps: str

target: str

time: str

update_request(upd: dict)[source]

Add or replace heys in a mars request.

Parameters:: upd (dict) – a dict with keys and values

add_additional_data_to_all(tag: str, steps: List[str], members_dict: Dict[str, List[int]], additional_data: Dict[str, bytes])[source]

Add additional common data to all files defined by tag, step and member.

The additional_data dict is expected to have the following structure: {

“data_key1”: b”Data to add to all files”, “data_key2”: b”Data to add to all files”, …

}

Parameters:

tag (str) – Name of tag
steps (List[int]) – Steps used to construct filename.
members_dict (Dict[str, List[int]]) – Dictionary with members used to construct filename.
additional_data (Dict[str, bytes]) – Dictionary containing the data to add.

Raises:

FileNotFoundError – If a file does not exist when trying to append data to it.

add_additional_file_specific_data(additional_data: Dict[str, bytes])[source]

Add additional file specific data.

The additional_data dict is expected to have the following structure, where the “common_data” key is optional: {

“common_data”: b”Common data to add to all files”, “file1”: b”Data to add to file1”, “file2”: b”Data to add to file2”, …

}

If the “common_data” key is present, the data will be added to a file before the file-specific data is added.

Parameters:: additional_data (Dict[str, bytes]) – Dictionary containing the data to add (value) to a given file (key).
Raises:: FileNotFoundError – If a file does not exist when trying to append data to it.

check_data_available(basetime, mars)[source]

Check if there is data for basetime for choosen expver.

Parameters:

basetime (str) – basetime
mars (dict) – mars config section

Raises:

ValueError – No data for this date.

get_and_remove_data(file_name: str) → bytes[source]

Read in and subsequently remove binary data.

Parameters:: file_name (str) – The name of the file containing the data to be read.
Returns:: The binary data read from the file.
Return type:: bytes

get_domain_data(config)[source]

Read and return domain data.

Parameters:: config (deode.ParsedConfig) – Configuration from which we get the domain data
Returns:: String containing the domain info for MARS

get_mars_keys(source, key_filter='-w shortName:s=z')[source]: Get the basic MARS settings from a static file, to correct a request.

get_steplist(bd_offset, fc_range, bdint)[source]

Get the list of steps for Mars request.

Parameters:

bd_offset (timedelta) – first boundary time
fc_range (timedelta) – forecast range
bdint (timedelta) – frequency of boundary files

Returns:

list of steps

Return type:

steps (List[int])

get_steps_and_members_to_retrieve(steps: List[int], path: Path, file_name: str, members: List[int]) → Tuple[List[int], Dict[str, List[int | None]]][source]

Check which mars file already exist and returns steps and members which missing.

Parameters:

steps (List[int]) – list of steps
path (pathlib.Path) – path to global files
file_name (string) – name of file to check
members (List[int]) – members to retrieve

Returns:

steps to retrieve members_dict (Dict[str, List[int | None]]): members to retrieve

Return type:

steps (List[int])

get_value_from_dict(dict_, key_orig)[source]

Check value according to key.

If a string returns the value itself
If key is a date search for the most suitable match in value
Else return the value matching the key.

Parameters:

dict (str, BaseConfig object) – Values to select
key_orig (str) – key for value checking

Returns:

Found value

Return type:

value (str)

Raises:

ValueError – Exception

move_files(tag: str, steps: List[int], members_dict: Dict[str, List[int]], target_dir: Path)[source]

Move files to the final location.

Parameters:

tag (str) – Name of tag
steps (list) – steps to process the files
members_dict (Dict[str, List[int]]) – dict with members to procces the files
target_dir (Path) – target directory to move the files to

write_mars_req(req, name, method)[source]

Write a request for MARS.

Parameters:

req (BaseRequest) – namelist object to write
name (string) – request file name
method (string) – selected method, retrieve or read

deode.namelist module

Namelist handling for MASTERODB w/SURFEX.

exception InvalidNamelistKindError[source]

Bases: ValueError

Custom exception.

exception InvalidNamelistTargetError[source]

Bases: ValueError

Custom exception.

class NamelistComparator(config)[source]

Bases: object

Helper class for namelist generation and integration.

compare_dicts(dbase, dcomp, action)[source]

Compare two dictionaries, recursively if needed.

The dict must only consist of dict, list, str, float, int, bool

Parameters:

dbase – base dict
dcomp – comparison dict
action (str) – one of ‘intersection’, ‘diff’ or ‘union’

Returns:

result dict from the specified comparison action

Return type:

dout

Raises:

SystemExit # noqa – DAR401

compare_lists(libase, licomp, sib, sic, sio, action)[source]

Compare two lists, recursively if needed.

The list must only consist of dict, list, str, float, int, bool

Parameters:

libase – base list
licomp – comparison list
sib – list of start indices for libase
sic – list of start indices for licomp
sio – list of start indices for liout (modified)
action (str) – one of ‘intersection’, ‘diff’ or ‘union’

Returns:

result list from the specified comparison action

Return type:

liout

Raises:

SystemExit # noqa – DAR401

class NamelistConverter[source]

Bases: object

Helper class to convert namelists between cycles, based on thenamelisttool.

static apply_tnt_directives_to_ftn_namelist(tnt_directive_filename, input_ftn)[source]

Apply the tnt directives to a fotran namelist using tnt.

Parameters:

tnt_directive_filename – the tnt directive filename
input_ftn – the namelist fortran file

Raises:

SystemExit – when conversion failed

static apply_tnt_directives_to_namelist_dict(tnt_directive_filename, namelist_dict)[source]

Apply the tnt directives to a namelist as dictionary.

Parameters:

tnt_directive_filename – the tnt directive filename
namelist_dict – the namelist dictionary

Returns:

the converted namelist dictionary

Return type:

new_namelist

Raises:

SystemExit – when conversion failed

static convert_ftn(input_ftn, output_ftn, from_cycle, to_cycle)[source]

Convert a namelist in fortran file between two cycles.

Parameters:

input_ftn – the input fortran filename
output_ftn – the output fortran filename
from_cycle – the input cycle
to_cycle – the target cycle

static convert_yml(input_yml, output_yml, from_cycle, to_cycle)[source]

Convert a namelist in yml file between two cycles.

Parameters:

input_yml – the input yaml filename
output_yml – the output yaml filename
from_cycle – the input cycle
to_cycle – the target cycle

Raises:

SystemExit – when conversion failed

static get_known_cycles()[source]: Return the cycles handled by the converter.

static get_tnt_files_list(from_cycle, to_cycle)[source]: Return the list of tnt directive files required for the conversion.

static get_to_next_version_tnt_filenames()[source]: Return the tnt file names between get_known_cycles().

class NamelistGenerator(config, kind, substitute=True)[source]

Bases: object

Fortran namelist generator based on hierarchical merging of (yaml) dicts.

assemble_namelist(target)[source]

Generate the namelists for ‘target’.

Parameters:: target (str) – task to generate namelists for
Returns:: Assembled namelist
Return type:: nlres (f90nml.Namelist)

fn_config(arg, default=None)[source]: Resolve namelist function cfg.

fn_stepfreq(arg)[source]: Resolve namelist function stepfreq.

fn_steplist(time_intervals)[source]: Resolve namelist function steplist.

generate_namelist(target, output_file)[source]

Generate the namelists for ‘target’.

Parameters:

target (str) – task to generate namelists for
output_file – where to write the result (fort.4 or EXSEG1.nam typically)

load(target)[source]

Generate the namelists for ‘target’.

Parameters:: target (str) – task to generate namelists for
Raises:: InvalidNamelistTargetError # noqa – DAR401
Returns:: Assembled namelist
Return type:: nlres (dict)

load_user_namelist()[source]

Read user provided namelist.

Returns:: Logics if namelist is found or not nldict : Namelist as dictionary cndict : Rules for dictionary
Return type:: found (boolean)

resolve_macros(cn)[source]: Resolve all macros in a nested dict (both keys and values!).

update(nldict, cndict_tag)[source]

Update with additional namelist dict.

Parameters:

nldict (dict) – additional namelist dict
cndict_tag – name to be used for recognition

update_from_config(target)[source]

Update with additional namelist dict from config.

Parameters:: target (str) – task to generate namelists for

write_namelist(nml, output_file)[source]

Write namelist using f90nml.

Parameters:

nml (f90nml.Namelist) – namelist to write
output_file (str) – namelist file name

class NamelistIntegrator(config)[source]

Bases: object

Helper class to read fortran namelists and store as yaml, in a more compact form.

Reduces duplication if several namelists have similar settings.

static dict2yml(nmldict, ymlfile, ordered_sections=None)[source]: Write dict as yaml file.

ftn2dict(ftnfile)[source]: Read fortran namelist file with f90nml and return as plain dict.

static yml2dict(ymlfile)[source]: Read yaml namelist file and return as dict.

find_value(s)[source]: Purpose: un-quote (list of) numbers and booleans.

flatten_cn(li)[source]: Recursively flatten a list of lists (of lists).

list_set_at_index(li, ix, val)[source]: Handle ‘sparse’ lists (that may contain null values).

represent_ordereddict(dumper, data)[source]: YAML representer, simplifies working with namelists read by f90nml.

to_dict(x)[source]: Recursively modify OrderedDict etc. to normal dict (required for OmegaConf).

write_namelist(nml, output_file)[source]

Write namelist using f90nml.

Parameters:

nml (f90nml.Namelist) – namelist to write
output_file (str) – namelist file name

deode.os_utils module

Utilities for simple tasks on OS level.

class Search[source]

Bases: object

Search class.

static find_files(directory, prefix='', postfix='', pattern='', recursive=True, onlyfiles=True, fullpath=False, olderthan=None, inorder=False) → list[source]

Find files in a directory.

Parameters:

directory (str) – Directory to search in.
prefix (str, optional) – Only find files with this prefix. Defaults to “”.
postfix (str, optional) – Only find files with the postfix. Defaults to “”.
pattern (str, optional) – Only find files with matching pattern. Defaults to “”.
recursive (bool, optional) – Go into directories recursively. Defaults to True.
onlyfiles (bool, optional) – Show only files. Defaults to True.
fullpath (bool, optional) – Give full path. Defaults to False. If recursive=True, fullpath is given automatically.
olderthan (int, optional) – Match only files older than X seconds from now. Defaults to None.
inorder (bool, optional) – Return sorted list of filenames. Defaults to False.

Returns:

List containing file names that matches criterias

Return type:

list

Examples

>>> files = find_files(
                '/foo/', prefix="", postfix="", recursive=False,
                onlyfiles=True, fullpath=True, olderthan=86400*100
            )

deodemakedirs(path: str | Path, unixgroup='', exist_ok=True, def_dir_mode=493)[source]

Create directories and change unix group as required.

For a given path the top directory that does not yet exist is searched for, created and unix group is set, if required. Permissions are set such that all subdirectories and new files inherit the unix group.

Parameters:

path (str | Path) – directory path that should be created if it doesn’t already exist.
unixgroup (str, optional) – unix group the newly created dirs should belong to.
exist_ok (boolean, optional) – Define whether directories may already exist or whether an error should be raised.
def_dir_mode (int, optional) – Default directory persmission mode. Defaults to 0o755

Raises:

OSError – If cannot create the directory.

filepath_iterator(paths, filename_pattern='*')[source]

Return iterator of paths to files given a list of file or directory paths.

Given a path or list of paths, yield Path objects corresponding to them. If a path points to a directory, then the directory is searched recursively and the paths to the files found in this process will be yielded.

Parameters:

paths (Union[pathlib.Path, List[pathlib.Path], str, List[str]]) – A single path or a collection of paths.
filename_pattern (str, optional) – Pattern used in the recursive glob in order to select the names of the files to be yielded. Defaults to “*”.

Yields:

pathlib.Path – Path to located files.

list_files_join(folder, f_pattern)[source]

Read and return file names based on given pattern.

Parameters:

folder – path with file location
f_pattern – glob pattern

Returns:

list of files that should be joined

ping(host)[source]

Ping host.

Parameters:: host (str) – Host to ping
Returns:: True if host responded
Return type:: (boolean)

remove_empty_dirs(src, dry_run=False)[source]

Remove directories.

Recursively and permanently removes the specified directory, and all of its empty subdirectories.

Parameters:

src (str or Path) – Top search directory
dry_run (boolean) – Flag for execution of cleaning or not

Returns:

True if any files found

Return type:

found_files (boolean)

resolve_path_relative_to_package(path: Path, ignore_errors: bool = False) → Path[source]

Resolve path relative to package directory.

If the path exists as is, return it. If not, check if it exists in the package directory and return path relative to package

Parameters:

path (Path) – Path to resolve.
ignore_errors (bool, optional) – Option to ignore errors. Defaults to False.

Raises:

FileNotFoundError – If it was impossible to determine path relative to package.
FileNotFoundError – If file does not exist locally or in the package directory.

Returns:

Original path (if exists locally), or resolved path relative to: package directory.

Return type:

Path

strip_off_mount_path(path: str | Path) → Path[source]

Strip off the mount path from a given path.

Assumptions:

the path contains the user name as a directory.
the parent of the user directory is of the format “<new-dir-name>” or “_<new-dir-name>_”, where “*” contains no underscore(s) and where the <new-dir-name> will be used as the new parent directory name relative to the user directory.

Parameters:: path (Union[str, Path]) – Path to strip off the mount path from.
Returns:: Path with the mount path stripped off.
Return type:: path
Raises:: ValueError – If the parent of the user directory does not contain 0 or 2 underscores.

Example

>>> strip_off_mount_path("/etc/ecmwf/nfs/dh1_home_b/$USER/Deode-Workflow/deode")
Path("/home/$USER/Deode-Workflow/deode")

deode.plugin module

Plug-in functionality.

class DeodePlugin(name, path)[source]

Bases: object

Deode plugin.

class DeodePluginFromConfigFile(config_file)[source]

Bases: DeodePlugin

Deode plugin.

static get_plugin_config(config_file)[source]: Get the registry config.

class DeodePluginRegistry(config=None)[source]

Bases: object

Registry of plugins.

deode_plugin()[source]: Base DEODE plugin.

get_registry_config()[source]: Get the registry config.

load_plugin(plugin)[source]

Load plugin.

Parameters:: plugin (DeodePlugin) – Deode plugin

load_plugins()[source]: Load all registered plugins.

plugin_exists(plugin)[source]

Check if plugin exists.

Parameters:: plugin (DeodePlugin) – Deode plugin
Returns:: True if already exists in registry.
Return type:: bool

register_plugin(plugin)[source]

Register plugin.

Parameters:: plugin (DeodePlugin) – Deode plugin

save_registry(config_file)[source]

Save registry config.

Parameters:: config_file (str) – Filename

class DeodePluginRegistryFromConfig(config)[source]

Bases: DeodePluginRegistry

Create a registry from a deode config file.

class DeodePluginRegistryFromFile(config_file)[source]

Bases: DeodePluginRegistry

Registry file of plugins.

deode.scheduler module

Scheduler module.

class EcflowClient(server, task)[source]

Bases: object

An ecflow client.

Encapsulate communication with the ecflow server. This will automatically call the child command init()/complete(), for job start/finish. It will also handle exceptions and signals, by calling the abort child command. ONLY one instance of this class, should be used. Otherwise zombies will be created.

static at_time()[source]

Generate time stamp.

Returns:: Time stamp.
Return type:: str

signal_handler(signum, extra=None)[source]

Signal handler.

Parameters:

signum (_type_) – _description_
extra (_type_, optional) – _description_. Defaults to None.

class EcflowLogServer(config)[source]

Bases: object

Ecflow log server.

class EcflowServer(config, start_command=None)[source]

Bases: Server

Ecflow server.

begin_suite(suite_name)[source]

Begin the suite.

Parameters:: suite_name (str) – Nam eof the suite.

force_aborted(task)[source]

Force the task aborted.

Parameters:: task (scheduler.EcflowTask) – Task to force aborted.

force_complete(task)[source]

Force the task complete.

Parameters:: task (scheduler.EcflowTask) – Task to force complete.

remove_suites(suite_list)[source]

Remove suites selected from a list.

Parameters:: suite_list (list) – Suite names.

replace(suite_name, def_file)[source]

Replace the suite name from def_file.

Parameters:

suite_name (str) – Suite name.
def_file (str) – Definition file.

Raises:

RuntimeError – If suite cannot be replaced.

start_server()[source]

Start the server.

Raises:: RuntimeError – Server is not running or Could not restart server.

class EcflowTask(ecf_name, ecf_tryno, ecf_pass, ecf_rid, ecf_timeout=20)[source]

Bases: object

Ecflow scheduler task.

class Server(config)[source]

Bases: ABC

Base server/scheduler class.

abstract begin_suite(suite_name)[source]

Begin the suite in a server specific way.

Parameters:: suite_name (str) – Name of the suite
Raises:: NotImplementedError – Must be implemented by the child server object.

abstract replace(suite_name, def_file)[source]

Create or change the suite definition.

Parameters:

suite_name (str) – Name of the suite.
def_file (str) – Name of the definition file.

Raises:

NotImplementedError – Must be implemented by the child server object.

abstract start_server()[source]

Start the server.

Raises:: NotImplementedError – Must be implemented by the child server object.

start_suite(suite_name, def_file, begin=True)[source]

Start the suite.

All the servers have these methods implemented and can start the server in a server specific way.

Parameters:

suite_name (str) – Name of the suite
def_file (str) – Name of the definition file.
begin (bool, optional) – If the suite should begin. Defaults to True.

deode.submission module

Module to handle submissions.

class NoSchedulerSubmission(task_settings)[source]

Bases: object

Create and submit job without a scheduler.

submit(task: str, config: ParsedConfig, template_job: str, task_job: Path, output: Path, member: int | None = None, troika: str = 'troika')[source]

Submit task.

Parameters:

task (str) – Task name
config (deode.ParsedConfig) – Config
template_job (str) – Task template job file
task_job (Path) – Task job file
output (Path) – Output file
member (int, optional) – Member number for which to submit job. Defaults to None.
troika (str, optional) – troika binary. Defaults to “troika”.

Raises:

RuntimeError – Submission failure.

class ProcessorLayout(kwargs: dict)[source]

Bases: object

Set processor information.

get_proc_dict()[source]: Generate a processor dict.

get_wrapper()[source]: Get and potentially parse the wrapper.

class TaskSettings(config)[source]

Bases: object

Set the task specific setttings.

get_settings(task)[source]

Get the settings.

Parameters:: task (_type_) – _description_
Returns:: _description_
Return type:: _type_

get_task_settings(task, key=None, variables=None, ecf_micro='%')[source]

Get task settings.

Parameters:

task (_type_) – _description_
key (_type_, optional) – _description_. Defaults to None.
variables (_type_, optional) – _description_. Defaults to None.
ecf_micro (str, optional) – _description_. Defaults to “%”.

Returns:

_description_

Return type:

_type_

parse_job(task: str, config: ParsedConfig, input_template_job: str, task_job: Path, member: int | None = None, variables=None, ecf_micro: str = '%', scheduler: str = 'ecflow')[source]

Read default job and change interpretor.

Parameters:

task (str) – Task name
config (deode.config) – The configuration
input_template_job (str) – Input container template.
task_job (Path) – Task container
member (int, optional) – Member number for which to parse job. Defaults to None.
variables (_type_, optional) – _description_. Defaults to None.
ecf_micro (str, optional) – _description_.
scheduler (str, optional) – Scheduler. Defaults to ecflow.

Raises:

RuntimeError – In case of missing module env file

parse_submission_defs(task)[source]

Parse the submssion definitions.

Parameters:: task (str) – The name of the task
Returns:: Parsed settings
Return type:: dict
Raises:: RuntimeError – Undefined submit type

recursive_items(dictionary)[source]

Recursive loop of dict.

Parameters:: dictionary (_type_) – _description_
Yields:: _type_ – _description_

update_task_setting(dic, upd)[source]

Update task settings dictionary.

Parameters:

dic (dict) – Dictionary to update
upd (dict) – Values to update

Returns:

updated dictionary

Return type:

dict

deode.toolbox module

Toolbox handling e.g. input/output.

exception ArchiveError[source]

Bases: Exception

Error raised when there are problems archiving data.

class ArchiveProvider(config, pattern, fetch=True)[source]

Bases: Provider

Data from ECFS.

create_resource(resource)[source]

Create the resource.

Parameters:: resource (Resource) – Resource.
Returns:: True if success
Return type:: bool

class ECFS(config, pattern, fetch=True)[source]

Bases: ArchiveProvider

Data from ECFS.

create_resource(resource)[source]

Create the resource.

Parameters:: resource (Resource) – Resource.
Returns:: True if success
Return type:: bool

class FDB(config, pattern, fetch=True)[source]

Bases: ArchiveProvider

Dummy FDB class.

check_expver_restrictions(expver)[source]

Check if user is allowed to archive this expver.

Parameters:: expver (str) – FDB experiment identifier
Raises:: RuntimeError – If user is not allowed to archive for this expver

create_resource(resource)[source]

Create the resource.

Parameters:: resource (Resource) – Resource.
Raises:: RuntimeError – If expver not set
Returns:: True if success
Return type:: bool

class FileManager(config)[source]

Bases: object

FileManager class.

Default DEODE provider.

Platform specific.

create_list(basetime, forecast_range, input_template, output_settings)[source]

Create list of files to process.

Parameters:

basetime (datetime.datetime) – Base time,
forecast_range (datetime.datetime) – forecast range,
input_template (str) – Input template,
output_settings (str) – Output settings

Returns:

dict of validates and grib fiels

Return type:

dict

get_input(target, destination, basetime=None, validtime=None, check_archive=False, provider_id='symlink')[source]

Set input data to deode.

Parameters:

target (str) – Input file pattern
destination (str) – Destination file pattern
basetime (datetime.datetime, optional) – Base time. Defaults to None.
validtime (datetime.datetime, optional) – Valid time. Defaults to None.
check_archive (bool, optional) – Also check archive. Defaults to False.
provider_id (str, optional) – Provider ID. Defaults to “symlink”.

Raises:

ProviderError – “No provider found for {target}”
NotImplementedError – “Checking archive not implemented yet”

Returns:

provider, resource

Return type:

tuple

get_output(target, destination, basetime=None, validtime=None, archive=False, provider_id='move')[source]

Set output data from deode.

Parameters:

target (str) – Input file pattern
destination (str) – Destination file pattern
basetime (datetime.datetime, optional) – Base time. Defaults to None.
validtime (datetime.datetime, optional) – Valid time. Defaults to None.
archive (bool, optional) – Also archive data. Defaults to False.
provider_id (str, optional) – Provider ID. Defaults to “move”.

Returns:

provider, aprovider, resource

Return type:

tuple

Raises:

ArchiveError – Could not archive data
NotImplementedError – Archive = True is not implemented

input(target, destination, basetime=None, validtime=None, check_archive=False, provider_id='symlink')[source]

Set input data to deode.

Parameters:

target (str) – Input file pattern
destination (str) – Destination file pattern
basetime (datetime.datetime, optional) – Base time. Defaults to None.
validtime (datetime.datetime, optional) – Valid time. Defaults to None.
check_archive (bool, optional) – Also check archive. Defaults to False.
provider_id (str, optional) – Provider ID. Defaults to “symlink”.

input_data_iterator(input_data_definition, basetime=None, validtime=None, provider_id='symlink')[source]

Handle input data spec dict.

Loop through the defined data types and fetch them.

Parameters:

input_data_definition (dict) – Input data spec
basetime (datetime.datetime, optional) – Base time. Defaults to None.
validtime (datetime.datetime, optional) – Valid time. Defaults to None.
provider_id (str) – Provider id. Defaults to “symlink”

Raises:

RuntimeError – “Invalid data handle type”

output(target, destination, basetime=None, validtime=None, archive=False, provider_id='move')[source]

Set output data from deode.

Parameters:

target (str) – Input file pattern
destination (str) – Destination file pattern
basetime (datetime.datetime, optional) – Base time. Defaults to None.
validtime (datetime.datetime, optional) – Valid time. Defaults to None.
archive (bool, optional) – Also archive data. Defaults to False.
provider_id (str, optional) – Provider ID. Defaults to “move”.

set_resources_from_dict(res_dict)[source]

Set resources from dict.

Parameters:: res_dict (_type_) – _description_
Raises:: ValueError – If the passed file type is neither ‘input’ nor ‘output’.

class LocalFileOnDisk(config, pattern, basetime=None, validtime=None)[source]

Bases: Resource

Local file on disk.

class LocalFileSystemCopy(config, pattern, fetch=True)[source]

Bases: Provider

Local file system copy.

create_resource(resource)[source]

Create the resource.

Parameters:: resource (Resource) – Resource.
Returns:: True if success
Return type:: bool

class LocalFileSystemMove(config, pattern, fetch=False)[source]

Bases: Provider

Local file system move.

create_resource(resource)[source]

Create the resource.

Parameters:: resource (Resource) – Resource.
Returns:: True if success
Return type:: bool

class LocalFileSystemSymlink(config, pattern, fetch=True)[source]

Bases: Provider

Local file system.

create_resource(resource)[source]

Symlink the resource.

Parameters:: resource (Resource) – Resource.
Returns:: True if success
Return type:: bool

class Platform(config)[source]

Bases: object

Platform.

evaluate(command_string: str, object_: str | object) → Any[source]

Evaluate command string, by applying corresponding command of object.

Parameters:

command_string (str) – Command string to evaluate
object (Union[str, object]) – Object to apply command from (if command is function of object). If str, the object is assumed to be a module. If a class, the command is assumed to be a method of the class.

Raises:

ModuleNotFoundError – If module object_ not found
AttributeError – If module/class object_ has no attribute named func
TypeError – If object is not a class or a string
TypeError – If the command to evaluate is not a function

Returns:

Return original command string if it is not a function call,: otherwise return the result of the function call.

Return type:

any

expand_macros(macros)[source]

Check and expand macros.

Parameters:: macros (dict) – Input macros
Returns:: Stored macros
Return type:: out_macros (dict)

fill_each_macro(macro_config)[source]: Fill each of the macros.

fill_macros()[source]: Fill the macros.

get_gen_macros()[source]

Get the environment macros.

Returns:: Environment macros to be used.
Return type:: dict

get_os_macros()[source]

Get the environment macros.

Returns:: Environment macros to be used.
Return type:: dict

get_platform()[source]

Get the platform.

Returns:: Platform specifc values.
Return type:: dict

get_platform_value(role, alt=None)[source]

Get the path.

Parameters:

role (str) – Type of variable to substitute
alt (str) – Alternative return value

Returns:

Value from platform.[role]

Return type:

str

get_provider(provider_id, target, fetch=True)[source]

Get the needed provider.

Parameters:

provider_id (str) – The intent of the provider.
target (Resource) – The target.
fetch (boolean) – Fetch the file or store it. Default to True.

Returns:

Provider

Return type:

Provider

Raises:

NotImplementedError – If provider not defined.

get_system_macros()[source]

Get the macros.

Returns:: Macros to define.
Return type:: dict

get_system_value(role)[source]

Get the system value.

Parameters:: role (str) – Type of variable to substitute
Returns:: Value from system.[role]
Return type:: str

get_value(setting, alt=None)[source]

Get the config value with substition.

Parameters:

setting (str) – Type of variable to substitute
alt (str) – Alternative return value

Returns:

Value from config with substituted variables

Return type:

str

resolve_macros(config_dict, keyval=None)[source]: Resolve all macros in a nested dict (both keys and values!).

store_macro(key, val)[source]

Check and store a macro.

Parameters:

key (str) – Macro key
val (str) – Macro value

Raises:

RuntimeError – If macro already exists

sub_str_dict(input_dict, basetime=None, validtime=None)[source]

Substitute strings in dictionary.

Parameters:

input_dict (dict) – Dict to be parsed
basetime (datetime.datetime, optional) – Base time. Defaults to None.
validtime (datetime.datetime, optional) – Valid time. Defaults to None.

Returns:

Updated dict

Return type:

d (dict)

sub_value(pattern, key, value, micro='@', ci=True)[source]

Substitute the value case-insensitively.

Parameters:

pattern (str) – Input string
key (str) – Key to replace
value (str) – Value to replace
micro (str, optional) – Micro character. Defaults to “@”.
ci (bool, optional) – Case insensitive. Defaults to True.

Returns:

Replaces string

Return type:

str

substitute(pattern, basetime=None, validtime=None, keyval=None)[source]

Substitute pattern.

Parameters:

pattern (str) – _description_
basetime (datetime.datetime, optional) – Base time. Defaults to None.
validtime (datetime.datetime, optional) – Valid time. Defaults to None.
keyval (str) – Key associated with pattern

Returns:

Substituted string.

Return type:

str

Raises:

RuntimeError – In case of erroneous macro

substitute_datetime(pattern, datetime, suffix='')[source]

Substitute datetime related properties.

Parameters:

pattern (str) – _description_
datetime (DateTime object) – datetime to treat
suffix (str) – Add on to key

Returns:

Substituted string.

Return type:

str

class Provider(config, identifier, fetch=True)[source]

Bases: object

Base provider class.

create_missing_dir(target)[source]

Create a directory if missing.

Parameters:: target (str) – Name of target

create_resource(resource)[source]

Create the resource.

Parameters:: resource (Resource) – The resource to be created
Raises:: NotImplementedError – Should be implemented

exception ProviderError[source]

Bases: Exception

Error raised when there are provider-related problems.

class Resource(_config, identifier)[source]

Bases: object

Resource container.

class SCP(config, pattern, fetch=True)[source]

Bases: ArchiveProvider

Transfer data with SCP.

create_resource(resource)[source]

Create the resource.

Parameters:: resource (Resource) – Resource.
Raises:: RuntimeError – If directory is not created
Returns:: True if success
Return type:: bool

compute_georef(domain_config)[source]: Computes georef from domain_config.