deode.suites package

Empty __init__.py.

Submodules

deode.suites.base module

Ecflow suites base class.

class EcflowNode(name, node_type, parent, ecf_files, variables=None, trigger: EcflowSuiteTriggers | List[EcflowNode] | EcflowNode | None = None, mirror_config=None, add_var_trigger=None, remote_path=None, def_status=None, ecf_files_remotely=None, cron=None, limit=None)

Bases: object

A Node class is the abstract base class for Suite, Family and Task.

Every Node instance has a name, and a path relative to a suite. The trigger of a node can either be another EcflowNode, a list of EcflowNode objects or an EcflowSuiteTriggers object.

class EcflowNodeContainer(name, node_type, parent, ecf_files, variables=None, trigger=None, def_status=None, ecf_files_remotely=None, cron=None, limit=None, add_var_trigger=None, remote_path=None)

Bases: EcflowNode

Ecflow node container.

class EcflowSuite(name, ecf_files, variables=None, dry_run=False, def_status=None, ecf_files_remotely=None)

Bases: EcflowNodeContainer

EcflowSuite.

save_as_defs(def_file)

Save defintion file.

Parameters:

def_file (str) – Name of the definition file.

class EcflowSuiteCron(days_of_week, time)

Bases: object

EcFlow Cron in a suite.

class EcflowSuiteFamily(name, parent, ecf_files, variables=None, trigger=None, def_status=None, ecf_files_remotely=None, cron=None, limit=None, add_var_trigger=None, remote_path=None)

Bases: EcflowNodeContainer

A family in ecflow.

class EcflowSuiteLimit(limit_name, max_jobs)

Bases: object

Ecflow limit for active jobs in family.

class EcflowSuiteTask(name, parent, config, task_settings: TaskSettings, ecf_files, input_template=None, parse=True, variables=None, ecf_micro='%', trigger=None, mirror=None, mirror_config=None, def_status=None, ecf_files_remotely=None, cron=None, add_var_trigger=None, remote_path=None)

Bases: EcflowNode

A task in an ecflow suite/family.

class EcflowSuiteTrigger(node, mode='complete')

Bases: object

EcFlow Trigger in a suite.

class EcflowSuiteTriggers(triggers, mode='AND')

Bases: object

Triggers to an ecflow suite.

add_triggers(triggers, mode='AND')

Add triggers.

Parameters:

• triggers (EcflowSuiteTriggers) – The triggers

• mode (str, optional) – Cat mode. Defaults to “AND”.

static create_string(triggers, mode)

Create the trigger string.

Parameters:

• triggers (list) – List of trigger objects

• mode (str) – Concatenation type.

Returns:

The trigger string based on trigger objects.

Return type:

str

Raises:

• ValueError – If there are no triggers to be processed

• TypeError – If trigger is not an EcflowSuiteTrigger object

class SuiteDefinition(config, dry_run=False)

Bases: object

Definition of suite.

save_as_defs(def_file)

Save definition file.

Parameters:

def_file (str) – Name of definition file

deode.suites.clean_old_data module

Ecflow suites.

class DeodeCleaningSuiteDefinition(config, dry_run=False)

Bases: SuiteDefinition

Definition of suite.

deode.suites.deode module

Ecflow suites.

class DeodeSuiteDefinition(config, dry_run=False)

Bases: SuiteDefinition

Definition of suite for the Deode Workflow.

deode.suites.deode_suite_components module

Module to create the different parts of the DEODE ecFlow suite.

class CycleFamily(parent, config, task_settings: TaskSettings, input_template, ecf_files, trigger=None, ecf_files_remotely=None)

Bases: EcflowSuiteFamily

Class for creating the Cycle ecFlow family.

class E923MonthlyFamily(parent, config, task_settings: TaskSettings, input_template, ecf_files, trigger=None, ecf_files_remotely=None, dry_run: bool = False, limit=None)

Bases: EcflowSuiteFamily

Class for creating the E923Monthly ecFlow family.

class ForecastFamily(parent, config, task_settings: TaskSettings, input_template, ecf_files, trigger=None, ecf_files_remotely=None)

Bases: EcflowSuiteFamily

Class for creating the Forecast ecFlow family.

class InitializationFamily(parent, config, task_settings: TaskSettings, input_template, ecf_files, trigger=None, ecf_files_remotely=None)

Bases: EcflowSuiteFamily

Class for creating the Initialization ecFlow family.

class InputDataFamily(parent, config, task_settings: TaskSettings, input_template, ecf_files, trigger=None, ecf_files_remotely=None, external_marsprep_trigger_node=None, add_var_trigger=None, remote_path=None, member=0)

Bases: EcflowSuiteFamily

Class for creating the InputDataFamily ecFlow family.

class InterpolationFamily(parent, config, task_settings: TaskSettings, input_template, ecf_files, cycles: Cycles, trigger=None, ecf_files_remotely=None, do_prep: bool = True, dry_run: bool = False, add_var_trigger=None, remote_path=None, member=0)

Bases: EcflowSuiteFamily

Class for creating the Interpolation ecFlow family.

class LBCFamily(parent, config, task_settings: TaskSettings, input_template, ecf_files, cycles: Cycles, trigger=None, lbc_family_trigger=None, ecf_files_remotely=None, dry_run: bool = False, member=0)

Bases: EcflowSuiteFamily

Class for creating the ecFlow LBCFamily.

class LBCSubFamilyGenerator(parent, config, task_settings: TaskSettings, input_template, ecf_files, bdint: timedelta, lbc_time_generator: Generator[Tuple[List[int], List[datetime]], None, None], trigger=None, ecf_files_remotely=None, is_first_cycle: bool = True, limit: EcflowSuiteLimit | None = None, member=0, do_slaf=False)

Bases: EcflowSuiteFamily

Class for creating the ecFlow LBCSubFamilyGenerator.

When iterating over this class, it will create a new LBC family for each time in the lbc_time_generator.

slaf_worker(task_name, old_trigger, bdshift, bd_index_time_dict)

Logic to determine worker and additional info for a boundary file batch.

Parameters:

• task_name (str) – task name of real job

• old_trigger – possibly existing trigger

• bdshift (timedelta) – SLAF boundary shift for this file

• bd_index_time_dict – for this batch

Returns:

member that does the actual interpolation part: slaf part for doer new_trigger: (updated) trigger for dependant task (addpert)

Return type:

doer

Raises:

RuntimeError – if SLAF planning gave inconsistent worker set

class MergeSQLitesFamily(parent, config, task_settings: TaskSettings, ecf_files, trigger=None, input_template=None, ecf_files_remotely=None)

Bases: EcflowSuiteFamily

Class for creating the MergeSQLites ecFlow family.

class MirrorFamily(parent, config, task_settings: TaskSettings, input_template, ecf_files, trigger=None, ecf_files_remotely=None, cycle_valid=None)

Bases: EcflowSuiteFamily

Class for creating the InputDataFamily ecFlow family.

class PgdInputFamily(parent, config, task_settings: TaskSettings, input_template, ecf_files, trigger=None, ecf_files_remotely=None)

Bases: EcflowSuiteFamily

Class for creating the PGD input ecFlow family.

class PgdNode(node_name, parent, config, task_settings: TaskSettings, input_template, ecf_files, trigger=None, ecf_files_remotely=None, limit=None)

Bases: EcflowSuiteFamily, EcflowSuiteTask

Class for creating the PGD ecFlow family and task.

class PostCycleFamily(parent, config, task_settings: TaskSettings, input_template, ecf_files, trigger=None, external_cycle_cleaning_trigger=None, ecf_files_remotely=None)

Bases: EcflowSuiteFamily

Class for creating the PostCycle ecFlow family.

class PrepFamily(parent, config, task_settings: TaskSettings, input_template, ecf_files, trigger=None, ecf_files_remotely=None)

Bases: EcflowSuiteFamily

Class for creating the PrepFamily ecFlow family.

class SLAFpartFamily(name, subtask, parent, config, task_settings: TaskSettings, input_template, ecf_files, trigger=None, variables=None, ecf_files_remotely=None)

Bases: EcflowSuiteFamily

Helper class for constructing SLAF perturbations (for EPS).

class StaticDataFamily(parent, config, task_settings: TaskSettings, input_template, ecf_files, trigger=None, ecf_files_remotely=None, dry_run: bool = False)

Bases: EcflowSuiteFamily

Class for creating the StaticData ecFlow family.

class StaticDataMemberGenerator(parent, config, task_settings: TaskSettings, input_template, ecf_files, ecf_files_remotely, dry_run)

Bases: object

Class for creating the StaticData ecFlow family generator.

get_member_family(member: int)

Get member specific family with tasks to produce static data.

Parameters:

member (int) – The member number.

Returns:

The member family that contains tasks to

produce static data for the given member.

Return type:

EcflowSuiteFamily

class StaticDataTasks(parent, config, task_settings: TaskSettings, input_template, ecf_files, ecf_files_remotely, limit: EcflowSuiteLimit | None = None)

Bases: object

Class for creating the StaticData ecFlow tasks.

class SubTaskFamily(parent, config, task_settings: TaskSettings, input_template, ecf_files, taskname, ntasks: int, extra_args=None, trigger=None, ecf_files_remotely=None)

Bases: EcflowSuiteFamily

Class for creating a subtask ecFlow family.

class TimeDependentFamily(parent, config, task_settings: TaskSettings, input_template, ecf_files, trigger: EcflowSuiteTask | StaticDataFamily | None = None, ecf_files_remotely=None, do_prep: bool = True, dry_run: bool = False)

Bases: EcflowSuiteFamily

Class for creating the time dependent part of a DW suite.

active_mirror(config, tag)

Detects active mirror from a given tag.

has_mirror_family(config)

Detects if we have any active mirrors.

property last_node

Return the last family node of self.

deode.suites.discover_suite module

Discover suites.

available_suites(reg: DeodePluginRegistry)

Create a list of available tasks.

Parameters:

reg (DeodePluginRegistry) – Deode plugin registry

Returns:

Suite objects

Return type:

known_types (list)

discover(package, base)

Discover SuiteDefinition classes.

Plugin classes are discovered in a given namespace package, deriving from a given base class. The base class itself is ignored, as are classes imported from another module (based on cls.__module__). Each discovered class is identified by the class name by changing it to lowercase and stripping the name of the base class, if it appears as a suffix.

Parameters:

• package (types.ModuleType) – Namespace package containing the plugins

• base (type) – Base class for the plugins

Returns:

type): Discovered plugin classes

Return type:

(dict of str

discover_modules(package, what='plugin')

Discover plugin modules.

Parameters:

• package (types.ModuleType) – Namespace package containing the plugins

• what (str, optional) – String describing what is supposed to be discovered. Defaults to “plugin”.

Yields:

tuple – str: Name of the imported module types.ModuleType: The imported module

get_suite(name, config)

Create a deode.suites.SuiteDefinition object from configuration.

Parameters:

• name (_type_) – _description_

• config (_type_) – _description_

Returns:

_description_

Return type:

_type_

Raises:

NotImplementedError – If SuiteDefinition name is not amongst the known SuiteDefinition names.

deode.suites.suite_utils module

Module for utility functions used in the suite definition scripts.

class Cycle(*, day: str, time: str, validtime: str, basetime: str)

Bases: object

Class for representing a cycle.

basetime: str

day: str

time: str

validtime: str

class Cycles(first_cycle: str, last_cycle: str, cycle_length: str)

Bases: object

Class for generating and iterating over Cycle objects.

property current_cycle: Cycle

Return the current Cycle object.

property current_index: int

Return the current cycle index.

cycle_length: str

property end_of_month: bool

Return True if the next cycle is in a different month.

Returns:

True if the next cycle is in a different month

Return type:

bool

first_cycle: str

last_cycle: str

property next_cycle: Cycle

Return the next Cycle object.

Raises:

StopIteration – If there are no more cycles.

lbc_times_generator(basetime: datetime, endtime: datetime, step: timedelta, mode: str = 'start', is_first_cycle: bool = True, do_interpolsstsic: bool = False, lbc_per_task: int = 1) → Generator[Dict[int, str], None, None]

Generate lbc times.

For each of them there will be LBC[NN] family.

Parameters:

• basetime – The base time.

• endtime – The end time.

• step – The step size.

• mode – The mode of the workflow.

• is_first_cycle – Whether this is the first cycle.

• do_interpolsstsic – Whether to do SST/SIC interpolation.

• lbc_per_task – Number of LBC assigned to each task. Default 1.

Returns:

The time period for which the last LBC will be computed.

Return type:

datetime

Yields:

datetime – The time period for which the next LBC will be computed.

slaf_planner(config, lbc_time_generator, me) → dict

Distribution of work in case several EPS members need the same boundary files.

This is typically the case with the SLAF method. Note that every member of the EPS calls this routine in the present setup (which is strictly not necessary).

Parameters:

• config (BaseConfig) – configuration object

• lbc_time_generator (Generator) – as above

• me (int) – my EPS member number

Returns:

Holds, for each unique boundary file, information

about which member will do the actual work and in which SLAF part (0, 1 or 2).

Return type:

doer (dict)

Raises:

RuntimeError – If the planning logic fails.