deode.tasks package
Tasks submodule.
Submodules
deode.tasks.archive module
ARCHIVEHOUR // ARCHIVESTATIC.
- class ArchiveFDB(config)[source]
Bases:
ArchiveTask
Archving task for time dependent data dedicated for FDB.
- class ArchiveHour(config)[source]
Bases:
ArchiveTask
Archving task for time dependent data.
- class ArchiveMergedSQLites(config)[source]
Bases:
ArchiveTask
Archving task for time dependent data.
- class ArchiveStatic(config)[source]
Bases:
ArchiveTask
Archving task for static data.
- class ArchiveStaticMember(config)[source]
Bases:
ArchiveTask
Archving task for static data.
deode.tasks.base module
Base site class.
- class Task(config, name)[source]
Bases:
object
Base Task class.
- archive_logs(files, target=None)[source]
Archive files in a log directory.
- Parameters:
files (str,list) – File(s) to be archived
target (str) – Target directory for archiving
- get_binary(binary_name)[source]
Determine binary path from task or system config section.
- Parameters:
binary_name (str) – Name of binary
- Returns:
full path to binary
- Return type:
bindir (str)
- get_task_setting(setting)[source]
Get task setting.
- Parameters:
setting (str) – Setting to find in task.{self.name}
- Returns:
Found setting
- Return type:
value
deode.tasks.batch module
Batch process.
deode.tasks.c903 module
C903.
deode.tasks.canari module
Canari.
deode.tasks.clean_old_data module
Clean old data.
- class CleanEhypeData(config)[source]
Bases:
CleanOldData
Clean old data from scratch.
- class CleanIFSData(config)[source]
Bases:
CleanOldData
Clean IFS data.
- class CleanOldData(config)[source]
Bases:
Task
Interface class to the cleaning of the old data.
- cutoff(delay)[source]
Calculates the time to determine the old data.
- Parameters:
delay (int) – delay in days
- Returns:
time in seconds
- Return type:
cutoff (int)
- get_old(path, format_, cutoff_time, ignore=None)[source]
Get directories which are older than cutoff.
- Parameters:
path (str) – Path to start of searching old directories
format (str) – regex for path
cutoff_time (int) – Directories or files older then cutoff
ignore (list) – List of directories to ignore
- Returns:
List of directories for remove
- Return type:
list_to_remove (list)
- class CleanScratchData(config)[source]
Bases:
CleanOldData
Clean old data from scratch.
- class CleanSuites(config)[source]
Bases:
CleanOldData
Clean old suites.
deode.tasks.cleaning_tasks module
Clean deode file systems.
deode.tasks.collectlogs module
CollectLogs.
- class CollectLogs(config, config_label='staticlogs')[source]
Bases:
Task
Collect task and scheduler logfiles and store them as tarfiles.
- scan_logs(tarlog, parent, target, pattern='', exclude='')[source]
Search for files matching a pattern and add them to a tar file.
- Parameters:
tarlog (tarfile object) – Tar file used
parent (str) – Top search directory
target (str) – Main search directory
pattern (str) – Optional search pattern
exclude (str) – Optional string for files to be excluded
- Raises:
FileNotFoundError – If parent directory does not exist
- class CollectLogsHour(config)[source]
Bases:
CollectLogs
Collectlog task for static data.
- class CollectLogsStatic(config)[source]
Bases:
CollectLogs
Collectlog task for static data.
deode.tasks.creategrib module
CreateGrib.
- class GlGrib(config, name='')[source]
Bases:
Task
Create grib files.
- convert2grib(infile, outfile, filetype)[source]
Convert FA to grib.
Namelist arguments are given in the creategrib.NAME config part per filetype :param infile: Input file :type infile: str :param outfile: Output file :type outfile: str :param filetype: filetype to look for in rules :type filetype: str
- Raises:
FileNotFoundError – Could not find file
deode.tasks.discover_task module
Discover tasks.
- available_tasks(reg)[source]
Create a list of available tasks.
- Parameters:
reg (DeodePluginRegistry) – Deode plugin registry
- Returns:
Task objects
- Return type:
known_types (list)
- discover(package, base)[source]
Discover task classes.
Plugin classes are discovered in a given namespace package, deriving from a given base class. The base class itself is ignored, as are classes imported from another module (based on
cls.__module__
). Each discovered class is identified by the class name by changing it to lowercase and stripping the name of the base class, if it appears as a suffix.- Parameters:
package (types.ModuleType) – Namespace package containing the plugins
base (type) – Base class for the plugins
- Returns:
type): Discovered plugin classes
- Return type:
(dict of str
- discover_modules(package, what='plugin')[source]
Discover plugin modules.
- Parameters:
package (types.ModuleType) – Namespace package containing the plugins
what (str, optional) – String describing what is supposed to be discovered. Defaults to “plugin”.
- Yields:
tuple – str: Name of the imported module types.ModuleType: The imported module
- get_task(name, config) Task [source]
Create a deode.tasks.Task object from configuration.
- Parameters:
name (_type_) – _description_
config (_type_) – _description_
- Raises:
NotImplementedError – If task name is not amongst the known task names.
- Returns:
The task object with name name. The task object has to be a subclass of Task to be retrievable.
- Return type:
deode.tasks.e923 module
E923.
- class E923(config)[source]
Bases:
Task
Methods for the e923 work.
- constant_part(constant_file)[source]
Run the constant part of e923.
- Parameters:
constant_file – filename of the resulting file
- monthly_part(constant_file)[source]
Run the monthly part of e923.
- Parameters:
constant_file – filename of the input constant file
- myexec(cmd, i)[source]
Execute binary task.
- Parameters:
cmd (str) – Command to run
i (int) – sequence number
deode.tasks.e923_update module
E923Update.
deode.tasks.e927 module
E927.
deode.tasks.forecast module
Forecast.
- class Forecast(config)[source]
Bases:
Task
Forecast task.
- accelerator_device_input()[source]
Copy the input files for GPU execution.
parallel_method: input file with parallelisation technique for each algorithm
synchost: input file defining optional device-to-host memory transfers
select_gpu: wrapper file for sbatch, binding GPU to MPI rank
- archive_output(filetype, periods)[source]
Archive forecast model output.
- Parameters:
filetype (str) – Filename template
periods (str) – Output list
- merge_output(filetype, periods)[source]
Merge distributed forecast model output.
- Parameters:
filetype (str) – File type (history, surfex, fullpos)
periods (str) – Output list
Final result is the expected output file name in the working directory (as if there was no IO server). NOTE: This function has been replaced by io_merge tasks. It is only called if n_io_merge=0.
deode.tasks.gmtedsoil module
GMTED and SOILGRID.
- class Gmted(config)[source]
Bases:
Task
GMTED.
- define_gmted_input(domain_properties: dict) tuple [source]
Define GMTED input files.
- Parameters:
domain_properties (dict) – Domain properties
- Returns:
GMTED input files
- Return type:
tuple
- get_domain_properties(config) dict [source]
Get domain properties.
- Parameters:
config (Config) – Config object
- Returns:
Domain properties
- Return type:
dict
- gmted_header_coordinates(east: float, west: float, south: float, north: float) tuple [source]
Get GMTED header coordinates.
- Parameters:
east (float) – East
west (float) – West
south (float) – South
north (float) – North
- Returns:
Header coordinates
- Return type:
tuple
- static tif2bin(gd, bin_file) None [source]
Convert tif file to binary file used by surfex.
- Parameters:
gd – gdal dataset
bin_file (str) – Binary file
- static write_gmted_header_file(header_file, hdr_north, hdr_south, hdr_west, hdr_east, hdr_rows, hdr_cols) None [source]
Write header file.
- Parameters:
header_file (str) – Header file
hdr_north (float) – North
hdr_south (float) – South
hdr_west (float) – West
hdr_east (float) – East
hdr_rows (int) – Number of rows
hdr_cols (int) – Number of columns
- class Soil(config)[source]
Bases:
Task
Prepare soil data task for PGD.
- static check_domain_validity(domain_properties: dict) None [source]
Check if domain is valid.
- Parameters:
domain_properties (dict) – Dict with domain properties
- Raises:
ValueError – If domain is outside soilgrid data area
- static coordinates_for_cutting_dataset(domain_properties: dict, halo: float = 5.0) tuple [source]
Get coordinates for cutting dataset.
- Parameters:
domain_properties (dict) – Dict with domain properties
halo (float) – Halo. Defaults to 5.0.
- Returns:
Coordinates for cutting dataset
- Return type:
tuple
- execute()[source]
Run task.
Define run sequence.
- Raises:
FileNotFoundError – If no tif files are found.
- get_domain_properties(config) dict [source]
Get domain properties.
- Parameters:
config (deode.ParsedConfig) – Configuration
- Returns:
Domain properties
- Return type:
dict
- static write_soil_header_file(header_file, soiltype, hdr_north, hdr_south, hdr_west, hdr_east, hdr_rows, hdr_cols, nodata=0, bits=8, write_fact=False, fact=10) None [source]
Write header file.
- Parameters:
header_file (str) – Header file
soiltype (str) – Soil type
hdr_north (float) – North
hdr_south (float) – South
hdr_west (float) – West
hdr_east (float) – East
hdr_rows (int) – Number of rows
hdr_cols (int) – Number of columns
nodata (int) – No data value. Defaults to 0.
bits (int) – Number of bits. Defaults to 8.
write_fact (bool) – Write factor. Defaults to False.
fact (int) – Factor. Defaults to 10
deode.tasks.gribmodify module
AddCalculatedFields.
- class AddCalculatedFields(config)[source]
Bases:
Task
Create grib files.
- add_field_to_grib(fnames, params, operation, output_params, layer_weights, physical_range)[source]
Add fields to the same grib following specified modification.
- Parameters:
fnames (list) – list of grib files
params (list) – list of parameter dictionaries of input parameters
operation (str) – operation to compute new field
output_params (dict) – list of parameter dictionaries of output parameters
layer_weights (list) – list of weights for each layer
physical_range (list) – list of physical range for each parameter
- Raises:
NotImplementedError – Operation not implemented yet”
- calc_patch_averaging(params, layer_weights, values_list, bitmap_list, physical_range, nature_weighting, surface=False)[source]
Calculate patch averaging.
Bitmaps are used to mask out invalid values. This is necessary, as the values for a parameter may be missing in some tiles despite the fact that the tile fraction is not 0. This is solved by updating the layer weights to always sum to 1 while excluding missing data.
- find_in_files(param, fname, additional_files)[source]
Check if field with specified parameters exists.
Checks the main file and all additional files.
- Parameters:
param – parameter dictionary,
fname – main grib file
additional_files – list of additional grib files
- Returns:
True if field exists in any file
- Return type:
bool
- find_par(param, fname)[source]
Check if field with specified parameters already exists in grib.
- Parameters:
param – parameter dictionary,
fname – grib file
- Returns:
True if field exists
- Return type:
bool
- get_unique_values(dicts, key, exclude=None)[source]
Get unique values of a key in a list of dictionaries.
- get_value_for_params(params, match_dict, values_list, bitmap_list, apply_bitmap=False, return_bitmap=False)[source]
Get the index of the dict in the list that matches all key-value pairs.
deode.tasks.impacts module
Impact model classes.
- class Ehype(name: str, *_args, **_kwargs)[source]
Bases:
ImpactModel
EHYPE specific methods.
- name: str = 'ehype'
- class ImpactModel(name: str, *_args, **_kwargs)[source]
Bases:
ABC
Abstract class for impact models.
- config: BasicConfig
- dump(to_dump)[source]
Write config to selected format.
- Parameters:
to_dump (dict) – config dict to write
- Raises:
TypeError – Unknown output file type
- filename: str | None = ''
- name: str
- class ImpactModels(config, taskname=None)[source]
Bases:
Task
Create info to and start impact models.
- class StartImpactModels(config)[source]
Bases:
ImpactModels
Starts the impact models.
deode.tasks.iomerge module
IO merge task.
- class IOmerge(config)[source]
Bases:
Task
IO merge task.
- check_fc_path()[source]
Check if fc_path exists.
- Raises:
FileNotFoundError – If directory is not present
- merge_output(lt, filetype)[source]
Merge distributed forecast model output.
- Parameters:
lt (timeDelta) – lead time
filetype (str) – Type of file
- static wait_for_file(filename, age_limit=15)[source]
Wait until a file is at least N seconds old.
- Parameters:
filename (str) – full path to file
age_limit (int) – minimum age of last modification
- wait_for_io(filetype, lt, maxtries=2)[source]
Wait for all io_server output to be stable.
- Parameters:
filetype (str) – kind of output file
lt (timeDelta) – lead time
maxtries (int) – maximum number of file search trials
- Returns:
List of files expected
- Return type:
file_list (list)
- Raises:
RuntimeError – In case of erroneous number of files
deode.tasks.marsprep module
Marsprep.
- class Marsprep(config)[source]
Bases:
Task
Marsprep task.
- execute()[source]
Run task.
Define run sequence.
- Raises:
RuntimeError – If there is an issue with the work folder.
- get_gg_data(tag: str, steps: List[int], members_dict: Dict[str, List[int]])[source]
Get gridpoint surface data.
- get_gg_snow_data(tag: str, steps: List[int], members_dict: Dict[str, List[int]], param)[source]
Get soil gridpoint data.
- get_ua_data(tag: str, steps: List[int], members_dict: Dict[str, List[int]])[source]
Get upper air data.
- property mars: dict
Get mars selection.
- static mars_selection(selection: str, config: ParsedConfig) dict [source]
Copy default settings if requested.
- Parameters:
selection (str) – The selection to use.
config (deode.ParsedConfig) – Configuration object
- Returns:
mars config section
- Return type:
mars (dict)
- update_data_request(request: BaseRequest, prefetch: bool, specify_domain: bool, bdmembers: List[int] | None = None, grid: str | None = None, source: str | None = None)[source]
Create ECMWF MARS system request.
- Parameters:
request – BaseRequest object to update
prefetch – Retrieve or stage
specify_domain – Use lat/lon and rotation or use global (default)
bdmembers – Boundary members to retrieve in case of eps.
grid – Specific grid for some request. Default None.
source – Sorce for retrieve data from disk. Defaults None.
deode.tasks.prep_run module
Preparatory task for a deode suite.
deode.tasks.serial module
Serial task.
deode.tasks.sfx module
Surfex tasks.
- class InputDataFromNamelist(nml, input_data, program, platform, basetime=None, validtime=None, one_decade=False)[source]
Bases:
object
Binary input data for offline executables.
- extend_macro(key, val, macros, sep='#')[source]
Extend entries from macro.
- Parameters:
key (_type_) – _description_
val (_type_) – _description_
macros (dict) – Macros
sep (str, optional) – Namelist key separator. Defaults to “#”.
- Raises:
NotImplementedError – _description_
NotImplementedError – _description_
- Returns:
Key, value dictionary
- Return type:
dict
- static get_nml_value(nml, block, key, indices=None)[source]
Get namelist value.
- Parameters:
nml (nmlf90.Namelist) – Namelist
block (str) – Namelist block
key (str) – Namelist key
indices (list, optional) – Indices to read. Defaults to None.
- Returns:
Namelist setting
- Return type:
setting (any)
- static get_nml_value_from_string(nml, string, sep='#', indices=None)[source]
Get namelist value from a string.
- Parameters:
nml (nmlf90.Namelist) – Namelist
string (str) – Namelist identifier
sep (str, optional) – _description_. Defaults to “#”.
indices (list, optional) – Indices to read. Defaults to None.
- Returns:
Namelist setting
- Return type:
setting (any)
- matching_value(data, val, sep='#', indices=None)[source]
Match the value. Possibly also read namelist value.
- Parameters:
data (dict) – Data to check keys for
val (str) – Key to find
sep (str, optional) – Namelist separator. Defaults to “#”.
indices (list, optional) – Indices in namelist
- Raises:
RuntimeError – “Malformed input data”
- Returns:
Matching entry in data.
- Return type:
dict
- process_data(sep='#')[source]
Process input definitions on files to map.
- Parameters:
sep (str, optional) – Namelist separator. Defaults to “#”.
- Returns:
A dict with mapped local names and target files.
- Return type:
mapped_data (dict)
- process_macro(key, val, macros, sep='#', indices=None)[source]
Process macro.
- Parameters:
key (str) – Key
val (str) – Value
macros (dict) – Macros
sep (str, optional) – Namelist key separator. Defaults to “#”.
indices (list, optional) – Process macro from namelist indices.
- Raises:
NotImplementedError – Only 2 dimensions are implemented
- Returns:
Key, value dictionary
- Return type:
dict
- read_macro_setting(macro_defs, key, default=None, sep='#')[source]
Read a macro setting.
- Parameters:
macro_defs (dict) – Macro definition
key (str) – Macro setting to get.
default (str, optional) – Default value. Defaults to None.
sep (str, optional) – Namelist key separator. Defaults to “#”.
- Returns:
setting (any)
deode.tasks.sqlite module
ExtractSQLite.