Changelog

All notable changes to this project will be documented in this file.

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.

Full Changelog

Unreleased

Added

  • Add path in config to prep input file to allow better control of file location. #1212(@uandrae)

  • Add merging of sqlite files in cases of more than 1 ensemble member. #1193(@mafdmi)

  • Add georef-calculation and determination of stream-type to deode case command #1213(@johtoblan)

  • Introduce first CSC dependent EPS settings. #1217(@uandrae)

Fixed

  • Set ensemble member number in namelist by default. #1216(@uandrae)

  • Specify tagged version 0.1.2 or above of pyfdb in pyproject.toml. #1205(@KristianHMoller)

  • Fix grib2 encodings of radar product to allow ensemble production. #1219 (@sbnielsen)

  • Fixed issue with the user having to specify all members when using the dict syntax in eps configuration. #1220(@mafdmi)

  • Fix bug in creation of merged sqlite files which contained member 0 in all data columns #1223(@svianaj)

[0.17.0] - 2025-06-13

Fixed

  • Allow using duration notation for both general.times.start and end. #1209 (@uandrae)

Added

  • Add option to write output only at first output time to creategrib and gribmodify. #1156(@KristianHMoller)

  • Added FDB paths and expver for LUMI. #1181 (@KristianHMoller)

Changed

  • Clarify procedure for creating a hotfix #1076(@KristianHMoller)

  • Updated to using ecCodes 2.41 on ATOS. #1187(@KristianHMoller)

  • Remove param 231049 from FDB exclusion. #1187(@KristianHMoller)

Fixed

  • Avoid generating a task when checking for active impact models. #1196 (@uandrae)

  • Allow FDB archiving without excluding fields. #1198 (@KristianHMoller)

  • Correct fullpos merge method. Removes dependency of merge order. #1195 (@uandrae)

  • Fix c903 climate file for lagged boundary conditions (bdshift). #1194 (@adeckmyn)

  • Fix arome nudging settings to remove spikes in norms of pressure-departure. #1180 (@kastelecn)

  • Synced develop with master #1186 (@mafdmi)

  • Correct time unit when changing productDefinitionTemplateNumber. #1156(@KristianHMoller)

  • Output surface geopotential only at analysis time. #1156(@KristianHMoller)

  • Remove duplicated radiation parameters from AQ output. #1156(@KristianHMoller)

  • Fix GRIB2 encoding for tile attribute properties. #1156(@KristianHMoller)

  • Fix lagged boundary conditions (bd_shift). #1175 (@adeckmyn)

  • Moved incorrectly placed entries in changelog. #1192(@KristianHMoller)

  • Corrected paths in config files for running demo cases as an ensemble. #1191 (@mafdmi)

  • Fixed pytest removing data on ATOS. #1043 (@mafdmi)

  • Make config file saved to wrk dir for every task. Removed member env in standalone task. \1155 (@mafdmi)

  • Revert test command in tests.yaml workflow. #1201 (@mafdmi)

[0.16.1] - 2025-05-23

Fixed

  • Reverted referencing pyproject.toml relative to package directory, since, when installing deode as a package, it does not include pyproject.toml file. #1177 (@mafdmi)

[0.16.0] - 2025-05-21

Added

  • Json schema check on expver for FDB archiving.#1168 (@uandrae)

  • Add posibility to run ALARO cy49t2 with surfex.#1128 (@kastelecn)

  • Add more control of expver for FDB archiving.#1120 (@uandrae)

  • Introduce fixed global orography on gaussian grid for marsprep speedup. #1143 (@adeckmyn)

  • Set start date for snow data. #1097 (@kastelecn)

  • Introduce subtasks for CreateGrib and AddCalculatedFields to reduce time to solution.#1138 (@uandrae)

  • Introduced EPS in the workflow. By default, every run is now treated as an ensemble. #1031

  • Introduce caching of parameters to speedup the search in GRIB files in gribmody.py.#1136 (@uandrae)

  • Add generation and storage of expanded config file for dcmbd usage.#1126 (@uandrae)

Changed

  • Switch on FDB archiving in operations and updated the FDB documentation. #1168 (@uandrae)

  • Make log collection settings configurable. #1167 (@uandrae)

  • Use module related environment variable for FDB5_HOME on atos_bologna.#1120 (@uandrae)

  • Deactivate TOMS computations to allow execution of forecast model in SP.#1135 (@uandrae)

  • Allow archiving to be called from any task. Allows log collection to run after archiving.#1117 (@uandrae)

  • Rename PreCleaning to RunPrep as it also handles storing of config files.#1126 (@uandrae)

  • Update GRIB encoding of root depth parameter for AQ.#1134 (@KristianHMoller)

  • Simplifies config files, having one for each cyle, one for each csc, and one for each host, instead of the combinations of all of these.#1088 (@pardallio)

Fixed

  • Restore gathering of task specific log files. #1167 (@uandrae)

  • Fix LBC filenames in case of mode = “restart”. #1162 (@kastelecn)

  • Added @MEMBER_STR@ to archive outpath where missing. #1147 (@mafdmi)

  • Made host resolvement for impact models happen run-time instead of when generating config. #1148 (@mafdmi)

  • Fix error with parent node for “CreateGribStatic” task. #1152 (@mafdmi)

  • Revert fetching ecFlow variables from ecFlow server. #1142 (@mafdmi)

[0.15.0] - 2025-04-24

Added

  • Add wfp configuration for all cscs and event-type for windfarm #999 (@jacobsnoeijer)

  • Prepare for activation of FDB archiving in operations and add a new FDB archiving task. #1096. FDB archiving switch off due to memory problems in #1127. (@uandrae)

  • Add grib conversion of static files #1035 (@uandrae)

  • Modified gribmodify to use a json file for configuration #1049 (@KristianHMoller)

  • Updated gribmodify to do patch averaging for AQ needs #1049 (@KristianHMoller)

  • Added config files for running CY49 in leonardo #1064 (@pardallio)

  • Added config files for running CY49 in SP #1064 (@pardallio)

Changed

  • Move Ecflow limit from E923 Family to StaticData family. [#1121] (https://github.com/destination-earth-digital-twins/Deode-Workflow/pull/1121) (@kastelecn)

  • Change walltime for Forecast task on LUMI for cy48t3 #1113 (FlorianW-ZAMG)

  • Change bindir for CY49t2 to match new name convention \1122(@uandrae)

  • Change default balance between PROCs used for Forecast and IO for default (small domain) configuration \1114(@jacobsnoeijer)

  • Change default behaviour of the deode show namelist command. Now produces unparsed namelists. #1099 (@uandrae)

  • Let the IOmerge tasks cleanup the Forecast working directories#1102 (@uandrae)

Fixed

  • Fix trigger bug introduced in #1102. #1125 (@uandrae)

  • Fix forecast archive bug introduced in #1102. #1124 (@uandrae)

  • Fix name of suite to ignore in cleaning. #1116 (@kastelecn)

  • Fixed some details in Mars requests depending on Mars client version and data origin [#1066] (https://github.com/destination-earth-digital-twins/Deode-Workflow/pull/1066) (@adeckmyn)

  • Remove erroneous macro warnings for non-existent impact model #1091 (@uandrae)

  • Made git branch structure figures appear in documentation. Resolved sphinx warnings #1077 (@mafdmi)

  • Make sure the IOmerge tasks fails if the Forecast task fails #1090(@uandrae)

  • Updated documentation of FDB as georef is added, also remove backgroundProcess = “99” as default grib_set #1094 (@johtoblan)

[0.14.0] - 2025-03-27

Added

  • Check on output frequency for fields calculated in gribmodify #1085 (@uandrae)

  • Added documentation for installation on leonardo. #1062(@uandrae)

  • Added setup for running CY48t3_arome and CY49t2_arome on Leonardo, for both 60x80 and large domains. #1057(@mippoliti-cin)

  • Uniform the fullpos namelists for all CSC’s in cy49 #1028 (@egregow)

  • Add function for evaluate relative dates from config file. #1085 (@kastelecn)

Changed

  • Update surfex namelist settings for CY49. #1050(@uandrae)

  • Add support to configure simulations for eventtype eclipse #1063(@jacobsnoeijer)

  • Change macro configuration to be more flexible #1032(@uandrae)

Fixed

  • Restore problematic deode_home and ecf_host changed in v0.13.0 #1068(@uandrae)

  • Use 32 bit space filling curve for geohash #1042 (@johtoblan)

  • Reduce ntasks for C903 on lumi. #1080 (@kastelecn)

  • Add snowfields to IFSENS MARS request #1055(@uandrae)

  • Specify timeIncrement to hours for 60M precipitation variables in faFieldNames.def#1052(@KristianHMoller)

[0.13.2] - 2025-03-21

Changed

  • Reintroduced calculation of wind gust to gribmodify and removed from fullpos #1070 (@KristianHMoller)

[0.13.1] - 2025-02-27

Changed

  • Use gl binary built with correct eccodes version #1029 (@uandrae))

[0.13.0] - 2025-02-26

Added

  • Add possibility to set start to yesterday. #1047 (@kastelecn)

  • Print deode version and config file used for each task #1011 (@uandrae)

  • Configure dependabot for frequent dependency updates. #989 (@observingClouds)

Changed

  • Set lumi_DT as default ifs selection on LUMI, seperate option for running on debug partition. #1034(@kastelen)

  • Name convention for LBC families #984(@uandrae)

  • Increase walltime for CreateGrib on atos_bologna #1009 (@uandrae)

  • Use bddir_sfx for MARS latlon file for more flexibilty on output location #1012 (@uandrae)

  • Sqlite extraction increased & limited to standard pressure levels for RH,S,T,D [#1013] (https://github.com/destination-earth-digital-twins/Deode-Workflow/pull/1013) (@svianaj)

  • Updated station list for sqlite extraction with new ones from WOW 2023-2024 period [#1030] (https://github.com/destination-earth-digital-twins/Deode-Workflow/pull/1030) (@svianaj)

Fixed

  • Remove BDMEMBER from all config files. #1024(@kastelecn)

  • Restored lost bdcycle setting #984(@uandrae)

  • Fix json schema for impact models. #1020(@kastelecn)

[0.12.1] - 2025-02-21

###Fixed

  • Fix operational setting for ehype. #1015 (@kastelecn)

[0.12.0] - 2025-02-21

Added

  • Add possibility to retrieve several EPS members at the same time.#988(@kastelecn)

  • Introduce possibility to generate ensemble config file. #865 (@mafdmi)

  • Introduce possibility for multiple search path for config files. #776 (@uandrae)

  • Added support for CMake compiled binaries on LUMI #980 (@pardallio)

Changed

  • Clean Marsprep slurm settings for atos_bologna #1001 (@uandrae)

  • Changed binary location o LUMI to CI/CD compiled binaries #980 (@pardallio)

  • Update ehype config settings. #967(@uandrae)

Fixed

  • Correct EcCodes module inconsistency for atos_bologna #1006 (@uandrae)

[0.11.0] - 2025-02-13

  • Add Harmonie-Arome configurations and update namelist creation for cy49t2 #969 (@romick-knmi)

  • Detach selection of orography filtering method from CSC. #960(@uandrae)

  • Namelist changes for c903 CY49t2, retrieve snow parameters on SOIL levels from mars.#978(@kastelecn)

  • Add large domain submission rules for CY49 on atos_bologna #945 (@uandrae)

  • Add synthetic satellite fields for cy49t2 #954 (@FlorianW-ZAMG)

  • Add albedo to AQ event output. #920 (@sbnielsen)

  • Add documentation for archiving with FDB on ATOS #768 (@johtoblan)

  • Track fields not archived to fdb#754 (@uandrae, @sbnielsen, @KristianHMoller)

  • Add geohash algorithm to identify starting area in FDB #973 (@johtoblan)

  • Add model version mapping to FDB grib_set parameters #975 (@KristianHMoller)

Changed

  • Update AROME dynamics settings #966(@uandrae)

  • Update fullpos selection for CY49 #941(@uandrae)

  • Increased walltime from 15 to 30 minutes for AddCalculatedFields #948 (@KristianHMoller)

  • Updated FA2GRIB2 translations #754 (@uandrae, @sbnielsen, @KristianHMoller)

  • Add EcCodes as a python dependency #937 (@johtoblan)

  • Make version of poethepoet more flexible #950 (@johtoblan)

  • Updated README.md for poetry >= v2.0.0 shell plugin installation. #935 (@BertvanUlft)

  • Externalise sqlite_utils to grib2sqlite. [#949] (https://github.com/destination-earth-digital-twins/Deode-Workflow/pull/949) (@adeckmyn)

  • Make submission groups more flexible. #867 (@uandrae)

  • Add pandas 2.0 compatible versions to allowed dependencies. #942 (@observingClouds)

Fixed

  • Correct orography truncation for linear grid. #960(@uandrae)

  • Correct merge of fullpos selection #941(@uandrae)

  • Load ecmwf-toolbox/2024.11.0.0 for AddCalculateFields #958(@KristianHMoller)

  • Semi-functional settings for running DW on LUMI (GMK binary, Cray 16).#947(@kastelecn)

  • Remove erroneous domain specification from flooding event #943 (@uandrae)

  • Remove 10wdir from gribmodify.#934 (@kastelecn)

  • Add FA2GRIB translation for gust wind speed, makes the gribmodify part obsolete. #928 (@uandrae)

  • Bump python version to 3.10 in github linting action. #933 (@johtoblan)

[0.10.0] - 2024-12-17

Added

  • Add namelists to create needed output for postprocessing #923 (@egregow)

  • Add unit test for standalone template file #890 (@pardallio)

  • Add wind fields to grib output. #916 (@kastelecn)

  • Add lightning to ALARO. #907 (@kastelecn)

  • Output surface geopotential (CY46h1) using gl to separate file in air quality events #854 (@sbnielsen)

Fixed

  • Fixed get_decadal_list #924(@natalieth)

  • Fixed fullpos.selection for subhourly output and added CLSMEAN.RAD.TEMP to alaro fullpus namelist. #893 (@kastelecn)

  • Fixes to make deode case -e produce a correct config file #897 (@uandrae)

  • Only run gl if there is an actual gl namelist #854 (@sbnielsen)

  • Correctly mask surfex values where missing in CY48t3 #854 (@sbnielsen)

  • Output surface geopotential (CY48t3) and land-sea-mask (CY48t3 and CY46h1) at analysis instead of forecast in air quality events #854 (@sbnielsen)

Changed

  • Updated git branch structure of repository to standardize branch names and workflows. #885 (@mafdmi)

  • Additional fixes for ALARO lightning, finall fixes for ALARO namelist, add graupel to total precipitation for all CSC. #915(@kastelecn)

  • Add MF_87 vertical levels for ALARO (instead of MF_90). #914 (@kastelecn)

  • Updated docs with accept/decline procedure for reviewers; all accepting reviewers now have to approve before merge. #872 (@mafdmi)

  • Updated air quality event output in cy46h1 to use surfex parameters from tile 3 instead of 1 to fix issue since transition to 3 patches. #912 (@sbnielsen)

  • Added WOW’s station lists and others for verification #860 (@svianaj)

[0.9.1] - 2024-11-22

Fixed

  • Archiving of config file in operational setting #896 (@FlorianW-ZAMG)

[0.9.0] - 2024-11-20

Added

  • Add mars settings for expver iit7 (eps example) #879 (@kastelecn)

  • Add xml dumper impact models configs. #837 (@uandrae)

  • Intercept standard logging in loguru log handler. #855

  • Add end time for expver i4ql (ATOS_DT) #866 (@kastelecn)

  • Airquality event type configuration file. #818 (@sbnielsen)

  • Temporary SURFEX grib2 definitions for CY6h1 and CY48t3. #818 (@sbnielsen)

  • Canopy water and land-sea-mask to fullpos aq selection. #818 (@sbnielsen)

Changed

  • Replace batches with ecflow limit for LBC tasks. #850(@kastelecn)

  • deodemakedirs will use default linux permission unless unixgroup is specified. #879 (@trygveasp)

  • Refactored the DEODE ecFlow suite generation scripts. Bumped python version to 3.10 in both test environment and pyproject.toml. #831 (@mafdmi)

  • Changed impact model config layout #838 (@uandrae)

  • Updated operational settings #859 (@kastelecn)

  • Added LACE’s station lists for verification [#860] (https://github.com/destination-earth-digital-twins/Deode-Workflow/pull/860) (@svianaj)

  • Creategrib task has been changed so that it is more flexible and can handle several conversions. [#818] (https://github.com/destination-earth-digital-twins/Deode-Workflow/pull/818) (@sbnielsen)

  • Grib definitions of momentum fluxes and canopy water. [#818] (https://github.com/destination-earth-digital-twins/Deode-Workflow/pull/818) (@sbnielsen)

  • Updated descriptions for setting up environment and installation in README.md. [#894] (https://github.com/destination-earth-digital-twins/Deode-Workflow/pull/894) (@ole-dmi)

Fixed

  • Added find_value function to fn_steplist to decode string into list. #883 (@pardallio)

  • Added empty modification submission files for alaro on atos_bologna to silent erroneous warnings. #887 (@kastelecn)

  • Not abort on missing logs directory. #878 (@trygveasp)

  • Fixed bug in clean_old_data.toml. #882 (@kastelecn)

  • Set NPATCH=3 explicitly for HARMONIE-AROME. \871 (@uandrae)

  • Made it impossible to parse any string to output_settings. Only empty string, string of ISO8601 format, or list of strings of ISO8601 format are possible. #830 (@mafdmi)

  • Bug that made it impossible to merge namelists with more than one hybrid level with master selection in CY48t3 #818 (@sbnielsen)

  • Change the behaviour of deode run to only require the --task argument. The other arguments will default to names based on the task name. #836 (@uandrae)

[0.8.0] - 2024-10-31

Added

  • Added a operational config toml to be activated after further inspection. #829 (@uandrae)

  • Added the same macros to general.times.[start|end|validtime] as for basetime. #829 (@uandrae)

  • Add settings for suite CleanOldData, which clean scratch data, suites in ecflow server and IFS data. It include cron. #825 (@kastelecn)

  • Add posibility to read, and remove, suite definition file. #799 #832 (@uandrae)

  • Add functionality for user macros and to expand macros in config file. #824 (@uandrae)

Changed

  • Changed the test procedure to require using deode/data/config_files/modifications/test_settings.toml. #829 (@uandrae)

  • Renamed the Norrkoping domain to 500m tempalate and changed useage accordingly. #829 (@uandrae)

  • Change mars setings to work on lumi with lumi_DT selection. #817 (@kastelecn)

  • Remove used suite definition file by default. #799 (@uandrae)

  • Change default ifs.selection to point to the valid DT on the current host. With this the default selection on atos is coming from expver=iekm . #826 (@uandrae)

Removed

  • Removed the requirement to set general.times.end. #829 (@uandrae)

  • Removed the requirement to set domain.[xlat0|xlon0]. #829 (@uandrae)

[0.7.1] - 2024-10-17

Added

  • Add total precipitation to grib and add total rain and snow to grib (ALARO). #804 (@kastelecn)

[0.7.0] - 2024-10-09

Added

  • Introduce a general method for starting impact models in general and EHYPE in particular #793 (@uandrae)

  • Automatic name convention for config file output from deode case #785 #798 (@uandrae)

  • Allow host specification by environment variable DEODE_HOST #774 (@uandrae)

Changed

  • Updated runtime thresholds for CreateGrib task to 2 hours #809 (@tbnc)

  • Separate fullpos selections by CSC where required #808 (@uandrae)

  • Updated binary paths to include the “latest” installations of IAL and gl #807 (@pardallio)

  • Change fullpos selection syntax from a list to a dictionary of lists to allow better merge functionality #792 (@uandrae)

  • Remove domain name from the stored grib files #788 (@uandrae)

  • Change URL for troika #795 (@uandrae)

  • Change default ifs_delection to ATOS_DT #775 (@kastelecn)

Fixed

  • Various configuration settings lost in translation #813 (@uandrae)

  • Store config file with a fixed name rather than the user one #808 (@uandrae)

  • Repeat calls to ecflow host fixing failures due to unreacheable ecflow host #805 (@uandrae)

  • Fix lost target run functionality #803 (@uandrae)

  • Correct usage of start date in the case name #801 (@uandrae)

  • Correct setting of ECCODES_DEFINITION_PATH depending on ECcodes version #766 (@uandrae)

  • Correct notation and activation of the CreateGrib task #770 (@uandrae)

  • Respect input types in namelist config parsing #784 (@uandrae)

  • Removed duplicated parsing #783 (@uandrae)

  • Fix ecf_host selector not selecting the correct naming convention on Atos #781 (@adam-otruba)

##[0.6.2] - 2024-09-26

Fixed

  • Fix ecf_host selector not selecting the correct naming convention on Atos #781 (@adam-otruba)

[0.6.1] - 2024-09-24

  • change binary version for CY48t3

  • set ATOS_DT as default ifs.selection

##[bugfix_v0.6.2] - 2024-09-26

Fixed

  • Fix ecf_host selector not selecting the correct naming convention on Atos #781 (@adam-otruba)

[bugfix_v0.6.1] - 2024-09-24

  • change binary version for CY48t3

  • set ATOS_DT as default ifs.selection

[0.6.0] - 2024-09-19

Added

  • Add missing unit tests for creategrib #770 (@uandrae)

  • Add switch LWTHRESHMOIST in CY46h1 following implementation in HARMONIE repo #757 (@natalieth)

  • Introduce the possibility for mulitple simultaneous archiving methods #752(@uandrae)

  • Use of config files and mod files shipped with Deode-Workflow is now possible, when installing Deode-Workflow as a package in another repository #671(@mafdmi)

  • Changes default archiving storage on ATOS to ec: #753(@FlorianW-ZAMG)

  • Added FDB-archiving on LUMI with pyfdb. #577

  • Introduced CY49t2 namelist and config files for all three CSCs #698 (@uandrae)

  • Updated documentation of ecflow server settings #659 (@FlorianW-ZAMG)

  • Mars works on LUMI, added selection LUMI_DT #647 (@kastelcn)

  • Introduced IOmerge task[s] that can run the merging of IOSERVER output in parallel while the forecast is running, rather than after #677 (@adeckmyn)

  • Introduced a ecf_host selection function to handle the old and new ecflow server name conventions on atos#675 (@uandrae)

  • Introduced a short description of available tasks #658 (@uandrae)

  • Introduction of Leonardo machine #645 (@dhaumont)

  • Adding 49t2 configuration files #643 (@dhaumont)

  • Thenamelisttool integration to convert a namelist from one cycle to another #613 (@dhaumont)

  • Added option to pick binaries from the different repositories in the same task. #630 (@kastelecn)

  • Functionality to run CY46h1 on LUMI. #562 (@tbnc)

  • Introduce config files for SMHI laptop. #606 (@uandrae)

  • Introduce config files for SMHI laptop. #606 (@uandrae)

  • Introduce config files for the SMHI HPC freja@NSC. #595 (@uandrae)

  • Additional way to initialise a module environment. #597 (@uandrae)

  • Function for setting ecf_port by userid. #598 (@uandrae)

  • Possibility for case setup and configurations. #557 (@trygveasp)

  • Possibility for cleaning of experiment. #587 #637 (@milennimh, @uandrae)

  • Add mean winds grib2 definitions. #585 (@sbnielsen)

  • Add continous integration workflow for testing installation process on Atos and LUMI #437 (@khintz)

Changed

  • Introduce OmegaConf for namelist configuration handling. Move namelist data to /data sub-directory. #759 (@adeckmyn)

  • Move bdcycle ro mars_settings, fixes for mars on LUMI #765 (@kastelecn)

  • Force user to set expver manually for FDB archiving #763 (@uandrae)

  • Updates path to Cycle 48t3 and 46h1 binaries to comply with CI/CD #760 (@pardallio)

  • Centralise definition of various package related directories #758(@uandrae)

  • Changed default DEODE_HOME to ./Deode-Workflow/deode #671(@mafdmi)

  • Updates path to Cycle 48t3 binaries to comply with CI/CD #755 (@pardallio)

  • Documentation for binary seclection #736 (@uandrae)

  • Update HARMONIE-AROME to harmonie-46h1.1 binaries. #693 (@uandrae)

  • Always use the class name as the task name to be consistent with submission settings. #701 (@kastelecn, @uandrae)

  • Make io-merge processing more verbose and configurable w.r.t waiting times. #694 (@uandrae)

  • Geopotential z in latlon grid for surfex input retrieves from ICMSH global file instead of global_sfcdir. #697 (@kastelecn)

  • Changed location of the binary fa_sfx2clim. #672 (@kastelecn)

  • Update the default case name to include the domain name. #655 (@uandrae)

  • Adapt settings to binary /scratch/project_465000527/ospaniel/executable_cy48t3/ works. #618 (@kastelecn)

  • Externalise the selection of static input data to the forecast. #619 (@uandrae)

  • Improve error message on erroneous command line usage. #629 (@uandrae)

  • README content and cosmetics corrections. #600 (@uandrae)

Fixed

  • Creation of remote directories with scp #763 (@uandrae)

  • Adjust sqlite template and path to fix missing ecfs archiving #748 (@uandrae)

  • Erroneous submission section on leonardo #736 (@uandrae)

  • Changing LUMI’s common_de330 data area. #738 (@draelsaid)

  • Removes platform-specific namelist settings from the Cy46h1 master namelist file following Jul 2024 binaries update. #742 (@tbnc)

  • Fix mars retrieve for latlon z. #687 (@kastelecn)

  • Correct the namelist update functionality. #673 (@uandrae)

  • Fix bug in SQLite extraction at other times than midnight #680 (@adeckmyn)

  • Minor change to CY46h1/master_namelists.yml for simulated radiance calculation #670 (@fbaordo)

  • Restore large domain settings and arome -> arome config template. #642 (@uandrae)

  • Make sure schema files are included for validation when creating new config files. #601 (@uandrae)

  • Allow macro parsing of path to troika. #596 (@uandrae)

  • Handle shift in forcing model start time. #627 (@uandrae)

  • Allow macro parsing of path to troika. #596 (@uandrae)

  • Introduce support for scp as a provider method in the file manager. #579 (@uandrae)

  • Include missing json schema validation for troika. #594 (@uandrae)

  • Amendments to [#557] with updated documentation and changes to allow all tests to run on both atos and LUMI. #586 (@uandrae)

  • Updated grib2 tablesVersions to 32 in faFieldName.def. #585 (@sbnielsen)

  • Updated grib2 definitions of following fields: SURFLIFTCONDLEV, SURFEQUILIBRLEV, SURFFREECONVLEV. #585 (@sbnielsen)

  • Remove scaleFactorOfFirstFixedSurface=0 for all fields with typeOfFirstFixedSurface=1 in grib2 definitions in agreement with eccodes standards. #585 (@sbnielsen)

  • Change CI-HPC workflow to only run on develop and master branch to adhere to security concerns. #681 (@khintz)

  • Fix bug where if do-cleaning was False in suite-configuration, suites could not be started #665 (@johtoblan)

  • Replace all occurences of “prototype” in documentation with “workflow” and fix all the resultant broken code and links #737 (@adam-otruba)

  • Replace unnecessarily external links within documentation with internal links #737 (@adam-otruba)

Removed

  • Obsolete config file for CY48t3 submission on lumi #736 (@uandrae)

  • All occurences of output variables with stepType!=instant, i.e. accumulated/min/max, at t=0 to conform to fdb strict grib encoding standards#590(@uandrae)

[0.5.0] - 2024-05-06

Added

  • Option for sub hour and sub minute bdint#565(@kastelecn)

  • Suites and tasks treated as plug-ins. #526 (@trygveasp)

  • Add option to run marsprep before each c903/Prep task separately. #546

  • Add extra variable, mean radiant temperature in both CY46h1 and CY48t3. #548 and #537

Changed

  • Updated CY48t3 binaries and switch on hybrid parallelisation. #548 (@kastelecn)

  • Updated CY46h1 binaries and switch on hybrid parallelisation. #541 (@uandrae)

  • Separate the subhourly fullpos output conig to allow more fine grained control. #524 (@uandrae)

  • Improve task detection. Avoids erroneous log messages #498 (@uandrae)

Fixed

  • Recover lost coverage #575 (@trygveasp)

  • Correct path construction in the sql extraction #566 (@uandrae)

  • Better error message in case of missing initial files #560 (@uandrae)

  • Decrease NPROC and NPROC_IO for forecast on the small domain to work with _target domain. #558 (@kastelecn)

  • Use old microphysics routine on LUMI as a temperorary workaround, fixes issue #536. #547 (@uandrae)

  • Correct task recognition for Forecast. Makes the usage in the submission config more intuitive. #549 (@uandrae)

  • Fix issue #483, broken AQ output selection. #524 (@uandrae)

  • Fix SQLite issues #485 (Failure when no obs. stations in domain) and #521 (Memory leak). #520 (@adeckmyn)

[0.4.0] - 2024-03-01

Breaking changes

  • Deode new calls (flags removed) #416 (@draelsaid)

Added

  • Implemented new de330-dev server (for more information see docs/markdown/lumi.md) #510(@draelsaid)

  • Implemented e923Update to improve the ALARO forecast (for more information see doc/markdown/e923_update.md) #510(@kastelecn)

  • Add ALARO forecast #261 (@adeckmyn)

  • Sqlite improvements #413 (@adeckmyn)

  • Add namelists and config files for ALARO forecast #261 (@adeckmyn)

  • Allow recursive references in the namelist config #412 (@uandrae)

  • SQLite extraction: add parameters requiring multiple GRIB fields, fix speed issues #413 (@adeckmyn)

  • Add fullpos radiation selection for solar renewable output streams #425 (@sbnielsen)

  • Add Paris RDP global DT experiment MARS request #430 (@uandrae)

  • Add option to use Open Street Map data to create PGD for Paris region #434 (@natalieth)

  • Add submission files for experiments with t3999 (Increasing the number of nodes to increase memory) #456 (@kastelecn)

  • LUMI Ecflow suite #464 (@draelsaid)

  • Add renewables wind selection #482 (@uandrae)

  • Add submission settings for c903 on LUMI #497 (@kastelecn)

  • Config updates that brings the default setup for AROME@CY48t3 running under ECFLOW on LUMI #502 (@uandrae)

  • Added poethepoet as part of the pyproject.toml, such that we can specify the version for local and github #496 (@johtoblan)

  • Dummy FDB archiving methods #495 (@uandrae)

  • Submission of bash wrapper scripts #492 (@trygveasp)

  • Introduce troika config for lumi prod/dev users and support for macro parsing of the troika file #491 (@uandrae)

Changed

  • Update Alaro to work properly #403 (@kastelecn)

  • Use OpenMP in CY48t3 forecasts on large domains #427 (@FlorianW-ZAMG)

  • Extend submission MODULE scope #432 (@uandrae)

  • Add more parameters to be extracted for verification #439 (@uandrae)

  • Add SQLite extraction of wind direction at pressure levels to config file #447 (@adeckmyn)

  • Fixed setup for forecasts using the global DT at 12UTC for boundary data #442 (@tbnc)

  • ALARO Forecast on LUMI - CPU and GPU version #452 (@dhaumont)

  • Update platform paths #454 (@uandrae)

  • Change the start date of the HRES data #455 (@kastelecn)

  • Added MARS settings for January 2017 period (winter AQ runs) using the global DT for boundary data #463 (@tbnc)

  • Add IWIDTH paranmeter to sfx namelist for CY48t3 #473 (@kastelecn)

  • Reading surface fields from sfcdir with higher resolution #476 (@kastelecn)

  • Activate ccsds packing for CY48t3 #479 (@uandrae)

  • Speeding up LUMI>Ecflow server file copy #542 (@draelsaid)

Removed

  • Remove mars_expver variable #406 (@tbnc)

  • Remove duplicated ArchiveStatic #448 (@uandrae)

  • Remove msdeode as default account and simplify account setting #487 (@uandrae)

Fixed

  • Replace hardcoded value for NRTFP3S with ${vertical_levels.nlev} #411 (@kastelecn)

  • Avoid STDOUT/STDERR decoding errors #426 (@uandrae)

  • Correct sqlite_model_name #429 (@uandrae)

  • Fix/pgd multidecades issue and climate file generation for longer periods #443 (@FlorianW-ZAMG)

  • fix missing satellite fields in grib-files at full hours #444 (@FlorianW-ZAMG)

  • Correct MARS config and json settings #446 (@uandrae)

  • Submission schema typo: Set Default value of lfttw to True #475 (@kastelecn)

  • reintroduce wind.u/v.phy in faFieldName.def, accidentally removed #477 (@FlorianW-ZAMG)

  • Fix setting of ECCODES_DEFINITION_PATH #480 (@uandrae)

  • Separate scheduler and stand alone arguments in bash script used in submission #501 (@trygveasp)

  • Correct erroneous usage of bdshift and fixed marsprep unit testing #489 (@uandrae)

  • Fix wind direction in SQLite extraction: use uvRelativeToGrid #449 (@adeckmyn)

Maintenance

  • Update documentation #576 (@draelsaid)

  • Update documentation #441 (@uandrae)

  • Lumi - improve submission settings #457 (@dhaumont)

  • Add pull-request template #465 (@leifdenby)

  • Draelsaid/latest lumi #467 (@draelsaid)

  • Fix typo and add info on ecflow on LUMI in README #472 (@tbnc)

  • LUMI: Ecflow Job Submission, Troika etc #478 (@draelsaid)

  • LUMI: deode flag removal and readme testing #493 (@draelsaid)

  • LUMI readme updates #503 (@draelsaid)

  • LUMI: Minor updates to readme #506 (@FlorianW-ZAMG)

[0.3.0] - 2023-11-28

Added

  • Configurable archiving on ecfs, default switched on

  • First sqlite extraction for verification. Default switched on but with a limited set of parameters

  • Support for reading IFSENS data

Changed

  • Extraction from MARS configurable from config. More data streams added and corrected resolution for global DT data.

  • Changed config structure. E.g. model timestep is now found in the domain part.

  • Updated config settings. E.g. settings for large domains running on atos. Boundary settings reflecting availability of HRES.

  • Json schema validation and documentation updates.

  • Change reflecting stricter linting.

Fixed

  • Stop writing log files to the user’s home directory. This was reintroduced by mistake.

  • Correct windfarm fullpos activation

  • Remove explicit python code from the submission files

  • Fixes to allow to run the forecast as a stand alone task for the default config

[0.2.0] - 2023-10-27

Added

  • Adjustment of coupling zone with depending on resolution

  • Introduce support for cold_start, start, restart mode using suite_control.mode see documentation for usage. Replaces the cold_start flag.

  • Fullpos output templates for air quality and hydrology applications. Add gust to the general fullpos output.

  • Speedup of the PGD generation by setting pgd.one_decade = true. Using false retains to old behaviour.

  • Allow setting of unix group for all created directories using platform.unix_group. The current default for atos is msdeode.

  • poetry devtools command

    To help developers with tasks such as linting and pre-push checks

Changed

  • New default binaries for CY46h1.

  • General improvements in documentation about installation, development practices and configuration.

  • Remove less frequently used settings from the example config files and update the json schema files, and documentation accordingly. Impose stricter output frequency settings.

  • Minimum required python version is now v3.9

    • Requires reinstaling the environment. See instructions in the README file.

  • deode doc config command now shows info defined in the schema only.

    Users can run deode show config when they wish to know what is used in the config.

  • Migrate most linting checks from flake8 to ruff. It is faster.

Removed

  • deode toml-formatter command Use a separate library for this

Fixed

  • Correct the unit testing of MARS tasks.

  • Updated GRIB2 definitions to conform to WMO standards.

  • Surface scheme settings for CY48t3. Fixes problems with erroneous soil data input and crashes in PREP when coupling AROME to AROME.

  • Configuration fixes for the default AROME to AROME coupling.

  • Format of online config doc in github pages.

  • Fixes throughout doc files and online docs.

[0.1.0] - 2023-10-11

Added

v0.1.0 of deode is able to perform forecast generation for an arbitrary European domain tailored to high-resolution simulations (with AROME CY48t3 and HARMONIE-AROME CY46h1) on the ECMWF ATOS HPC. Basic functionality for archiving on ECFS@ATOS is availiable but switched off by default.