====== ESMValTool v2 on Mistral ======

Most of this document is bases on the instructions in [[https://esmvaltool.readthedocs.io/en/latest/|the official documentation]].

Prepared for the MPI-M LunchBytes seminar on Nov 18 by Sebastian Milinski <sebastian.milinski@mpimet.mpg.de>.


===== Installing ESMValTool on Mistral =====
The easiest way to install ESMValTool on Mistral is using Anaconda. It makes sense to move the environments to a different location than the home directory because each environment may use 4-5GB.

First, load the anaconda module:

  module load anaconda3/bleeding_edge

<alert type="warning" dismiss="false">
If there is a problem loading the anaconda module, try:
  module unload netcdf_c
</alert>



To change the default environment directory, edit the ''~/.condarc'' file:

  envs_dirs:
    - /work/your_project/m123456/conda-envs


To create a new environment and install ESMValTool in it, run:
  
  conda create --name esmvaltool_fixed -c esmvalgroup -c conda-forge julia esmvalcore==2.0.0b1 esmvaltool==2.0.0b1

<alert type="danger" dismiss="false">
On Nov 18 2019, the official installation instructions result in an installation that is incompatible with many examples. This configuration uses a slightly older ESMValTool version and explicitly installs julia.</alert>

Anaconda will print a list of packages that will be installed. Confirm with enter and get a coffee... (this can take a while).

Some useful commands for working with Anaconda can be found in the [[https://docs.conda.io/projects/conda/en/4.6.0/_downloads/52a95608c49671267e40c689e0bc00ca/conda-cheatsheet.pdf|conda cheat sheet]].

===== Configure ESMValTool =====

A config file with some adjustments for Mistral can be copied from here:
  /home/mpim/m300265/scripts/lunchbytes/esmvaltool/config-user-lunchbytes.yml

===== Run ESMValTool =====

A very simple example can be copied from here:
  /home/mpim/m300265/scripts/lunchbytes/esmvaltool/examples/recipe_python.yml

In the ''config-user-lunchbytes.yml'', all output will go into the directory ''./esmvaltool_output'' in your current working directory. You might want to change your directory before the next command.

Before running ESMValTool, make sure that the Conda environment that you created before is activated:
  source activate esmvaltool

<alert type="warning" dismiss="true">
Depending on your shell configuration, you can use //conda activate// instead of //source activate//:
  Conda activate esmvaltool
</alert>

Finally, to run this example, type:
  esmvaltool -c config-user-lunchbytes.yml recipe_python.yml
  
When you are done with the ESMValTool runs and want to deactivate the Conda environment to return to your normal software tree or a different Conda environment, type:
  source deactivate

<alert type="warning" dismiss="true">
Depending on your shell configuration, you can use //conda deactivate// instead of //source deactivate//:
  Conda deactivate
</alert>

===== Understanding the output =====

===== Modifying the preprocessor and diagnostics =====

===== Additional ressources =====
  * CMIP6 data availability: [[https://pcmdi.llnl.gov/CMIP6/ArchiveStatistics/esgf_data_holdings/]]
  * [[https://esmvaltool.readthedocs.io/en/latest/recipes/index.html| recipes in the ESMValTool documentation]]
  * [[https://github.com/ESMValGroup/ESMValTool/tree/version2_development/esmvaltool/recipes|More recipes on github]]