====== intake-esm ======

Prepared for the MPI-M LunchBytes online seminar on March 26 by Aaron Spring.

<alert type="warning" dismiss="true">
For now on, please use the official catalogs folder:
  /work/ik1017/Catalogs
</alert>

===== Installing intake-esm on Mistral =====

The easiest way to install ''intake-esm'' on Mistral is using Anaconda. It makes sense to move the environments to a different location than the home directory because each environment may use more than 1GB.

First, load the anaconda module:

  module load anaconda3/bleeding_edge

<alert type="warning" dismiss="true">
If there is a problem loading the anaconda module, try before:
  module unload netcdf_c
</alert>

To change the default environment directory, edit the ''~/.condarc'' file:

  envs_dirs:
    - /work/your_project/m123456/conda-envs

To create a new environment and install ''intake-esm'', clone the tutorial repository from ''gitlab.dkrz.de'' to your ''$HOME'' directory:

  git clone https://gitlab.dkrz.de/m300524/lunchbytes_intake-esm.git
  cd lunchbytes_intake-esm
  conda env create -f intake-esm.yaml

Confirm the list of packages that will be installed and wait...

Some useful commands for working with Anaconda can be found in the [[https://docs.conda.io/projects/conda/en/4.6.0/_downloads/52a95608c49671267e40c689e0bc00ca/conda-cheatsheet.pdf|conda cheat sheet]].

===== Run Jupyter =====

We will access the conda environment ''intake-esm'' interactively via ''jupyter''. We use the script ''start-jupyter'' from [[https://www.dkrz.de/up/systems/mistral/programming/jupyter-notebook| DKRZ]].

Clone the tutorial repository also locally:

  git clone https://gitlab.dkrz.de/m300524/lunchbytes_intake-esm.git
  cd lunchbytes_intake-esm

Before starting ''jupyter'' from your local machine, ensure that no other ''jupyter'' instance is running locally and remotely.

Start ''jupyter'' on mistral with the DKRZ script from your local laptop:

  ./utils/start-jupyter -u mXXXXXX -i lunchbytes_intake-esm/utils/jupyter_preload -c lab -A your_project -p compute

In case you need to wait too long to get access or you get memory issues during using, try specifying a different [[https://www.dkrz.de/up/systems/mistral/configuration|''node type'']], such as ''-p compute(2)''. You can also try ''-p shared'', but then I got too little memory for ''dask'' to work fast.

If you get asked for ''username'' and ''password'' each time, [[local_it:use_software:unix_linux:faq|consider setting up your ssh key]].

===== Tutorial =====

Open the notebook inside ''jupyter'' in the repository ''lunchbytes_intake-esm'' and play.

I thought everyone has access to my home directory. If not, please ''clone'' ''intake-esm-data-store'':

  git clone https://gitlab.dkrz.de/m300524/intake-esm-datastore

And change the paths in your notebook.

Note: We can only access the CMIP6 files already downloaded to ''/work/ik1017/CMIP6/data/CMIP6'' at the time the ''intake-esm'' ''catalog'' was built last time. This might be less output than what you find on [[https://pcmdi.llnl.gov/CMIP6/ArchiveStatistics/esgf_data_holdings/|ESGF]].

If you ultimately fail to get anything working, a very similar rnotebook in the [[https://binder.pangeo.io/v2/gh/aaronspring/LunchBytes_intake-esm_cloud/master?urlpath=lab?filepath=notebooks%2Fintake-esm_cloud.ipynb|cloud]]. Note that no changes are saved if you log off. Click on the notebook file to save the ''.ipynb'' to your local computer.



===== Tasks =====

work on the notebook [[https://gitlab.dkrz.de/m300524/lunchbytes_intake-esm/-/blob/master/tasks.ipynb|tasks.ipynb]].
===== Additional ressources =====

  * search github
  * ask your peer
  * google
  * [[https://intake-esm.readthedocs.io/en/latest/|intake-esm]]
  * local CMIP6 download: ''/work/ik1017/CMIP6/data/CMIP6''
  * CMIP6 data availability: [[https://pcmdi.llnl.gov/CMIP6/ArchiveStatistics/esgf_data_holdings/]]
  * [[https://intake-xarray.readthedocs.io/en/latest/|intake-xarray]]
  * [[analysis:pot_pourri:python:intake|intake on mistral wiki post]]