intake-esm

Prepared for the MPI-M LunchBytes online seminar on March 26 by Aaron Spring.

Installing intake-esm on Mistral

The easiest way to install intake-esm on Mistral is using Anaconda. It makes sense to move the environments to a different location than the home directory because each environment may use more than 1GB.

First, load the anaconda module: 

module load anaconda3/bleeding_edge

To change the default environment directory, edit the ~/.condarc file: 

envs_dirs:
  - /work/your_project/m123456/conda-envs

To create a new environment and install intake-esm, clone the tutorial repository from gitlab.dkrz.de to your $HOME directory: 

git clone https://gitlab.dkrz.de/m300524/lunchbytes_intake-esm.git
cd lunchbytes_intake-esm
conda env create -f intake-esm.yaml

Confirm the list of packages that will be installed and wait…

Some useful commands for working with Anaconda can be found in the conda cheat sheet.

Run Jupyter

We will access the conda environment intake-esm interactively via jupyter. We use the script start-jupyter from DKRZ.

Clone the tutorial repository also locally: 

git clone https://gitlab.dkrz.de/m300524/lunchbytes_intake-esm.git
cd lunchbytes_intake-esm

Before starting jupyter from your local machine, ensure that no other jupyter instance is running locally and remotely.

Start jupyter on mistral with the DKRZ script from your local laptop: 

./utils/start-jupyter -u mXXXXXX -i lunchbytes_intake-esm/utils/jupyter_preload -c lab -A your_project -p compute

In case you need to wait too long to get access or you get memory issues during using, try specifying a different ''node type'', such as -p compute(2). You can also try -p shared, but then I got too little memory for dask to work fast.

If you get asked for username and password each time, consider setting up your ssh key.

Tutorial

Open the notebook inside jupyter in the repository lunchbytes_intake-esm and play.

I thought everyone has access to my home directory. If not, please clone intake-esm-data-store: 

git clone https://gitlab.dkrz.de/m300524/intake-esm-datastore

And change the paths in your notebook.

Note: We can only access the CMIP6 files already downloaded to /work/ik1017/CMIP6/data/CMIP6 at the time the intake-esm catalog was built last time. This might be less output than what you find on ESGF.

If you ultimately fail to get anything working, a very similar rnotebook in the cloud. Note that no changes are saved if you log off. Click on the notebook file to save the .ipynb to your local computer.

Tasks

work on the notebook tasks.ipynb.

Additional ressources