Prepared for the MPI-M LunchBytes online seminar on March 26 by Aaron Spring.
/work/ik1017/Catalogs
The easiest way to install intake-esm on Mistral is using Anaconda. It makes sense to move the environments to a different location than the home directory because each environment may use more than 1GB.
First, load the anaconda module:
module load anaconda3/bleeding_edge
module unload netcdf_c
To change the default environment directory, edit the ~/.condarc file:
envs_dirs: - /work/your_project/m123456/conda-envs
To create a new environment and install intake-esm, clone the tutorial repository from gitlab.dkrz.de to your $HOME directory:
git clone https://gitlab.dkrz.de/m300524/lunchbytes_intake-esm.git cd lunchbytes_intake-esm conda env create -f intake-esm.yaml
Confirm the list of packages that will be installed and wait…
Some useful commands for working with Anaconda can be found in the conda cheat sheet.
We will access the conda environment intake-esm interactively via jupyter. We use the script start-jupyter from DKRZ.
Clone the tutorial repository also locally:
git clone https://gitlab.dkrz.de/m300524/lunchbytes_intake-esm.git cd lunchbytes_intake-esm
Before starting jupyter from your local machine, ensure that no other jupyter instance is running locally and remotely.
Start jupyter on mistral with the DKRZ script from your local laptop:
./utils/start-jupyter -u mXXXXXX -i lunchbytes_intake-esm/utils/jupyter_preload -c lab -A your_project -p compute
In case you need to wait too long to get access or you get memory issues during using, try specifying a different ''node type'', such as -p compute(2). You can also try -p shared, but then I got too little memory for dask to work fast.
If you get asked for username and password each time, consider setting up your ssh key.
Open the notebook inside jupyter in the repository lunchbytes_intake-esm and play.
I thought everyone has access to my home directory. If not, please clone intake-esm-data-store:
git clone https://gitlab.dkrz.de/m300524/intake-esm-datastore
And change the paths in your notebook.
Note: We can only access the CMIP6 files already downloaded to /work/ik1017/CMIP6/data/CMIP6 at the time the intake-esm catalog was built last time. This might be less output than what you find on ESGF.
If you ultimately fail to get anything working, a very similar rnotebook in the cloud. Note that no changes are saved if you log off. Click on the notebook file to save the .ipynb to your local computer.
work on the notebook tasks.ipynb.
/work/ik1017/CMIP6/data/CMIP6