Bash Script for postprocessing with CDO

The shell (bash) script below is an example for postprocessing using multiple CDO commands including the following general parts:
- Selecting time periods
- Selecting variables
- Temporal resampling (e.g. to daily means, if the data is already at daily means, then this step would be idle but harmless)
- Interpolation to a longitude-latitude grid (see section “Regridding the data” using CDO within the shell)

The script assumes the existence of a grid-description and a weight-file (see also “Regridding the data”).

Since the required computational performance may exceed what is available at the interactive prepost nodes, it may be useful to read the Wiki introduction section about SLURM workload manager and allocating an exclusive computational environment.

As an example, the script selects surface temperatre (ts) for the months January-March of the 1 year-long coupled dyamond simulation), calculates daily mean values and interpolates the data onto a 1°x1° longitude-latitude grid. This script makes use of the SLURM workload manager and can be submitted via sbatch scriptname.sh. If it is intended to execute the script without SLURM, just delete the 'SBATCH … lines at the top (leave the #!/bin/sh) and execute the script via bash scriptname.sh.

#!/bin/sh 
#SBATCH --job-name=myjobname
#SBATCH --partition=gpu
#SBATCH --nodes=1
#SBATCH --time=01:00:00
#SBATCH --mail-type=FAIL
#SBATCH --account=project_name_to_be_charged
#SBATCH --output=job_log.o%j
#SBATCH --error=job_log.e%j
 
set -xe
 
## example of how to postprocess 2d (lon-lat and time) atmospheric variable. 
 
### START USER INPUT
CDO="/home/mpim/m214003/local/bin/cdo" #path for cdo version (here Uwes local version)
var="ts" #variable
years="2020"
exp="dpp0016"
grid="1" #in degree (i.e. 1x1): other option: grid="005" for 0.05x0.05 or define own grid description file and weight file
months="01 02 03"
model_files="${exp}_atm_2d_ml_" #adjust for variable of interest
path_to_scratch="$SCRATCH/${exp}"
path_to_model_raw_data="/work/mh0287/k203123/GIT/icon-aes-dyw_albW/experiments/${exp}"
griddes_file="griddes_${grid}.txt" #specifiy grid description text file here (see Wiki on this: https://wiki.mpimet.mpg.de/doku.php?id=analysis:postprocessing_icon:regridding:shell:start)
weight_file="weight_${exp}_${grid}.nc" #specifiy weight file here (see Wiki on this: https://wiki.mpimet.mpg.de/doku.php?id=analysis:postprocessing_icon:regridding:shell:start)
out_dir="$SCRATCH/${exp}" # this is the path where the final file will be stored. In this example, this is also the $SCRATCH dir of the user, but could be changed to the $WORK dir of the user for instance
griddes_file="griddes_${grid}.txt" #specifiy grid description text file here (see Wiki on this: https://wiki.mpimet.mpg.de/doku.php?id=analysis:postprocessing_icon:regridding:shell:start)
weight_file="weight_${exp}_${grid}.nc" #specifiy weight file here (see Wiki on this: https://wiki.mpimet.mpg.de/doku.php?id=analysis:postprocessing_icon:regridding:shell:start)
### END USER INPUT
 
#####################################
### creating directory structure on scratch (if non existing)
 
for yy in ${years}; do
   for mm in ${months}; do
 
      ${CDO} -O -P 24 -remap,${griddes_file},${weight_file} -daymean -cat -apply,"-selname,${var}" [ ${path_to_model_raw_data}/${model_files}*${yy}${mm}*.nc ] ${path_to_scratch}/${var}_tmp_${mm}.nc
 
   done  #mm
done #yrs
 
## cat and del monthly chunk files
${CDO} -O -P 24 cat ${path_to_scratch}/${var}_tmp_*.nc ${out_dir}/${exp}_${var}_${grid}x${grid}.nc
rm ${path_to_scratch}/${var}_tmp_*.nc