Transfer to HPSS file system

After I presented the scripts for compressing output and restarts, I show you how to end the storage task.

Search only in this Namespaces below. For a global search, use the field in the upper right corner. More tips: how_to_use_the_wikisearch

HPSS and pftp

The HPSS tape system is accessible from within MISTRAL through pftp program. When you issue pftp command in the shell, you are connected to the HPSS server and you are ready to transfer the data. Before you enter to HPSS server, just change to the place where your files to be stored are located and issue the pftp command. Then you can use the put command to upload the individual files.

However, with the scripts we have created a directory structure that contain all the files that we want to upload. In the previous examples, they were all stored in /scratch/x/x000000/foobar/experiments/. With this structure we do not need to upload file by file, but use the recursive mput command to upload recursively the directory. If we start at /scratch/x/x000000/foobar, then we need to do the following 

x000000@mistral.mlogin104:/scratch/x/x000000/foobar$ pftp      # Access HPSS server
Using /sw/rhel6-x64/hpss/pftp-7.5.1.2-krb5/etc/HPSS.conf
Connected to lhp.
220-
#####################################
# HPSS production system at DKRZ
#####################################
# HPSS 7.5 Parallel FTP Daemon on lhp
# coming from mistral.dkrz.de
#####################################

220 lhp FTP server (HPSS 7.5.1 Parallel FTP Daemon) ready.
334 Using authentication type GSSAPI; ADAT must follow
GSSAPI accepted as authentication type
GSSAPI authentication failed
Name (tape.dkrz.de:x000000): 
331 Password required for x000000.
Password:****************                                      # Give your password
230 User x000000 logged in as x000000@PHPSS.DKRZ.DE
Remote system type is UNIX.
215 Parallel protocol is PDATA_ONLY
215 SAN3P support disabled
ftp> cd yy0000/x000000                                         # Change to your directory
250 CWD command successful.
ftp> mkdir mpiesm-1.2.01p1-fixvar                              # Create your base directory
257 MKD command successful.
ftp> cd mpiesm-1.2.01p1-fixvar                                 # Change to the base directory
250 CWD command successful.
ftp> pwd                                                       # Check where you are
257 "/hpss/arch/yy0000/x000000/mpiesm-1.2.01p1-fixvar" is current directory.
ftp>

With the former lines, we have created the main directory where the experiments will be uploaded. Then we enter the recursive mode and upload recursively the whole experiments directory. 

ftp> !pwd                                                      # Check where you are in MISTRAL
/scratch/x/x000000/foobar
ftp> prompt                                                    # Deactivate interactive mode
Interactive mode off.
ftp> recursive mput experiments                                # Recursively put the experiments subdirectory of /scratch/x/x000000/foobar
                                                               # in /hpss/arch/yy0000/x000000/mpiesm-1.2.01p1-fixvar/experiments

You should wait a considerable amount of time for the transfer to end (might be useful to use a screen to do this). Operation ended. Enjoy!

NB: The command with the ! is executed in MISTRAL.