Transferring Files Between SciServer and MPCDF Viper/Raven

This guide explains how to transfer files between SciServer and the Viper/Raven computing clusters. These are external systems not directly connected to our platform. Access is not automatic and requires an active MPCDF account.

Important Notice & Disclaimer

  • Viper vs Raven: Although we focus on Viper in these instructions, accessing Raven follows a very similar path. Consult the official MPCDF documentation for the most up to date instructions.

  • Official Documentation: This guide is intended as a getting started reference. Always consult the official Viper User Guide.

  • Support Limitations: Please exercise caution when running commands. If you encounter issues, please conduct your own research or consult the official Viper documentation before contacting support.

  • Access Control: Viper is an external system not directly connected to our platform. Access is not automatic. If you do not have Viper access, you must contact your MPE Project Lead and ask about it. This process is outside the control of sciserver.

  • Slurm job submission: Viper uses Slurm for job scheduling, which has a learning curve for running Python scripts. Examples are available in MPCDF Viper’s official documentation. Slurm being one of the most widely used workload manager for batch job scheduling means that you can also find plenty of online tutorials.

Viper Filesystem Overview

Before proceeding, please understand the Viper storage structure:

  • Home Directory (/u/<mpcdf_username>): Permanent storage (GPFS). Use this for source files, config files, and final results.

    • Example: If your MPCDF username is jgil, your home is /u/jgil/.

  • Temporary Storage (/ptmp/<mpcdf_username>): High-performance batch I/O (12 PB). No system backups. Files not accessed for more than 12 weeks are automatically removed.

    • Example: Your temp folder is /ptmp/jgil/.

Prerequisites

  • An active SciServer account with a running container.

  • Viper access granted by your project lead. If you don’t have Viper access yet, contact your project lead before proceeding.

1. Persist your SSH configuration

Container pods are ephemeral - if your container restarts, anything stored outside of your mounted workspace will be lost. To avoid recreating SSH keys every time, store the .ssh folder in your persistent storage and symlink it.

# create the ssh folder in your persistent storage
mkdir -p /home/idies/workspace/Storage/$SCISERVER_USER_NAME/persistent/ssh

# if you already have a ~/.ssh folder, copy it first
cp -r ~/.ssh/* /home/idies/workspace/Storage/$SCISERVER_USER_NAME/persistent/ssh/

# remove the ephemeral one and create a symlink
rm -rf ~/.ssh
ln -s /home/idies/workspace/Storage/$SCISERVER_USER_NAME/persistent/ssh ~/.ssh

# verify
ls -la ~ | grep .ssh
# should show: .ssh -> /home/idies/workspace/Storage/$SCISERVER_USER_NAME/persistent/ssh

2. Generate SSH keys

If you don’t have SSH keys yet:

# check first
ls ~/.ssh/
# if id_ed25519 and id_ed25519.pub are already there, skip this step

# generate a new key pair
ssh-keygen -t ed25519 -f ~/.ssh/id_ed25519 -N ""

3. Configure SSH ControlMaster

MPCDF network requires 2FA every time you establish a connection. This can become annoying when doing trials. Using ssh’s ControlMaster allows you to authenticate to Viper once (including 2FA) and reuse that session for all subsequent connections without re-authenticating.

Create ~/.ssh/config:

cat > ~/.ssh/config << EOF
Host viper
    HostName viper.mpcdf.mpg.de
    User <MPCDF_USERNAME>
    ControlMaster auto
    ControlPath ~/.ssh/control-%r@%h:%p
    ControlPersist yes
EOF

chmod 600 ~/.ssh/config

4. Establish the ControlMaster session

Open a terminal in your SciServer container and connect to Viper. This is the only time you will need to enter your credentials and 2FA:

ssh viper
# enter your Viper password and 2FA when prompted
# keep this terminal open

Verify the control socket is active in a second terminal:

ssh -O check viper
# should return: Master running (pid=XXXX)

5. Transfer files

With the ControlMaster session active, you can now transfer files without re-authenticating.

Copy a single file to Viper (Temporary Scratch):

scp ~/workspace/Storage/$SCISERVER_USER_NAME/persistent/myfile.csv viper:/ptmp/<MPCDF_USERNAME>/

Copy a folder to Viper (Temporary Scratch):

rsync -av ~/workspace/Storage/$SCISERVER_USER_NAME/persistent/myfolder/ viper:/ptmp/<MPCDF_USERNAME>/myfolder/

Copy results back from Viper (Permanent Home):

rsync -av viper:/u/<MPCDF_USERNAME>/results/ ~/workspace/Storage/$SCISERVER_USER_NAME/persistent/results/

Tips:

  • Prefer rsync over scp for large or many files - it is resumable and skips already-transferred files.

  • Viper scratch (/ptmp) is temporary. Copy your results back to SciServer or Viper Home (/u) when your job is done.

  • Plan your data transfers carefully. Copying hundreds of GB is possible but time consuming - consider working with subsets of your data where possible.