Transferring Files Between SciServer and MPCDF Viper/Raven

This guide explains how to transfer files between SciServer and the Viper/Raven computing clusters. These are external systems that require an active MPCDF account.

Important Notice & Disclaimer

  • Official Documentation: This guide is a getting started reference. Always consult the official Viper User Guide.

  • Access Control: Access to Viper is not automatic. If you do not have access, please contact your MPE Project Lead.

  • Slurm Scheduling: Viper uses Slurm for job scheduling. Refer to the official documentation for batch script examples.

Viper Filesystem Overview

  • Home Directory (/u/<mpcdf_username>): Permanent storage (GPFS). Use this for source files and final results.

  • Temporary Storage (/ptmp/<mpcdf_username>): High-performance batch I/O. No system backups. Files not accessed for 12 weeks are automatically removed.

Prerequisites

Crucial: Persistence Setup Because containers are ephemeral, you must have your SSH configuration persisted to avoid re-generating keys and config files every session.

Before proceeding, ensure you have completed:

  1. The Identity and Access guide (to create and store your .ssh folder persistently).

  2. The Shell Automation guide (to ensure your .ssh folder is automatically linked on boot).

  3. A valid MPCDF account with username, passsword, and 2FA already configured.

1. Configure SSH ControlMaster

MPCDF network requires 2FA every time you establish a connection. To avoid repeating 2FA for every single file transfer, we use ControlMaster to reuse a single authenticated session.

Create (or update) your SSH config file ~/.ssh/config, using your own MPCDF username:

Host viper
    HostName viper.mpcdf.mpg.de
    User <MPCDF_USERNAME>
    ControlMaster auto
    ControlPath ~/.ssh/control-%r@%h:%p
    ControlPersist yes

Make sure the config file has the right permissions:

chmod 600 ~/.ssh/config

2. Establish the ControlMaster session

Open a terminal in your SciServer container and connect to Viper. This is the only time you will need to enter your credentials and 2FA:

ssh viper
# enter your Viper password and 2FA when prompted
# keep this terminal open

Verify the control socket is active in a second terminal:

ssh -O check viper
# should return: Master running (pid=XXXX)

3. Transfer files

With the ControlMaster session active, you can now transfer files without re-authenticating.

Copy a single file to Viper (Temporary Scratch):

scp ~/workspace/Storage/$SCISERVER_USER_NAME/persistent/myfile.csv viper:/ptmp/<MPCDF_USERNAME>/

Copy a folder to Viper (Temporary Scratch):

rsync -av ~/workspace/Storage/$SCISERVER_USER_NAME/persistent/myfolder/ viper:/ptmp/<MPCDF_USERNAME>/myfolder/

Copy results back from Viper (Permanent Home):

rsync -av viper:/u/<MPCDF_USERNAME>/results/ ~/workspace/Storage/$SCISERVER_USER_NAME/persistent/results/

Tips for Data Transfer

  • Use rsync: Prefer rsync over scp for large transfers—it is resumable and more efficient.

  • Storage Warning: Remember that Viper scratch (/ptmp) is temporary. Always copy final results back to SciServer or your Viper Home (/u) directory.