Compute Containers

TL;DR – The containers you launch are meant for interactive development, not for heavy‑weight HPC jobs. They run on dedicated Kubernetes nodes (20 CPU / 250 GiB RAM) but each container is limited to 8 CPU, 50 GiB RAM and a modest storage quota.

What a “Compute Container” is (in plain language)

  • It is a JupyterLab notebook server that starts in a few seconds.

  • The container gives you a ready‑to‑use Python environment (the Sciserver images) and direct access to the platform’s shared data volumes.

  • Think of it as a personal, cloud‑based workstation – convenient for prototyping, data exploration, and light analyses.

  • Containers are ephemeral by nature. Make sure to get comfortable using your personal storage and python environments to make the most out of containers.

Resource limits you should know

Resource

What the container gets

CPU

8 vCPU (hard limit)

Memory

50 GiB (hard limit)

Storage

≈4 GiB per container (size‑limit)

You do not need to configure any of this – it is applied automatically when you create “New Container”.

Parallelization

Important: Heavy CPU/GPU use or long‑running batch tasks should run in Compute Jobs or, preferably, on the MPCDF High‑Performance Computing (HPC) cluster, which has vastly more resources.

Parallelization in Python can speed up processing by using multiple cores, but in SciServer all containers share finite CPU and memory with other users. Intensive parallel work may slow down colleagues’ jobs. Moreover, you have access to exponentially more CPUs and Memory using available alternatives.

If your workflow already needs parallelization, you’re likely ready to move to HPC. Contact us and we’ll guide you on integrating your data and code into the cluster.

Why the storage limit matters

The container’s root filesystem is a temporary layer that disappears when the session ends. If you install a lot of Python packages (e.g., via mamba install …) the cache can fill the ~4 GiB quota and the installation will fail with “No space left on device”.

Best practice: keep the package cache outside the container, on your personal persistent storage.

Quick checklist for new users

  1. Launch a Compute container from the portal (choose the image you need).

  2. Set the cache location (run the three commands above).

  3. Install additional packages with mamba install ….

  4. Work in JupyterLab as usual – your notebooks and data are saved in the mounted /workspace/Storage/<username>/persistent/ directory, so they survive container restarts.

  5. When you’re done, simply close the session; the container is terminated but your files and the mamba cache remain.

Managing Python Environments

By default, software installed inside a container disappears when the container stops. To keep your Python packages and libraries, you must store them in your Persistent User Storage.

We recommend two workflows depending on your needs:

  1. Persistent Environments (Recommended): Store the actual environment folder on your storage. This is fastest for daily work.

  2. Reproducible Environments: Export an environment.yml file. This is best for sharing code or ensuring exact version matches across projects.

Where to get help

Upstream docs: https://www.sciserver.org/about/compute/

For help, see the Support & Contact page.