Compute Containers

TL;DR – The containers you launch are meant for interactive development, not for heavy‑weight HPC jobs. They run on dedicated Kubernetes nodes (20 CPU / 250 GiB RAM) but each container is limited to 8 CPU, 50 GiB RAM and a modest storage quota.

What a “Compute Container” is (in plain language)

It is a JupyterLab notebook server that starts in a few seconds.
The container gives you a ready‑to‑use Python environment (the Sciserver images) and direct access to the platform’s shared data volumes.
Think of it as a personal, cloud‑based workstation – convenient for prototyping, data exploration, and light analyses.

Resource limits you should know

Resource	What the container gets
CPU	8 vCPU (hard limit)
Memory	50 GiB (hard limit)
Storage	≈4 GiB per container (size‑limit)

You do not need to configure any of this – it is applied automatically when you create “New Container”.

Parallelization

Important: Heavy CPU/GPU use or long‑running batch tasks should run in Compute Jobs or, preferably, on the MPCDF High‑Performance Computing (HPC) cluster, which has vastly more resources.

Parallelization in Python can speed up processing by using multiple cores, but in SciServer all containers share finite CPU and memory with other users. Intensive parallel work may slow down colleagues’ jobs.

If your workflow already needs parallelization, you’re likely ready to move to HPC. Contact us and we’ll guide you on integrating your data and code into the cluster.

Why the storage limit matters

The container’s root filesystem is a temporary layer that disappears when the session ends. If you install a lot of Python packages (e.g., via mamba install …) the cache can fill the ~4 GiB quota and the installation will fail with “No space left on device”.

Best practice: keep the package cache outside the container, on your personal persistent storage.

Move the mamba cache to your personal storage (Optional)

Your personal storage is automatically mounted at:

~/workspace/Storage/<username>/persistent/
~/workspace/Temporary/<username>/scratch/

Create a directory for the cache on either of those and tell mamba to use it:

# 1. Create a cache folder (run once)
mkdir -p ~/workspace/Storage/<username>/persistent/mamba_cache

# 2. Export the variable for the current session (add to ~/.bashrc if you like)
export MAMBA_ROOT_PREFIX=~/workspace/Storage/<username>/persistent/mamba_cache

# 3. Verify
mamba info | grep 'pkgs dir'

From now on every mamba install … will write packages to the persistent volume, so the container’s own filesystem stays small and you can reuse the cache across containers.

Quick checklist for new users

Launch a Compute container from the portal (choose the image you need).
Set the cache location (run the three commands above).
Install additional packages with mamba install ….
Work in JupyterLab as usual – your notebooks and data are saved in the mounted /workspace/Storage/<username>/persistent/ directory, so they survive container restarts.
When you’re done, simply close the session; the container is terminated but your files and the mamba cache remain.

Working accross containers (Python Dependencies Management)

Note: This is a starting point for working with environments inside SciServer images. It is not a complete or guaranteed solution. For full support, consult the conda environments documentation or contact the Support & Contact team.

Quick Start

If you’ve never saved your Python environment or dependencies, now is the time to start! This ensures your workflows can be recreated on the new infrastructure.

1. Save Your Environment

For conda/mamba users (most common in SciServer): Run this in your container:
```
mamba env export > environment.yml
```
This creates a file listing all installed packages and versions.
For pip users: Run this in your container:
```
pip freeze > requirements.txt
```
This saves a list of pip-installed packages.

⚠️ Note: If you use both conda and pip, the environment.yml file will include pip dependencies automatically.

2. Test Your File

Create a new container (or test in a local environment) and try recreating your setup:
```
mamba env create -f environment.yml
# OR
pip install -r requirements.txt
```
Fix any errors before the migration.

3. What If You’re Not Sure?

Start with mamba env export — it’s the most comprehensive for SciServer workflows.
If you’re missing files or get errors, reach out — I can help you debug.

📘 Learn More

Quick Reference Cheat-Sheet (copy-paste)

Generate an environment file based on your currently active environment:

# Inside a Sciserver compute container
mamba env create -f environment-converted.yaml
mamba activate my-project

It should look something like this:

# environment-converted.yaml
name: my-project
channels:
  - conda-forge          # only this channel is needed
dependencies:
  - python=3.11          # matches the current base image
  - numpy
  - pandas
  - scikit-learn
  - jupyterlab
  - pip:
    - my-private-pkg

Where to get help

Upstream docs: https://www.sciserver.org/about/compute/ For help, see the Support & Contact page.