Skip to main content

Singularity Practice

Singularity Practice (x86)​

Creating a Docker image with a Miniconda environment and converting it to a Singularity image for use in a high-performance computing (HPC) environment is a practical solution, especially for scenarios that need to ensure consistency and reproducibility of scientific research environments. Here are the detailed steps:

Step 1: Create and Configure a Docker Image​

1.1 Start a Basic Ubuntu Docker Container

docker run -it --name my_conda_env ubuntu:latest /bin/bash

1.2 Install Miniconda

Inside the container, download and install Miniconda. Note that the Miniconda version may be updated, please visit the Miniconda official website for the latest installation link.

apt-get update && apt-get install -y wget
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
bash Miniconda3-latest-Linux-x86_64.sh -b
export PATH=/root/miniconda3/bin:$PATH

1.3 Create and Configure a Conda Environment

Create a new Conda environment and install the packages you need. For example, creating an environment named myenv and installing numpy:

conda create -n myenv numpy -y

Activate the environment and perform any additional configuration:

conda activate myenv
# Perform any additional environment configuration here

1.4 Exit and Save the Docker Image

After configuration is complete, exit the container:

exit

Create a new Docker image from your container:

docker commit my_conda_env my_conda_env_image:latest

1.5 Export the Docker Image

docker save -o my_conda_env_image.tar my_conda_env_image:latest

Step 2: Convert the Docker Image to a Singularity Image​

Make sure Singularity is installed on your system, then execute the following command:

singularity build my_conda_env_image.sif docker-archive://my_conda_env_image.tar
singularity build singularity_llama_factory_test.sif docker-archive://singularity_llama_factory.tar

Step 3: Use the Singularity Image in an HPC Environment​

3.1 Run the Singularity Container

In the HPC environment, load the Singularity module (if needed) and run your Singularity image:

module load singularity
singularity shell my_conda_env_image.sif

3.2 Activate the Conda Environment

Inside the Singularity container, you need to reinitialize the Conda environment. This is because Conda's initialization script does not run by default in non-interactive shells.

. /miniconda3/etc/profile.d/conda.sh
conda activate myenv

3.3 Run Your Model

In the activated Conda environment, you can now run your large model or any other required commands.

Notes​

  • When creating the Docker image, make sure to include all necessary dependencies and configurations to ensure your environment can run seamlessly on the HPC system.
  • Due to the special nature of HPC environments, specific optimizations may be needed, such as using specific libraries or adjusting environment settings to meet high-performance computing requirements.
  • Follow your institution's regulations and best practices regarding the use of containers and HPC resources.

This solution will help you create a Docker image with a Miniconda environment, convert it to a Singularity image, and use that environment in the HPC environment to run large models.

M2 Chip Singularity Practice​

When using a Mac with an M2 chip (based on ARM architecture), if you want to pull an x86_64 architecture (sometimes also called amd64 architecture) Ubuntu image through Docker, you can use the --platform option to specify the target platform. This feature allows you to pull images that differ from your physical hardware architecture. Make sure your Docker version supports this option.

Here is the command to pull an x86_64 architecture Ubuntu image:

docker pull --platform linux/amd64 ubuntu:latest

This command tells Docker to attempt to pull the ubuntu:latest image for the x86_64 architecture, even though you are using an ARM-based Mac.

Notes

  • Docker Version: Make sure your Docker Desktop version is up to date to support the --platform parameter.
  • Performance and Compatibility: Running x86_64 architecture containers on ARM architecture machines may be achieved through emulation, which may affect performance. Additionally, not all software can run perfectly in an emulated environment, and compatibility issues may arise.
  • Use Case: This method is particularly suitable for situations where you need to ensure that the development and testing environment is consistent with the production environment (which may be x86_64 architecture servers).

By using the --platform option, you can use Docker more flexibly on a Mac with an M2 chip, including developing and testing applications for different hardware architectures.

Check Ubuntu Architecture

arch

Other best practices are the same as for x86.