How to build and run computation on OCCAM

The very first step

Obtaining an account

First of all, you need an account on our facility: you can ask for an account through the contact form located in this site. Are admmitted only the Researchers from the departments/centers/organizations/firms who subscribed to C3S; students are not allowed. Every Researcher should use the institutional email account. After contacting C3S staff through the web form, you will be enabled to access the gitlab service, which is the main interface to the facility. We will refer to your username in the scope of these instructions as YOURUSERNAME.

Store your ssh keys

The only allowed method to access OCCAM computation resources is via ssh (NO access via login/password).

In order to be able to connect via ssh on the access node (ssh://occam.c3s.unito.it), and for all the OCCAM machinery to know you, you are required to upload an ssh key to you user profile. Please go to https://gitlab.c3s.unito.it/ then from the personal menu, click on "Edit Profile" and then on "SSH Keys" on the lateral menubar. There you will find detailed instructions, or ask to google if you are in trouble with your personal setup: there is a plenty of howto on ssh credential creation.

Software Containers

OCCAM works differently from most HPC facilities: application software is autonomously packaged by users in containers, which are then run as isolated Virtual Workstations or Virtual Clusters.

The OCCAM user interface is based on GitLab; we encourage use of the OCCAM GitLab repository for your code, but you need only to use the image registry if you don't want to do so.

OCCAM uses docker containers and gitlab uses git commands: all that may help you to better know OCCAM but you can use OCCAM resources without any docker and/or git tool installed on your computer.

Insights

In OCCAM, all user software is run in containers. Even if this is not technically completely correct, you can think of a container as a bag of software that holds the full software stack you need for your computation: OS, libraries, executables, files, except the Linux kernel that is shared with the physical host on which the container runs. Of course it is possible to pass configuration files and command line arguments to executables inside the container at runtime.

There are several implementations of software containers; in OCCAM we currently use Docker containers, an industry-standard tool to build, manage and run containers on Linux. You can find the full documentation at the Docker site , e.g. here: https://docs.docker.com/learn/, but you will need only very basic knowledge of it (pointers to relevant documentation will be provided wherever needed).

Jargon alert: what you will build and upload is not a container but an image; a container is a running instance of an image, more or less like a process is a running program.

Please note, this is not a full Docker tutorial but only a step-by-step example on how to build, upload and run a container on OCCAM. Please refer to the Docker official documentation for more information.

In general, in order to build and upload an image, you would install Docker in your machine. Please download and install from https://store.docker.com/search?offering=community&type=edition; yes, you can build and run Linux containers on MacOS, Windows or Linux (any), and they will run on OCCAM.

Dockerfile

The Dockerfile is the "core" of a container building.

According to official Docker documentation, a Dockerfile is: "a text document that contains all the commands a user could call on the command line to assemble an image. Using docker build users can create an automated build that executes several command-line instructions in succession."

That means all the information and commands related to the container creation have to be inserted in the Dockerfile. (a.e. OS, libraries, folders, files, etc. )

(Take a look to the official references at https://docs.docker.com/engine/reference/builder/ )

How to run computation on OCCAM

For eager users: please familiarize yourself with containers! To do so, please take a look at a "ready to use" matlab container:

a.e. https://gitlab.c3s.unito.it/matlab/r2021a

The use is self-explained and showed in the Readme.MD file in the page.

OCCAM is able to run container with pre-builded images, pay attention to the simple Dockerfile in another ready-to-use image at Occam, the CERN's ROOT application:

https://gitlab.c3s.unito.it/cern/root

Se also the R-Project page:

https://gitlab.c3s.unito.it/r-project/

Remarks: in order to use computation resources, you will always need to access to occam logging in with your ssh-keys and your ssh client.

Plese refer to the Appendix for an example project that can be user as a starting point for building your own project .

Please note that directly running a container is currently supported only for single-node, single-step computations. Multi-node MPI applications and pipelines need some more massaging.

OCCAM workflow

The basic workflow is the following (in parenthesis the Docker command used):

You create a gitlab project that will store all artifacts needed for running your computation on OCCAM;
You create (build) and test the image on your machine by writing a Dockerfile and using standard Docker commands;
You upload (push) the image (possibly also during the build) on the OCCAM image registry located in your gitlab project (only images from our internal registry can be run on OCCAM);
You login (login) to the OCCAM access node and copy all input files somewhere in your home directory;
You use OCCAM special commands to run your container in the cluster.

In the following, you will be guided through these steps to build and run the example image.

Create a project on gitlab

A project on gitlab is needed to store all the artifacts, files, Dockerfile, docker images that will be required to run your computation environment. It's simple and straightforward:

using a web browser, Sign in to https://gitlab.c3s.unito.it with the web credentials obtained from our staff;
click on the New Project green button;
write down a name for your project in the Project name field; we'll refer to this name in the scope of these instructions as PNAME
for the first project, you can now got to the end of the page and click on the Create Project green button;
all done.

Building an image

Building an image means picking a base image and adding all the pieces and configurations needed to run your applications, much like you would do on a new workstation. The base image typically contains just the OS, but there exist images with preinstalled software (for example, R). You can find official Docker images you can use as base images at the Docker Store https://store.docker.com/search?q=&source=verified&type=image or from nVidia project https://github.com/NVIDIA/nvidia-docker/wiki if you need to use our GPUs.

On OCCAM, you can build the image, you would just enter in the project folder you created with the Dockerfile, then launch the OCCAM command to build the project:

occam-build -t gitlab.c3s.unito.it/<YOURUSERNAME>/<PNAME> .

or, simply:

occam-build -t <YOURUSERNAME>/<PNAME> .

In some cases (a.e. with some internet connection), you might need to indicate the port 5000 in the command:

occam-build -t gitlab.c3s.unito.it:5000/<YOURUSERNAME>/<PNAME> .

Please note the last dot '.' in the build command is mandatory !!

The occam-build command will also push on the OCCAM image registry located in your gitlab project the image described by the commands in the Dockerfile.

Insights: standalone tests

If you want to learn a little bit more on Docker on your computer please refer to the Appendix at the link below. Please note the choice of a Linux distribution is not important, we suggest to use CentOS 7. This is the OS we choose in the Appendix: Mandelbrot Example

Access to OCCAM computational resources

To use OCCAM computational resources you need to login to the access node occam.c3s.unito.it with your ssk keys and your ssh client. There you'll run an image on the cluster. If everything is set up correctly, your ssh key has already been propagated to your home directory on the system, and you should be able to log in without a password:

ssh <username>@occam.c3s.unito.it

The only way to login into occam access node is through the use of the ssh-keys uploaded to gitlab web interface; login/password access to ssh is disabled

If the logon went ok, then you will see something like this:

Last login: Tue May 16 11:18:37 CEST 2017 on pts/1
Setup (once) your occam private registry
Please login to the Occam Registry with the occam-login command before doing computations.

Running on OCCAM

Following the suggestion, start using our first occam- command

occam-login

The command output will be as follow:

When prompted for a username/login, please insert your gitlab credentials

Info: creating repo: /archive/home/YOURUSERNAME/.udocker
Info: installing from tarball 1.0.1
username: *enter YOUR USERNAME ON gitlab.c3s.unito.it*
password: *enter YOUR PASSWORD ON gitlab.c3s.unito.it*

this needs to be done only once, since it will store a token in your home dir.

WARNING: The gitlab web interface accept as username either the effective username or the email, but in docker login you should give in input your username, not your email!

If everythis went ok, create a directory to work in:

mkdir docker-example
cd docker-example

Please note that for security reasons you are not allowed to directly run Docker on the system; however, some extra machinery under the hood will do so for you.

Now you can use the occam-run command to run your container:

occam-run <YOURUSERNAME>/<PNAME>

You will see some extra output generated by the machinery that re-builds the container in a special format and run it on one of the nodes. If you don't specify anything, like in the example above, it will use one of the "light" nodes that was set apart for testing.

You are logged-in to: Occam Registry
Current Directory is: /archive/home/YOURUSERNAME/docker-example
Container Image is: YOURUSERNAME/PNAME
Finished building personalized image for: YOURUSERNAME
Other args: 
LogFile will be: /archive/home/YOURUSERNAME/docker-example/node22-152708.log
ErrFile will be: /archive/home/YOURUSERNAME/docker-example/node22-152708.err
When job will finish, the file  /archive/home/srabelli/node22-152708.done will be created
Remote ID for occam-kill command parameter: node22-152708
Running detached from terminal, bye.

The Remote ID (in this case node22-152708) is the unique handle you can use to interact with the running container. Any output to stderr and stdout will go, respectively, to <id>.log and <id>.err files. When the job will finish, an additional file <id>.done will be created along the other two files to signal that all works are done.

As soon as the container starts on the node, the occam-run command exits: "Running detached from terminal, bye." here means "done dispatching the container", not "done running". Of course in this example the container runs only for a very short time, but you can check whether it is still running or killing it with

occam-inspect
occam-kill node22-152708

At the end of the execution (which in this example means almost immediately) you will again find a mandelbrot.png file in the current directory. Of course, you can pass arguments and options to your container through the occam-run command line:

occam-run <YOURUSERNAME>/PNAME 10 10. 1001 1001

That's all folks!

Now if you are confident on your understanding, go to OCCAM command reference and discover all the OCCAM commands parameters/functions.

What about the Credit System aka C3SCoins?

Now that you are confindent with Docker, images, containers and projects, you would to use some nodes of OCCAM. For doing your computation, you need to book one (or more) node(s) through the Resource Booking System that is available on this site, look at the main menu.

A booking reservation has an associated cost, automatically calculated by the system, so you need to get a C3SCoin credit to finalize a reservation. The credit exchange capabilities embedded into the Booking System allows credits exchange to/from everyone, but usually you should ask your credits to your departmental coordinator. A list of coordinators can be found at the end of the Credit System howto.

So, first of all surf into the Resource Booking System, then contact you coordinator and ask to them some credits. If your coordinator accept your request, a credit exchange will follow; with these credits you will book the nodes you need autonomously, choosing node type, time slots and duration of the reservation.

Appendix

Mandelbrot Example

An example project is provided on OCCAM's GitLab instance that can be used as a starting point for building your own project. Please find the project at https://gitlab.c3s.unito.it/bagnasco/docker-example

Please note that directly running a container is currently supported only for single-node, single-step computations. Multi-node MPI applications and pipelines need some more massaging.

First, you will need to copy all the relevant files to a directory in your machine, then write a Dockerfile to tell Docker how to build the container. In this example, the Dockerfile is provided and there is only one file, the Python script that computes the Mandelbrot set, with the rather unimaginative name of mandelbrot.py.

If you're familiar with git, you can get both the files (script and Dockerfile) by cloning this project's repository:

git clone git(at)gitlab.c3s.unito.it:bagnasco/docker-example.git cd docker-example

Otherwise, you can simply download them with wget or curl, or even with a browser, for example

mkdir docker-example cd docker-example (Linux/MacOS) wget https://gitlab.c3s.unito.it/bagnasco/docker-example/raw/master/Dockerfile (Linux/MacOS) wget https://gitlab.c3s.unito.it/bagnasco/docker-example/raw/master/mandelbrot.py (Windows) use a browser and point to https://gitlab.c3s.unito.it/bagnasco/docker-example/raw/master/Dockerfile and save the page as Dockerfile into the docker-example directory just created (Windows) use a browser and point to https://gitlab.c3s.unito.it/bagnasco/docker-example/raw/master/mandelbrot.py and save the page as mandelbrot.py into the docker-example directory just created

Take your time to peruse the Dockerfile before building; the full docker commands reference is at https://docs.docker.com/engine/reference/builder/.

Now you can build the image with

docker build -t gitlab.c3s.unito.it:5000/<YOURUSERNAME>/<PNAME> .

The last dot in the build command is mandatory !! You will see that Docker downloads the base image, then step-by-step does all the operations described in the Dockerfile. <username>/mandelbrot is the name you give to the image. Please replace \<YOURUSERNAME> with your OCCAM username and PNAME with your gitlab project name. Docker caches everything that can be cached, so if you change something in the Dockerfile and build again it will not, for example, download again the base image.

At this point, you can use the command docker images to see the newly-built image (as well as all other previously built images, if any), and docker run to test the container:

docker images

The output should be something like the following lines, where YOURUSERNAME and PNAME are replaced with the values used in the build command:

REPOSITORY TAG IMAGE ID CREATED SIZE gitlab.c3s.unito.it:5000/<YOURUSERNAME>/<PNAME> latest 43d6299dbca8 2 weeks ago 240 MB

Now let's try to run it on your desktop/laptop to see docker in action:

(Linux/MacOS) docker run -ti --rm --volume $PWD:/data --workdir /data gitlab.c3s.unito.it:5000/<YOURUSERNAME>/<PNAME> (Windows) docker run -ti --rm --volume %cd%:/data --workdir /data gitlab.c3s.unito.it:5000/<YOURUSERNAME>/<PNAME>

In our convention, the software must read and write to the current directory from which the container is launched. This is exposed inside the container by the --volume $PWD:$PWD --workdir $PWD options to the command. You can find the full docker run reference at https://docs.docker.com/engine/reference/run/.

The output should be something like

Computing... Drawing... Done.

and you should find a new file mandelbrot.png in the current directory. You can pass arguments to the script, that will override the defaults defined in the Dockerfile, by adding them to the command line:

(Linux/MacOS) docker run -ti --rm --volume $PWD:/data --workdir /data <YOURUSERNAME>/<PNAME> 10 10. 101 301 (Windows) docker run -ti --rm --volume %cd%:/data --workdir /data <YOURUSERNAME>/<PNAME> 10 10. 101 301

Uploading the image to occam registry

An image registry is a store where the images can be pushed/pulled when needed; every gitlab project own a private registry useful for storing our project images and occam is able to pull from there you "bag of tools" for doing computation. So, in order to run on OCCAM, you need to push the image to the image registry.

First you will need to login to the GitLab repository, using a PERSONAL ACCESS TOKEN created from your profile in https://gitlab.c3s.unito.it/-/user_settings/personal_access_tokens :

echo "<TOKEN>" | docker login gitlab.c3s.unito.it:5000 -u <YOURUSERNAME> --password-stdin

You should see something like:

WARNING! Your password will be stored unencrypted in /home/YOURUSERNAME/.docker/config.json ... Login Succeeded

Ignore the warning or configure a credential helper, searching how to do it on the internet...

Then you can push it with a name including the registry name and push it:

docker push gitlab.c3s.unito.it:5000/<YOURUSERNAME>/<PNAME>

If everything went well, you can go to the registry page at https://gitlab.c3s.unito.it/\<YOURUSERNAME>/\<PNAME>/container_registry and you will see your image entry. It will be named "latest": you can add version tags to your containers like <YOURUSERNAME>/<PNAME>:v1, but the "latest" tag will always be an alias for the, well, latest version built.

Please refer to the other HowTo sections to further information.

(last revision: august 2022 by Thomas Cocco)