CERM Cluster User's guide

The default shell is the bash shell. To change it use the chsh command. At login /etc/motd file is displayed: please take care of reading it, because information about system are usally written there.

A basic default environment is already set up by means of system login configuration files, this includes variables and paths for the all the compilers and their MPI wrappers of the MPI standard, and OpenPBS/Torque batch queuing system with the MAUI scheduler.

Check your environment with the env command. You should be careful modifying the shell customization files (.cshrc .profile .login .bashrc), since they could overwrite the default values altering the behaviour of the compilers and of the batch queuing system.

The Portable Batch System, PBS, is a workload management system for Linux clusters. It supplies command to submit, monitor, and delete jobs. It has the following components.

Below are the steps needed to run production code:

Below are some of the commonly used PBS options in a job script file. The options start with "#PBS".

Note

(*) File staging can specify which files should be copied onto the execution host before the job starts and which files should be copied off the execution host when it completes. The file_list regardless of the direction of copy, is of the following form, where the name local_file is the name of the file on the system where the job executes, and the remote_file is the destination name on the host specified by hostname: local_file@hostname:remote_file. stagein=my.input@frontend-0:/home/login_name/my.input stageout=my.output@frontend-0:/home/login_name/my.output

There are some quite useful Maui commands:

There are a number of predefined environment variables. These include the following:

The following environment variables relate to the submission machine:

The following variables relate to the environment where the job is executing:

The following job script template should be modified for the need of the job.

A job script may consist of PBS directives, comments and executable statements. A PBS directive provides a way of specifying job attributes in addition to the command line options. For example:

#PBS -N Job_name
#PBS -l walltime=10:30,mem=320kb
#
step1 arg1 arg2
step2 arg3 arg4

Use the qsub command to submit the job script (in this example the name of the job script is run).

$ qsub run

PBS assigns a job a unique job identifier once it is submitted (e.g. 123.athlon). After a job has been queued, it is selected for execution based on the time it has been in the queue, wall-clock time limit, and number of processors.

Below are commands for monitoring a job:

Suppose for instance you want to run your a test.x interactively on four processors then you could use the following sequence of commands:

$ qsub -l nodes=2:ppn=2,walltime=0:30:00 -I 

at this point (if there are free resources) you will enter in the batch interactive session, and you could run your test with:

$ lamboot -v $PBS_NODEFILE
$ cd testdir 

$ mpirun -n 4 -no-shmem test.x
$ mpirun -np 4 

Example of an interactive execution:

$ qsub -l nodes=2:ppn=2,walltime=0:30:00 -I
$ cd testdir 

$ mpirun -n 4 test.x 

The following job script template should be modified for the need of the job.

#!/bin/bash -f
#PBS -l nodes=2:ppn=2
#PBS -k oe
LAMSTART="lamboot $PBS_NODEFILE"
LAMSTOP="lamhalt $PBS_NODEFILE"
HOME="/home_n01/guest"
LAUNCH="mpirun -np 4 cpmd.x"
WORKDIR="${HOME}/cp_test"
export PP_LIBRARY_PATH=${WORKDIR}
cd ${WORKDIR}
${LAMSTART}
${LAUNCH} au_surf_job1.in > au_surf_job1.out
${LAMSTOP}
#
exit

The following job scripts should be used for GROMACS parallel calculations. The first one is for pre-minimization and the second one is to launch the dinamic calculation.

#!/bin/bash -f
#PBS -k oe
#PBS -m n
PBS_O_WORKDIR="/home_n11/hetdyn/GROMACS/SPI_1ns_cluster"
lamboot
LAUNCH="./SPI_MINI.csh"
cd ${PBS_O_WORKDIR}
${LAUNCH}
exit

#!/bin/bash -f
#PBS -k oe
#PBS -m n
PBS_O_WORKDIR="/home_n11/hetdyn/GROMACS/SPI_1ns_cluster"
lamboot
LAUNCH="./SPI_MD_5PR_1ns.csh"
cd ${PBS_O_WORKDIR}
${LAUNCH}
exit