9.9. Launching with PBS / Torque

PRRTE supports PBS, PBS Pro, Torque, and other related resource managers.

9.9.1. Verify PBS/Torque support

The prte_info command can be used to determine whether or not an installed Open MPI includes Torque/PBS Pro support:

shell$ prte_info | grep ras

If the PRRTE installation includes support for PBS/Torque, you should see a line similar to that below. Note the MCA version information varies depending on which version of Open MPI is installed.

MCA ras: tm (MCA v2.1.0, API v2.0.0, Component v3.0.0)

9.9.2. Launching

When properly configured, PRRTE obtains both the list of hosts and how many processes to start on each host from Torque / PBS Pro directly. Hence, it is unnecessary to specify the --hostfile, --host, or -n options to prterun. PRRTE will use PBS/Torque-native mechanisms to launch and kill processes (ssh is not required).

For example:

# Allocate a PBS job with 4 nodes
shell$ qsub -I -lnodes=4

# Now run a job on all the nodes allocated by PBS/Torque
shell$ prterun mpi-hello-world

This will run the application processes on the nodes that were allocated by PBS/Torque. Or, if submitting a script:

shell$ cat my_script.sh
#!/bin/sh
prterun mpi-hello-world
shell$ qsub -l nodes=4 my_script.sh

Warning

Do not modify $PBS_NODEFILE!

We’ve had reports from some sites that system administrators modify the $PBS_NODEFILE in each job according to local policies. This will currently cause PRRTE to behave in an unpredictable fashion. As long as no new hosts are added to the hostfile, it usually means that PRRTE will incorrectly map processes to hosts, but in some cases it can cause PRRTE to fail to launch processes altogether.