soprano.hpc.submitter.queues

Definition of QueueInterface class.

Classes

QueueInterface(sub_cmd, list_cmd, kill_cmd, ...)

QueueInterface object

class soprano.hpc.submitter.queues.QueueInterface(sub_cmd, list_cmd, kill_cmd, sub_outre, list_outre, list_user_opt=None)

Bases: object

QueueInterface object

A class meant to simplify interfacing in a basic way with a Queue system. Contains commands to submit to the queue, list the job IDs, and kill them if necessary. Will contain Regexps to parse for IDs and additional information as returned upon submission and listing. It is important that the regular expressions used employ NAMED GROUPS to parse the various fields. In particular, a job_id group must ALWAYS be present. The class also provides some static variables implementing standard interfaces for common queueing systems. These can be retrieved by using QueueInterface.<NAME>. The currently implemented names are the following:

• LSF (IBM’s managing system, using the command bsub)

• GridEngine (Sun’s managing system, also available in an open version,

  using the command qsub)

• PBS (another managing system using qsub)

Initialize the QueueInterface.

Args:

sub_cmd (str): command used to submit a script to the queue

list_cmd (str): command used to list all queued jobs for the user

kill_cmd (str): command used to kill a job given its id

sub_outre (str): regular expression used to parse the output of

sub_cmd. Must contain at least a job_id named

group

list_outre (str): regular expression used to parse the output of

list_cmd. Must contain at least a job_id named

group

list_user_opt (Optional[str]): name of the option for passing a

specific user name when listing

jobs. For example, on LSF this is

-u. By default it’s None, and the

default of the queueing system is

used.

kill(job_id)

Kill the job with the given ID

Args:

job_id (str): ID of the job to kill

list(user='$USER')

List all jobs found in the queue

Returns:

jobs (dict): a dict of jobs classified by ID containing all info

that can be matched through list_outre

user (Optional[str]): user for whom jobs should be listed. Will

not have any effect if list_user_opt has not

been specified. Default is $USER.

property remote_target

RemoteTarget if a remote Host is set

set_remote_host(host=None, timeout=1.0)

Set a remote host for use by this QueueInterface

If a remote host is set, this Interface will try using the Paramiko library to connect to it via SSH. Check the docstring of soprano.hpc.submitter.utils.RemoteTarget to see what the limitations are for this usage.

Args:

host (Optional[str]): host name of the remote machine to connect

to. Further information must be contained in

the user’s ~/.ssh/config file. If left empty

the remote host is UNSET and therefore the

interface goes back to local use.

timeout (Optional[float]): connection timeout in seconds (default

is 1 second)

submit(script, cwd=None)

Submit a job to the queue.x

Args:

script (str): content of the submission script

cwd (Optional[str]): path to the desired working directory

Returns:

job_id (str): the job ID assigned by the queue system and parsed

with sub_outre