parsl.executors.WorkQueueExecutor

class parsl.executors.WorkQueueExecutor(label='WorkQueueExecutor', working_dir='.', managed=True, project_name=None, project_password=None, project_password_file=None, port=0, env=None, shared_fs=False, source=False, init_command='', full_debug=True, see_worker_output=False)

Executor to use Work Queue batch system

The WorkQueueExecutor system utilizes the Work Queue framework to efficiently delegate Parsl apps to remote machines in clusters and grids using a fault-tolerant system. Users can run the work_queue_worker program on remote machines to connect to the WorkQueueExecutor, and Parsl apps will then be sent out to these machines for execution and retrieval.

label: str

A human readable label for the executor, unique with respect to other Work Queue master programs

working_dir: str

Location for Parsl to perform app delegation to the Work Queue system

managed: bool

If this executor is managed by the DFK or externally handled

project_name: str

Work Queue process name

project_password: str

Optional password for the Work Queue project

project_password_file: str

Optional password file for the work queue project

port: int

TCP port on Parsl submission machine for Work Queue workers to connect to. Workers will specify this port number when trying to connect to Parsl

env: dict{str}

Dictionary that contains the environmental variables that need to be set on the Work Queue worker machine

shared_fs: bool

Define if working in a shared file system or not. If Parsl and the Work Queue workers are on a shared file system, Work Queue does not need to transfer and rename files for execution

source: bool

Choose whether to transfer parsl app information as source code. (Note: this increases throughput for @python_apps, but the implementation does not include functionality for @bash_apps, and thus source=False must be used for programs utilizing @bash_apps.)

init_command: str

Command to run before constructed Work Queue command

see_worker_output: bool

Prints worker standard output if true

__init__(label='WorkQueueExecutor', working_dir='.', managed=True, project_name=None, project_password=None, project_password_file=None, port=0, env=None, shared_fs=False, source=False, init_command='', full_debug=True, see_worker_output=False)

Initialize self. See help(type(self)) for accurate signature.

Methods

__init__([label, working_dir, managed, …])	Initialize self.
create_name_tuple(parsl_file_obj, in_or_out)	Returns a tuple containing information about an input or output file to a Parsl app.
create_new_name(file_name)	Returns a unique file name for an input file name.
run_dir([value])	Path to the run directory.
scale_in(count)	Scale in method.
scale_out(*args, **kwargs)	Scale out method.
scaling_enabled()	Specify if scaling is enabled.
shutdown(*args, **kwargs)	Shutdown the executor.
start()	Create submit process and collector thread to create, send, and retrieve Parsl tasks within the Work Queue system.
submit(func, *args, **kwargs)	Processes the Parsl app by its arguments and submits the function information to the task queue, to be executed using the Work Queue system.

Attributes

hub_address	Address to the Hub for monitoring.
hub_port	Port to the Hub for monitoring.