parsl.executors.WorkQueueExecutor

class parsl.executors.WorkQueueExecutor(label='WorkQueueExecutor', working_dir='.', managed=True, project_name=None, project_password=None, project_password_file=None, port=0, env=None, shared_fs=False, source=False, init_command='', full_debug=True, see_worker_output=False)[source]

Executor to use Work Queue batch system

The WorkQueueExecutor system utilizes the Work Queue framework to efficiently delegate Parsl apps to remote machines in clusters and grids using a fault-tolerant system. Users can run the work_queue_worker program on remote machines to connect to the WorkQueueExecutor, and Parsl apps will then be sent out to these machines for execution and retrieval.

label: str
A human readable label for the executor, unique with respect to other Work Queue master programs
working_dir: str
Location for Parsl to perform app delegation to the Work Queue system
managed: bool
If this executor is managed by the DFK or externally handled
project_name: str
Work Queue process name
project_password: str
Optional password for the Work Queue project
project_password_file: str
Optional password file for the work queue project
port: int
TCP port on Parsl submission machine for Work Queue workers to connect to. Workers will specify this port number when trying to connect to Parsl
env: dict{str}
Dictionary that contains the environmental variables that need to be set on the Work Queue worker machine
shared_fs: bool
Define if working in a shared file system or not. If Parsl and the Work Queue workers are on a shared file system, Work Queue does not need to transfer and rename files for execution
source: bool
Choose whether to transfer parsl app information as source code. (Note: this increases throughput for @python_apps, but the implementation does not include functionality for @bash_apps, and thus source=False must be used for programs utilizing @bash_apps.)
init_command: str
Command to run before constructed Work Queue command
see_worker_output: bool
Prints worker standard output if true
__init__(label='WorkQueueExecutor', working_dir='.', managed=True, project_name=None, project_password=None, project_password_file=None, port=0, env=None, shared_fs=False, source=False, init_command='', full_debug=True, see_worker_output=False)[source]

Initialize self. See help(type(self)) for accurate signature.

Methods

__init__([label, working_dir, managed, …]) Initialize self.
create_name_tuple(parsl_file_obj, in_or_out) Returns a tuple containing information about an input or output file to a Parsl app.
create_new_name(file_name) Returns a unique file name for an input file name.
run_dir([value]) Path to the run directory.
scale_in(count) Scale in method.
scale_out(*args, **kwargs) Scale out method.
scaling_enabled() Specify if scaling is enabled.
set_bad_state_and_fail_all(exception) Allows external error handlers to mark this executor as irrecoverably bad and cause all tasks submitted to it now and in the future to fail.
shutdown(*args, **kwargs) Shutdown the executor.
start() Create submit process and collector thread to create, send, and retrieve Parsl tasks within the Work Queue system.
status() Return the status of all jobs/blocks currently known to this executor.
submit(func, *args, **kwargs) Processes the Parsl app by its arguments and submits the function information to the task queue, to be executed using the Work Queue system.

Attributes

bad_state_is_set Returns true if this executor is in an irrecoverable error state.
executor_exception Returns an exception that indicates why this executor is in an irrecoverable state.
hub_address Address to the Hub for monitoring.
hub_port Port to the Hub for monitoring.
provider
tasks Contains a dictionary mapping task IDs to the corresponding Future objects for all tasks that have been submitted to this executor.