parsl.executors.ThreadPoolExecutor

class parsl.executors.ThreadPoolExecutor(label: str = 'threads', max_threads: int = 2, thread_name_prefix: str = '', storage_access: List[Any] = None, working_dir: Optional[str] = None, managed: bool = True)[source]

A thread-based executor.

Parameters
  • max_threads (int) – Number of threads. Default is 2.

  • thread_name_prefix (string) – Thread name prefix

  • storage_access (list of Staging) – Specifications for accessing data this executor remotely.

  • managed (bool) – If True, parsl will control dynamic scaling of this executor, and be responsible. Otherwise, this is managed by the user.

__init__(label: str = 'threads', max_threads: int = 2, thread_name_prefix: str = '', storage_access: List[Any] = None, working_dir: Optional[str] = None, managed: bool = True)[source]

Methods

__init__([label, max_threads, ...])

create_monitoring_info(status)

Create a monitoring message for each block based on the poll status.

handle_errors(error_handler, status)

This method is called by the error management infrastructure after a status poll.

monitor_resources()

Resource monitoring sometimes deadlocks when using threads, so this function returns false to disable it.

scale_in(blocks)

Scale in the number of active blocks by specified amount.

scale_out([workers])

Scales out the number of active workers by 1.

set_bad_state_and_fail_all(exception)

Allows external error handlers to mark this executor as irrecoverably bad and cause all tasks submitted to it now and in the future to fail.

shutdown([block])

Shutdown the ThreadPool.

start()

Start the executor.

status()

Return the status of all jobs/blocks currently known to this executor.

submit(func, resource_specification, *args, ...)

Submits work to the thread pool.

Attributes

bad_state_is_set

Returns true if this executor is in an irrecoverable error state.

error_management_enabled

Indicates whether worker error management is supported by this executor.

executor_exception

Returns an exception that indicates why this executor is in an irrecoverable state.

hub_address

Address to the Hub for monitoring.

hub_port

Port to the Hub for monitoring.

label

provider

run_dir

Path to the run directory.

scaling_enabled

Specify if scaling is enabled.

status_polling_interval

Returns the interval, in seconds, at which the status method should be called.

tasks

Contains a dictionary mapping task IDs to the corresponding Future objects for all tasks that have been submitted to this executor.

monitor_resources()[source]

Resource monitoring sometimes deadlocks when using threads, so this function returns false to disable it.

scale_in(blocks)[source]

Scale in the number of active blocks by specified amount.

This method is not implemented for threads and will raise the error if called.

Raises

NotImplemented exception

scale_out(workers=1)[source]

Scales out the number of active workers by 1.

This method is notImplemented for threads and will raise the error if called.

Raises

NotImplemented exception

property scaling_enabled[source]

Specify if scaling is enabled.

The callers of ParslExecutors need to differentiate between Executors and Executors wrapped in a resource provider

shutdown(block=False)[source]

Shutdown the ThreadPool.

Kwargs:
  • block (Bool): To block for confirmations or not

start()[source]

Start the executor.

Any spin-up operations (for example: starting thread pools) should be performed here.

submit(func, resource_specification, *args, **kwargs)[source]

Submits work to the thread pool.

This method is simply pass through and behaves like a submit call as described here Python docs: