parsl.executors.swift_t.TurbineExecutor

class parsl.executors.swift_t.TurbineExecutor(label='turbine', storage_access=None, working_dir=None, managed=True)

The Turbine executor.

Bypass the Swift/T language and run on top off the Turbine engines in an MPI environment.

Here is a diagram

             |  Data   |  Executor   |   IPC      | External Process(es)
             |  Flow   |             |            |
        Task | Kernel  |             |            |
      +----->|-------->|------------>|outgoing_q -|-> Worker_Process
      |      |         |             |            |    |         |
Parsl<---Fut-|         |             |            |  result   exception
          ^  |         |             |            |    |         |
          |  |         |   Q_mngmnt  |            |    V         V
          |  |         |    Thread<--|incoming_q<-|--- +---------+
          |  |         |      |      |            |
          |  |         |      |      |            |
          +----update_fut-----+

__init__(label='turbine', storage_access=None, working_dir=None, managed=True)

Initialize the thread pool.

Trying to implement the emews model.

Methods

__init__([label, storage_access, …])	Initialize the thread pool.
create_monitoring_info(status)	Create a monitoring message for each block based on the poll status.
handle_errors(error_handler, status)	This method is called by the error management infrastructure after a status poll.
monitor_resources()	Should resource monitoring happen for tasks on running on this executor?
scale_in(blocks)	Scale in the number of active blocks by specified amount.
scale_out([blocks])	Scales out the number of active workers by 1.
set_bad_state_and_fail_all(exception)	Allows external error handlers to mark this executor as irrecoverably bad and cause all tasks submitted to it now and in the future to fail.
shutdown()	Shutdown method, to kill the threads and workers.
start()	Start the executor.
status()	Return the status of all jobs/blocks currently known to this executor.
submit(func, *args, **kwargs)	Submits work to the the outgoing_q.
weakref_cb([q])	We do not use this yet.

Attributes

bad_state_is_set	Returns true if this executor is in an irrecoverable error state.
error_management_enabled	Indicates whether worker error management is supported by this executor.
executor_exception	Returns an exception that indicates why this executor is in an irrecoverable state.
hub_address	Address to the Hub for monitoring.
hub_port	Port to the Hub for monitoring.
provider
run_dir	Path to the run directory.
scaling_enabled	Specify if scaling is enabled.
status_polling_interval	Returns the interval, in seconds, at which the status method should be called.
tasks	Contains a dictionary mapping task IDs to the corresponding Future objects for all tasks that have been submitted to this executor.

label: str

scale_in(blocks)

Scale in the number of active blocks by specified amount.

This method is not implemented for turbine and will raise an error if called.

Raises

NotImplementedError –

scale_out(blocks=1)

Scales out the number of active workers by 1.

This method is not implemented for threads and will raise the error if called. This would be nice to have, and can be done

Raises

NotImplementedError –

property scaling_enabled

Specify if scaling is enabled.

The callers of ParslExecutors need to differentiate between Executors and Executors wrapped in a resource provider

shutdown()

Shutdown method, to kill the threads and workers.

start()

Start the executor.

Any spin-up operations (for example: starting thread pools) should be performed here.

submit(func, *args, **kwargs)

Submits work to the the outgoing_q.

The outgoing_q is an external process listens on this queue for new work. This method is simply pass through and behaves like a submit call as described here Python docs:

Parameters

• func (-) – Callable function

• args (-) – List of arbitrary positional arguments.

Kwargs:

• kwargs (dict) : A dictionary of arbitrary keyword args for func.

Returns

Future

weakref_cb(q=None)

We do not use this yet.