parsl.executors.swift_t.TurbineExecutor¶
- class parsl.executors.swift_t.TurbineExecutor(label='turbine', storage_access=None, working_dir=None)[source]¶
The Turbine executor.
Bypass the Swift/T language and run on top off the Turbine engines in an MPI environment.
Here is a diagram
| Data | Executor | IPC | External Process(es) | Flow | | | Task | Kernel | | | +----->|-------->|------------>|outgoing_q -|-> Worker_Process | | | | | | | Parsl<---Fut-| | | | result exception ^ | | | | | | | | | Q_mngmnt | | V V | | | Thread<--|incoming_q<-|--- +---------+ | | | | | | | | | | | | +----update_fut-----+
- __init__(label='turbine', storage_access=None, working_dir=None)[source]¶
Initialize the thread pool.
Trying to implement the emews model.
Methods
__init__
([label, storage_access, working_dir])Initialize the thread pool.
create_monitoring_info
(status)Create a monitoring message for each block based on the poll status.
handle_errors
(error_handler, status)This method is called by the error management infrastructure after a status poll.
monitor_resources
()Should resource monitoring happen for tasks on running on this executor?
scale_in
(blocks)Scale in the number of active blocks by specified amount.
scale_out
([blocks])Scales out the number of active workers by 1.
set_bad_state_and_fail_all
(exception)Allows external error handlers to mark this executor as irrecoverably bad and cause all tasks submitted to it now and in the future to fail.
shutdown
()Shutdown method, to kill the threads and workers.
start
()Start the executor.
status
()Return the status of all jobs/blocks currently known to this executor.
submit
(func, *args, **kwargs)Submits work to the the outgoing_q.
weakref_cb
([q])We do not use this yet.
Attributes
bad_state_is_set
Returns true if this executor is in an irrecoverable error state.
error_management_enabled
Indicates whether worker error management is supported by this executor.
executor_exception
Returns an exception that indicates why this executor is in an irrecoverable state.
hub_address
Address to the Hub for monitoring.
hub_port
Port to the Hub for monitoring.
label
provider
radio_mode
run_dir
Path to the run directory.
status_polling_interval
Returns the interval, in seconds, at which the status method should be called.
- scale_in(blocks)[source]¶
Scale in the number of active blocks by specified amount.
This method is not implemented for turbine and will raise an error if called.
- Raises
- scale_out(blocks=1)[source]¶
Scales out the number of active workers by 1.
This method is not implemented for threads and will raise the error if called. This would be nice to have, and can be done
- Raises
- start()[source]¶
Start the executor.
Any spin-up operations (for example: starting thread pools) should be performed here.
- submit(func, *args, **kwargs)[source]¶
Submits work to the the outgoing_q.
The outgoing_q is an external process listens on this queue for new work. This method is simply pass through and behaves like a submit call as described here Python docs:
- Parameters
func (-) – Callable function
args (-) – List of arbitrary positional arguments.
- Kwargs:
kwargs (dict) : A dictionary of arbitrary keyword args for func.
- Returns
Future