parsl.data_provider.data_manager.DataManager¶
- class parsl.data_provider.data_manager.DataManager(dfk: DataFlowKernel)[source]¶
The DataManager is responsible for transferring input and output data.
- __init__(dfk: DataFlowKernel) None [source]¶
Initialize the DataManager.
- Parameters
dfk (-) – The DataFlowKernel that this DataManager is managing data for.
Methods
__init__
(dfk)Initialize the DataManager.
optionally_stage_in
(input, func, executor)replace_task
(file, func, executor)This will give staging providers the chance to wrap (or replace entirely!) the task function.
replace_task_stage_out
(file, func, executor)This will give staging providers the chance to wrap (or replace entirely!) the task function.
stage_in
(file, input, executor)Transport the input from the input source to the executor, if it is file-like, returning a DataFuture that wraps the stage-in operation.
stage_out
(file, executor, app_fu)Transport the file from the local filesystem to the remote Globus endpoint.
- replace_task(file: parsl.data_provider.files.File, func: Callable, executor: str) Callable [source]¶
This will give staging providers the chance to wrap (or replace entirely!) the task function.
- replace_task_stage_out(file: parsl.data_provider.files.File, func: Callable, executor: str) Callable [source]¶
This will give staging providers the chance to wrap (or replace entirely!) the task function.
- stage_in(file: parsl.data_provider.files.File, input: Any, executor: str) Any [source]¶
Transport the input from the input source to the executor, if it is file-like, returning a DataFuture that wraps the stage-in operation.
If no staging in is required - because the
file
parameter is not file-like, then return that parameter unaltered.- Parameters
self (-) –
input (-) – input to stage in. If this is a File or a DataFuture, stage in tasks will be launched with appropriate dependencies. Otherwise, no stage-in will be performed.
executor (-) – an executor the file is going to be staged in to.
- stage_out(file: parsl.data_provider.files.File, executor: str, app_fu: concurrent.futures._base.Future) Optional[concurrent.futures._base.Future] [source]¶
Transport the file from the local filesystem to the remote Globus endpoint.
This function returns either a Future which should complete when the stageout is complete, or None, if no staging needs to be waited for.
- Parameters
self (-) –
file (-) –
executor (-) –
app_fu (-) – complete before stageout begins.