parsl.data_provider.data_manager.DataManager¶
-
class
parsl.data_provider.data_manager.
DataManager
(dfk: DataFlowKernel)[source]¶ The DataManager is responsible for transferring input and output data.
-
__init__
(dfk: DataFlowKernel) → None[source]¶ Initialize the DataManager.
- Parameters
dfk (-) – The DataFlowKernel that this DataManager is managing data for.
Methods
__init__
(dfk)Initialize the DataManager.
optionally_stage_in
(input, func, executor)replace_task
(file, func, executor)This will give staging providers the chance to wrap (or replace entirely!) the task function.
replace_task_stage_out
(file, func, executor)This will give staging providers the chance to wrap (or replace entirely!) the task function.
stage_in
(file, input, executor)Transport the input from the input source to the executor, if it is file-like, returning a DataFuture that wraps the stage-in operation.
stage_out
(file, executor, app_fu)Transport the file from the local filesystem to the remote Globus endpoint.
-
replace_task
(file: parsl.data_provider.files.File, func: Callable, executor: str) → Callable[source]¶ This will give staging providers the chance to wrap (or replace entirely!) the task function.
-
replace_task_stage_out
(file: parsl.data_provider.files.File, func: Callable, executor: str) → Callable[source]¶ This will give staging providers the chance to wrap (or replace entirely!) the task function.
-
stage_in
(file: parsl.data_provider.files.File, input: Any, executor: str) → Any[source]¶ Transport the input from the input source to the executor, if it is file-like, returning a DataFuture that wraps the stage-in operation.
If no staging in is required - because the
file
parameter is not file-like, then return that parameter unaltered.- Parameters
self (-) –
input (-) – input to stage in. If this is a File or a DataFuture, stage in tasks will be launched with appropriate dependencies. Otherwise, no stage-in will be performed.
executor (-) – an executor the file is going to be staged in to.
-
stage_out
(file: parsl.data_provider.files.File, executor: str, app_fu: concurrent.futures._base.Future) → Optional[concurrent.futures._base.Future][source]¶ Transport the file from the local filesystem to the remote Globus endpoint.
This function returns either a Future which should complete when the stageout is complete, or None, if no staging needs to be waited for.
- Parameters
self (-) –
file (-) –
executor (-) –
app_fu (-) – complete before stageout begins.
-