parsl.data_provider.rsync.RSyncStaging

class parsl.data_provider.rsync.RSyncStaging(hostname)[source]

This staging provider will execute rsync on worker nodes to stage in files from a remote location.

Worker nodes must be able to authenticate to the rsync server without interactive authentication - for example, worker initialization could include an appropriate SSH key configuration.

The submit side will need to run an rsync-compatible server (for example, an ssh server with the rsync binary installed)

__init__(hostname)[source]

Methods

__init__(hostname)

can_stage_in(file)

Given a File object, decide if this staging provider can stage the file.

can_stage_out(file)

Like can_stage_in, but for staging out.

replace_task(dm, executor, file, f)

For a file to be staged in, optionally return a replacement app function, which usually should be the original app function wrapped in staging code.

replace_task_stage_out(dm, executor, file, f)

For a file to be staged out, optionally return a replacement app function, which usually should be the original app function wrapped in staging code.

stage_in(dm, executor, file, parent_fut)

This call gives the staging provider an opportunity to prepare for stage-in and to launch arbitrary tasks which must complete as part of stage-in.

stage_out(dm, executor, file, parent_fut)

This call gives the staging provider an opportunity to prepare for stage-out and to launch arbitrary tasks which must complete as part of stage-out.

can_stage_in(file)[source]

Given a File object, decide if this staging provider can stage the file. Usually this is be based on the URL scheme, but does not have to be. If this returns True, then other methods of this Staging object will be called to perform the staging.

can_stage_out(file)[source]

Like can_stage_in, but for staging out.

replace_task(dm, executor, file, f)[source]

For a file to be staged in, optionally return a replacement app function, which usually should be the original app function wrapped in staging code.

replace_task_stage_out(dm, executor, file, f)[source]

For a file to be staged out, optionally return a replacement app function, which usually should be the original app function wrapped in staging code.

stage_in(dm, executor, file, parent_fut)[source]

This call gives the staging provider an opportunity to prepare for stage-in and to launch arbitrary tasks which must complete as part of stage-in.

This call will be made with a fresh copy of the File that may be modified for the purposes of this particular staging operation, rather than the original application-provided File. This allows staging specific information (primarily localpath) to be set on the File without interfering with other stagings of the same File.

The call can return a:
  • DataFuture: the corresponding task input parameter will be replaced by the DataFuture, and the main task will not run until that DataFuture is complete. The DataFuture result should be the file object as passed in.

  • None: the corresponding task input parameter will be replaced by a suitable automatically generated replacement that container the File fresh copy, or is the fresh copy.

stage_out(dm, executor, file, parent_fut)[source]

This call gives the staging provider an opportunity to prepare for stage-out and to launch arbitrary tasks which must complete as part of stage-out.

Even though it should set up stageout, it will be invoked before the task executes. Any work which needs to happen after the main task should depend on app_fu.

For a given file, either return a Future which completes when stageout is complete, or return None to indicate that no stageout action need be waited for. When that Future completes, parsl will mark the relevant output DataFuture complete.

Note the asymmetry here between stage_in and stage_out: this can return any Future, while stage_in must return a DataFuture.