parsl.providers.AWSProvider

class parsl.providers.AWSProvider(image_id, init_blocks=1, min_blocks=0, max_blocks=10, nodes_per_block=1, parallelism=1, worker_init='', instance_type='t2.small', region='us-east-2', spot_max_bid=0, key_name=None, key_file=None, profile=None, iam_instance_profile_arn='', state_file=None, walltime='01:00:00', linger=False, launcher=SingleNodeLauncher())[source]

A provider for using Amazon Elastic Compute Cloud (EC2) resources.

One of 3 methods are required to authenticate: keyfile, profile or environment variables. If neither keyfile or profile are set, the following environment variables must be set: AWS_ACCESS_KEY_ID (the access key for your AWS account), AWS_SECRET_ACCESS_KEY (the secret key for your AWS account), and (optionaly) the AWS_SESSION_TOKEN (the session key for your AWS account).

Parameters:
  • image_id (str) – Identification of the Amazon Machine Image (AMI).
  • worker_init (str) – String to append to the Userdata script executed in the cloudinit phase of instance initialization.
  • walltime (str) – Walltime requested per block in HH:MM:SS.
  • key_file (str) – Path to json file that contains ‘AWSAccessKeyId’ and ‘AWSSecretKey’.
  • nodes_per_block (int) – This is always 1 for ec2. Nodes to provision per block.
  • profile (str) – Profile to be used from the standard aws config file ~/.aws/config.
  • nodes_per_block – Nodes to provision per block. Default is 1.
  • init_blocks (int) – Number of blocks to provision at the start of the run. Default is 1.
  • min_blocks (int) – Minimum number of blocks to maintain. Default is 0.
  • max_blocks (int) – Maximum number of blocks to maintain. Default is 10.
  • instance_type (str) – EC2 instance type. Instance types comprise varying combinations of CPU, memory, . storage, and networking capacity For more information on possible instance types,. see here Default is ‘t2.small’.
  • region (str) – Amazon Web Service (AWS) region to launch machines. Default is ‘us-east-2’.
  • key_name (str) – Name of the AWS private key (.pem file) that is usually generated on the console to allow SSH access to the EC2 instances. This is mostly used for debugging.
  • spot_max_bid (float) – Maximum bid price (if requesting spot market machines).
  • iam_instance_profile_arn (str) – Launch instance with a specific role.
  • state_file (str) – Path to the state file from a previous run to re-use.
  • walltime – Walltime requested per block in HH:MM:SS. This option is not currently honored by this provider.
  • launcher (Launcher) – Launcher for this provider. Possible launchers include SingleNodeLauncher (the default), SrunLauncher, or AprunLauncher
  • linger (Bool) – When set to True, the workers will not halt. The user is responsible for shutting down the node.
__init__(image_id, init_blocks=1, min_blocks=0, max_blocks=10, nodes_per_block=1, parallelism=1, worker_init='', instance_type='t2.small', region='us-east-2', spot_max_bid=0, key_name=None, key_file=None, profile=None, iam_instance_profile_arn='', state_file=None, walltime='01:00:00', linger=False, launcher=SingleNodeLauncher())[source]

Initialize self. See help(type(self)) for accurate signature.

Methods

__init__(image_id[, init_blocks, …]) Initialize self.
cancel(job_ids) Cancel the jobs specified by a list of job ids.
config_route_table(vpc, internet_gateway) Configure route table for Virtual Private Cloud (VPC).
create_session() Create a session.
create_vpc() Create and configure VPC
get_instance_state([instances]) Get states of all instances on EC2 which were started by this file.
goodbye()
initialize_boto_client() Initialize the boto client.
read_state_file(state_file) Read the state file, if it exists.
security_group(vpc) Create and configure a new security group.
show_summary() Print human readable summary of current AWS state to log and to console.
shut_down_instance([instances]) Shut down a list of instances, if provided.
spin_up_instance(command, job_name) Start an instance in the VPC in the first available subnet.
status(job_ids) Get the status of a list of jobs identified by their ids.
submit([command, blocksize, tasks_per_node, …]) Submit the command onto a freshly instantiated AWS EC2 instance.
teardown() Teardown the EC2 infastructure.
write_state_file() Save information that must persist to a file.
xstr(s)

Attributes

current_capacity Returns the current blocksize.
scaling_enabled The callers of ParslExecutors need to differentiate between Executors and Executors wrapped in a resource provider