parsl.providers.AWSProvider

class parsl.providers.AWSProvider(image_id, key_name, init_blocks=1, min_blocks=0, max_blocks=10, nodes_per_block=1, parallelism=1, worker_init='', instance_type='t2.small', region='us-east-2', spot_max_bid=0, key_file=None, profile=None, iam_instance_profile_arn='', state_file=None, walltime='01:00:00', linger=False, launcher=SingleNodeLauncher())[source]

A provider for using Amazon Elastic Compute Cloud (EC2) resources.

One of 3 methods are required to authenticate: keyfile, profile or environment variables. If neither keyfile or profile are set, the following environment variables must be set: AWS_ACCESS_KEY_ID (the access key for your AWS account), AWS_SECRET_ACCESS_KEY (the secret key for your AWS account), and (optionaly) the AWS_SESSION_TOKEN (the session key for your AWS account).

Parameters

  • image_id (str) – Identification of the Amazon Machine Image (AMI).

  • worker_init (str) – String to append to the Userdata script executed in the cloudinit phase of instance initialization.

  • walltime (str) – Walltime requested per block in HH:MM:SS.

  • key_file (str) – Path to json file that contains ‘AWSAccessKeyId’ and ‘AWSSecretKey’.

  • nodes_per_block (int) – This is always 1 for ec2. Nodes to provision per block.

  • profile (str) – Profile to be used from the standard aws config file ~/.aws/config.

  • nodes_per_block – Nodes to provision per block. Default is 1.

  • init_blocks (int) – Number of blocks to provision at the start of the run. Default is 1.

  • min_blocks (int) – Minimum number of blocks to maintain. Default is 0.

  • max_blocks (int) – Maximum number of blocks to maintain. Default is 10.

  • instance_type (str) – EC2 instance type. Instance types comprise varying combinations of CPU, memory, . storage, and networking capacity For more information on possible instance types,. see here Default is ‘t2.small’.

  • region (str) – Amazon Web Service (AWS) region to launch machines. Default is ‘us-east-2’.

  • key_name (str) – Name of the AWS private key (.pem file) that is usually generated on the console to allow SSH access to the EC2 instances. This is mostly used for debugging.

  • spot_max_bid (float) – Maximum bid price (if requesting spot market machines).

  • iam_instance_profile_arn (str) – Launch instance with a specific role.

  • state_file (str) – Path to the state file from a previous run to re-use.

  • walltime – Walltime requested per block in HH:MM:SS. This option is not currently honored by this provider.

  • launcher (Launcher) – Launcher for this provider. Possible launchers include SingleNodeLauncher (the default), SrunLauncher, or AprunLauncher

  • linger (Bool) – When set to True, the workers will not halt. The user is responsible for shutting down the node.

__init__(image_id, key_name, init_blocks=1, min_blocks=0, max_blocks=10, nodes_per_block=1, parallelism=1, worker_init='', instance_type='t2.small', region='us-east-2', spot_max_bid=0, key_file=None, profile=None, iam_instance_profile_arn='', state_file=None, walltime='01:00:00', linger=False, launcher=SingleNodeLauncher())[source]

Initialize self. See help(type(self)) for accurate signature.

Methods

__init__(image_id, key_name[, init_blocks, …])

Initialize self.

cancel(job_ids)

Cancel the jobs specified by a list of job ids.

config_route_table(vpc, internet_gateway)

Configure route table for Virtual Private Cloud (VPC).

create_session()

Create a session.

create_vpc()

Create and configure VPC

get_instance_state([instances])

Get states of all instances on EC2 which were started by this file.

goodbye()

initialize_boto_client()

Initialize the boto client.

read_state_file(state_file)

Read the state file, if it exists.

security_group(vpc)

Create and configure a new security group.

show_summary()

Print human readable summary of current AWS state to log and to console.

shut_down_instance([instances])

Shut down a list of instances, if provided.

spin_up_instance(command, job_name)

Start an instance in the VPC in the first available subnet.

status(job_ids)

Get the status of a list of jobs identified by their ids.

submit([command, tasks_per_node, job_name])

Submit the command onto a freshly instantiated AWS EC2 instance.

teardown()

Teardown the EC2 infastructure.

write_state_file()

Save information that must persist to a file.

xstr(s)

Attributes

cores_per_node

Number of cores to provision per node.

current_capacity

Returns the current blocksize.

label

Provides the label for this provider

mem_per_node

Real memory to provision per node in GB.

status_polling_interval

Returns the interval, in seconds, at which the status method should be called.