Connection

Module describing a connection.

class qarnot.connection.Connection(fileconf=None, client_token=None, cluster_url=None, cluster_unsafe=False, cluster_timeout=None, storage_url=None, storage_unsafe=False, retry_count=5, retry_wait=1.0, cluster_custom_certificate=None, storage_custom_certificate=None)

Bases: object

Represents the couple cluster/user to which submit tasks.

__init__(fileconf=None, client_token=None, cluster_url=None, cluster_unsafe=False, cluster_timeout=None, storage_url=None, storage_unsafe=False, retry_count=5, retry_wait=1.0, cluster_custom_certificate=None, storage_custom_certificate=None)

Create a connection to a cluster with given config file, options or environment variables. Available environment variable are QARNOT_CLUSTER_URL, QARNOT_CLUSTER_UNSAFE, QARNOT_CLUSTER_TIMEOUT and QARNOT_CLIENT_TOKEN.

Parameters:

• fileconf (str or dict) – path to a qarnot configuration file or a corresponding dict
• client_token (str) – API Token
• cluster_url (str) – (optional) Cluster url.
• cluster_unsafe (bool) – (optional) Disable certificate check
• cluster_timeout (int) – (optional) Timeout value for every request
• storage_url (str) – (optional) Storage service url.
• storage_unsafe (bool) – (optional) Disable certificate check
• retry_count (int) – (optional) ConnectionError retry count. Default to 5.
• retry_wait (float) – (optional) Retry on error wait time, progressive. (wait * (retry_count - retry_num). Default to 1s

Configuration sample:

[cluster]
# url of the REST API
url=https://localhost
# No SSL verification ?
unsafe=False
[client]
# auth string of the client
token=login
[storage]
url=https://storage
unsafe=False

s3client

Pre-configured s3 client object.

Return type:list(S3.Client) Returns:A list of ObjectSummary resources

s3resource

Pre-configured s3 resource object.

Return type:list(S3.ServiceResource) Returns:A list of ObjectSummary resources

user_info

Get information of the current user on the cluster.

Return type:

UserInfo

Returns:

Requested information.

Raises:

• qarnot.exceptions.UnauthorizedException – invalid credentials
• qarnot.exceptions.QarnotGenericException – API general error, see message for details

buckets()

Get the list of buckets.

Return type:list(class:~qarnot.bucket.Bucket). Returns:List of buckets

pools(summary=True, tags_intersect=None)

Get the list of pools stored on this cluster for this user.

Parameters:

• summary (bool) – only get the summaries.
• tags_intersect (list of str, optional) – Desired filtering tags, all of them

Return type:

List of Pool.

Returns:

Pools stored on the cluster owned by the user.

Raises:

• qarnot.exceptions.UnauthorizedException – invalid credentials
• qarnot.exceptions.QarnotGenericException – API general error, see message for details

tasks(tags=None, summary=True, tags_intersect=None)

Get the list of tasks stored on this cluster for this user.

Parameters:

• tags (list of str, optional) – Desired filtering tags, any of them
• summary (bool) – only get the summaries.
• tags_intersect (list of str, optional) – Desired filtering tags, all of them

Return type:

List of Task.

Returns:

Tasks stored on the cluster owned by the user.

Raises:

• qarnot.exceptions.UnauthorizedException – invalid credentials
• qarnot.exceptions.QarnotGenericException – API general error, see message for details

jobs()

Get the list of jobs stored on this cluster for this user.

Raises:

• qarnot.exceptions.UnauthorizedException – invalid credentials
• qarnot.exceptions.QarnotGenericException – API general error, see message for details

retrieve_pool(uuid)

Retrieve a qarnot.pool.Pool from its uuid

Parameters:

uuid (str) – Desired pool uuid

Return type:

Pool

Returns:

Existing pool defined by the given uuid

Raises:

• qarnot.exceptions.MissingPoolException – pool does not exist
• qarnot.exceptions.UnauthorizedException – invalid credentials
• qarnot.exceptions.QarnotGenericException – API general error, see message for details

retrieve_task(uuid)

Retrieve a qarnot.task.Task from its uuid

Parameters:

uuid (str) – Desired task uuid

Return type:

Task

Returns:

Existing task defined by the given uuid

Raises:

• qarnot.exceptions.MissingTaskException – task does not exist
• qarnot.exceptions.UnauthorizedException – invalid credentials
• qarnot.exceptions.QarnotGenericException – API general error, see message for details

retrieve_job(uuid)

Retrieve a qarnot.job.Job from its uuid

Parameters:

uuid (str) – Desired job uuid

Return type:

Job

Returns:

Existing job defined by the given uuid

Raises:

• qarnot.exceptions.MissingJobException – job does not exist
• qarnot.exceptions.UnauthorizedException – invalid credentials
• qarnot.exceptions.QarnotGenericException – API general error, see message for details

retrieve_or_create_bucket(uuid)

Retrieve a Bucket from its description, or create a new one.

Parameters:uuid (str) – the bucket uuid (name) Return type:Bucket Returns:Existing or newly created bucket defined by the given name

retrieve_bucket(uuid)

Retrieve a Bucket from its uuid (name)

Parameters:uuid (str) – Desired bucket uuid (name) Return type:Bucket Returns:Existing bucket defined by the given uuid (name) Raises:botocore.exceptions.ClientError: Bucket does not exist, or invalid credentials

create_pool(name, profile, instancecount=1, shortname=None)

Create a new Pool.

Parameters:

• name (str) – given name of the pool
• profile (str) – which profile to use with this pool
• instancecount (int) – number of instances to run for the pool
• shortname (str) – optional unique friendly shortname of the pool

Return type:

Pool

Returns:

The created Pool.

Note

See available profiles with profiles().

create_elastic_pool(name, profile, minimum_total_slots=0, maximum_total_slots=1, minimum_idle_slots=0, minimum_idle_time_seconds=0, resize_factor=1, resize_period=90, shortname=None)

Create a new Pool.

Parameters:

• name (str) – given name of the pool
• profile (str) – which profile to use with this pool
• minimum_total_slots (int) – minimum number of instances to run for the pool
• maximum_total_slots (int) – maximum number of instances to run for the pool
• minimum_idle_slots (int) – the number of instances that can be idle before considering shrinking the pool
• minimum_idle_time_seconds (int) – the number of seconds before considering shrinking the pool
• resize_factor (float) – the speed with which we grow the pool to meet the demand
• resize_period (int) – the time between the load checks that decide if the pool grows or shrinks
• shortname (str) – optional unique friendly shortname of the pool

Return type:

Pool

Returns:

The created Pool.

Note

See available profiles with profiles().

create_task(name, profile_or_pool=None, instancecount_or_range=1, shortname=None, job=None)

Create a new Task.

Parameters:

• name (str) – given name of the task
• profile_or_pool (str or Pool or None) – which profile to use with this task, or which Pool to run task, or which job to attach it to
• instancecount_or_range (int or str) – number of instances, or ranges on which to run task. Defaults to 1.
• shortname (str) – optional unique friendly shortname of the task
• job (Job) – which job to attach the task to

Return type:

Task

Returns:

The created Task.

Note

See available profiles with profiles().

submit_tasks(tasks)

Submit a list of Task.

:param List of Task. :raises qarnot.exceptions.QarnotGenericException: API general error, see message for details

Note

Will ensure all added files are on the resource bucket regardless of their uploading mode.

profiles()

Get list of profiles available on the cluster.

Return type:

list of Profile

Raises:

• qarnot.exceptions.UnauthorizedException – invalid credentials
• qarnot.exceptions.QarnotGenericException – API general error, see message for details

retrieve_profile(name)

Get details of a profile from its name.

Return type:

Profile

Raises:

• qarnot.exceptions.UnauthorizedException – invalid credentials
• qarnot.exceptions.QarnotGenericException – API general error, see message for details

create_bucket(name)

Create a new Bucket. If the bucket already exist, retrieve the existing bucket.

Parameters:name (str) – bucket name Return type:qarnot.bucket.Bucket Returns:The created or existing Bucket.

create_job(name, pool=None, shortname=None, useDependencies=False)

Create a new Job.

Parameters:

• name (str) – given name of the job
• pool (Pool or None) – which Pool to submit the job in,
• shortname (str) – userfriendly job name
• use_dependencies – allow dependencies between tasks in this job

Returns:

The created Job.

class qarnot.connection.UserInfo(info)

Bases: object

Information about a qarnot user.

__init__(info)

x.__init__(…) initializes x; see help(type(x)) for signature

email = None

Type:str

User email address.

max_bucket = None

Type:int

Maximum number of buckets allowed (resource and result buckets).

quota_bytes_bucket = None

Type:int

Total storage space allowed for the user’s buckets (in Bytes).

used_quota_bytes_bucket = None

Type:int

Total storage space used by the user’s buckets (in Bytes).

task_count = None

Type:int

Total number of tasks belonging to the user.

max_task = None

Type:int

Maximum number of tasks the user is allowed to create.

running_task_count = None

Type:int

Number of tasks currently in ‘Submitted’ state.

max_running_task = None

Type:int

Maximum number of running tasks.

max_instances = None

Type:int

Maximum number of instances.

class qarnot.connection.Profile(info)

Bases: object

Information about a profile.

__init__(info)

x.__init__(…) initializes x; see help(type(x)) for signature

name = None

Type:str

Name of the profile.

constants = None

Type:List of (str, str)

List of couples (name, value) representing constants for this profile and their default values.

__repr__() <==> repr(x)