Task

Module to handle a task.

class qarnot.task.Task(connection, name, profile_or_pool=None, instancecount_or_range=1, shortname=None, job=None)[source]

Bases: object

Represents a Qarnot task.

Note

A Task must be created with qarnot.connection.Connection.create_task() or retrieved with qarnot.connection.Connection.tasks() or qarnot.connection.Connection.retrieve_task().

__init__(connection, name, profile_or_pool=None, instancecount_or_range=1, shortname=None, job=None)[source]

Create a new Task.

Parameters:
  • connection (qarnot.connection.Connection) – the cluster on which to send the task
  • name (str) – given name of the task
  • profile_or_pool (str or Pool or None) – which profile to use with this task, or which Pool to run task,
  • instancecount_or_range (int or str) – number of instances or ranges on which to run task
  • shortname (str) – userfriendly task name
  • job (Job) – which job to attach the task to
run(output_dir=None, job_timeout=None, live_progress=False, results_progress=None)[source]

Submit a task, wait for the results and download them if required.

Parameters:
  • output_dir (str) – (optional) path to a directory that will contain the results
  • job_timeout (float) – (optional) Number of seconds before the task abort() if it is not already finished
  • live_progress (bool) – (optional) display a live progress
  • results_progress (bool or function(float, float, str)) – (optional) can be a callback (read,total,filename) or True to display a progress bar
Raises:

Note

Will ensure all added file are on the resource bucket regardless of their uploading mode.

Note

If this function is interrupted (script killed for example), but the task is submitted, the task will still be executed remotely (results will not be downloaded)

Warning

Will override output_dir content.

resume(output_dir, job_timeout=None, live_progress=False, results_progress=None)[source]

Resume waiting for this task if it is still in submitted mode. Equivalent to wait() + download_results().

Parameters:
  • output_dir (str) – path to a directory that will contain the results
  • job_timeout (float) – Number of seconds before the task abort() if it is not already finished
  • live_progress (bool) – display a live progress
  • results_progress (bool or function(float, float, str)) – can be a callback (read,total,filename) or True to display a progress bar
Raises:

Note

Do nothing if the task has not been submitted.

Warning

Will override output_dir content.

submit()[source]

Submit task to the cluster if it is not already submitted.

Raises:

Note

Will ensure all added files are on the resource bucket regardless of their uploading mode.

Note

To get the results, call download_results() once the job is done.

abort()[source]

Abort this task if running.

Raises:
update_resources()[source]

Update resources for a running task.

The typical workflow is as follows:
  1. Upload new files on your resource bucket,
  2. Call this method,
  3. The new files will appear on all the compute nodes in the $DOCKER_WORKDIR folder

Note: There is no way to know when the files are effectively transfered. This information is available on the compute node only. Note: The update is additive only: files deleted from the bucket will NOT be deleted from the task’s resources directory.

Raises:
delete(purge_resources=False, purge_results=False)[source]

Delete this task on the server.

Parameters:
  • purge_resources (bool) – parameter value is used to determine if the bucket is also deleted. Defaults to False.
  • purge_results (bool) – parameter value is used to determine if the bucket is also deleted. Defaults to False.
Raises:
update(flushcache=False)[source]

Update the task object from the REST Api. The flushcache parameter can be used to force the update, otherwise a cached version of the object will be served when accessing properties of the object. Some methods will flush the cache, like submit(), abort(), wait() and instant(). Cache behavior is configurable with auto_update and update_cache_time.

Raises:
classmethod from_json(connection, json_task, is_summary=False)[source]

Create a Task object from a json task.

Parameters:
Returns:

The created Task.
commit()[source]

Replicate local changes on the current object instance to the REST API

Raises:

Note

When updating buckets’ properties, auto update will be disabled until commit is called.

wait(timeout=None, live_progress=False)[source]

Wait for this task until it is completed.

Parameters:
  • timeout (float) – maximum time (in seconds) to wait before returning (None => no timeout)
  • live_progress (bool) – display a live progress
Return type:

bool
Returns:

Is the task finished
Raises:
snapshot(interval)[source]

Start snapshooting results. If called, this task’s results will be periodically updated, instead of only being available at the end.

Snapshots will be taken every interval second from the time the task is submitted.

Parameters:

interval (int) – the interval in seconds at which to take snapshots
Raises:

Note

To get the temporary results, call download_results().

instant()[source]

Make a snapshot of the current task.

Raises:

Note

To get the temporary results, call download_results().

state
Type:str
Getter:return this task’s state

State of the task.

Value is in
  • UnSubmitted
  • Submitted
  • PartiallyDispatched
  • FullyDispatched
  • PartiallyExecuting
  • FullyExecuting
  • UploadingResults
  • DownloadingResults
  • Cancelled
  • Success
  • Failure

Warning

this is the state of the task when the object was retrieved, call update() for up to date value.

resources
Type:list(Bucket)
Getter:Returns this task’s resources bucket
Setter:Sets this task’s resources bucket

Represents resource files.

results
Type:Bucket
Getter:Returns this task’s results bucket
Setter:Sets this task’s results bucket

Represents results files.

download_results(output_dir, progress=None)[source]

Download results in given output_dir.

Parameters:
  • output_dir (str) – local directory for the retrieved files.
  • progress (bool or function(float, float, str)) – can be a callback (read,total,filename) or True to display a progress bar
Raises:

Warning

Will override output_dir content.

stdout()[source]

Get the standard output of the task since the submission of the task.

Return type:

str
Returns:

The standard output.
Raises:

Note

The buffer is circular, if stdout is too big, prefer calling fresh_stdout() regularly.

fresh_stdout()[source]

Get what has been written on the standard output since last time this function was called or since the task has been submitted.

Return type:

str
Returns:

The new output since last call.
Raises:
stderr()[source]

Get the standard error of the task since the submission of the task.

Return type:

str
Returns:

The standard error.
Raises:

Note

The buffer is circular, if stderr is too big, prefer calling fresh_stderr() regularly.

fresh_stderr()[source]

Get what has been written on the standard error since last time this function was called or since the task has been submitted.

Return type:

str
Returns:

The new error messages since last call.
Raises:
uuid
Type:str
Getter:Returns this task’s uuid

The task’s uuid.

Automatically set when a task is submitted.

name
Type:str
Getter:Returns this task’s name
Setter:Sets this task’s name

The task’s name.

Can be set until task is submitted.

shortname
Type:str
Getter:Returns this task’s shortname
Setter:Sets this task’s shortname

The task’s shortname, must be DNS compliant and unique, if not provided, will default to uuid.

Can be set until task is submitted.

tags
Type::class:list(str)
Getter:Returns this task’s tags
Setter:Sets this task’s tags

Custom tags.

pool
Type:Pool
Getter:Returns this task’s pool
Setter:Sets this task’s pool

The pool to run the task in.

Can be set until run() is called.

Warning

This property is mutually exclusive with profile

profile
Type:str
Getter:Returns this task’s profile
Setter:Sets this task’s profile

The profile to run the task with.

Can be set until run() or submit() is called.

Warning

This property is mutually exclusive with pool

instancecount
Type:int
Getter:Returns this task’s instance count
Setter:Sets this task’s instance count

Number of instances needed for the task.

Can be set until run() or submit() is called.

Raises:AttributeError – if advanced_range is not None when setting this property

Warning

This property is mutually exclusive with advanced_range

running_core_count
Type:int
Getter:Returns this task’s running core count

Number of core running inside the task.

running_instance_count
Type:int
Getter:Returns this task’s running instance count

Number of instances running inside the task.

advanced_range
Type:str
Getter:Returns this task’s advanced range
Setter:Sets this task’s advanced range

Advanced instances range selection.

Allows to select which instances will be computed. Should be None or match the following extended regular expression “([0-9]+|[0-9]+-[0-9]+)(,([0-9]+|[0-9]+-[0-9]+))+” eg: 1,3-8,9,12-19

Can be set until run() is called.

Raises:AttributeError – if instancecount is not 0 when setting this property

Warning

This property is mutually exclusive with instancecount

snapshot_whitelist
Type:str
Getter:Returns this task’s snapshot whitelist
Setter:Sets this task’s snapshot whitelist

Snapshot white list (regex) for snapshot() and instant()

Can be set until task is submitted.

snapshot_blacklist
Type:str
Getter:Returns this task’s snapshot blacklist
Setter:Sets this task’s snapshot blacklist

Snapshot black list (regex) for snapshot() instant()

Can be set until task is submitted.

results_whitelist
Type:str
Getter:Returns this task’s results whitelist
Setter:Sets this task’s results whitelist

Results whitelist (regex)

Can be set until task is submitted.

results_blacklist
Type:str
Getter:Returns this task’s results blacklist
Setter:Sets this task’s results blacklist

Results blacklist (regex)

Can be set until task is submitted.

status
Type:qarnot.status.Status
Getter:Returns this task’s status

Status of the task

completed_instances
Type:list(CompletedInstance)
Getter:Return this task’s completed instances
creation_date
Type:str
Getter:Returns this task’s creation date

Creation date of the task (UTC Time)

errors
Type:list(Error)
Getter:Returns this task’s errors if any.

Error reason if any, empty string if none

constants
Type:dictionary{str : str}
Getter:Returns this task’s constants dictionary.
Setter:set the task’s constants dictionary.

Update the constants if needed Constants are the parametrazer of the profils. Use them to adjust your profile parametter.

constraints
Type:dictionary{str : str}
Getter:Returns this task’s constraints dictionary.
Setter:set the task’s constraints dictionary.

Update the constraints if needed advance usage

wait_for_pool_resources_synchronization
Type:bool or None
Getter:Returns this task’s wait_for_pool_resources_synchronization.
Setter:set the task’s wait_for_pool_resources_synchronization.
Raises:qarnot.exceptions.AttributeError – can’t set this attribute on a launched task
auto_update
Type:bool
Getter:Returns this task’s auto update state
Setter:Sets this task’s auto update state

Auto update state, default to True When auto update is disabled properties will always return cached value for the object and a call to update() will be required to get latest values from the REST Api.

update_cache_time
Type:int
Getter:Returns this task’s auto update state
Setter:Sets this task’s auto update state

Cache expiration time, default to 5s

set_task_dependencies_from_uuids(uuids)[source]

Setter for the task dependencies using uuid

set_task_dependencies_from_tasks(tasks)[source]

Setter for the task dependencies using tasks

auto_delete

Autodelete this Task if it is finished and your max number of task is reach

Can be set until run() or submit() is called.

Type:bool
Getter:Returns is this task must autodelete
Setter:Sets this task’s autodelete
Default_value:“False”
Raises:AttributeError – if you try to reset the auto_delete after the task is submit
completion_ttl

The task will be auto delete completion_ttl after it is finished

Can be set until run() or submit() is called.

Getter:Returns this task’s completed time to live.
Type:str
Setter:Sets this task’s this task’s completed time to live.
Type:str or datetime.timedelta
Default_value:“”
Raises:AttributeError – if you try to set it after the task is submitted

The completion_ttl must be a timedelta or a time span format string (example: ‘d.hh:mm:ss’ or ‘hh:mm:ss’)

__repr__() <==> repr(x)[source]
class qarnot.task.CompletedInstance(json)[source]

Bases: object

Completed Instance Information

Note

Read-only class

__init__(json)[source]

x.__init__(…) initializes x; see help(type(x)) for signature

instance_id = None
Type:int

Instance number.

state = None
Type:str

Instance final state.

wall_time_sec = None
Type:float

Instance wall time in seconds.

exec_time_sec = None
Type:float

Execution time in seconds.

exec_time_sec_ghz = None
Type:float

Execution time in seconds GHz.

peak_memory_mb = None
Type:int

Peak memory size in MB.

average_ghz = None
Type:float

Instance execution time GHz

results = None
Type::class:list(str)

Instance produced results

__repr__() <==> repr(x)[source]
class qarnot.task.BulkTaskResponse(json)[source]

Bases: object

Bulk Task Response Information

Note

Read-only class

__init__(json)[source]

x.__init__(…) initializes x; see help(type(x)) for signature

status_code = None
Type:int

Status code.

uuid = None
Type:str

Created Task Uuid.

message = None
Type:str

User friendly error message.

is_success()[source]

Check that the task submit has been successful.

Return type:bool
Returns:The task creation success(depending on received uuid and the status code).
__repr__() <==> repr(x)[source]