Storage

Storage prototype

class qarnot.storage.Storage

Bases: object

Common architecture for storage providers

get_all_files(output_dir, progress=None)

Get all files from the storage.

Parameters:

• output_dir (str) – local directory for the retrieved files.
• progress (bool or function(float, float, str)) – can be a callback (read,total,filename) or True to display a progress bar

Raises:

• qarnot.exceptions.MissingBucketException – the bucket is not on the server
• qarnot.exceptions.QarnotGenericException – API general error, see message for details
• qarnot.exceptions.UnauthorizedException – invalid credentials

Warning

Will override output_dir content.

__init__()

x.__init__(…) initializes x; see help(type(x)) for signature

list_files()

List files on the storage.

… note:

File object returned must have a key property.

Returns:List of the files on the storage.

get_file(remote, local=None, progress=None)

Get a file from the storage. Create needed subfolders.

Parameters:

• remote (str) – the name of the remote file
• local (str) – local name of the retrieved file (defaults to remote)
• progress (bool or function(float, float, str)) – can be a callback (read,total,filename) or True to display a progress bar

Return type:

str

Returns:

The name of the output file.

Raises:

ValueError – no such file

copy_file(source, dest)

Create a copy of a file

Parameters:

• source (str) – name of the existing file to duplicate
• dest (str) – name of the created file

add_directory(local, remote)

Add a directory to the storage. Does not follow symlinks. File hierarchy is preserved.

Parameters:

• local (str) – path of the local directory to add
• remote (str) – path of the directory on remote node (defaults to local)

Raises:

IOError – not a valid directory

add_file(local_or_file, remote)

Add a local file or a Python File on the storage.

Note

You can also use object[remote] = local

Parameters:

• local_or_file (str or File) – path of the local file or an opened Python File
• remote (str) – name of the remote file (defaults to local_or_file)

delete_file(remote)

Delete a file from the storage.

Parameters:remote (str) – the name of the remote file

update(flush=None)

Update object from remote endpoint

Parameters:flush (bool) – bypass cache

flush()

Ensure all background uploads are complete

__getitem__(filename)

x.__getitem__(y) <==> x[y]

__setitem__(remote, filename)

x.__setitem__(i, y) <==> x[i]=y

__delitem__(filename)

x.__delitem__(y) <==> del x[y]

__contains__(k) → True if D has a key k, else False

__iter__() <==> iter(x)

__eq__(other)

x.__eq__(y) <==> x == y

__ne__(other)

x.__ne__(y) <==> x != y