escale.relay package

This package defines an interface to a remote host as called by Manager and a few implementations.

class escale.relay.AbstractRelay(client, address, repository, logger=None, ui_controller=None)

Bases: escale.base.essential.Reporter

Send files to/from a remote host.

This class is an interface that groups together the methods called by escale.manager.Manager.

client

client identifier.

Type:str
address

address of the remote host.

Type:str
repository

path of the repository on the remote host.

Type:str
close()

Close the connection with the remote host.

Returns:True if successful, False if failed.
Return type:bool
delete(remote_file)

Fake download and delete a remote file.

This operation should mimic pop but deletes the remote file instead of downloading it.

Parameters:remote_file (str) – relative path to the remote file.
Returns:True if successful, False if failed.
Return type:bool
getMetadata(remote_file, output_file=None, timestamp_format=None)

Download meta-information.

If no local file is specified but output_file is True, getMetadata makes and returns a temporary file instead of a Metadata object.

The returned temporary file should be manually unlinked once done with it.

Example:

import os

temporary_file = relay.getMetadata(path_to_remote_file, output_file=True)

with open(temporary_file, 'r') as f:
    # do something with `f`

os.unlink(temporary_file)
Parameters:
  • remote_file (str) – path to regular file (no placeholder or message).
  • output_file (str or bool) – path to local file.
  • timestamp_format (str) – backward-compatibility option for the former metadata format.
Returns:

metadata or path of a local file.
Return type:

Metadata or str

in 0.5.1: getMetaInfo renamed as getMetadata

new in 0.5.1: timestamp_format; getMetadata returns a Metadata object

listCorrupted(remote_dir='', recursive=True)

List the files on the remote host that are corrupted. All paths are relative to the repository root.

Corrupted files are files with a lock owned by self, as identified by the client attribute. If client evaluates to False, corrupted files are returned as an empty list.

Corrupted files are represented by the unreleased locks.

Parameters:
  • remote_dir (str) – remote directory to “ls”.
  • recursive (bool) – whether to list subdirectories or not.
Returns:

list of locks.
Return type:

list of LockInfo
listReady(remote_dir='', recursive=True)

List the files on the remote host that are ready for download. All paths are relative to the repository root.

Parameters:
  • remote_dir (str) – remote directory to “ls”.
  • recursive (bool) – whether to list subdirectories or not.
Returns:

list of paths.
Return type:

list of str
listTransferred(remote_dir='', end2end=True, recursive=True)

List the files on the remote host that have been transferred. All paths are relative to the repository root.

Parameters:
  • remote_dir (str) – remote directory to “ls”.
  • end2end (bool) – if True, list only files which content is no longer available on the remote host.
  • recursive (bool) – whether to list subdirectories or not.
Returns:

list of paths.
Return type:

list of str
open()

Establish the connection with the remote host.

Returns:True if successful, False if failed.
Return type:bool
pop(remote_file, local_dest, placeholder=True, blocking=True, **kwargs)

Download a file from the remote host and unlinks remote copy if relevant.

Parameters:
  • remote_file (str) – relative path to the remote file.
  • local_dest (str) – path to the target local file.
  • placeholder (bool or int) – whether to generate a placeholder file. If an int is given, it specifies the number of pullers (downloading clients).
  • blocking (bool) – if target exists and is locked, whether should we block until the lock is released or skip the file.
Returns:

True if successful, False if failed.
Return type:

bool
purge(remote_dest='')

Delete a remote directory or collection, and its content.

This method is called by test routines with default argument to purge the entire repository on the relay host.

Parameters:remote_dir (str) – path to the remote directory to be removed.
push(local_file, remote_dest, last_modified=None, checksum=None, blocking=True)

Upload a file to the remote host.

Parameters:
  • local_file (str) – path to the local file to be sent.
  • remote_dest (str) – path to the target remote file.
  • last_modified (str) – meta information to be recorded for the remote copy.
  • checksum (str-like) – checksum of the encrypted content of local_file.
  • blocking (bool) – if target exists and is locked, whether should we block until the lock is released or skip the file.
Returns:

True if successful, False if failed.
Return type:

bool

new in 0.5.1: checksum

remoteListing()

Crawl the relay repository for later calls to list* methods.

remoteListing should be optional and all the list* methods should work with or without past calls to this method.

repair(lock, local_file, checksum=None)

Attempt to repair a corrupted file.

Parameters:
  • lock (LockInfo) – lock information for the corrupted file; the target attribute refers to the file on the remote host.
  • local_file (escale.manager.Accessor) – accessor for local file.
  • checksum (str-like) – checksum of the encrypted content of local_file.

new in 0.5.1: checksum

storageSpace()

Query for how much space is available on the remote host.

Returns:first argument is available space in megabytes (MB), second argument is disk quota in megabytes (MB).
Return type:(int or float, int or None)
class escale.relay.Relay(client, address, repository, logger=None, ui_controller=None, lock_timeout=True, timestamped_messages=False, **ignored)

Bases: escale.relay.relay.AbstractRelay

Send files to/from a remote relay.

This class is a partial implementation of AbstractRelay. Especially, it implements independent placeholders and locks.

Placeholders and locks are named after the corresponding regular files. Extra strings are prepended and appended to the regular filename.

Messages are similar to placeholders and locks. They act as actual placeholders for regular files when these regular files have to be deleted before they could be downloaded by all the pullers.

Messages are an experimental feature and the clearing mechanism is not fully implemented yet.

Placeholders, locks and messages are represented as filenames. They can be referred to as special files, while transferred files are referred to as regular files.

Any derivative class should implement:

_temporary_file

list of paths to existing temporary files.

Type:list
_placeholder_prefix

prefix for placeholder files.

Type:str
_placeholder_suffix

suffix for placeholder files.

Type:str
_lock_prefix

prefix for lock files.

Type:str
_lock_suffix

suffix for lock files.

Type:str
lock_timeout

maximum age of an unclaimed lock in seconds.

Type:bool or int
_message_hash

generates a message subextension (str); takes the path to the corresponding local regular file as input argument (str); a subextension should not contain ‘.’.

Type:function or None
_message_prefix

prefix for message files.

Type:str
_message_suffix

suffix for message files.

Type:str
placeholder_cache

dictionnary of cached placeholders.

Type:dict

new in 0.5.1: placeholder_cache

as of 0.7.6: default lock_timeout is 3 days

_get(remote_file, local_dest, makedirs=True)

Download a file and do NOT delete it from the remote host.

Parameters:
  • remote_file (str) – path to a file on the remote host.
  • local_dest (str) – path to a local directory.
  • makedirs (bool) – make directories if missing.
Returns:

True if transfer was successful, False otherwise.
Return type:

bool or nothing
_list(remote_dir='', recursive=True, stats=[])

List all files, including hidden files, relative to remote_dir. All paths are relative to the repository root.

Parameters:
  • remote_dir (str) – directory on the remote host.
  • recursive (bool) – if True, list files in subdirectories as well.
  • stats (list) – can be [ ‘mtime’ ].
Returns:

list of paths.
Return type:

iterator or list of str
_pop(remote_file, local_dest, makedirs=True)

Download a file and delete it from the remote host.

Note

_pop() can be implemented with an extra _unlink keyword argument that is not supported by default and makes the default implementation for _get() valid.

Parameters:
  • remote_file (str) – path to a file on the remote host.
  • local_dest (str) – path to a local directory.
  • makedirs (bool) – make directories if missing.
  • _unlink (bool, optional) – if False, do not delete the file from the remote host. This keyword argument may not be recognized at all!
Returns:

True if transfer was successful, False otherwise.
Return type:

bool or nothing
_push(local_file, remote_dest)

Send a local file to the remote host.

Parameters:
  • local_file (str) – path to a local file.
  • remote_dest (str) – path to a file on the remote host.
Returns:

True if transfer was successful, False otherwise.
Return type:

bool or nothing
acquireLock(remote_file, mode=None, blocking=True)

This method treats locks as files.

close()

Close the connection with the remote host.

Returns:True if successful, False if failed.
Return type:bool
delTemporaryFile(f)

Delete an existing temporary file.

Parameters:file (str) – path to temporary file.
delete(remote_file, blocking=True, **kwargs)

Fake download and delete a remote file.

This operation should mimic pop but deletes the remote file instead of downloading it.

Parameters:remote_file (str) – relative path to the remote file.
Returns:True if successful, False if failed.
Return type:bool
getLockInfo(remote_file)

This method treats locks as files.

It should NOT raise any exception with missing locks, but return an empty LockInfo instead.

getMetadata(remote_file, output_file=None, timestamp_format=None)

This method treats placeholders as files.

hasLock(remote_file)

Checks for lock presence.

The default implementation manipulates locks as individual files.

Parameters:remote_file (str) – relative path to a regular file on the remote host.
Returns:
True if there exists a lock for remote_file,
False otherwise.
Return type:bool
hasPlaceholder(remote_file)

Checks for placeholder presence.

The default implementation manipulates placeholders as individual files.

Parameters:remote_file (str) – relative path to a regular file on the remote host.
Returns:
True if there exists a placeholder for remote_file,
False otherwise.
Return type:bool
listCorrupted(remote_dir='', recursive=True)

The default implementation manipulates locks as individual files.

listReady(remote_dir='', recursive=True)

The default implementation manipulates placeholders and locks as individual files.

It caches last modification times of placeholders for future getMetadata calls.

listTransferred(remote_dir='', end2end=True, recursive=True)

The default implementation manipulates placeholders and locks as individual files.

markAsRead(remote_file, local_placeholder=None)

This method treats placeholders as files.

Compatible with both old-style and new-style placeholders.

newTemporaryFile()

Make a new temporary file.

Returns:path to temporary file.
Return type:str
open()

Establish the connection with the remote host.

Returns:True if successful, False if failed.
Return type:bool
pop(remote_file, local_dest, placeholder=True, blocking=True, **kwargs)

Download a file from the remote host and unlinks remote copy if relevant.

Parameters:
  • remote_file (str) – relative path to the remote file.
  • local_dest (str) – path to the target local file.
  • placeholder (bool or int) – whether to generate a placeholder file. If an int is given, it specifies the number of pullers (downloading clients).
  • blocking (bool) – if target exists and is locked, whether should we block until the lock is released or skip the file.
Returns:

True if successful, False if failed.
Return type:

bool
push(local_file, remote_dest, last_modified=None, checksum=None, blocking=True)

Upload a file to the remote host.

Parameters:
  • local_file (str) – path to the local file to be sent.
  • remote_dest (str) – path to the target remote file.
  • last_modified (str) – meta information to be recorded for the remote copy.
  • checksum (str-like) – checksum of the encrypted content of local_file.
  • blocking (bool) – if target exists and is locked, whether should we block until the lock is released or skip the file.
Returns:

True if successful, False if failed.
Return type:

bool

new in 0.5.1: checksum

releaseLock(remote_file)

This method treats locks as files.

releasePlace(remote_file, handle_missing=False)

This method treats placeholders as files.

remoteListing()

Crawl the relay repository for later calls to list* methods.

remoteListing should be optional and all the list* methods should work with or without past calls to this method.

repair(lock, local_file, checksum=None)

Attempt to repair a corrupted file.

Parameters:
  • lock (LockInfo) – lock information for the corrupted file; the target attribute refers to the file on the remote host.
  • local_file (escale.manager.Accessor) – accessor for local file.
  • checksum (str-like) – checksum of the encrypted content of local_file.

new in 0.5.1: checksum

size(remote_file)

Size of a file in bytes.

If the file does not exist, return None instead.

Parameters:remote_file (str) – relative path to a file on the remote host.
Returns:file size in bytes.
Return type:int or None
storageSpace()

Query for how much space is available on the remote host.

Returns:first argument is available space in megabytes (MB), second argument is disk quota in megabytes (MB).
Return type:(int or float, int or None)
touch(remote_file, content=None)

Create an empty file.

Warning

this is different from Unix touch and overwrites existing files instead of updating the last access time attribute.

unlink should raise an error on deleting missing files.

This implementation relies on _pop. As a consequence _pop should raise an error on trying to get a missing file.

updatePlaceholder(remote_file, last_modified=None, checksum=None)

Update a placeholder when the corresponding file is pushed.

This method treats placeholders as files.

To pop or get a file, use markAsRead() instead.

new in 0.5.1: checksum

class escale.relay.LocalMount(client, address, mount_point, **super_args)

Bases: escale.relay.relay.Relay

Add support for local file system (mounts).

_get(relay_file, local_file, makedirs=True)

Download a file and do NOT delete it from the remote host.

Parameters:
  • remote_file (str) – path to a file on the remote host.
  • local_dest (str) – path to a local directory.
  • makedirs (bool) – make directories if missing.
Returns:

True if transfer was successful, False otherwise.
Return type:

bool or nothing
_list(relay_dir='', recursive=True, stats=[])

List all files, including hidden files, relative to remote_dir. All paths are relative to the repository root.

Parameters:
  • remote_dir (str) – directory on the remote host.
  • recursive (bool) – if True, list files in subdirectories as well.
  • stats (list) – can be [ ‘mtime’ ].
Returns:

list of paths.
Return type:

iterator or list of str
_push(local_file, relay_dest, makedirs=True)

Send a local file to the remote host.

Parameters:
  • local_file (str) – path to a local file.
  • remote_dest (str) – path to a file on the remote host.
Returns:

True if transfer was successful, False otherwise.
Return type:

bool or nothing
open()

Establish the connection with the remote host.

Returns:True if successful, False if failed.
Return type:bool
purge(relay_dir='')

Delete a remote directory or collection, and its content.

This method is called by test routines with default argument to purge the entire repository on the relay host.

Parameters:remote_dir (str) – path to the remote directory to be removed.
storageSpace()

Query for how much space is available on the remote host.

Returns:first argument is available space in megabytes (MB), second argument is disk quota in megabytes (MB).
Return type:(int or float, int or None)

unlink should raise an error on deleting missing files.

This implementation relies on _pop. As a consequence _pop should raise an error on trying to get a missing file.

class escale.relay.FTP(client, address, repository, username=None, password=None, protocol=None, encoding='utf-8', account=None, keyfile=None, certfile=None, context=None, certificate=None, verify_ssl=None, ssl_version=None, **super_args)

Bases: escale.relay.relay.Relay

Add support for FTP remote hosts on top of the ftplib standard library.

Tested with pure-ftpd, vsftpd and proftpd.

username

FTP username.

Type:str
password

FTP password.

Type:str
protocol

either ‘ftp’ or ‘ftps’ (case insensitive).

Type:str
encoding

encoding for file names.

Type:str
account

acct argument for ftplib.FTP and ftplib.FTP_TLS.

Type:?
certificate

path to .pem certificate file, or pair of paths (.cert.pem, .key.pem).

Type:str
certfile

path to .cert.pem certificate file.

Type:str
keyfile

path to .key.pem private key file; requires certfile to be defined.

Type:str
ssl_version

SSL version as supported by parse_ssl_version().

Type:int or str
verify_ssl

if True check server’s certificate; if None check certificate if any; if False do not check certificate.

Type:bool
_get(remote_file, local_file, makedirs=True)

Download a file and do NOT delete it from the remote host.

Parameters:
  • remote_file (str) – path to a file on the remote host.
  • local_dest (str) – path to a local directory.
  • makedirs (bool) – make directories if missing.
Returns:

True if transfer was successful, False otherwise.
Return type:

bool or nothing
_list(remote_dir='', recursive=True, stats=[])

List all files, including hidden files, relative to remote_dir. All paths are relative to the repository root.

Parameters:
  • remote_dir (str) – directory on the remote host.
  • recursive (bool) – if True, list files in subdirectories as well.
  • stats (list) – can be [ ‘mtime’ ].
Returns:

list of paths.
Return type:

iterator or list of str
_push(local_file, remote_dest, makedirs=True)

Send a local file to the remote host.

Parameters:
  • local_file (str) – path to a local file.
  • remote_dest (str) – path to a file on the remote host.
Returns:

True if transfer was successful, False otherwise.
Return type:

bool or nothing
_request(callback, *args, **kwargs)

Wrap method calls on object of type ftplib.FTP. Reconnect to server if connection has timed out.

There is no need to wrap successive calls unless some significant amount of time can be spent between these calls. Wrap only the first call.

close()

Close the connection with the remote host.

Returns:True if successful, False if failed.
Return type:bool
open()

Establish the connection with the remote host.

Returns:True if successful, False if failed.
Return type:bool
purge(remote_dir='')

Delete a remote directory or collection, and its content.

This method is called by test routines with default argument to purge the entire repository on the relay host.

Parameters:remote_dir (str) – path to the remote directory to be removed.
size(remote_file, fail=False)

Size of a file in bytes.

If the file does not exist, return None instead.

Parameters:remote_file (str) – relative path to a file on the remote host.
Returns:file size in bytes.
Return type:int or None
storageSpace()

Query for how much space is available on the remote host.

Returns:first argument is available space in megabytes (MB), second argument is disk quota in megabytes (MB).
Return type:(int or float, int or None)

unlink should raise an error on deleting missing files.

This implementation relies on _pop. As a consequence _pop should raise an error on trying to get a missing file.

class escale.relay.SSH(client, address, repository, logger=None, **ignored)

Bases: escale.relay.relay.Relay

NOT IMPLEMENTED YET

SSH support is suspended as commented in issue #3.

escale.relay.GoogleDrive

alias of escale.relay.google.drive.DriveGoogle

class escale.relay.RClone(client, remote, repository, rclone_bin=None, config={}, **super_args)

Bases: escale.relay.relay.Relay

Implements Relay for the various protocols supported by rclone.

_list(remote_dir='', recursive=True, stats=[])
_pop(remote_file, local_file, makedirs=True, _unlink=True)

Download a file and delete it from the remote host.

Note

_pop() can be implemented with an extra _unlink keyword argument that is not supported by default and makes the default implementation for _get() valid.

Parameters:
  • remote_file (str) – path to a file on the remote host.
  • local_dest (str) – path to a local directory.
  • makedirs (bool) – make directories if missing.
  • _unlink (bool, optional) – if False, do not delete the file from the remote host. This keyword argument may not be recognized at all!
Returns:

True if transfer was successful, False otherwise.
Return type:

bool or nothing
_push(local_file, remote_file, makedirs=True)

makedirs is ignored (always True).

purge(remote_dir='')

Delete a remote directory or collection, and its content.

This method is called by test routines with default argument to purge the entire repository on the relay host.

Parameters:remote_dir (str) – path to the remote directory to be removed.
storageSpace()

unlink should raise an error on deleting missing files.

This implementation relies on _pop. As a consequence _pop should raise an error on trying to get a missing file.

escale.relay.info module

class escale.relay.info.LockInfo(version=None, owner=None, target=None, mode=None)

Bases: object

mode
owner
target
version
class escale.relay.info.Metadata(version=None, target=None, pusher=None, timestamp=None, timestamp_format=None, checksum=None, parts=None, pullers=[], **ignored)

Bases: object

checksum
fileModified(local_file=None, last_modified=None, checksum=None, hash_function=None, remote=False, debug=None)

Tell whether a file has been modified.

Parameters:
  • local_file (str) – local file path; file must have a valid last modification time.
  • last_modified (int) – last modification time (local).
  • checksum (str-like) – checksum of file content (local).
  • hash_function (callable) – hash function that can be applied to the content of the local_file file if checksum is not defined.
  • remote (bool) – if True, fileModified tells whether or not the remote copy of the file is a modified version of the local file; if False, fileModified tells whether or not the local file is a modified version of the remote copy of the file; if None, fileModified tells whether there is any difference.
Returns:

True if file has been modified.
Return type:

bool
header
ignored
part_count
parts
pullers
pusher
reader_count
target
timestamp
timestamp_format
version
escale.relay.info.parse_lock_file(file, target=None)
escale.relay.info.parse_metadata(lines, target=None, timestamp_format=None, log=None)

escale.relay.relay module

class escale.relay.relay.AbstractRelay(client, address, repository, logger=None, ui_controller=None)

Bases: escale.base.essential.Reporter

Send files to/from a remote host.

This class is an interface that groups together the methods called by escale.manager.Manager.

client

client identifier.

Type:str
address

address of the remote host.

Type:str
repository

path of the repository on the remote host.

Type:str
address
client
close()

Close the connection with the remote host.

Returns:True if successful, False if failed.
Return type:bool
delete(remote_file)

Fake download and delete a remote file.

This operation should mimic pop but deletes the remote file instead of downloading it.

Parameters:remote_file (str) – relative path to the remote file.
Returns:True if successful, False if failed.
Return type:bool
getMetadata(remote_file, output_file=None, timestamp_format=None)

Download meta-information.

If no local file is specified but output_file is True, getMetadata makes and returns a temporary file instead of a Metadata object.

The returned temporary file should be manually unlinked once done with it.

Example:

import os

temporary_file = relay.getMetadata(path_to_remote_file, output_file=True)

with open(temporary_file, 'r') as f:
    # do something with `f`

os.unlink(temporary_file)
Parameters:
  • remote_file (str) – path to regular file (no placeholder or message).
  • output_file (str or bool) – path to local file.
  • timestamp_format (str) – backward-compatibility option for the former metadata format.
Returns:

metadata or path of a local file.
Return type:

Metadata or str

in 0.5.1: getMetaInfo renamed as getMetadata

new in 0.5.1: timestamp_format; getMetadata returns a Metadata object

listCorrupted(remote_dir='', recursive=True)

List the files on the remote host that are corrupted. All paths are relative to the repository root.

Corrupted files are files with a lock owned by self, as identified by the client attribute. If client evaluates to False, corrupted files are returned as an empty list.

Corrupted files are represented by the unreleased locks.

Parameters:
  • remote_dir (str) – remote directory to “ls”.
  • recursive (bool) – whether to list subdirectories or not.
Returns:

list of locks.
Return type:

list of LockInfo
listReady(remote_dir='', recursive=True)

List the files on the remote host that are ready for download. All paths are relative to the repository root.

Parameters:
  • remote_dir (str) – remote directory to “ls”.
  • recursive (bool) – whether to list subdirectories or not.
Returns:

list of paths.
Return type:

list of str
listTransferred(remote_dir='', end2end=True, recursive=True)

List the files on the remote host that have been transferred. All paths are relative to the repository root.

Parameters:
  • remote_dir (str) – remote directory to “ls”.
  • end2end (bool) – if True, list only files which content is no longer available on the remote host.
  • recursive (bool) – whether to list subdirectories or not.
Returns:

list of paths.
Return type:

list of str
open()

Establish the connection with the remote host.

Returns:True if successful, False if failed.
Return type:bool
pop(remote_file, local_dest, placeholder=True, blocking=True, **kwargs)

Download a file from the remote host and unlinks remote copy if relevant.

Parameters:
  • remote_file (str) – relative path to the remote file.
  • local_dest (str) – path to the target local file.
  • placeholder (bool or int) – whether to generate a placeholder file. If an int is given, it specifies the number of pullers (downloading clients).
  • blocking (bool) – if target exists and is locked, whether should we block until the lock is released or skip the file.
Returns:

True if successful, False if failed.
Return type:

bool
purge(remote_dest='')

Delete a remote directory or collection, and its content.

This method is called by test routines with default argument to purge the entire repository on the relay host.

Parameters:remote_dir (str) – path to the remote directory to be removed.
push(local_file, remote_dest, last_modified=None, checksum=None, blocking=True)

Upload a file to the remote host.

Parameters:
  • local_file (str) – path to the local file to be sent.
  • remote_dest (str) – path to the target remote file.
  • last_modified (str) – meta information to be recorded for the remote copy.
  • checksum (str-like) – checksum of the encrypted content of local_file.
  • blocking (bool) – if target exists and is locked, whether should we block until the lock is released or skip the file.
Returns:

True if successful, False if failed.
Return type:

bool

new in 0.5.1: checksum

remoteListing()

Crawl the relay repository for later calls to list* methods.

remoteListing should be optional and all the list* methods should work with or without past calls to this method.

repair(lock, local_file, checksum=None)

Attempt to repair a corrupted file.

Parameters:
  • lock (LockInfo) – lock information for the corrupted file; the target attribute refers to the file on the remote host.
  • local_file (escale.manager.Accessor) – accessor for local file.
  • checksum (str-like) – checksum of the encrypted content of local_file.

new in 0.5.1: checksum

repository
storageSpace()

Query for how much space is available on the remote host.

Returns:first argument is available space in megabytes (MB), second argument is disk quota in megabytes (MB).
Return type:(int or float, int or None)
class escale.relay.relay.IRelay(client, address, repository, logger=None, ui_controller=None, lock_timeout=True, timestamped_messages=False, **ignored)

Bases: escale.relay.relay.Relay

Extend Relay with alternative implementation for hasPlaceholder() and hasLock().

This class is useful if placeholders (respectively locks) are available together in a list.

“I” stands for “individual” because placeholders and locks are individual entities and can be manipulated as a file, in addition to be records in a list.

The listPlaceholders() and listLocks() methods should be overloaded.

hasLock(remote_file)

Checks for lock presence.

The default implementation manipulates locks as individual files.

Parameters:remote_file (str) – relative path to a regular file on the remote host.
Returns:
True if there exists a lock for remote_file,
False otherwise.
Return type:bool
hasPlaceholder(remote_file)

Checks for placeholder presence.

The default implementation manipulates placeholders as individual files.

Parameters:remote_file (str) – relative path to a regular file on the remote host.
Returns:
True if there exists a placeholder for remote_file,
False otherwise.
Return type:bool
listLocks(remote_dir)

List locks.

Parameters:remote_dir (str) – path to a remote directory.
Returns:list of paths to locks.
Return type:list of str
listPlaceholders(remote_dir)

List placeholders.

Parameters:remote_dir (str) – path to a remote directory.
Returns:list of paths to placeholders.
Return type:list of str
class escale.relay.relay.PRelay(client, address, repository, logger=None, ui_controller=None, lock_timeout=True, timestamped_messages=False, **ignored)

Bases: escale.relay.relay.Relay

Extend Relay with alternative implementation for hasPlaceholder() and hasLock().

This class is useful if “placeheld” (respectively locked) files are available together in a list.

“P” stands for “page” and refers to the fact that placeholders and locks live as records. They may not be actual files.

The listPlaceheld() and listLocked() methods should be overloaded.

hasLock(remote_file)

Checks for lock presence.

The default implementation manipulates locks as individual files.

Parameters:remote_file (str) – relative path to a regular file on the remote host.
Returns:
True if there exists a lock for remote_file,
False otherwise.
Return type:bool
hasPlaceholder(remote_file)

Checks for placeholder presence.

The default implementation manipulates placeholders as individual files.

Parameters:remote_file (str) – relative path to a regular file on the remote host.
Returns:
True if there exists a placeholder for remote_file,
False otherwise.
Return type:bool
listLocked(remote_dir)

List locked files.

Parameters:remote_dir (str) – path to a remote directory.
Returns:list of paths to locked files.
Return type:list of str
listPlaceheld(remote_dir)

List placeheld files.

Parameters:remote_dir (str) – path to a remote directory.
Returns:list of paths to placeheld files.
Return type:list of str
class escale.relay.relay.Relay(client, address, repository, logger=None, ui_controller=None, lock_timeout=True, timestamped_messages=False, **ignored)

Bases: escale.relay.relay.AbstractRelay

Send files to/from a remote relay.

This class is a partial implementation of AbstractRelay. Especially, it implements independent placeholders and locks.

Placeholders and locks are named after the corresponding regular files. Extra strings are prepended and appended to the regular filename.

Messages are similar to placeholders and locks. They act as actual placeholders for regular files when these regular files have to be deleted before they could be downloaded by all the pullers.

Messages are an experimental feature and the clearing mechanism is not fully implemented yet.

Placeholders, locks and messages are represented as filenames. They can be referred to as special files, while transferred files are referred to as regular files.

Any derivative class should implement:

_temporary_file

list of paths to existing temporary files.

Type:list
_placeholder_prefix

prefix for placeholder files.

Type:str
_placeholder_suffix

suffix for placeholder files.

Type:str
_lock_prefix

prefix for lock files.

Type:str
_lock_suffix

suffix for lock files.

Type:str
lock_timeout

maximum age of an unclaimed lock in seconds.

Type:bool or int
_message_hash

generates a message subextension (str); takes the path to the corresponding local regular file as input argument (str); a subextension should not contain ‘.’.

Type:function or None
_message_prefix

prefix for message files.

Type:str
_message_suffix

suffix for message files.

Type:str
placeholder_cache

dictionnary of cached placeholders.

Type:dict

new in 0.5.1: placeholder_cache

as of 0.7.6: default lock_timeout is 3 days

_fromLock(filename)
_fromMessage(filename)
_fromPlaceholder(filename)
_fromSpecial(filename)
_get(remote_file, local_dest, makedirs=True)

Download a file and do NOT delete it from the remote host.

Parameters:
  • remote_file (str) – path to a file on the remote host.
  • local_dest (str) – path to a local directory.
  • makedirs (bool) – make directories if missing.
Returns:

True if transfer was successful, False otherwise.
Return type:

bool or nothing
_isLock(filename)
_isMessage(filename)
_isPlaceholder(filename)
_isSpecial(filename)
_list(remote_dir='', recursive=True, stats=[])

List all files, including hidden files, relative to remote_dir. All paths are relative to the repository root.

Parameters:
  • remote_dir (str) – directory on the remote host.
  • recursive (bool) – if True, list files in subdirectories as well.
  • stats (list) – can be [ ‘mtime’ ].
Returns:

list of paths.
Return type:

iterator or list of str
_lock(filename)
_lock_prefix
_lock_suffix
_message(filename, filepath)
_message_hash
_message_prefix
_message_suffix
_placeholder(filename)
_placeholder_prefix
_placeholder_suffix
_pop(remote_file, local_dest, makedirs=True)

Download a file and delete it from the remote host.

Note

_pop() can be implemented with an extra _unlink keyword argument that is not supported by default and makes the default implementation for _get() valid.

Parameters:
  • remote_file (str) – path to a file on the remote host.
  • local_dest (str) – path to a local directory.
  • makedirs (bool) – make directories if missing.
  • _unlink (bool, optional) – if False, do not delete the file from the remote host. This keyword argument may not be recognized at all!
Returns:

True if transfer was successful, False otherwise.
Return type:

bool or nothing
_push(local_file, remote_dest)

Send a local file to the remote host.

Parameters:
  • local_file (str) – path to a local file.
  • remote_dest (str) – path to a file on the remote host.
Returns:

True if transfer was successful, False otherwise.
Return type:

bool or nothing
_safeLock(filename)
_safeMessage(filename, filepath)
_safePlaceholder(filename)
_temporary_files
acquireLock(remote_file, mode=None, blocking=True)

This method treats locks as files.

close()

Close the connection with the remote host.

Returns:True if successful, False if failed.
Return type:bool
delTemporaryFile(f)

Delete an existing temporary file.

Parameters:file (str) – path to temporary file.
delete(remote_file, blocking=True, **kwargs)

Fake download and delete a remote file.

This operation should mimic pop but deletes the remote file instead of downloading it.

Parameters:remote_file (str) – relative path to the remote file.
Returns:True if successful, False if failed.
Return type:bool
exists(filename, dirname=None)
fromLock(path)
fromMessage(path)
fromPlaceholder(path)
fromSpecial(path)
get(remote_file, local_dest, placeholder=True, blocking=True, **kwargs)
getLockInfo(remote_file)

This method treats locks as files.

It should NOT raise any exception with missing locks, but return an empty LockInfo instead.

getMetadata(remote_file, output_file=None, timestamp_format=None)

This method treats placeholders as files.

hasLock(remote_file)

Checks for lock presence.

The default implementation manipulates locks as individual files.

Parameters:remote_file (str) – relative path to a regular file on the remote host.
Returns:
True if there exists a lock for remote_file,
False otherwise.
Return type:bool
hasPlaceholder(remote_file)

Checks for placeholder presence.

The default implementation manipulates placeholders as individual files.

Parameters:remote_file (str) – relative path to a regular file on the remote host.
Returns:
True if there exists a placeholder for remote_file,
False otherwise.
Return type:bool
isLock(path)
isMessage(path)
isPlaceholder(path)
isSpecial(path)
listCorrupted(remote_dir='', recursive=True)

The default implementation manipulates locks as individual files.

listReady(remote_dir='', recursive=True)

The default implementation manipulates placeholders and locks as individual files.

It caches last modification times of placeholders for future getMetadata calls.

listTransferred(remote_dir='', end2end=True, recursive=True)

The default implementation manipulates placeholders and locks as individual files.

listing_cache
lock(path)
lock_timeout
markAsRead(remote_file, local_placeholder=None)

This method treats placeholders as files.

Compatible with both old-style and new-style placeholders.

message(path)
newTemporaryFile()

Make a new temporary file.

Returns:path to temporary file.
Return type:str
open()

Establish the connection with the remote host.

Returns:True if successful, False if failed.
Return type:bool
placeholder(path)
placeholder_cache
pop(remote_file, local_dest, placeholder=True, blocking=True, **kwargs)

Download a file from the remote host and unlinks remote copy if relevant.

Parameters:
  • remote_file (str) – relative path to the remote file.
  • local_dest (str) – path to the target local file.
  • placeholder (bool or int) – whether to generate a placeholder file. If an int is given, it specifies the number of pullers (downloading clients).
  • blocking (bool) – if target exists and is locked, whether should we block until the lock is released or skip the file.
Returns:

True if successful, False if failed.
Return type:

bool
push(local_file, remote_dest, last_modified=None, checksum=None, blocking=True)

Upload a file to the remote host.

Parameters:
  • local_file (str) – path to the local file to be sent.
  • remote_dest (str) – path to the target remote file.
  • last_modified (str) – meta information to be recorded for the remote copy.
  • checksum (str-like) – checksum of the encrypted content of local_file.
  • blocking (bool) – if target exists and is locked, whether should we block until the lock is released or skip the file.
Returns:

True if successful, False if failed.
Return type:

bool

new in 0.5.1: checksum

releaseLock(remote_file)

This method treats locks as files.

releasePlace(remote_file, handle_missing=False)

This method treats placeholders as files.

remoteListing()

Crawl the relay repository for later calls to list* methods.

remoteListing should be optional and all the list* methods should work with or without past calls to this method.

repair(lock, local_file, checksum=None)

Attempt to repair a corrupted file.

Parameters:
  • lock (LockInfo) – lock information for the corrupted file; the target attribute refers to the file on the remote host.
  • local_file (escale.manager.Accessor) – accessor for local file.
  • checksum (str-like) – checksum of the encrypted content of local_file.

new in 0.5.1: checksum

size(remote_file)

Size of a file in bytes.

If the file does not exist, return None instead.

Parameters:remote_file (str) – relative path to a file on the remote host.
Returns:file size in bytes.
Return type:int or None
storageSpace()

Query for how much space is available on the remote host.

Returns:first argument is available space in megabytes (MB), second argument is disk quota in megabytes (MB).
Return type:(int or float, int or None)
touch(remote_file, content=None)

Create an empty file.

Warning

this is different from Unix touch and overwrites existing files instead of updating the last access time attribute.

unlink should raise an error on deleting missing files.

This implementation relies on _pop. As a consequence _pop should raise an error on trying to get a missing file.

updatePlaceholder(remote_file, last_modified=None, checksum=None)

Update a placeholder when the corresponding file is pushed.

This method treats placeholders as files.

To pop or get a file, use markAsRead() instead.

new in 0.5.1: checksum

escale.relay.relay.with_path(path, fun, *args, **kwargs)

Helper function that applies a string manipulation function to the filename part of a path.

escale.relay.index module

class escale.relay.index.AbstractIndexRelay(client, address, repository, logger=None, ui_controller=None)

Bases: escale.relay.relay.AbstractRelay

acquirePageLock(page, mode=None)
clearIndex(page=None)
consumeUpdate(page, terminate)
getPageIndex(page)
getUpdate(page, terminate=None, full_index=False)
getUpdateData(page, destination)
getUpdateIndex(page, sync=True)
hasIndex(page)
hasUpdate(page)
listPages()
loaded(page)
releasePageLock(page)
repairUpdates()
requestMissing(page)
setUpdate(page)
setUpdateData(page, data)
setUpdateIndex(page, index, sync=True)
class escale.relay.index.IndexRelay(*args, **kwargs)

Bases: escale.relay.index.AbstractIndexRelay

Index-based relay.

RelayIndex plays both AbstractRelay and (part of) Manager roles.

Repository content is indexed. It consists of a comprehensive index and an update.

An update stores files in a compressed archive and is accompanied by an index that lists the content of the archive.

The total archive is encrypted instead of the individual files.

Indexing potentially supports paging. See also the page() method.

_force(operation, target, func, *args, **kwargs)
acquirePageLock(page, mode)
address
allPages()
clearIndex(page=None)
client
close()

Close the connection with the remote host.

Returns:True if successful, False if failed.
Return type:bool
consumeUpdate(page, terminate=None)
getIndexChanges(page, sync=True, check_mtime=False)
getMetadata(remote_file, *args, **kwargs)

Download meta-information.

If no local file is specified but output_file is True, getMetadata makes and returns a temporary file instead of a Metadata object.

The returned temporary file should be manually unlinked once done with it.

Example:

import os

temporary_file = relay.getMetadata(path_to_remote_file, output_file=True)

with open(temporary_file, 'r') as f:
    # do something with `f`

os.unlink(temporary_file)
Parameters:
  • remote_file (str) – path to regular file (no placeholder or message).
  • output_file (str or bool) – path to local file.
  • timestamp_format (str) – backward-compatibility option for the former metadata format.
Returns:

metadata or path of a local file.
Return type:

Metadata or str

in 0.5.1: getMetaInfo renamed as getMetadata

new in 0.5.1: timestamp_format; getMetadata returns a Metadata object

getPageIndex(page)
getUpdateData(page, destination)
getUpdateIndex(page, sync=True)
hasIndex(page)
hasLock(page)
hasUpdate(page)
indexed(remote_file)
listCorrupted(remote_dir='', recursive=True)

List the files on the remote host that are corrupted. All paths are relative to the repository root.

Corrupted files are files with a lock owned by self, as identified by the client attribute. If client evaluates to False, corrupted files are returned as an empty list.

Corrupted files are represented by the unreleased locks.

Parameters:
  • remote_dir (str) – remote directory to “ls”.
  • recursive (bool) – whether to list subdirectories or not.
Returns:

list of locks.
Return type:

list of LockInfo
listPages(remote_dir='', recent_only=False)
listReady(remote_dir='', recursive=True)

List the files on the remote host that are ready for download. All paths are relative to the repository root.

Parameters:
  • remote_dir (str) – remote directory to “ls”.
  • recursive (bool) – whether to list subdirectories or not.
Returns:

list of paths.
Return type:

list of str
listTransferred(remote_dir='', end2end=True, recursive=True)

List the files on the remote host that have been transferred. All paths are relative to the repository root.

Parameters:
  • remote_dir (str) – remote directory to “ls”.
  • end2end (bool) – if True, list only files which content is no longer available on the remote host.
  • recursive (bool) – whether to list subdirectories or not.
Returns:

list of paths.
Return type:

list of str
listing_cache
loaded(page, mtime=None, check_mtime=True)
logger
open()

Establish the connection with the remote host.

Returns:True if successful, False if failed.
Return type:bool
page(remote_file)

Page key/name.

Hash function for paging of files by remote path.

Parameters:remote_file (str) – path to file.
Returns:page key/name.
Return type:str
persistentIndex(page)
refreshListing(remote_dir='', force=False)
releasePageLock(page)
remoteListing()

new in 0.7.13: remoteListing returns the list of recently modified entries (more recent then in cache) if any, else the entire listing.

repair(*args, **kwargs)

Attempt to repair a corrupted file.

Parameters:
  • lock (LockInfo) – lock information for the corrupted file; the target attribute refers to the file on the remote host.
  • local_file (escale.manager.Accessor) – accessor for local file.
  • checksum (str-like) – checksum of the encrypted content of local_file.

new in 0.5.1: checksum

repairUpdates()
repository
requestMissing(page, remote_files)
sanityChecks(page)
setPageIndex(page, index)
setUpdateData(page, data)
setUpdateIndex(page, index, sync=True)
skipIndexRelated(files_or_locks)
storageSpace()

Query for how much space is available on the remote host.

Returns:first argument is available space in megabytes (MB), second argument is disk quota in megabytes (MB).
Return type:(int or float, int or None)
tryAcquirePageLock(page, mode)

Non-blocking equivalent of acquirePageLock.

ui_controller
updateData(page, mode=None)
updateIndex(page, mode=None)
updateRelated(page, filename)
updateTimestamp(page, mode=None)
class escale.relay.index.IndexUpdate(relay, page, mode)

Bases: _abcoll.MutableMapping

_abc_cache = <_weakrefset.WeakSet object>
_abc_negative_cache = <_weakrefset.WeakSet object>
_abc_negative_cache_version = 31
_abc_registry = <_weakrefset.WeakSet object>
logger
class escale.relay.index.TopDirectoriesIndex(*args, **kwargs)

Bases: escale.relay.index.IndexRelay

allPages()
page(resource)

Page key/name.

Hash function for paging of files by remote path.

Parameters:remote_file (str) – path to file.
Returns:page key/name.
Return type:str
class escale.relay.index.UpdateRead(relay, page, terminate, full_index=False)

Bases: escale.relay.index.IndexUpdate

_abc_cache = <_weakrefset.WeakSet object>
_abc_negative_cache = <_weakrefset.WeakSet object>
_abc_negative_cache_version = 31
_abc_registry = <_weakrefset.WeakSet object>
class escale.relay.index.UpdateWrite(relay, page)

Bases: escale.relay.index.IndexUpdate

_abc_cache = <_weakrefset.WeakSet object>
_abc_negative_cache = <_weakrefset.WeakSet object>
_abc_negative_cache_version = 31
_abc_registry = <_weakrefset.WeakSet object>
escale.relay.index.read_index(filename, compress=False, groupby=[], debug=None)

Read index from file.

escale.relay.index.write_index(filename, metadata, pullers=[], compress=False, groupby=[])

Write index to file.

escale.relay.localmount module

class escale.relay.localmount.LocalMount(client, address, mount_point, **super_args)

Bases: escale.relay.relay.Relay

Add support for local file system (mounts).

exists(relay_file, dirname=None)
open()

Establish the connection with the remote host.

Returns:True if successful, False if failed.
Return type:bool
purge(relay_dir='')

Delete a remote directory or collection, and its content.

This method is called by test routines with default argument to purge the entire repository on the relay host.

Parameters:remote_dir (str) – path to the remote directory to be removed.
storageSpace()

Query for how much space is available on the remote host.

Returns:first argument is available space in megabytes (MB), second argument is disk quota in megabytes (MB).
Return type:(int or float, int or None)

unlink should raise an error on deleting missing files.

This implementation relies on _pop. As a consequence _pop should raise an error on trying to get a missing file.

escale.relay.ftp module

class escale.relay.ftp.FTP(client, address, repository, username=None, password=None, protocol=None, encoding='utf-8', account=None, keyfile=None, certfile=None, context=None, certificate=None, verify_ssl=None, ssl_version=None, **super_args)

Bases: escale.relay.relay.Relay

Add support for FTP remote hosts on top of the ftplib standard library.

Tested with pure-ftpd, vsftpd and proftpd.

username

FTP username.

Type:str
password

FTP password.

Type:str
protocol

either ‘ftp’ or ‘ftps’ (case insensitive).

Type:str
encoding

encoding for file names.

Type:str
account

acct argument for ftplib.FTP and ftplib.FTP_TLS.

Type:?
certificate

path to .pem certificate file, or pair of paths (.cert.pem, .key.pem).

Type:str
certfile

path to .cert.pem certificate file.

Type:str
keyfile

path to .key.pem private key file; requires certfile to be defined.

Type:str
ssl_version

SSL version as supported by parse_ssl_version().

Type:int or str
verify_ssl

if True check server’s certificate; if None check certificate if any; if False do not check certificate.

Type:bool
certfile
close()

Close the connection with the remote host.

Returns:True if successful, False if failed.
Return type:bool
encoding
exists(remote_file, dirname=None)
keyfile
login()
open()

Establish the connection with the remote host.

Returns:True if successful, False if failed.
Return type:bool
purge(remote_dir='')

Delete a remote directory or collection, and its content.

This method is called by test routines with default argument to purge the entire repository on the relay host.

Parameters:remote_dir (str) – path to the remote directory to be removed.
size(remote_file, fail=False)

Size of a file in bytes.

If the file does not exist, return None instead.

Parameters:remote_file (str) – relative path to a file on the remote host.
Returns:file size in bytes.
Return type:int or None
storageSpace()

Query for how much space is available on the remote host.

Returns:first argument is available space in megabytes (MB), second argument is disk quota in megabytes (MB).
Return type:(int or float, int or None)

unlink should raise an error on deleting missing files.

This implementation relies on _pop. As a consequence _pop should raise an error on trying to get a missing file.

escale.relay.webdav module

escale.relay.google module

escale.relay.google.GoogleDrive

alias of escale.relay.google.drive.DriveGoogle

escale.relay.google.drive module

class escale.relay.google.drive.DriveGoogle(client, mount_point, repository, drive_bin=None, config={}, **super_args)

Bases: escale.relay.relay.Relay

Implements Relay for Google Drive with drive.

This backend is shiped together with the escale.cli.config.googledrive configuration helper.

close(enforce=False)

Close the connection with the remote host.

Returns:True if successful, False if failed.
Return type:bool
exists(remote_file, dirname=None)
mount_point
open()

Establish the connection with the remote host.

Returns:True if successful, False if failed.
Return type:bool
pull_extra_arguments
purge(remote_dir='')

Delete a remote directory or collection, and its content.

This method is called by test routines with default argument to purge the entire repository on the relay host.

Parameters:remote_dir (str) – path to the remote directory to be removed.
push_extra_arguments
storageSpace()

Warning

reads used and total spaces of the entire cloud space!

unlink should raise an error on deleting missing files.

This implementation relies on _pop. As a consequence _pop should raise an error on trying to get a missing file.

escale.relay.google.drive.drive_binary(name=None)

escale.relay.generic module

class escale.relay.generic.RClone(client, remote, repository, rclone_bin=None, config={}, **super_args)

Bases: escale.relay.relay.Relay

Implements Relay for the various protocols supported by rclone.

exists(remote_file, dirname=None)
purge(remote_dir='')

Delete a remote directory or collection, and its content.

This method is called by test routines with default argument to purge the entire repository on the relay host.

Parameters:remote_dir (str) – path to the remote directory to be removed.
remote
storageSpace()

unlink should raise an error on deleting missing files.

This implementation relies on _pop. As a consequence _pop should raise an error on trying to get a missing file.

escale.relay.generic.rclone module

class escale.relay.generic.rclone.RClone(client, remote, repository, rclone_bin=None, config={}, **super_args)

Bases: escale.relay.relay.Relay

Implements Relay for the various protocols supported by rclone.

exists(remote_file, dirname=None)
purge(remote_dir='')

Delete a remote directory or collection, and its content.

This method is called by test routines with default argument to purge the entire repository on the relay host.

Parameters:remote_dir (str) – path to the remote directory to be removed.
remote
storageSpace()

unlink should raise an error on deleting missing files.

This implementation relies on _pop. As a consequence _pop should raise an error on trying to get a missing file.

escale.relay.generic.rclone.rclone_binary(name=None)