escale.manager package¶

escale.manager.get_client_name(repository, config={})¶

Read client name from config or fall back to default.

Parameters:	repository (str) – repository name. config (dict) – option-value configuration dictionnary.
Returns:	client name.
Return type:	str

class escale.manager.Manager(relay, repository=None, address=None, directory=None, encryption=<escale.encryption.encryption.Plain object>, timestamp=True, refresh=True, clientname=None, filetype=[], include=None, exclude=None, tq_controller=None, count=None, checksum=True, checksum_cache=None, includedirectory=None, excludedirectory=None, waitonerror=[], verbosity=1, **relay_args)¶

Bases: escale.base.essential.Reporter

Makes the glue between the local file system and the relay and encryption layers.

This class manages the meta information, file modifications and sleep times.

repository¶

local file access controller.

Type:	escale.manager.AccessController

address¶

remote host address.

Type:	str

directory¶

path to the repository on the remote host.

Type:	str

encryption¶

encryption layer.

Type:	escale.encryption.Cipher

relay¶

communication layer.

Type:	escale.relay.AbstractRelay

timestamp¶

if True (recommended), manages file modification times. If str, in addition determines the timestamp format as supported by time.strftime().

Type:	bool or str

checksum¶

hash algorithm name; see also the hashlib library.

Type:	str

checksum_cache¶

path to a checksum cache file (str) or cache (dict) of checksums for local files for relative filepaths as keys and (last modification time, checksum) pairs as elements. If bool, then the cache path will be automatically determined.

Type:	bool or str or dict

filetype¶

list of file extensions.

Type:	list of str

include¶

regular expressions to include files by name.

Type:	list of str

exclude¶

regular expressions to exclude files by name.

Type:	list of str

tq_controller¶

time and quota controller.

Type:	escale.manager.TimeQuotaController

count¶

puller count.

Type:	int

includedirectory¶

regular expressions to include directories by name.

Type:	list of str

excludedirectory¶

regular expressions to exclude directories by name.

Type:	list of str

waitonerror¶

error codes that trigger wait and recovery.

Type:	list of int

verbosity¶

2 or higher makes Escale so verbose that it can make the entire OS freeze.

Type:	int

relay_args¶

extra keyword arguments for pop().

Type:	dict

new in 0.7.1: checksum_cache new in 0.7.4: wait_on_error new in 0.7.6: verbosity

download()¶: Finds out which files are to be downloaded and download them.

filter(files)¶

Applies filters on a list of file paths.

Parameters:	files (list) – list of file paths.
Returns:	list of selected file paths from `files`.
Return type:	list

localFiles(path=None)¶

Transitional method.

Use self.repository.readableFiles instead.

run()¶

Runs the manager.

Example:

from escale.manager import Manager
from escale.relay   import WebDAV

Manager(WebDAV,
    remote_host,
    path_to_local_repository,
    path_to_remote_repository
    ).run()

sanityChecks()¶: Performs sanity checks and fixes the corrupted files.

upload()¶: Finds out which files are to be uploaded and upload them.

class escale.manager.Accessor(exists=None, delete=None)¶

Bases: object

Interface to a single local file.

This class aims at hiding the actual file so that access to it is strictly controlled, while still featuring basic file manipulation methods.

class escale.manager.AccessController(repository, path=None, persistent=None, ui_controller=None, push_only=False, pull_only=False, mode=None, create=False, unsafe=False, verbosity=1, **ignored)¶

Bases: escale.base.essential.Reporter

Manage access to the files in a local repository.

persistent¶

path to persistent data.

Type:	str

repository¶

repository identifier.

Type:	str

path¶

path to repository root.

Type:	str

mode¶

any of download, upload, conservative or share/shared (default).

Type:	str

create¶

if True create persistent data if missing.

Type:	str

unsafe¶

skip all the checks that require accessing the file system.

Type:	bool

verbosity¶

verbosity level; if greater than 2, may cause the OS to freeze.

Type:	int

When push_only (resp. pull_only) is True , mode is upload (resp. download).

When mode is download, upload or shared, and the persistent attributes do not exist at init time (database not created), any external change (e.g. with escalectl) will not be taken into account, unless create is True.

confirmPull(filename)¶: Return a context manager so that permissions can be updated at the beginning or at completion of the transfer.

confirmPush(filename)¶: Return a context manager so that permissions can be updated at the beginning or at completion of the transfer.

listFiles(path=None, basename=None, dirname=None, absolute=False)¶

List all visible files in the local repository.

Files which name begins with “.” are ignored.

Parameters:	path (str) – relative path to recursively crawl from. basename (boolean function) – returns True if the input file basename qualifies. dirname (boolean function) – returns True if the input file directory name qualifies. absolute (bool) – if True, the returned paths are absolute, else relative.
Returns:	list of local files.
Return type:	list of str

readable(files, unsafe=False)¶

Select filenames from the repository that can be uploaded.

Parameters:	files (list) – list of relative paths.
Returns:	uploadable filenames.
Return type:	list

writable(filename, absolute=True)¶

Get the local path corresponding to a remote resource if it can be downloaded.

Parameters:	filename (str) – path to remote file. absolute (bool) – if True, the returned path is absolute, else relative.
Returns:	local path, or `None` if the local resource is not writable.
Return type:	str or None

class escale.manager.TimeQuotaController(refresh=True, quota=None, quota_read_interval=None, quota_read_callback=None, logger=None)¶

Bases: object

Time and quota controller.

refresh (int): maximum refresh interval in seconds.

quota (int or float): maximum used space on relay host, in MB.

class escale.manager.History(repository=None, persistent=None, **super_args)¶: Bases: escale.manager.history.TimeQuotaController

escale.manager.config cache¶

class escale.manager.cache.ChecksumCache(cache)¶: Bases: dict

escale.manager.cache.asbinary(s)¶

escale.manager.cache.find_checksum_cache(section, config=None)¶

escale.manager.cache.read_checksum_cache(path, log=None)¶

deprecated

Rename old checksum cache files appending .old at the end. This function will convert the old cache to the new format.

escale.manager.cache.write_checksum_cache(path, cache, log=None)¶: deprecated

escale.manager.config module¶

escale.manager.config.get_cache_file(config=None, section=None, prefix='', previously=None)¶

Get the corresponding persistent data location for a given configuration file and section.

Parameters:	config (ConfigParser) – configuration object or filepath. section (str) – section/repository name. prefix (str) – prefixes the basename of the returned path. previously (str) – former section/repository name if it has been renamed.
Returns:	path to persistent data location.
Return type:	str

new in 0.5: moved from escale.base.config

escale.manager.config.get_client_name(repository, config={})¶

Read client name from config or fall back to default.

Parameters:	repository (str) – repository name. config (dict) – option-value configuration dictionnary.
Returns:	client name.
Return type:	str

escale.manager.config.get_dist_file(default_dirs={}, filename=None, config=None, section=None, options=None)¶

Get path of “distribution” files such as caches and locks.

Parameters:	default_dirs (dict) – mapping configuration filepath -> distribution filepath. filename (str) – basename. config (ConfigParser or str) – configuration filepath or object. section (str) – repository name. options (str or tuple or list) – configuration option(s).
Returns:	filepath.
Return type:	str

new in 0.5: moved from escale.base.config

The “dist” directory is determined from the configuration object if any, or from the configuration filepath combined with the default_dirs mapping.

If filename is not provided, it is derived from the basename of the configuration file.

escale.manager.config.get_pid_file(config=None)¶

Get the location of the pid (process id) file.

Parameters:	config (ConfigParser) – configuration object or filepath.
Returns:	path to pid file.
Return type:	str

new in 0.5: moved from escale.base.config

escale.manager.config.parse_address(addr, multi_path_protocols=[])¶

Parse host addresses or paths.

Parameters:	addr (str) – host address. multi_path_protocols (list) – list of multi-path protocols.
Returns:	(protocol, address, port, path)
Return type:	(str, str, str, str)

Addresses can have multiple formats:

file://absolutepath
protocol://servername[:port][/relativepath]

Note that absolutepath must start with ‘/’ or ‘~/’ to be identified as an absolute path.

absolutepath and servername are returned as second argument and relativepath as fourth output argument.

new in 0.5: multi_path_protocols; moved from escale.base.config

multi-path protocols admit paths such as:

protocol
protocol://relativepath where relativepath is returned as path
protocol://absolutepath where absolutepath is returned as address
protocol://absolutepath//relativepath

Note that absolutepath must start with ‘/’ or ‘~/’ to be identified as an absolute path.

Similarly relativepath should not start with any of ‘/’ or ‘~/’.

Example multi-path address:

googledrive:///home/user/MyGoogleDrive//Escale repository

where /home/user/MyGoogleDrive is the path of the “locally-mounted” storage space and Escale repository is the directory that Escale will use for synchronization.

The above example can be parsed as expected if ‘googledrive’ is provided in the multi_path_protocols input argument.

escale.manager.config.parse_section(config, repository, logger)¶

Make relay backend and parse related options.

Parameters:	config (ConfigParser) – configuration object. repository (str) – configuration section. logger (Logger) – logger.
Returns:	relay class and dictionnary of parameters.
Return type:	(escale.relay.AbstractRelay, dict)

escale.manager.access module¶

class escale.manager.access.AccessAttributes(location=None, dbm_mode='c')¶

Bases: object

getReadability(resource)¶

getWritability(resource)¶

isReadable(resource, make_explicit=None)¶: Determine whether a resource can be uploaded.

isWritable(resource, make_explicit=None)¶: Determine whether a resource can be downloaded.

location¶

setNotReadable(resource)¶

setNotWritable(resource)¶

setReadability(resource, r)¶

setReadable(resource)¶

setWritability(resource, w)¶

setWritable(resource)¶

table(resource)¶

unsetReadability(resource)¶

unsetWritability(resource)¶

class escale.manager.access.AccessController(repository, path=None, persistent=None, ui_controller=None, push_only=False, pull_only=False, mode=None, create=False, unsafe=False, verbosity=1, **ignored)¶

Bases: escale.base.essential.Reporter

Manage access to the files in a local repository.

persistent¶

path to persistent data.

Type:	str

repository¶

repository identifier.

Type:	str

path¶

path to repository root.

Type:	str

mode¶

any of download, upload, conservative or share/shared (default).

Type:	str

create¶

if True create persistent data if missing.

Type:	str

unsafe¶

skip all the checks that require accessing the file system.

Type:	bool

verbosity¶

verbosity level; if greater than 2, may cause the OS to freeze.

Type:	int

When push_only (resp. pull_only) is True , mode is upload (resp. download).

When mode is download, upload or shared, and the persistent attributes do not exist at init time (database not created), any external change (e.g. with escalectl) will not be taken into account, unless create is True.

absolute(filename)¶

accessor(filename)¶

confirmPull(filename)¶: Return a context manager so that permissions can be updated at the beginning or at completion of the transfer.

confirmPush(filename)¶: Return a context manager so that permissions can be updated at the beginning or at completion of the transfer.

getReadability(filename)¶

getWritability(filename)¶

listFiles(path=None, basename=None, dirname=None, absolute=False)¶

List all visible files in the local repository.

Files which name begins with “.” are ignored.

Parameters:	path (str) – relative path to recursively crawl from. basename (boolean function) – returns True if the input file basename qualifies. dirname (boolean function) – returns True if the input file directory name qualifies. absolute (bool) – if True, the returned paths are absolute, else relative.
Returns:	list of local files.
Return type:	list of str

mode

readable(files, unsafe=False)¶

Select filenames from the repository that can be uploaded.

Parameters:	files (list) – list of relative paths.
Returns:	uploadable filenames.
Return type:	list

setReadability(filename, r)¶

setWritability(filename, w)¶

writable(filename, absolute=True)¶

Get the local path corresponding to a remote resource if it can be downloaded.

Parameters:	filename (str) – path to remote file. absolute (bool) – if True, the returned path is absolute, else relative.
Returns:	local path, or `None` if the local resource is not writable.
Return type:	str or None

class escale.manager.access.Accessor(exists=None, delete=None)¶

Bases: object

Interface to a single local file.

This class aims at hiding the actual file so that access to it is strictly controlled, while still featuring basic file manipulation methods.

delete¶

exists¶

class escale.manager.access.ControllerProxy(controller)¶

Bases: object

controller¶

mode¶

persistent¶

exception escale.manager.access.PermissionError¶: Bases: exceptions.OSError

class escale.manager.access.Pull(controller, filename)¶

Bases: escale.manager.access.ControllerProxy

filename¶

class escale.manager.access.Push(controller, filename)¶

Bases: escale.manager.access.ControllerProxy

filename¶

class escale.manager.access.TableEntry(table, key, default=None)¶

Bases: object

default¶

delete()¶

get()¶

key¶

set(value)¶

table¶

escale.manager.access.asbinary(s)¶

escale.manager.history module¶

class escale.manager.history.History(repository=None, persistent=None, **super_args)¶: Bases: escale.manager.history.TimeQuotaController

class escale.manager.history.TimeQuotaController(refresh=True, quota=None, quota_read_interval=None, quota_read_callback=None, logger=None)¶

Bases: object

Time and quota controller.

refresh (int): maximum refresh interval in seconds.

quota (int or float): maximum used space on relay host, in MB.

pull(local_file)¶

push(local_file, callback=None)¶

wait()¶

escale.manager.index module¶

class escale.manager.index.IndexManager(relay, *args, **kwargs)¶

Bases: escale.manager.manager.Manager

download()¶: Finds out which files are to be downloaded and download them.

localFiles(path=None)¶

Transitional method.

Use self.repository.readableFiles instead.

reportTransferred(download_or_upload, transferred_files)¶

sanityChecks()¶: Performs sanity checks and fixes the corrupted files.

shuffle(_list, with_updates_first=False)¶

terminate(pullers)¶

upload()¶: Finds out which files are to be uploaded and upload them.

escale.manager.manager module¶

class escale.manager.manager.Manager(relay, repository=None, address=None, directory=None, encryption=<escale.encryption.encryption.Plain object>, timestamp=True, refresh=True, clientname=None, filetype=[], include=None, exclude=None, tq_controller=None, count=None, checksum=True, checksum_cache=None, includedirectory=None, excludedirectory=None, waitonerror=[], verbosity=1, **relay_args)¶

Bases: escale.base.essential.Reporter

Makes the glue between the local file system and the relay and encryption layers.

This class manages the meta information, file modifications and sleep times.

repository¶

local file access controller.

Type:	escale.manager.AccessController

address¶

remote host address.

Type:	str

directory¶

path to the repository on the remote host.

Type:	str

encryption¶

encryption layer.

Type:	escale.encryption.Cipher

relay¶

communication layer.

Type:	escale.relay.AbstractRelay

timestamp¶

if True (recommended), manages file modification times. If str, in addition determines the timestamp format as supported by time.strftime().

Type:	bool or str

checksum¶

hash algorithm name; see also the hashlib library.

Type:	str

checksum_cache¶

path to a checksum cache file (str) or cache (dict) of checksums for local files for relative filepaths as keys and (last modification time, checksum) pairs as elements. If bool, then the cache path will be automatically determined.

Type:	bool or str or dict

filetype¶

list of file extensions.

Type:	list of str

include¶

regular expressions to include files by name.

Type:	list of str

exclude¶

regular expressions to exclude files by name.

Type:	list of str

tq_controller¶

time and quota controller.

Type:	escale.manager.TimeQuotaController

count¶

puller count.

Type:	int

includedirectory¶

regular expressions to include directories by name.

Type:	list of str

excludedirectory¶

regular expressions to exclude directories by name.

Type:	list of str

waitonerror¶

error codes that trigger wait and recovery.

Type:	list of int

verbosity¶

2 or higher makes Escale so verbose that it can make the entire OS freeze.

Type:	int

relay_args¶

extra keyword arguments for pop().

Type:	dict

new in 0.7.1: checksum_cache new in 0.7.4: wait_on_error new in 0.7.6: verbosity

address

checksum(resource, return_mtime=False)

count

dir¶

download()¶: Finds out which files are to be downloaded and download them.

filter(files)¶

Applies filters on a list of file paths.

Parameters:	files (list) – list of file paths.
Returns:	list of selected file paths from `files`.
Return type:	list

localFiles(path=None)¶

Transitional method.

Use self.repository.readableFiles instead.

mode¶

path¶

remoteListing()¶

run()¶

Runs the manager.

Example:

from escale.manager import Manager
from escale.relay   import WebDAV

Manager(WebDAV,
    remote_host,
    path_to_local_repository,
    path_to_remote_repository
    ).run()

sanityChecks()¶: Performs sanity checks and fixes the corrupted files.

upload()¶: Finds out which files are to be uploaded and upload them.

escale.manager.migration module¶

escale.manager.migration.inter_relay_copy(src_relay, dest_relay, safe=True, overwrite=False, files=[])¶

Transfer files from a source relay to a destination relay.

Files are locally downloaded from the source and then uploaded to the destination one after the other.

Regular files in the source repository are all locked both on the source relay and destination relay before any transfer begins, and they are all released at the very end.

Note

relays should implement Relay instead of AbstractRelay

In a future release, this function will also translate the names of the special files according to prefixes and suffixes defined in the relay objects.

Parameters:	src_relay (escale.relay.Relay) – source opened relay. dest_relay (escale.relay.Relay) – destination opened relay. safe (bool or (bool, bool)) – if `True`, lock the regular files before operating; if a tuple, the first boolean applies to the source relay and the second one applies to the destination relay. overwrite (bool) – overwrite existing locks on destination relay; you should ensure yourself that the destination repository is inactive. files (list of str) – list of paths in the source repository; default behavior consists of transfering the entire repository tree.
Returns:	list of paths for files that could not be successfully transfered.
Return type:	list of str

Note

instead of setting overwrite to True, it would preferable to purge the destination repository.

escale.manager.migration.migrate_repositories_and_update_config(changes, config=None, safe=True, logger=None)¶

Migrate relay repositories and update configuration file.

Parameters:	changes (ConfigParser) – changes to apply to relays. config (str or ConfigParser) – configuration for active relays. safe (bool) – whether to lock all the resources or not. logger (Logger) – logger.

escale.manager.migration.migrate_repository(repository, changes, config=None, safe=True, logger=None, alt_repo=None)¶

Parse the configuration sections for the source and destination relays and transfer the entire repository from the source relay to the destination relay.

Expected options in changes are relay protocol (required), relay address, relay port, path of the relay repository and credentials.

Client name or the encryption parameters cannot be modified at migration time.

Parameters:	repository (str) – configuration section. changes (ConfigParser) – configuration object with a section corresponding to repository. config (str or ConfigParser) – configuration file or object for the source relay. safe (bool or (bool, bool)) – whether to lock all the resources before transfer; the argument is passed directly to inter_relay_copy. logger (Logger) – logger. alt_repo (str) – section to look for in changes instead of repositories.
Returns:	updated configuration object for the migrated and other active relays.
Return type:	ConfigParser

escale.manager.backup module¶

class escale.manager.backup.Backup(directory, **super_args)¶

Bases: escale.relay.localmount.LocalMount

acquireLock(**kwargs)¶: This method treats locks as files.

releaseLock(**kwargs)¶: This method treats locks as files.

escale.manager.backup.backup_manager(archive, repository, backup_or_restore='backup', safe=True, config=None, logger=None)¶

escale.manager.backup.backup_relay_repository(relay, archive, safe=True, logger=None)¶

escale.manager.backup.restore_relay_repository(relay, archive, safe=True, logger=None)¶

Table of Contents

Previous topic

Next topic

This Page

escale.manager package¶

escale.manager.config cache¶

escale.manager.config module¶

escale.manager.access module¶

escale.manager.history module¶

escale.manager.index module¶

escale.manager.manager module¶

escale.manager.migration module¶

escale.manager.backup module¶