escale.manager package¶
-
escale.manager.
get_client_name
(repository, config={})¶ Read client name from config or fall back to default.
Parameters: - repository (str) – repository name.
- config (dict) – option-value configuration dictionnary.
Returns: client name.
Return type: str
-
class
escale.manager.
Manager
(relay, repository=None, address=None, directory=None, encryption=<escale.encryption.encryption.Plain object>, timestamp=True, refresh=True, clientname=None, filetype=[], include=None, exclude=None, tq_controller=None, count=None, checksum=True, checksum_cache=None, includedirectory=None, excludedirectory=None, waitonerror=[], verbosity=1, **relay_args)¶ Bases:
escale.base.essential.Reporter
Makes the glue between the local file system and the
relay
andencryption
layers.This class manages the meta information, file modifications and sleep times.
-
repository
¶ local file access controller.
Type: escale.manager.AccessController
-
address
¶ remote host address.
Type: str
-
directory
¶ path to the repository on the remote host.
Type: str
-
encryption
¶ encryption layer.
Type: escale.encryption.Cipher
-
relay
¶ communication layer.
Type: escale.relay.AbstractRelay
-
timestamp
¶ if
True
(recommended), manages file modification times. If str, in addition determines the timestamp format as supported bytime.strftime()
.Type: bool or str
-
checksum
¶ hash algorithm name; see also the
hashlib
library.Type: str
-
checksum_cache
¶ path to a checksum cache file (str) or cache (dict) of checksums for local files for relative filepaths as keys and (last modification time, checksum) pairs as elements. If bool, then the cache path will be automatically determined.
Type: bool or str or dict
-
filetype
¶ list of file extensions.
Type: list of str
-
include
¶ regular expressions to include files by name.
Type: list of str
-
exclude
¶ regular expressions to exclude files by name.
Type: list of str
-
tq_controller
¶ time and quota controller.
Type: escale.manager.TimeQuotaController
-
count
¶ puller count.
Type: int
-
includedirectory
¶ regular expressions to include directories by name.
Type: list of str
-
excludedirectory
¶ regular expressions to exclude directories by name.
Type: list of str
-
waitonerror
¶ error codes that trigger wait and recovery.
Type: list of int
-
verbosity
¶ 2 or higher makes Escale so verbose that it can make the entire OS freeze.
Type: int
new in 0.7.1: checksum_cache new in 0.7.4: wait_on_error new in 0.7.6: verbosity
-
download
()¶ Finds out which files are to be downloaded and download them.
-
filter
(files)¶ Applies filters on a list of file paths.
Parameters: files (list) – list of file paths. Returns: list of selected file paths from files
.Return type: list
-
localFiles
(path=None)¶ Transitional method.
Use
self.repository.readableFiles
instead.
-
run
()¶ Runs the manager.
Example:
from escale.manager import Manager from escale.relay import WebDAV Manager(WebDAV, remote_host, path_to_local_repository, path_to_remote_repository ).run()
-
sanityChecks
()¶ Performs sanity checks and fixes the corrupted files.
-
upload
()¶ Finds out which files are to be uploaded and upload them.
-
-
class
escale.manager.
Accessor
(exists=None, delete=None)¶ Bases:
object
Interface to a single local file.
This class aims at hiding the actual file so that access to it is strictly controlled, while still featuring basic file manipulation methods.
-
class
escale.manager.
AccessController
(repository, path=None, persistent=None, ui_controller=None, push_only=False, pull_only=False, mode=None, create=False, unsafe=False, verbosity=1, **ignored)¶ Bases:
escale.base.essential.Reporter
Manage access to the files in a local repository.
-
persistent
¶ path to persistent data.
Type: str
-
repository
¶ repository identifier.
Type: str
-
path
¶ path to repository root.
Type: str
-
mode
¶ any of download, upload, conservative or share/shared (default).
Type: str
-
create
¶ if
True
create persistent data if missing.Type: str
-
unsafe
¶ skip all the checks that require accessing the file system.
Type: bool
-
verbosity
¶ verbosity level; if greater than 2, may cause the OS to freeze.
Type: int
When push_only (resp. pull_only) is
True
, mode is upload (resp. download).When mode is download, upload or shared, and the persistent attributes do not exist at init time (database not created), any external change (e.g. with escalectl) will not be taken into account, unless create is
True
.-
confirmPull
(filename)¶ Return a context manager so that permissions can be updated at the beginning or at completion of the transfer.
-
confirmPush
(filename)¶ Return a context manager so that permissions can be updated at the beginning or at completion of the transfer.
-
listFiles
(path=None, basename=None, dirname=None, absolute=False)¶ List all visible files in the local repository.
Files which name begins with “.” are ignored.
Parameters: - path (str) – relative path to recursively crawl from.
- basename (boolean function) – returns True if the input file basename qualifies.
- dirname (boolean function) – returns True if the input file directory name qualifies.
- absolute (bool) – if True, the returned paths are absolute, else relative.
Returns: list of local files.
Return type: list of str
-
readable
(files, unsafe=False)¶ Select filenames from the repository that can be uploaded.
Parameters: files (list) – list of relative paths. Returns: uploadable filenames. Return type: list
-
writable
(filename, absolute=True)¶ Get the local path corresponding to a remote resource if it can be downloaded.
Parameters: - filename (str) – path to remote file.
- absolute (bool) – if True, the returned path is absolute, else relative.
Returns: local path, or
None
if the local resource is not writable.Return type: str or None
-
-
class
escale.manager.
TimeQuotaController
(refresh=True, quota=None, quota_read_interval=None, quota_read_callback=None, logger=None)¶ Bases:
object
Time and quota controller.
refresh (int): maximum refresh interval in seconds.
quota (int or float): maximum used space on relay host, in MB.
-
class
escale.manager.
History
(repository=None, persistent=None, **super_args)¶
escale.manager.config cache¶
-
class
escale.manager.cache.
ChecksumCache
(cache)¶ Bases:
dict
-
escale.manager.cache.
asbinary
(s)¶
-
escale.manager.cache.
find_checksum_cache
(section, config=None)¶
-
escale.manager.cache.
read_checksum_cache
(path, log=None)¶ deprecated
Rename old checksum cache files appending .old at the end. This function will convert the old cache to the new format.
-
escale.manager.cache.
write_checksum_cache
(path, cache, log=None)¶ deprecated
escale.manager.config module¶
-
escale.manager.config.
get_cache_file
(config=None, section=None, prefix='', previously=None)¶ Get the corresponding persistent data location for a given configuration file and section.
Parameters: - config (ConfigParser) – configuration object or filepath.
- section (str) – section/repository name.
- prefix (str) – prefixes the basename of the returned path.
- previously (str) – former section/repository name if it has been renamed.
Returns: path to persistent data location.
Return type: str
new in 0.5: moved from
escale.base.config
-
escale.manager.config.
get_client_name
(repository, config={})¶ Read client name from config or fall back to default.
Parameters: - repository (str) – repository name.
- config (dict) – option-value configuration dictionnary.
Returns: client name.
Return type: str
-
escale.manager.config.
get_dist_file
(default_dirs={}, filename=None, config=None, section=None, options=None)¶ Get path of “distribution” files such as caches and locks.
Parameters: - default_dirs (dict) – mapping configuration filepath -> distribution filepath.
- filename (str) – basename.
- config (ConfigParser or str) – configuration filepath or object.
- section (str) – repository name.
- options (str or tuple or list) – configuration option(s).
Returns: filepath.
Return type: str
new in 0.5: moved from
escale.base.config
The “dist” directory is determined from the configuration object if any, or from the configuration filepath combined with the default_dirs mapping.
If filename is not provided, it is derived from the basename of the configuration file.
-
escale.manager.config.
get_pid_file
(config=None)¶ Get the location of the pid (process id) file.
Parameters: config (ConfigParser) – configuration object or filepath. Returns: path to pid file. Return type: str new in 0.5: moved from
escale.base.config
-
escale.manager.config.
parse_address
(addr, multi_path_protocols=[])¶ Parse host addresses or paths.
Parameters: - addr (str) – host address.
- multi_path_protocols (list) – list of multi-path protocols.
Returns: (protocol, address, port, path)
Return type: (str, str, str, str)
Addresses can have multiple formats:
file://absolutepath
protocol://servername[:port][/relativepath]
Note that
absolutepath
must start with ‘/’ or ‘~/’ to be identified as an absolute path.absolutepath
andservername
are returned as second argument andrelativepath
as fourth output argument.new in 0.5: multi_path_protocols; moved from
escale.base.config
multi-path protocols admit paths such as:
protocol
protocol://relativepath
whererelativepath
is returned as pathprotocol://absolutepath
whereabsolutepath
is returned as addressprotocol://absolutepath//relativepath
Note that
absolutepath
must start with ‘/’ or ‘~/’ to be identified as an absolute path.Similarly
relativepath
should not start with any of ‘/’ or ‘~/’.Example multi-path address:
googledrive:///home/user/MyGoogleDrive//Escale repository
where
/home/user/MyGoogleDrive
is the path of the “locally-mounted” storage space andEscale repository
is the directory that Escale will use for synchronization.The above example can be parsed as expected if ‘googledrive’ is provided in the multi_path_protocols input argument.
-
escale.manager.config.
parse_section
(config, repository, logger)¶ Make relay backend and parse related options.
Parameters: - config (ConfigParser) – configuration object.
- repository (str) – configuration section.
- logger (Logger) – logger.
Returns: relay class and dictionnary of parameters.
Return type: (escale.relay.AbstractRelay, dict)
escale.manager.access module¶
-
class
escale.manager.access.
AccessAttributes
(location=None, dbm_mode='c')¶ Bases:
object
-
getReadability
(resource)¶
-
getWritability
(resource)¶
-
isReadable
(resource, make_explicit=None)¶ Determine whether a resource can be uploaded.
-
isWritable
(resource, make_explicit=None)¶ Determine whether a resource can be downloaded.
-
location
¶
-
setNotReadable
(resource)¶
-
setNotWritable
(resource)¶
-
setReadability
(resource, r)¶
-
setReadable
(resource)¶
-
setWritability
(resource, w)¶
-
setWritable
(resource)¶
-
table
(resource)¶
-
unsetReadability
(resource)¶
-
unsetWritability
(resource)¶
-
-
class
escale.manager.access.
AccessController
(repository, path=None, persistent=None, ui_controller=None, push_only=False, pull_only=False, mode=None, create=False, unsafe=False, verbosity=1, **ignored)¶ Bases:
escale.base.essential.Reporter
Manage access to the files in a local repository.
-
persistent
¶ path to persistent data.
Type: str
-
repository
¶ repository identifier.
Type: str
-
path
¶ path to repository root.
Type: str
-
mode
¶ any of download, upload, conservative or share/shared (default).
Type: str
-
create
¶ if
True
create persistent data if missing.Type: str
-
unsafe
¶ skip all the checks that require accessing the file system.
Type: bool
-
verbosity
¶ verbosity level; if greater than 2, may cause the OS to freeze.
Type: int
When push_only (resp. pull_only) is
True
, mode is upload (resp. download).When mode is download, upload or shared, and the persistent attributes do not exist at init time (database not created), any external change (e.g. with escalectl) will not be taken into account, unless create is
True
.-
absolute
(filename)¶
-
accessor
(filename)¶
-
confirmPull
(filename)¶ Return a context manager so that permissions can be updated at the beginning or at completion of the transfer.
-
confirmPush
(filename)¶ Return a context manager so that permissions can be updated at the beginning or at completion of the transfer.
-
getReadability
(filename)¶
-
getWritability
(filename)¶
-
listFiles
(path=None, basename=None, dirname=None, absolute=False)¶ List all visible files in the local repository.
Files which name begins with “.” are ignored.
Parameters: - path (str) – relative path to recursively crawl from.
- basename (boolean function) – returns True if the input file basename qualifies.
- dirname (boolean function) – returns True if the input file directory name qualifies.
- absolute (bool) – if True, the returned paths are absolute, else relative.
Returns: list of local files.
Return type: list of str
-
mode
-
readable
(files, unsafe=False)¶ Select filenames from the repository that can be uploaded.
Parameters: files (list) – list of relative paths. Returns: uploadable filenames. Return type: list
-
setReadability
(filename, r)¶
-
setWritability
(filename, w)¶
-
writable
(filename, absolute=True)¶ Get the local path corresponding to a remote resource if it can be downloaded.
Parameters: - filename (str) – path to remote file.
- absolute (bool) – if True, the returned path is absolute, else relative.
Returns: local path, or
None
if the local resource is not writable.Return type: str or None
-
-
class
escale.manager.access.
Accessor
(exists=None, delete=None)¶ Bases:
object
Interface to a single local file.
This class aims at hiding the actual file so that access to it is strictly controlled, while still featuring basic file manipulation methods.
-
delete
¶
-
exists
¶
-
-
class
escale.manager.access.
ControllerProxy
(controller)¶ Bases:
object
-
controller
¶
-
mode
¶
-
persistent
¶
-
-
exception
escale.manager.access.
PermissionError
¶ Bases:
exceptions.OSError
-
class
escale.manager.access.
Pull
(controller, filename)¶ Bases:
escale.manager.access.ControllerProxy
-
filename
¶
-
-
class
escale.manager.access.
Push
(controller, filename)¶ Bases:
escale.manager.access.ControllerProxy
-
filename
¶
-
-
class
escale.manager.access.
TableEntry
(table, key, default=None)¶ Bases:
object
-
default
¶
-
delete
()¶
-
get
()¶
-
key
¶
-
set
(value)¶
-
table
¶
-
-
escale.manager.access.
asbinary
(s)¶
escale.manager.history module¶
-
class
escale.manager.history.
History
(repository=None, persistent=None, **super_args)¶
-
class
escale.manager.history.
TimeQuotaController
(refresh=True, quota=None, quota_read_interval=None, quota_read_callback=None, logger=None)¶ Bases:
object
Time and quota controller.
refresh (int): maximum refresh interval in seconds.
quota (int or float): maximum used space on relay host, in MB.
-
pull
(local_file)¶
-
push
(local_file, callback=None)¶
-
wait
()¶
-
escale.manager.index module¶
-
class
escale.manager.index.
IndexManager
(relay, *args, **kwargs)¶ Bases:
escale.manager.manager.Manager
-
download
()¶ Finds out which files are to be downloaded and download them.
-
localFiles
(path=None)¶ Transitional method.
Use
self.repository.readableFiles
instead.
-
reportTransferred
(download_or_upload, transferred_files)¶
-
sanityChecks
()¶ Performs sanity checks and fixes the corrupted files.
-
shuffle
(_list, with_updates_first=False)¶
-
terminate
(pullers)¶
-
upload
()¶ Finds out which files are to be uploaded and upload them.
-
escale.manager.manager module¶
-
class
escale.manager.manager.
Manager
(relay, repository=None, address=None, directory=None, encryption=<escale.encryption.encryption.Plain object>, timestamp=True, refresh=True, clientname=None, filetype=[], include=None, exclude=None, tq_controller=None, count=None, checksum=True, checksum_cache=None, includedirectory=None, excludedirectory=None, waitonerror=[], verbosity=1, **relay_args)¶ Bases:
escale.base.essential.Reporter
Makes the glue between the local file system and the
relay
andencryption
layers.This class manages the meta information, file modifications and sleep times.
-
repository
¶ local file access controller.
Type: escale.manager.AccessController
-
address
¶ remote host address.
Type: str
-
directory
¶ path to the repository on the remote host.
Type: str
-
encryption
¶ encryption layer.
Type: escale.encryption.Cipher
-
relay
¶ communication layer.
Type: escale.relay.AbstractRelay
-
timestamp
¶ if
True
(recommended), manages file modification times. If str, in addition determines the timestamp format as supported bytime.strftime()
.Type: bool or str
-
checksum
¶ hash algorithm name; see also the
hashlib
library.Type: str
-
checksum_cache
¶ path to a checksum cache file (str) or cache (dict) of checksums for local files for relative filepaths as keys and (last modification time, checksum) pairs as elements. If bool, then the cache path will be automatically determined.
Type: bool or str or dict
-
filetype
¶ list of file extensions.
Type: list of str
-
include
¶ regular expressions to include files by name.
Type: list of str
-
exclude
¶ regular expressions to exclude files by name.
Type: list of str
-
tq_controller
¶ time and quota controller.
Type: escale.manager.TimeQuotaController
-
count
¶ puller count.
Type: int
-
includedirectory
¶ regular expressions to include directories by name.
Type: list of str
-
excludedirectory
¶ regular expressions to exclude directories by name.
Type: list of str
-
waitonerror
¶ error codes that trigger wait and recovery.
Type: list of int
-
verbosity
¶ 2 or higher makes Escale so verbose that it can make the entire OS freeze.
Type: int
new in 0.7.1: checksum_cache new in 0.7.4: wait_on_error new in 0.7.6: verbosity
-
address
-
checksum
(resource, return_mtime=False)
-
count
-
dir
¶
-
download
()¶ Finds out which files are to be downloaded and download them.
-
filter
(files)¶ Applies filters on a list of file paths.
Parameters: files (list) – list of file paths. Returns: list of selected file paths from files
.Return type: list
-
localFiles
(path=None)¶ Transitional method.
Use
self.repository.readableFiles
instead.
-
mode
¶
-
path
¶
-
remoteListing
()¶
-
run
()¶ Runs the manager.
Example:
from escale.manager import Manager from escale.relay import WebDAV Manager(WebDAV, remote_host, path_to_local_repository, path_to_remote_repository ).run()
-
sanityChecks
()¶ Performs sanity checks and fixes the corrupted files.
-
upload
()¶ Finds out which files are to be uploaded and upload them.
-
escale.manager.migration module¶
-
escale.manager.migration.
inter_relay_copy
(src_relay, dest_relay, safe=True, overwrite=False, files=[])¶ Transfer files from a source relay to a destination relay.
Files are locally downloaded from the source and then uploaded to the destination one after the other.
Regular files in the source repository are all locked both on the source relay and destination relay before any transfer begins, and they are all released at the very end.
Note
relays should implement
Relay
instead ofAbstractRelay
In a future release, this function will also translate the names of the special files according to prefixes and suffixes defined in the relay objects.
Parameters: - src_relay (escale.relay.Relay) – source opened relay.
- dest_relay (escale.relay.Relay) – destination opened relay.
- safe (bool or (bool, bool)) – if
True
, lock the regular files before operating; if a tuple, the first boolean applies to the source relay and the second one applies to the destination relay. - overwrite (bool) – overwrite existing locks on destination relay; you should ensure yourself that the destination repository is inactive.
- files (list of str) – list of paths in the source repository; default behavior consists of transfering the entire repository tree.
Returns: list of paths for files that could not be successfully transfered.
Return type: list of str
Note
instead of setting overwrite to True, it would preferable to purge the destination repository.
-
escale.manager.migration.
migrate_repositories_and_update_config
(changes, config=None, safe=True, logger=None)¶ Migrate relay repositories and update configuration file.
Parameters: - changes (ConfigParser) – changes to apply to relays.
- config (str or ConfigParser) – configuration for active relays.
- safe (bool) – whether to lock all the resources or not.
- logger (Logger) – logger.
-
escale.manager.migration.
migrate_repository
(repository, changes, config=None, safe=True, logger=None, alt_repo=None)¶ Parse the configuration sections for the source and destination relays and transfer the entire repository from the source relay to the destination relay.
Expected options in changes are relay protocol (required), relay address, relay port, path of the relay repository and credentials.
Client name or the encryption parameters cannot be modified at migration time.
Parameters: - repository (str) – configuration section.
- changes (ConfigParser) – configuration object with a section corresponding to repository.
- config (str or ConfigParser) – configuration file or object for the source relay.
- safe (bool or (bool, bool)) – whether to lock all the resources before transfer; the argument is passed directly to inter_relay_copy.
- logger (Logger) – logger.
- alt_repo (str) – section to look for in changes instead of repositories.
Returns: updated configuration object for the migrated and other active relays.
Return type: ConfigParser
escale.manager.backup module¶
-
class
escale.manager.backup.
Backup
(directory, **super_args)¶ Bases:
escale.relay.localmount.LocalMount
-
acquireLock
(**kwargs)¶ This method treats locks as files.
-
releaseLock
(**kwargs)¶ This method treats locks as files.
-
-
escale.manager.backup.
backup_manager
(archive, repository, backup_or_restore='backup', safe=True, config=None, logger=None)¶
-
escale.manager.backup.
backup_relay_repository
(relay, archive, safe=True, logger=None)¶
-
escale.manager.backup.
restore_relay_repository
(relay, archive, safe=True, logger=None)¶