General usage¶
Command-line¶
escale¶
The simplest way to run Escale is to type:
escale
This, however, will fail the first time you run Escale, because you need to configure it first with the -i
option:
escale -i
You can alternatively select a specific conf file with the -c
option:
escale -c <path-to-conf-file>
Last but not least, you can run Escale as a daemon (in background) with the -d
option:
escale -d
If you run Escale as a daemon or with multiple clients (as defined in the configuration file), you may find it more convenient to start and stop Escale with the escalectl command.
Note
never run Escale as the super user. Escale does not need any special privilege.
If no configuration file is provided on the commandline, Escale will look for one of the following files: /etc/escale.conf (system-wide), ~/.escale/escale.conf, ~/.config/escale/escale.conf where ~ stands for the home directory.
escalectl¶
This command makes it easier to start and stop Escale:
escalectl start
escalectl stop
Access modifier edition¶
It also permits to set access modifiers for individual files:
escalectl access <filename> <modifiers>
where <modifiers>
is any contiguous combination of access modifiers.
An access modifier begins with either r
for read or w
for write and may end with any of +
(allowed, default), -
(forbidden) or ?
to unset the existing modifier if any.
Without <modifiers>
, escalectl access <filename>
shows the modifiers for file <filename>
.
See command-line help for more information:
escalectl access --help
Moving relays¶
escalectl can also migrate a relay repository from a host and another.
escalectl migrate -r <repository> <destination-relay>
where <repository>
is an existing section in the default configuration file
(optional if you maintain a single repository in Escale)
and <destination-relay>
is a relay address in the same format as requested during assisted
configuration with escale -i.
A relay address for example can be <protocol>://<servername>/<path-to-repository>
.
A new configuration file can also be provided instead. Beware that section names should match.
More options are listed in:
escalectl migrate --help
Backup¶
escalectl also features repository backup and restoration. For example:
# back-up the 'my-repository' repository into a 'my-backup.tar.bz2' archive
escalectl backup my-repository my-backup.tar.bz2 --fast
# restore the 'my-repository' repository from the 'my-backup.tar.bz2' archive
escalectl restore my-repository my-backup.tar.bz2 --fast
The --fast
option is recommended only if the repository will not undergo changes during backup or restoration.
Even if called with this option, the process may take a while. You are advised to copy yourself the remote repository with a native client (FTP, web, etc).
Recovery procedure¶
There is a mechanism to recover a lost relay repository, provided that all the clients were up-to-date.
Note
restrictions may apply if clients run in conservative mode.
escalectl can restore the placeholders with the fix
subcommand:
escalectl fix
A specific repository can be specified if several repositories are defined in the default configuration file.
Configuration file¶
A configuration file is a text file with key = value
and [section]
lines.
The key = value
lines that appear before the first [section]
line define global parameters that will be common to all the clients.
Each section refers to a client. It begins with a [section]
where section
can be replaced by the name that will appear in the logs. Each client should have a distinct name.
Parameters that can only be global are:
log file
: path to log filelog rotate
: number of rotated log files (default: 3)
Note
booleans can be either yes
, no
, 1
, 0
, true
, false
, on
or off
.
Note
regular expressions for filenames can be basic strings with wildcard *
as the only supported metacharacter, or full regular expressions as recognized by the re module if they begin and (optionally) end with the /
character.
Other parameters are:
local path
(orpath
): path to the local repositoryprotocol
: eitherftp
,ftps
,webdav
,http
,https
,file
,rclone
,dropbox
,googlecloud
,googledrive
,amazoncloud
,s3
,onedrive
,b2
,hubic
,sftp
orswift
. See Relay backendshost address
(orrelay address
,remote address
,address
): address of the remote hosthost directory
(or … +dir
variants): directory of the repository on the remote hostusername
: username on the remote hostpassword
orsecret file
orcredential
: password on the remote host or path to a file that contains the password or both the username and the password on a single line (username:password
)refresh
: synchronization interval in secondsmodification time
ormtime
ortimestamp
: seeManager
- either
push only
orpull only
: boolean that defines whether the client should only push or pull. By default a client both pushes and pulls. Supported aliases forpush only
andpull only
areread only
andwrite only
respectively encryption
: boolean that defines whether to encrypt/decrypt the files or not, or algorithm identifier (e.g.fernet
,blowfish
, etc). See Encryptionpassphrase
orkey
: passphrase or path to a file that contains the passphrase for the encryption algorithmcertificate
orcertfile
: path to the client certificatekeyfile
: path to the client private keyverify ssl
: boolean; checks the remote host’s certificatessl version
: eitherSSLv2
,SSLv3
,SSLv23
,TLS
,TLSv1
,TLSv1.1
orTLSv1.2
file extension
(orfile type
): a comma-separated list of file extensions (with or without the initial dot)include
(orinclude files
,pattern
,filter
): comma-separated list of regular expressions to filter in files by nameexclude
(orexclude files
): comma-separated list of regular expressions to filter out files by nameinclude directories
(orinclude directory
): comma-separated list of regular expressions to filter in directories by relative path; works properly only on top directoriesexclude directories
(orexclude directory
): comma-separated list of regular expressions to filter out directories by relative pathdisk quota
: a decimal number with storage space units such asKB
,MB
,GB
, etcmaintainer
: an email address; if a client aborts and an SMTP server is available on the client machine, a notice email can be sent to this addressmode
(orsynchronization mode
): eitherdownload
(synonym ofpull only = yes
),upload
(synonym ofpush only = yes
),conservative
/preservative
orshare
/shared
(default). See Synchronization modeslock timeout
: timeout for unclaimed locks, in secondspuller count
(orpullers
): number of puller nodes operating on the remote repository. See Multi-client and multi-puller regimeschecksum
(orhash algorithm
): boolean (default: true) or hash algorithm has supported byhashlib.new()
. See also hashlib.algorithms_availablechecksum cache
: boolean (default: true); makes the local checksum cache persistentindex
(orcompact
): boolean (default: false) or string; index-based relay repository management; see also Indexingmaxpagesize
(ormaxarchivesize
): a decimal number with optional storage space units such asKB
,MB
,GB
, etc (default value: 1 GB, default unit: MB)priority
: admits onlyupload
as a value; see also Synchronization modesallow page deletion
(orpage deletion
): boolean (default: false); in download mode, when all the files referenced on an index page have disappeared, report them as missing; default behaviour considers these situations as illegal and requests client restart instead of propagating the deletion upstream
Relay backends¶
Escale features FTP (ftp
, ftps
) and WebDAV (http
, https
, webdav
) native clients.
It relies on rclone (rclone
) for Dropbox, Google Cloud Storage, Google Drive, Amazon Cloud Storage, Amazon S3, Microsoft OneDrive and others.
There is also Google Drive client (googledrive
) that requires the drive utility.
This is governed by the protocol
configuration option.
In addition, a local directory (or mount; file
) can be used as a relay repository.
This is especially useful when no native client is available for a given service but third party software can mount the remote space into the file system.
For example Dropbox is not yet natively supported by Escale but the Dropbox proprietary client can synchronize a directory and Escale can use this or any synchronized subdirectory.
In the case of Dropbox, however, using protocol = rclone
instead or equivalently protocol = dropbox
is recommended.
Synchronization modes¶
The synchronization mode can be upload
, download
, shared
or conservative
:
upload
: local files are sent to the other clients; local files cannot be modified.download
: all new files or file modifications from other clients are admitted; local files are not be sent over the internet but can be modified.shared
: files are fully synchronized; all file additions and modifications are propagated and local files can be modified.conservative
: local files are sent to the relay repository but cannot be overwritten except if they originate from another client and have never been locally modified.
A one-way transfer link will typically define a client running in ‘upload’ mode and others running in ‘download’ mode.
Full synchronization of two clients will be achieved setting both clients to run in ‘shared’ mode.
The shared
and conservative
modes, together with indexing, admit the priority = upload
setting that makes upload take priority over download.
When many files are available for upload when an upload round begins, the client sends as many updates as necessary to upload all these files.
This excludes the files that are newly added during the upload round.
Letting upload take priority may be especially helpful if the local repository happens to be a bottleneck, for example repositories with millions of files available on an NFS mount.
This option is not recommended though.
This may lead to a deadlock if an update from another client is available on the relay.
It is recommended instead to set two separate clients, one in download
mode and the other in upload
mode.
Multi-client and multi-puller regimes¶
For now, multi-client/multi-puller scenarios may lead to conflicts or inconsistensies in the local and remote repositories.
Use it at your own risk.
Todo
make doc
Encryption¶
Two encryption algorithms are supported: fernet
from the cryptography library and blowfish
.
Some backends also support native
when the proper backend features an encryption mechanism.
See for example the googledrive
backend.
blowfish
has two backends: blowfish.cryptography
from the cryptography library (default if the library is available) and blowfish.blowfish
from the blowfish library.
Note that blowfish.cryptography
and blowfish.blowfish
cannot interoperate.
Both algorithms require a passphrase that follow a specific format. It is advised that the first node lets escale -i
generate a passphrase (available in the configuration directory) and then to communicate the generated passphrase to the other nodes.
Note
never send credentials or passphrases by plain email. Consider encrypted email or services like onetimesecret.com instead.
Indexing¶
The files in a default repository are represented as individual files on the relay. This is suitable for directory structures with limited number of subdirectories and files.
To synchronize thousands of small files, the indexing alternative is recommended.
It is driven by the ìndex
and maxpagesize
configuration attributes.
index = 1
sets indexing on.
A comprehensive index file is made available in the relay repository. When files are to be transferred, they are bundled into a compressed archive and propagated through the relay repository together with a limited index file that lists the content of the archive.
The archive is compressed (and encrypted if encryption is on) once the total size of the pending files reaches the value defined by the maxpagesize
configuration parameter, or no more files are to be sent.
Note that compression makes the actual uploaded data smaller than the maxpagesize
value.
One may increase this latter value at the risk of an update exceeding the maximum size.
Note that some relay services may not explicitly reject an oversized files and replace the expected data file by a zero-byte file instead.
This may happen again and again until the upload content results in a small-enough file.
See also the protocol section.