Files Operations¶
The files module handles filesystem state, file uploads and template generation.
files.directory
¶
Add/remove/update directories.
files.directory(
path, present=True, assume_present=False, user=None, group=None, mode=None,
recursive=False,
)
- path: path of the remote folder
- present: whether the folder should exist
- assume_present: whether to assume the directory exists
- user: user to own the folder
- group: group to own the folder
- mode: permissions of the folder
- recursive: recursively apply user/group/mode
Examples:
files.directory(
name='Ensure the /tmp/dir_that_we_want_removed is removed',
path='/tmp/dir_that_we_want_removed',
present=False,
)
files.directory(
name='Ensure /web exists',
path='/web',
user='myweb',
group='myweb',
)
# multiple directories
dirs = ['/netboot/tftp', '/netboot/nfs']
for dir in dirs:
files.directory(
name='Ensure the directory `{}` exists'.format(dir),
path=dir,
)
files.download
¶
Download files from remote locations using curl or wget.
files.download(
src, dest, user=None, group=None, mode=None, cache_time=None, force=False, sha256sum=None,
sha1sum=None, md5sum=None,
)
- src: source URL of the file
- dest: where to save the file
- user: user to own the files
- group: group to own the files
- mode: permissions of the files
- cache_time: if the file exists already, re-download after this time (in seconds)
- force: always download the file, even if it already exists
- **sha**256sum: sha256 hash to checksum the downloaded file against
- **sha**1sum: sha1 hash to checksum the downloaded file against
- **md**5sum: md5 hash to checksum the downloaded file against
Example:
files.download(
name='Download the Docker repo file',
src='https://download.docker.com/linux/centos/docker-ce.repo',
dest='/etc/yum.repos.d/docker-ce.repo',
)
files.file
¶
Add/remove/update files.
files.file(
path, present=True, assume_present=False, user=None, group=None, mode=None, touch=False,
create_remote_dir=True,
)
- path: name/path of the remote file
- present: whether the file should exist
- assume_present: whether to assume the file exists
- user: user to own the files
- group: group to own the files
- mode: permissions of the files as an integer, eg: 755
- touch: whether to touch the file
- create_remote_dir: create the remote directory if it doesn’t exist
create_remote_dir
:- If the remote directory does not exist it will be created using the same
user & group as passed to
files.put
. The mode will not be copied over, if this is required callfiles.directory
separately.
Example:
# Note: The directory /tmp/secret will get created with the default umask.
files.file(
name='Create /tmp/secret/file',
path='/tmp/secret/file',
mode='600',
user='root',
group='root',
touch=True,
create_remote_dir=True,
)
files.get
¶
Download a file from the remote system.
files.get(src, dest, add_deploy_dir=True, create_local_dir=False, force=False)
- src: the remote filename to download
- dest: the local filename to download the file to
- add_deploy_dir: dest is relative to the deploy directory
- create_local_dir: create the local directory if it doesn’t exist
- force: always download the file, even if the local copy matches
- Note:
- This operation is not suitable for large files as it may involve copying the remote file before downloading it.
Example:
files.get(
name='Download a file from a remote',
src='/etc/centos-release',
dest='/tmp/whocares',
)
files.line
¶
Ensure lines in files using grep to locate and sed to replace.
files.line(
path, line, present=True, replace=None, flags=None, backup=False,
interpolate_variables=False, assume_present=False,
)
- path: target remote file to edit
- line: string or regex matching the target line
- present: whether the line should be in the file
- replace: text to replace entire matching lines when
present=True
- flags: list of flags to pass to sed when replacing/deleting
- backup: whether to backup the file (see below)
- interpolate_variables: whether to interpolate variables in
replace
- Regex line matching:
- Unless line matches a line (starts with ^, ends $), pyinfra will wrap it such that
it does, like:
^.*LINE.*$
. This means we don’t swap parts of lines out. To change bits of lines, seefiles.replace
. - Regex line escaping:
- If matching special characters (eg a crontab line containing *), remember to escape
it first using Python’s
re.escape
. - Backup:
- If set to
True
, any editing of the file will place an old copy with the ISO date (taken from the machine runningpyinfra
) appended as the extension. If you pass a string value this will be used as the extension of the backed up file. - Append:
- If
line
is not in the file but we want it (present
set toTrue
), then it will be append to the end of the file.
Examples:
# prepare to do some maintenance
maintenance_line = 'SYSTEM IS DOWN FOR MAINTENANCE'
files.line(
name='Add the down-for-maintence line in /etc/motd',
path='/etc/motd',
line=maintenance_line,
)
# Then, after the mantenance is done, remove the maintenance line
files.line(
name='Remove the down-for-maintenance line in /etc/motd',
path='/etc/motd',
line=maintenance_line,
replace='',
present=False,
)
# example where there is '*' in the line
files.line(
name='Ensure /netboot/nfs is in /etc/exports',
path='/etc/exports',
line=r'/netboot/nfs .*',
replace='/netboot/nfs *(ro,sync,no_wdelay,insecure_locks,no_root_squash,'
'insecure,no_subtree_check)',
)
files.line(
name='Ensure myweb can run /usr/bin/python3 without password',
path='/etc/sudoers',
line=r'myweb .*',
replace='myweb ALL=(ALL) NOPASSWD: /usr/bin/python3',
)
# example when there are double quotes (")
line = 'QUOTAUSER=""'
files.line(
name='Example with double quotes (")',
path='/etc/adduser.conf',
line='^{}$'.format(line),
replace=line,
)
files.link
¶
Add/remove/update links.
files.link(
path, target=None, present=True, assume_present=False, user=None, group=None,
symbolic=True, create_remote_dir=True,
)
- path: the name of the link
- target: the file/directory the link points to
- present: whether the link should exist
- assume_present: whether to assume the link exists
- user: user to own the link
- group: group to own the link
- symbolic: whether to make a symbolic link (vs hard link)
- create_remote_dir: create the remote directory if it doesn’t exist
create_remote_dir
:- If the remote directory does not exist it will be created using the same
user & group as passed to
files.put
. The mode will not be copied over, if this is required callfiles.directory
separately. - Source changes:
- If the link exists and points to a different target, pyinfra will remove it and recreate a new one pointing to then new target.
Examples:
# simple example showing how to link to a file
files.link(
name='Create link /etc/issue2 that points to /etc/issue',
path='/etc/issue2',
target='/etc/issue',
)
# complex example demonstrating the assume_present option
from pyinfra.operations import apt, files
install_nginx = apt.packages(
name='Install nginx',
packages=['nginx'],
)
files.link(
name='Remove default nginx site',
path='/etc/nginx/sites-enabled/default',
present=False,
assume_present=install_nginx.changed,
)
files.put
¶
Upload a local file to the remote system.
files.put(
src, dest, user=None, group=None, mode=None, add_deploy_dir=True, create_remote_dir=True,
force=False, assume_exists=False,
)
- src: local filename to upload
- dest: remote filename to upload to
- user: user to own the files
- group: group to own the files
- mode: permissions of the files
- add_deploy_dir: src is relative to the deploy directory
- create_remote_dir: create the remote directory if it doesn’t exist
- force: always upload the file, even if the remote copy matches
- assume_exists: whether to assume the local file exists
create_remote_dir
:- If the remote directory does not exist it will be created using the same
user & group as passed to
files.put
. The mode will not be copied over, if this is required callfiles.directory
separately. - Note:
- This operation is not suitable for large files as it may involve copying the file before uploading it.
Examples:
# Note: This requires a 'files/motd' file on the local filesystem
files.put(
name='Update the message of the day file',
src='files/motd',
dest='/etc/motd',
mode='644',
)
files.replace
¶
Replace contents of a file using sed
.
files.replace(path, match, replace, flags=None, backup=False, interpolate_variables=False)
- path: target remote file to edit
- match: text/regex to match for
- replace: text to replace with
- flags: list of flags to pass to sed
- backup: whether to backup the file (see below)
- interpolate_variables: whether to interpolate variables in
replace
- Backup:
- If set to
True
, any editing of the file will place an old copy with the ISO date (taken from the machine runningpyinfra
) appended as the extension. If you pass a string value this will be used as the extension of the backed up file.
Example:
files.replace(
name='Change part of a line in a file',
path='/etc/motd',
match='verboten',
replace='forbidden',
)
files.rsync
¶
Use rsync
to sync a local directory to the remote system. This operation will actually call
the rsync
binary on your system.
files.rsync(src, dest, flags=['-ax', '--delete'])
Important
The files.rsync
operation is in alpha, and only supported using SSH
or @local
connectors.
Caution
When using SSH, the files.rsync
operation only supports the sudo
and sudo_user
global arguments.
files.sync
¶
Syncs a local directory with a remote one, with delete support. Note that delete will remove extra files on the remote side, but not extra directories.
files.sync(
src, dest, user=None, group=None, mode=None, delete=False, exclude=None, exclude_dir=None,
add_deploy_dir=True,
)
- src: local directory to sync
- dest: remote directory to sync to
- user: user to own the files and directories
- group: group to own the files and directories
- mode: permissions of the files
- delete: delete remote files not present locally
- exclude: string or list/tuple of strings to match & exclude files (eg *.pyc)
- exclude_dir: string or list/tuple of strings to match & exclude directories (eg node_modules)
Example:
# Sync local files/tempdir to remote /tmp/tempdir
files.sync(
name='Sync a local directory with remote',
src='files/tempdir',
dest='/tmp/tempdir',
)
files.template
¶
Generate a template using jinja2 and write it to the remote system.
files.template(src, dest, user=None, group=None, mode=None, create_remote_dir=True)
- src: local template filename
- dest: remote filename
- user: user to own the files
- group: group to own the files
- mode: permissions of the files
- create_remote_dir: create the remote directory if it doesn’t exist
create_remote_dir
:- If the remote directory does not exist it will be created using the same
user & group as passed to
files.put
. The mode will not be copied over, if this is required callfiles.directory
separately. - Notes:
Common convention is to store templates in a “templates” directory and have a filename suffix with ‘.j2’ (for jinja2).
For information on the template syntax, see the jinja2 docs.
Examples:
files.template(
name='Create a templated file',
src='templates/somefile.conf.j2',
dest='/etc/somefile.conf',
)
files.template(
name='Create service file',
src='templates/myweb.service.j2',
dest='/etc/systemd/system/myweb.service',
mode='755',
user='root',
group='root',
)
# Example showing how to pass python variable to template file.
# The .j2 file can use `{{ foo_variable }}` to be interpolated.
foo_variable = 'This is some foo variable contents'
files.template(
name='Create a templated file',
src='templates/foo.j2',
dest='/tmp/foo',
foo_variable=foo_variable,
)