Files Operations¶

The files module handles filesystem state, file uploads and template generation.

`files.directory`¶

Add/remove/update directories.

files.directory(
    path, present=True, assume_present=False, user=None, group=None, mode=None,
    recursive=False, force=False, force_backup=True, force_backup_dir=None,
)

path: path of the remote folder
present: whether the folder should exist
assume_present: whether to assume the directory exists
user: user to own the folder
group: group to own the folder
mode: permissions of the folder
recursive: recursively apply user/group/mode
force: if the target exists and is not a file, move or remove it and continue
force_backup: set to False to remove any existing non-file when force=True
force_backup_dir: directory to move any backup to when force=True

Examples:

files.directory(
    name='Ensure the /tmp/dir_that_we_want_removed is removed',
    path='/tmp/dir_that_we_want_removed',
    present=False,
)

files.directory(
    name='Ensure /web exists',
    path='/web',
    user='myweb',
    group='myweb',
)

# multiple directories
dirs = ['/netboot/tftp', '/netboot/nfs']
for dir in dirs:
    files.directory(
        name='Ensure the directory `{}` exists'.format(dir),
        path=dir,
    )

`files.download`¶

Download files from remote locations using curl or wget.

files.download(
    src, dest, user=None, group=None, mode=None, cache_time=None, force=False, sha256sum=None,
    sha1sum=None, md5sum=None,
)

src: source URL of the file
dest: where to save the file
user: user to own the files
group: group to own the files
mode: permissions of the files
cache_time: if the file exists already, re-download after this time (in seconds)
force: always download the file, even if it already exists
**sha**256sum: sha256 hash to checksum the downloaded file against
**sha**1sum: sha1 hash to checksum the downloaded file against
**md**5sum: md5 hash to checksum the downloaded file against

Example:

files.download(
    name='Download the Docker repo file',
    src='https://download.docker.com/linux/centos/docker-ce.repo',
    dest='/etc/yum.repos.d/docker-ce.repo',
)

`files.file`¶

Add/remove/update files.

files.file(
    path, present=True, assume_present=False, user=None, group=None, mode=None, touch=False,
    create_remote_dir=True, force=False, force_backup=True, force_backup_dir=None,
)

path: name/path of the remote file
present: whether the file should exist
assume_present: whether to assume the file exists
user: user to own the files
group: group to own the files
mode: permissions of the files as an integer, eg: 755
touch: whether to touch the file
create_remote_dir: create the remote directory if it doesn’t exist
force: if the target exists and is not a file, move or remove it and continue
force_backup: set to False to remove any existing non-file when force=True
force_backup_dir: directory to move any backup to when force=True

create_remote_dir:: If the remote directory does not exist it will be created using the same user & group as passed to files.put. The mode will not be copied over, if this is required call files.directory separately.

Example:

# Note: The directory /tmp/secret will get created with the default umask.
files.file(
    name='Create /tmp/secret/file',
    path='/tmp/secret/file',
    mode='600',
    user='root',
    group='root',
    touch=True,
    create_remote_dir=True,
)

`files.get`¶

Download a file from the remote system.

files.get(src, dest, add_deploy_dir=True, create_local_dir=False, force=False)

src: the remote filename to download
dest: the local filename to download the file to
add_deploy_dir: dest is relative to the deploy directory
create_local_dir: create the local directory if it doesn’t exist
force: always download the file, even if the local copy matches

Note:: This operation is not suitable for large files as it may involve copying the remote file before downloading it.

Example:

files.get(
    name='Download a file from a remote',
    src='/etc/centos-release',
    dest='/tmp/whocares',
)

`files.line`¶

Ensure lines in files using grep to locate and sed to replace.

files.line(
    path, line, present=True, replace=None, flags=None, backup=False,
    interpolate_variables=False, escape_regex_characters=False, assume_present=False,
)

path: target remote file to edit
line: string or regex matching the target line
present: whether the line should be in the file
replace: text to replace entire matching lines when present=True
flags: list of flags to pass to sed when replacing/deleting
backup: whether to backup the file (see below)
interpolate_variables: whether to interpolate variables in replace
assume_present: whether to assume a matching line already exists in the file
escape_regex_characters: whether to escape regex characters from the matching line

Regex line matching:: Unless line matches a line (starts with ^, ends $), pyinfra will wrap it such that it does, like: ^.*LINE.*$. This means we don’t swap parts of lines out. To change bits of lines, see files.replace.
Regex line escaping:: If matching special characters (eg a crontab line containing *), remember to escape it first using Python’s re.escape.
Backup:: If set to True, any editing of the file will place an old copy with the ISO date (taken from the machine running pyinfra) appended as the extension. If you pass a string value this will be used as the extension of the backed up file.
Append:: If line is not in the file but we want it (present set to True), then it will be append to the end of the file.

Examples:

# prepare to do some maintenance
maintenance_line = 'SYSTEM IS DOWN FOR MAINTENANCE'
files.line(
    name='Add the down-for-maintence line in /etc/motd',
    path='/etc/motd',
    line=maintenance_line,
)

# Then, after the mantenance is done, remove the maintenance line
files.line(
    name='Remove the down-for-maintenance line in /etc/motd',
    path='/etc/motd',
    line=maintenance_line,
    replace='',
    present=False,
)

# example where there is '*' in the line
files.line(
    name='Ensure /netboot/nfs is in /etc/exports',
    path='/etc/exports',
    line=r'/netboot/nfs .*',
    replace='/netboot/nfs *(ro,sync,no_wdelay,insecure_locks,no_root_squash,'
    'insecure,no_subtree_check)',
)

files.line(
    name='Ensure myweb can run /usr/bin/python3 without password',
    path='/etc/sudoers',
    line=r'myweb .*',
    replace='myweb ALL=(ALL) NOPASSWD: /usr/bin/python3',
)

# example when there are double quotes (")
line = 'QUOTAUSER=""'
files.line(
    name='Example with double quotes (")',
    path='/etc/adduser.conf',
    line='^{}$'.format(line),
    replace=line,
)

`files.link`¶

Add/remove/update links.

files.link(
    path, target=None, present=True, assume_present=False, user=None, group=None,
    symbolic=True, create_remote_dir=True, force=False, force_backup=True,
    force_backup_dir=None,
)

path: the name of the link
target: the file/directory the link points to
present: whether the link should exist
assume_present: whether to assume the link exists
user: user to own the link
group: group to own the link
symbolic: whether to make a symbolic link (vs hard link)
create_remote_dir: create the remote directory if it doesn’t exist
force: if the target exists and is not a file, move or remove it and continue
force_backup: set to False to remove any existing non-file when force=True
force_backup_dir: directory to move any backup to when force=True

create_remote_dir:: If the remote directory does not exist it will be created using the same user & group as passed to files.put. The mode will not be copied over, if this is required call files.directory separately.
Source changes:: If the link exists and points to a different target, pyinfra will remove it and recreate a new one pointing to then new target.

Examples:

# simple example showing how to link to a file
files.link(
    name='Create link /etc/issue2 that points to /etc/issue',
    path='/etc/issue2',
    target='/etc/issue',
)


# complex example demonstrating the assume_present option
from pyinfra.operations import apt, files

install_nginx = apt.packages(
    name='Install nginx',
    packages=['nginx'],
)

files.link(
    name='Remove default nginx site',
    path='/etc/nginx/sites-enabled/default',
    present=False,
    assume_present=install_nginx.changed,
)

`files.put`¶

Upload a local file to the remote system.

files.put(
    src, dest, user=None, group=None, mode=None, add_deploy_dir=True, create_remote_dir=True,
    force=False, assume_exists=False,
)

src: local filename to upload
dest: remote filename to upload to
user: user to own the files
group: group to own the files
mode: permissions of the files
add_deploy_dir: src is relative to the deploy directory
create_remote_dir: create the remote directory if it doesn’t exist
force: always upload the file, even if the remote copy matches
assume_exists: whether to assume the local file exists

dest:: If this is a directory that already exists on the remote side, the local file will be uploaded to that directory with the same filename.
create_remote_dir:: If the remote directory does not exist it will be created using the same user & group as passed to files.put. The mode will not be copied over, if this is required call files.directory separately.
Note:: This operation is not suitable for large files as it may involve copying the file before uploading it.

Examples:

# Note: This requires a 'files/motd' file on the local filesystem
files.put(
    name='Update the message of the day file',
    src='files/motd',
    dest='/etc/motd',
    mode='644',
)

`files.replace`¶

Replace contents of a file using sed.

files.replace(path, match, replace, flags=None, backup=False, interpolate_variables=False)

path: target remote file to edit
match: text/regex to match for
replace: text to replace with
flags: list of flags to pass to sed
backup: whether to backup the file (see below)
interpolate_variables: whether to interpolate variables in replace

Backup:: If set to True, any editing of the file will place an old copy with the ISO date (taken from the machine running pyinfra) appended as the extension. If you pass a string value this will be used as the extension of the backed up file.

Example:

files.replace(
    name='Change part of a line in a file',
    path='/etc/motd',
    match='verboten',
    replace='forbidden',
)

`files.rsync`¶

Use rsync to sync a local directory to the remote system. This operation will actually call the rsync binary on your system.

files.rsync(src, dest, flags=['-ax', '--delete'])

Important

The files.rsync operation is in alpha, and only supported using SSH or @local connectors.

Caution

When using SSH, the files.rsync operation only supports the sudo and sudo_user global arguments.

`files.sync`¶

Syncs a local directory with a remote one, with delete support. Note that delete will remove extra files on the remote side, but not extra directories.

files.sync(
    src, dest, user=None, group=None, mode=None, delete=False, exclude=None, exclude_dir=None,
    add_deploy_dir=True,
)

src: local directory to sync
dest: remote directory to sync to
user: user to own the files and directories
group: group to own the files and directories
mode: permissions of the files
delete: delete remote files not present locally
exclude: string or list/tuple of strings to match & exclude files (eg *.pyc)
exclude_dir: string or list/tuple of strings to match & exclude directories (eg node_modules)
add_deploy_dir: interpret src as relative to deploy directory instead of current directory

Example:

# Sync local files/tempdir to remote /tmp/tempdir
files.sync(
    name='Sync a local directory with remote',
    src='files/tempdir',
    dest='/tmp/tempdir',
)

Note: exclude and exclude_dir use fnmatch behind the scenes to do the filtering.

exclude matches against the filename.
exclude_dir matches against the path of the directory, relative to src. Since fnmatch does not treat path separators (/ or \) as special characters, excluding all directories matching a given name, however deep under src they are, can be done for example with exclude_dir=["__pycache__", "*/__pycache__"]

`files.template`¶

Generate a template using jinja2 and write it to the remote system.

files.template(src, dest, user=None, group=None, mode=None, create_remote_dir=True)

src: local template filename
dest: remote filename
user: user to own the files
group: group to own the files
mode: permissions of the files
create_remote_dir: create the remote directory if it doesn’t exist

create_remote_dir:

If the remote directory does not exist it will be created using the same user & group as passed to files.put. The mode will not be copied over, if this is required call files.directory separately.

Notes:

Common convention is to store templates in a “templates” directory and have a filename suffix with ‘.j2’ (for jinja2).

For information on the template syntax, see the jinja2 docs.

Examples:

files.template(
    name='Create a templated file',
    src='templates/somefile.conf.j2',
    dest='/etc/somefile.conf',
)

files.template(
    name='Create service file',
    src='templates/myweb.service.j2',
    dest='/etc/systemd/system/myweb.service',
    mode='755',
    user='root',
    group='root',
)

# Example showing how to pass python variable to template file. You can also
# use dicts and lists. The .j2 file can use `{{ foo_variable }}` to be interpolated.
foo_variable = 'This is some foo variable contents'
foo_dict = {
    "str1": "This is string 1",
    "str2": "This is string 2"
}
foo_list = [
    "entry 1",
    "entry 2"
]
files.template(
    name='Create a templated file',
    src='templates/foo.yml.j2',
    dest='/tmp/foo.yml',
    foo_variable=foo_variable,
    foo_dict=foo_dict,
    foo_list=foo_list
)

# templates/foo.j2
name: "{{ foo_variable }}"
dict_contents:
    str1: "{{ foo_dict.str1 }}"
    str2: "{{ foo_dict.str2 }}"
list_contents:
{% for entry in foo_list %}
    - "{{ entry }}"
{% endfor %}

Files Operations¶

files.directory¶

files.download¶

files.file¶

files.get¶

files.line¶

files.link¶

files.put¶

files.replace¶

files.rsync¶

files.sync¶

files.template¶

`files.directory`¶

`files.download`¶

`files.file`¶

`files.get`¶

`files.line`¶

`files.link`¶

`files.put`¶

`files.replace`¶

`files.rsync`¶

`files.sync`¶

`files.template`¶