Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Docs] .skyignore doc #4114

Merged
merged 2 commits into from
Oct 18, 2024
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
53 changes: 28 additions & 25 deletions docs/source/examples/syncing-code-artifacts.rst
Original file line number Diff line number Diff line change
Expand Up @@ -46,31 +46,7 @@ VMs. The task is invoked under that working directory (so that it can call
scripts, access checkpoints, etc.).

.. note::

**Exclude files from syncing**

For large, multi-gigabyte workdirs, uploading may be slow because they
are synced to the remote VM(s). To exclude large files in
your workdir from being uploaded, add them to a :code:`.skyignore` file
under your workdir. :code:`.skyignore` follows RSYNC filter rules.

Example :code:`.skyignore` file:

.. code-block::

# Files that match pattern under ONLY CURRENT directory
/hello.py
/*.txt
/dir

# Files that match pattern under ALL directories
*.txt
hello.py

# Files that match pattern under a directory ./dir/
/dir/*.txt

Do NOT use ``.`` to indicate local directory (e.g. ``./hello.py``).
To exclude large files from being loaded, see :ref:`exclude-uploading-files`.
yika-luo marked this conversation as resolved.
Show resolved Hide resolved

.. note::

Expand Down Expand Up @@ -140,6 +116,33 @@ file_mount may be slow because they are processed by ``rsync``. Use
:ref:`SkyPilot bucket mounting <sky-storage>` to efficiently handle
large files.

.. _exclude-uploading-files:

Exclude uploading files
--------------------------------------
By default, SkyPilot uses your existing :code:`.gitignore` and :code:`.git/info/exclude` to exclude files from syncing.

Alternatively, you can use :code:`.skyignore` if you want to separate SkyPilot's syncing behavior from Git's.
If you use a :code:`.skyignore` file, SkyPilot will only exclude files based on that file without using the default Git files.

Any :code:`.skyignore` file under either your workdir or source paths of file_mounts is respected.

:code:`.skyignore` follows RSYNC filter rules, e.g.

.. code-block::

# Files that match pattern under CURRENT directory
/file.txt
/dir
/*.jar
/dir/*.jar

# Files that match pattern under ALL directories
*.jar
file.txt

Do _not_ use ``.`` to indicate local directory (e.g., instead of ``./file``, write ``/file``).

.. _downloading-files-and-artifacts:

Downloading files and artifacts
Expand Down
4 changes: 2 additions & 2 deletions docs/source/reference/yaml-spec.rst
Original file line number Diff line number Diff line change
Expand Up @@ -22,8 +22,8 @@ Available fields:
# If a relative path is used, it's evaluated relative to the location from
# which `sky` is called.
#
# To exclude files from syncing, add them to a .skyignore file under your working directory.
yika-luo marked this conversation as resolved.
Show resolved Hide resolved
# Details: https://skypilot.readthedocs.io/en/latest/examples/syncing-code-artifacts.html#uploading-code-and-project-files
# To exclude files from syncing, see
# https://skypilot.readthedocs.io/en/latest/examples/syncing-code-artifacts.html#exclude-uploading-files
workdir: ~/my-task-code
# Number of nodes (optional; defaults to 1) to launch including the head node.
Expand Down
Loading