.ebignore includes symlinked files in ignored directories

[tomchkk]

Description

Having specified .ebignore to ignore a whole directory, if that directory contains a symlink, that symlink is not ignored and included in the source bundle that eb cli creates.

Steps to reproduce

With an eb application and environment set up - where the project contains a directory called ignore and, within that, one standard directory dir and one symlinked directory symlink; and the project's .ebignore file contains an entry to ignore that whole directory from the created artifact - run eb deploy env-name.

.ebignore file:

/bin/ignore/

Observed result

The artifact created and uploaded to S3 will contain symlinked directory ignore/symlink, but not ignore/dir.

Structure of example project eb-bug:

Actual (partial) output from eb deploy env-name --debug shows no entry at all for ./bin/ignore/symlink:

...
2021-03-21 08:28:27,522 (INFO) eb : Found .ebignore, using system zip.
2021-03-21 08:28:27,522 (DEBUG) ebcli.core.fileoperations : Project root found at: /path/2/eb-bug
2021-03-21 08:28:27,522 (INFO) eb : Zipping up folder at location: /path/2/eb-bug
2021-03-21 08:28:27,523 (INFO) eb :  +adding: .//
2021-03-21 08:28:27,523 (INFO) eb :   +adding: ./.envrc
2021-03-21 08:28:27,524 (INFO) eb :   +adding: ./Issue.md
2021-03-21 08:28:27,524 (INFO) eb :   +adding: ./.gitignore
2021-03-21 08:28:27,524 (INFO) eb :   -skipping: ./.ebignore
2021-03-21 08:28:27,524 (INFO) eb :  +adding: ./bin/
2021-03-21 08:28:27,525 (INFO) eb :   +adding: ./bin/cli
2021-03-21 08:28:27,525 (INFO) eb :   -skipping: ./bin/ignore/dir/file
2021-03-21 08:28:27,525 (INFO) eb :   -skipping: ./.elasticbeanstalk
2021-03-21 08:28:27,525 (INFO) eb :   -skipping: ./.elasticbeanstalk/app_versions
2021-03-21 08:28:27,525 (INFO) eb :  +adding: ./.git/
...

Expected result

The artifact created and uploaded to S3 should not contain directory ignore.

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

1. OS: macos 11.2.3
2. EBCLI version: EB CLI 3.19.3 (Python 3.9.2)

[Palakpatel67]

@tomchkk thanks for reporting this issue. We will look into it and post any updates here.

[IgorVanian]

This happened to us with /node_modules/.bin/ symlink folder being processed but since it's a leftover and modules don't exist in node_modules/ it exits with an error:

[vince1995]

Any progress on this? This error exists since years.

[vince1995]

I've looked into the code and noticed that here the symlinks are written into the zip without checking if they are in the .ebignore file.

[vince1995]

I've opened a PR with the fix for that.

[alexd2580-sf]

I second merging the PR. I haven't reviewed the PR, but this bug makes me want to self-host my stuff somewhere far away from AWS.

[llorenspujol]

Any updates on this issue?
IMO It should be a high priority issue. If one folder is ignored, it should be ignored entirely, included its symlinks to folders.

Without this fix, Elastic Beanstalk app.zip can end with a bunch of symlinks that point to undefined locations. For example, I can have this error on startup:

2022/04/21 09:11:08.715481 [ERROR] An error occurred during execution of command [app-deploy] - [StageApplication]. Stop running the command. Error: chown /var/app/staging/venv/dev/lib/python3.8/collections: no such file or directory

Because 'collections' is a symlink to a folder that do not exist in AL2.

All workarounds I am studying are quite painful… so it would be great to fix it.

[vince1995]

Yeah, really annoying that this is still not fixed. Look at my PR (which is more a workaround instead of a fix). You could also delete the symlinks before uploading to AWS and recreate them after the deployment.

[llorenspujol]

Yeah, I looked, thanks @vince1995 . Though it does not work really well in my case, since I have a great number of symlinks. What I ended doing is just deleting the slice of code that 'zips' symlinks.

[llorenspujol]

For anyone facing this issue, I created a branch that does NOT zip symlinks (even when they aren't ignored). It is a fork from the version 3.20.3.

If you have awsebcli in your requirements file, you can download de patched version directly from git:

(requirements.txt)
...
git+https://github.com/llorenspujol/[email protected]#egg=awsebcli
...

Hope this helps!

[vince1995]

Yes, that's something I thought of as well, but I wanted to create a solution that could be merged into the master branch because some developers may explicitly want the symlinks in the zip bundle.

[yohanthshetty]

I have fixed the symlink issue. The symlinks are being treated like just another file/directory.
The symlinks aren't lost either.

[d-rocham]

Symlinks are still being included, even if an explicit path to the problematic symlink is added to .ebignore. Could this issue be re-opened?

[yohanthshetty]

I will confirm if the symlinks are still being included in zip. Will re-open this issue if it still exists.

[cavasinf]

Same case here and still have the problem.

Having a sub-module Git that is linked with Composer (as a vendor), creates a symlink into the /vendor folder.
ATE, every deployment with eb deploy result as non execution of composer install because of:

You included a 'vendor' folder in your source bundle

Unziping the source zip in Windows doesn't do anything with that symlink, so no vendor folder appear.
BUT when unzinping from a Linux environment it ends like that:

  [...]
  inflating: [...]
  inflating: [...]
finishing deferred symbolic links:
  vendor/allsoftware/allsoftware-symfony-bundle -> ../../lib/allsoftware-symfony-bundle/

[mi-mazouz]

Any update ?

[yohanthshetty]

Working on a fix.

[yohanthshetty]

Hi All, Fix has been pushed to master. Pease pull the latest code to test. Broken/dangling symlinks are ignored. Folders with symlinks added are ignored entirely if added to .ebignore.

[yohanthshetty]

Hi All, New version for EBCLI has been released. Please update the ebcli version by referring docs.

[andraspatka]

It seems like it wasn't fixed yet.
Contents of .ebignore:

# ignore everything
*
# Don't ignore docker-compose.yml
!./docker-compose.yml

Error when running eb deploy:

ERROR: FileNotFoundError - [Errno 2] No such file or directory: './env/bin/python'

version is latest:

$ eb --version
EB CLI 3.20.6 (Python 3.10.6 (main, Mar 10 2023, 10:55:28) [GCC 11.3.0])

The culprit:

$ ls-la env/bin/python
lrwxr-xr-x 1 root root 7 Apr 12 13:16 env/bin/python -> python3

[itelescu]

any updates on this ?

[shawzhang]

I just add to ignore those links to my .gitignore file: Just add this line fixed it:
my_venv/bin/python*

[mdeyo]

I'm working with EB CLI 3.20.10 and this is still an issue.

My virtual environment folder (virt/) is included in the .ebignore, but the broken symlink virt/lib64 keeps being included with eb deploy.

Another observation:
If I delete the .ebignore and instead use .gitignore for the virt/ directory, then eb deploy works without including the broken symlink folder! However there are files that I want included on deploy that I want to Git to ignore, hence having different .gitignore and .ebignore files. Any updates available on if this issue is still a priority?

Our fix:

1. Ensure the problem virtual env directory is correctly included in the .ebignore file
2. Uninstall/remove awsebcli from your machine
3. Re-install awsebcli (we had success with both homebrew and pip installations on different machines)
4. eb deploy after double checking the .ebignore is correct