Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

.ebignore includes symlinked files in ignored directories #69

Open
tomchkk opened this issue Mar 21, 2021 · 23 comments · Fixed by #207 · May be fixed by #91
Open

.ebignore includes symlinked files in ignored directories #69

tomchkk opened this issue Mar 21, 2021 · 23 comments · Fixed by #207 · May be fixed by #91
Assignees

Comments

@tomchkk
Copy link

tomchkk commented Mar 21, 2021

Description

Having specified .ebignore to ignore a whole directory, if that directory contains a symlink, that symlink is not ignored and included in the source bundle that eb cli creates.

Steps to reproduce

With an eb application and environment set up - where the project contains a directory called ignore and, within that, one standard directory dir and one symlinked directory symlink; and the project's .ebignore file contains an entry to ignore that whole directory from the created artifact - run eb deploy env-name.

.ebignore file:

/bin/ignore/

Observed result

The artifact created and uploaded to S3 will contain symlinked directory ignore/symlink, but not ignore/dir.

Structure of example project eb-bug:

image

Actual (partial) output from eb deploy env-name --debug shows no entry at all for ./bin/ignore/symlink:

...
2021-03-21 08:28:27,522 (INFO) eb : Found .ebignore, using system zip.
2021-03-21 08:28:27,522 (DEBUG) ebcli.core.fileoperations : Project root found at: /path/2/eb-bug
2021-03-21 08:28:27,522 (INFO) eb : Zipping up folder at location: /path/2/eb-bug
2021-03-21 08:28:27,523 (INFO) eb :  +adding: .//
2021-03-21 08:28:27,523 (INFO) eb :   +adding: ./.envrc
2021-03-21 08:28:27,524 (INFO) eb :   +adding: ./Issue.md
2021-03-21 08:28:27,524 (INFO) eb :   +adding: ./.gitignore
2021-03-21 08:28:27,524 (INFO) eb :   -skipping: ./.ebignore
2021-03-21 08:28:27,524 (INFO) eb :  +adding: ./bin/
2021-03-21 08:28:27,525 (INFO) eb :   +adding: ./bin/cli
2021-03-21 08:28:27,525 (INFO) eb :   -skipping: ./bin/ignore/dir/file
2021-03-21 08:28:27,525 (INFO) eb :   -skipping: ./.elasticbeanstalk
2021-03-21 08:28:27,525 (INFO) eb :   -skipping: ./.elasticbeanstalk/app_versions
2021-03-21 08:28:27,525 (INFO) eb :  +adding: ./.git/
...

Expected result

The artifact created and uploaded to S3 should not contain directory ignore.

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

  1. OS: macos 11.2.3
  2. EBCLI version: EB CLI 3.19.3 (Python 3.9.2)
@Palakpatel67
Copy link

@tomchkk thanks for reporting this issue. We will look into it and post any updates here.

@Palakpatel67 Palakpatel67 self-assigned this Mar 30, 2021
@IgorVanian
Copy link

This happened to us with /node_modules/.bin/ symlink folder being processed but since it's a leftover and modules don't exist in node_modules/ it exits with an error:
image

@vince1995
Copy link

Any progress on this? This error exists since years.

@vince1995
Copy link

vince1995 commented Sep 13, 2021

I've looked into the code and noticed that here the symlinks are written into the zip without checking if they are in the .ebignore file.

@vince1995 vince1995 linked a pull request Sep 13, 2021 that will close this issue
@vince1995
Copy link

I've opened a PR with the fix for that.

@alexd2580-sf
Copy link

alexd2580-sf commented Nov 3, 2021

I second merging the PR. I haven't reviewed the PR, but this bug makes me want to self-host my stuff somewhere far away from AWS.

@llorenspujol
Copy link

Any updates on this issue?
IMO It should be a high priority issue. If one folder is ignored, it should be ignored entirely, included its symlinks to folders.

Without this fix, Elastic Beanstalk app.zip can end with a bunch of symlinks that point to undefined locations. For example, I can have this error on startup:

2022/04/21 09:11:08.715481 [ERROR] An error occurred during execution of command [app-deploy] - [StageApplication]. Stop running the command. Error: chown /var/app/staging/venv/dev/lib/python3.8/collections: no such file or directory

Because 'collections' is a symlink to a folder that do not exist in AL2.

All workarounds I am studying are quite painful… so it would be great to fix it.

@vince1995
Copy link

Yeah, really annoying that this is still not fixed. Look at my PR (which is more a workaround instead of a fix). You could also delete the symlinks before uploading to AWS and recreate them after the deployment.

@llorenspujol
Copy link

Yeah, I looked, thanks @vince1995 . Though it does not work really well in my case, since I have a great number of symlinks. What I ended doing is just deleting the slice of code that 'zips' symlinks.

@llorenspujol
Copy link

For anyone facing this issue, I created a branch that does NOT zip symlinks (even when they aren't ignored). It is a fork from the version 3.20.3.

If you have awsebcli in your requirements file, you can download de patched version directly from git:

(requirements.txt)
...
git+https://github.com/llorenspujol/[email protected]#egg=awsebcli
...

Hope this helps!

@vince1995
Copy link

Yes, that's something I thought of as well, but I wanted to create a solution that could be merged into the master branch because some developers may explicitly want the symlinks in the zip bundle.

This was referenced Sep 19, 2022
yohanthshetty added a commit that referenced this issue Sep 23, 2022
Fixed bug where a folder mentioned in .ebignore file was being included in the final code.zip on running eb deploy. This was due to the presence of a symlink within that folder.
@yohanthshetty
Copy link
Contributor

I have fixed the symlink issue. The symlinks are being treated like just another file/directory.
The symlinks aren't lost either.

@d-rocham
Copy link

Symlinks are still being included, even if an explicit path to the problematic symlink is added to .ebignore. Could this issue be re-opened?

@yohanthshetty
Copy link
Contributor

I will confirm if the symlinks are still being included in zip. Will re-open this issue if it still exists.

@yohanthshetty yohanthshetty reopened this Feb 2, 2023
@cavasinf
Copy link

cavasinf commented Feb 9, 2023

Same case here and still have the problem.

Having a sub-module Git that is linked with Composer (as a vendor), creates a symlink into the /vendor folder.
ATE, every deployment with eb deploy result as non execution of composer install because of:

You included a 'vendor' folder in your source bundle

Unziping the source zip in Windows doesn't do anything with that symlink, so no vendor folder appear.
BUT when unzinping from a Linux environment it ends like that:

  [...]
  inflating: [...]
  inflating: [...]
finishing deferred symbolic links:
  vendor/allsoftware/allsoftware-symfony-bundle -> ../../lib/allsoftware-symfony-bundle/

@mi-mazouz
Copy link

Any update ?

@yohanthshetty
Copy link
Contributor

Working on a fix.

@yohanthshetty
Copy link
Contributor

Hi All, Fix has been pushed to master. Pease pull the latest code to test. Broken/dangling symlinks are ignored. Folders with symlinks added are ignored entirely if added to .ebignore.

@yohanthshetty
Copy link
Contributor

Hi All, New version for EBCLI has been released. Please update the ebcli version by referring docs.

@andraspatka
Copy link

It seems like it wasn't fixed yet.
Contents of .ebignore:

# ignore everything
*
# Don't ignore docker-compose.yml
!./docker-compose.yml

Error when running eb deploy:

ERROR: FileNotFoundError - [Errno 2] No such file or directory: './env/bin/python'

version is latest:

$ eb --version
EB CLI 3.20.6 (Python 3.10.6 (main, Mar 10 2023, 10:55:28) [GCC 11.3.0])

The culprit:

$ ls-la env/bin/python
lrwxr-xr-x 1 root root 7 Apr 12 13:16 env/bin/python -> python3

@itelescu
Copy link

itelescu commented Jun 4, 2023

any updates on this ?

@shawzhang
Copy link

I just add to ignore those links to my .gitignore file: Just add this line fixed it:
my_venv/bin/python*

@mdeyo
Copy link

mdeyo commented Jul 29, 2024

I'm working with EB CLI 3.20.10 and this is still an issue.

My virtual environment folder (virt/) is included in the .ebignore, but the broken symlink virt/lib64 keeps being included with eb deploy.

Another observation:
If I delete the .ebignore and instead use .gitignore for the virt/ directory, then eb deploy works without including the broken symlink folder! However there are files that I want included on deploy that I want to Git to ignore, hence having different .gitignore and .ebignore files. Any updates available on if this issue is still a priority?

Our fix:

  1. Ensure the problem virtual env directory is correctly included in the .ebignore file
  2. Uninstall/remove awsebcli from your machine
  3. Re-install awsebcli (we had success with both homebrew and pip installations on different machines)
  4. eb deploy after double checking the .ebignore is correct

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet