Added symlink dereferencing in fast packaging and tests

[vchowdhary]

TL;DR

Adds a switch to enable dereferencing symlinks during fast packaging in pyflyte serialize fast workflows.
cc @wild-endeavor

Type

• Bug Fix
• Feature
• Plugin

Are all requirements met?

• Code completed
• Smoke tested
• Unit tests added
• Code documentation added --- added documentation for click switch
• Any pending items have an associated Issue

Complete description

Added a click switch to pyflyte serialize fast workflows that enables/disables dereferencing any symlinks encountered during packaging. This only applies to fast_package.

Tracking Issue

NA

No issue associated with this -- this is part of a request from Stripe. Our use case for this is as follows:
Our workflows often import code from utils files that are in sibling directories -- this makes the current pyflyte package not quite work for us. Our workaround is to use our bazel build target, which generates a runfiles directory that contains all the source code and dependencies, and then serialize from that folder. Some symlinks are generated during the build, so we needed to add dereference=True when opening the tar file so that the actual target contents are loaded and downloaded during execution.

This change should not effect any of the current DataPersistence plugins that are used since they are all remote and would require dereferencing symlinks anyways if they wanted to use the contents.

Follow-up issue

NA

[welcome]

Thank you for opening this pull request! 🙌

These tips will help get your PR across the finish line:

• Most of the repos have a PR template; if not, fill it out to the best of your knowledge.
• Sign off your commits (Reference: DCO Guide).

[codecov]

Codecov Report

Merging #1151 (9c133d8) into master (61bd02a) will increase coverage by 0.07%.
The diff coverage is 95.65%.

@@            Coverage Diff             @@
##           master    #1151      +/-   ##
==========================================
+ Coverage   68.43%   68.51%   +0.07%     
==========================================
  Files         288      288              
  Lines       26032    26085      +53     
  Branches     2915     2918       +3     
==========================================
+ Hits        17816    17871      +55     
+ Misses       7736     7735       -1     
+ Partials      480      479       -1     

Impacted Files	Coverage Δ
flytekit/clis/sdk_in_container/serialize.py	57.14% <66.66%> (+0.62%)	⬆️
flytekit/clis/sdk_in_container/package.py	96.29% <100.00%> (+0.14%)	⬆️
flytekit/clis/sdk_in_container/register.py	80.76% <100.00%> (+0.37%)	⬆️
flytekit/tools/fast_registration.py	89.06% <100.00%> (ø)
flytekit/tools/repo.py	73.68% <100.00%> (ø)
...ests/flytekit/unit/tools/test_fast_registration.py	100.00% <100.00%> (ø)
tests/flytekit/unit/core/test_type_engine.py	98.39% <0.00%> (+<0.01%)	⬆️
flytekit/core/type_engine.py	59.34% <0.00%> (+0.09%)	⬆️
tests/flytekit/unit/cli/pyflyte/test_run.py	99.20% <0.00%> (+0.32%)	⬆️
flytekit/clis/sdk_in_container/run.py	84.19% <0.00%> (+0.59%)	⬆️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

[eapolinario]

This change looks good. Can you also make sure that new flag is added to the other places where fast_package is used?

[vchowdhary]

This change looks good. Can you also make sure that new flag is added to the other places where fast_package is used?

Thanks for catching that! There was one place that I missed.

[eapolinario]

What about?

flytekit/flytekit/tools/repo.py

Line 98 in 3137609

archive_fname = fast_registration.fast_package(source, output_tmpdir)

[vchowdhary]

Ah my bad, missed that.

[welcome]

Congrats on merging your first pull request! 🎉

[vchowdhary]

@eapolinario
Any idea on when this would be released? We see that there are monthly releases usually and one hasn't happened yet for september.