Use joblib hashing to generate cache key to ensure repeatability

[wild-endeavor]

TL;DR

Thanks to @bethebunny for finding/fixing this.

When the type of a literal is a dict or Dict (i.e. no type specified for the value), the type engine converts it to a protobuf struct. If the protobuf struct contains more than one field, then the cache key generated under the old code which ran base64 encoded a cloudpickle dump would sometimes change between calls, which led to local caching not working.

Type

• Bug Fix
• Feature
• Plugin

Are all requirements met?

• Code completed
• Smoke tested
• Unit tests added
• Code documentation added
• Any pending items have an associated Issue

Complete description

How did you fix the bug, make the feature etc. Link to any design docs etc

Tracking Issue

https://github.com/flyteorg/flyte/issues/

Follow-up issue

NA
OR
https://github.com/flyteorg/flyte/issues/

[wild-endeavor]

nm ignore this for now. need to add more to this.

[wild-endeavor]

@eapolinario can you take a look at this when you get a chance please? I don't understand the unit tests that are failing and can't find any references to explain what's happening. Why do we not want ints to cache?

[codecov]

Codecov Report

Merging #1126 (d1bbf28) into master (0e830c5) will increase coverage by 0.06%.
The diff coverage is 88.88%.

@@            Coverage Diff             @@
##           master    #1126      +/-   ##
==========================================
+ Coverage   68.24%   68.31%   +0.06%     
==========================================
  Files         287      287              
  Lines       25808    25873      +65     
  Branches     2884     2890       +6     
==========================================
+ Hits        17612    17674      +62     
- Misses       7718     7722       +4     
+ Partials      478      477       -1     

Impacted Files	Coverage Δ
setup.py	0.00% <ø> (ø)
flytekit/core/local_cache.py	47.05% <55.55%> (+1.40%)	⬆️
tests/flytekit/unit/core/test_local_cache.py	99.62% <100.00%> (+0.03%)	⬆️
flytekit/extras/tasks/shell.py	77.71% <0.00%> (-0.45%)	⬇️
tests/flytekit/unit/core/test_node_creation.py	95.72% <0.00%> (-0.24%)	⬇️
flytekit/core/promise.py	51.77% <0.00%> (-0.02%)	⬇️
plugins/setup.py	0.00% <0.00%> (ø)
tests/flytekit/unit/core/test_type_engine.py	98.34% <0.00%> (+0.01%)	⬆️
tests/flytekit/unit/cli/pyflyte/test_run.py	98.80% <0.00%> (+0.09%)	⬆️
flytekit/core/type_engine.py	58.63% <0.00%> (+0.20%)	⬆️
... and 2 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.