Add TypeTransformer for TensorFlow tensor

[VPraharsha03]

This PR adds support for tf.Tensor as a native flyte type

Type

• Bug Fix
• Feature
• Plugin

Are all requirements met?

• Code completed
• Smoke tested
• Unit tests added
• Code documentation added
• Any pending items have an associated Issue

Complete description

• Adds a TypeTransformer for tf.Tensor in flytekit/extras/tensorflow/tensor/tensor.py
• Note : I'm working on the tests and will be soon adding them. Done.

Tracking Issue

flyteorg/flyte#2569

Follow-up issue

NA

[welcome]

Thank you for opening this pull request! 🙌

These tips will help get your PR across the finish line:

• Most of the repos have a PR template; if not, fill it out to the best of your knowledge.
• Sign off your commits (Reference: DCO Guide).

[pingsutw]

Thank you. Could we move these files to flytekit/extras/tensorflow/tensor? because we have another pr adding the TensorFlow model transformer to this directory.

btw, we need a test for TensorflowTensorTransformer

[codecov]

Codecov Report

Merging #1243 (327d802) into master (3d008e8) will increase coverage by 0.11%.
The diff coverage is 88.93%.

❗ Current head 327d802 differs from pull request most recent head 88fe3cc. Consider uploading reports for the commit 88fe3cc to get more accurate results

@@            Coverage Diff             @@
##           master    #1243      +/-   ##
==========================================
+ Coverage   68.63%   68.74%   +0.11%     
==========================================
  Files         288      292       +4     
  Lines       26279    26462     +183     
  Branches     2935     2492     -443     
==========================================
+ Hits        18036    18192     +156     
- Misses       7764     7789      +25     
- Partials      479      481       +2     

Impacted Files	Coverage Δ
flytekit/configuration/internal.py	16.43% <0.00%> (+0.43%)	⬆️
flytekit/core/container_task.py	29.54% <ø> (ø)
flytekit/core/task.py	35.71% <ø> (ø)
flytekit/tools/fast_registration.py	86.15% <0.00%> (-2.91%)	⬇️
tests/flytekit/unit/cli/pyflyte/test_run.py	99.20% <ø> (ø)
flytekit/deck/renderer.py	28.57% <33.33%> (+0.79%)	⬆️
flytekit/configuration/__init__.py	37.54% <50.00%> (+0.08%)	⬆️
flytekit/extras/tensorflow/__init__.py	55.55% <55.55%> (ø)
flytekit/core/local_cache.py	46.66% <66.66%> (-0.40%)	⬇️
flytekit/types/numpy/ndarray.py	50.00% <73.33%> (+8.33%)	⬆️
... and 24 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

[VPraharsha03]

make dev-requirements.txt fails with the following error:

Could not find a version that matches protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<3.20,<5.0.0dev,==3.20.3,>=3.19.5,>=3.9.2 (from -c requirements.txt (line 122))
Tried: 2.0.3, 2.3.0, 2.4.1, 2.5.0, 2.6.0, 2.6.1, 3.0.0, 3.0.0, 3.1.0, 3.1.0.post1, 3.1.0.post1, 3.2.0, 3.2.0, 3.3.0, 3.4.0, 3.4.0, 3.5.0.post1, 3.5.0.post1, 3.5.1, 3.5.1, 3.5.2, 3.5.2, 3.5.2.post1, 3.5.2.post1, 3.6.0, 3.6.0, 3.6.1, 3.6.1, 3.7.0, 3.7.0, 3.7.1, 3.7.1, 3.8.0, 3.8.0, 3.9.0, 3.9.0, 3.9.1, 3.9.1, 3.9.2, 3.9.2, 3.10.0, 3.10.0, 3.11.0, 3.11.0, 3.11.1, 3.11.1, 3.11.2, 3.11.2, 3.11.2, 3.11.3, 3.11.3, 3.11.3, 3.12.0, 3.12.1, 3.12.2, 3.12.2, 3.12.2, 3.12.4, 3.12.4, 3.12.4, 3.13.0, 3.13.0, 3.13.0, 3.14.0, 3.14.0, 3.14.0, 3.15.0, 3.15.0, 3.15.0, 3.15.1, 3.15.1, 3.15.1, 3.15.2, 3.15.2, 3.15.2, 3.15.3, 3.15.3, 3.15.3, 3.15.4, 3.15.4, 3.15.4, 3.15.5, 3.15.5, 3.15.5, 3.15.6, 3.15.6, 3.15.6, 3.15.7, 3.15.7, 3.15.7, 3.15.8, 3.15.8, 3.15.8, 3.16.0, 3.16.0, 3.16.0, 3.17.0, 3.17.0, 3.17.0, 3.17.1, 3.17.1, 3.17.1, 3.17.2, 3.17.2, 3.17.2, 3.17.3, 3.17.3, 3.17.3, 3.18.0, 3.18.0, 3.18.0, 3.18.1, 3.18.1, 3.18.1, 3.18.3, 3.18.3, 3.18.3, 3.19.0, 3.19.0, 3.19.0, 3.19.1, 3.19.1, 3.19.1, 3.19.2, 3.19.2, 3.19.2, 3.19.3, 3.19.3, 3.19.3, 3.19.4, 3.19.4, 3.19.4, 3.19.5, 3.19.5, 3.19.5, 3.19.6, 3.19.6, 3.19.6, 3.20.0, 3.20.0, 3.20.0, 3.20.1, 3.20.1, 3.20.1, 3.20.2, 3.20.2, 3.20.2, 3.20.3, 3.20.3, 3.20.3, 4.21.0, 4.21.0, 4.21.0, 4.21.0, 4.21.1, 4.21.1, 4.21.1, 4.21.1, 4.21.2, 4.21.2, 4.21.2, 4.21.2, 4.21.3, 4.21.3, 4.21.3, 4.21.3, 4.21.4, 4.21.4, 4.21.4, 4.21.4, 4.21.5, 4.21.5, 4.21.5, 4.21.5, 4.21.6, 4.21.6, 4.21.6, 4.21.6, 4.21.7, 4.21.7, 4.21.7, 4.21.7, 4.21.8, 4.21.8, 4.21.8, 4.21.8
Skipped pre-versions: 2.0.0b0, 3.0.0a2, 3.0.0a3, 3.0.0b1, 3.0.0b1.post1, 3.0.0b1.post2, 3.0.0b2, 3.0.0b2, 3.0.0b2.post1, 3.0.0b2.post1, 3.0.0b2.post2, 3.0.0b2.post2, 3.0.0b3, 3.0.0b4, 3.0.0b4, 3.2.0rc1, 3.2.0rc1, 3.2.0rc1.post1, 3.2.0rc1.post1, 3.2.0rc2, 3.2.0rc2, 3.7.0rc2, 3.7.0rc2, 3.7.0rc3, 3.7.0rc3, 3.8.0rc1, 3.8.0rc1, 3.9.0rc1, 3.9.0rc1, 3.10.0rc1, 3.10.0rc1, 3.11.0rc1, 3.11.0rc1, 3.11.0rc2, 3.11.0rc2, 3.12.0rc1, 3.12.0rc2, 3.13.0rc3, 3.13.0rc3, 3.13.0rc3, 3.14.0rc1, 3.14.0rc1, 3.14.0rc1, 3.14.0rc2, 3.14.0rc2, 3.14.0rc2, 3.14.0rc3, 3.14.0rc3, 3.14.0rc3, 3.15.0rc1, 3.15.0rc1, 3.15.0rc1, 3.15.0rc2, 3.15.0rc2, 3.15.0rc2, 3.16.0rc1, 3.16.0rc1, 3.16.0rc1, 3.16.0rc2, 3.16.0rc2, 3.16.0rc2, 3.17.0rc1, 3.17.0rc1, 3.17.0rc1, 3.17.0rc2, 3.17.0rc2, 3.17.0rc2, 3.18.0rc1, 3.18.0rc1, 3.18.0rc1, 3.18.0rc2, 3.18.0rc2, 3.18.0rc2, 3.19.0rc1, 3.19.0rc1, 3.19.0rc1, 3.19.0rc2, 3.19.0rc2, 3.19.0rc2, 3.20.0rc1, 3.20.0rc1, 3.20.0rc1, 3.20.0rc2, 3.20.0rc2, 3.20.0rc2, 3.20.1rc1, 3.20.1rc1, 3.20.1rc1, 4.0.0rc1, 4.0.0rc1, 4.0.0rc1, 4.0.0rc2, 4.0.0rc2, 4.0.0rc2, 4.21.0rc1, 4.21.0rc1, 4.21.0rc1, 4.21.0rc1, 4.21.0rc2, 4.21.0rc2, 4.21.0rc2, 4.21.0rc2
There are incompatible versions in the resolved dependencies:
  protobuf==3.20.3 (from -c requirements.txt (line 122))
  protobuf<3.20,>=3.9.2 (from tensorflow==2.10.0->-r dev-requirements.in (line 16))
  protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.19.5 (from google-cloud-bigquery==3.3.5->-r dev-requirements.in (line 12))
  protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.19.5 (from google-cloud-bigquery-storage==2.16.2->-r dev-requirements.in (line 13))

I've tried adding grpcio<=1.47.0 and grpcio-status<=1.47.0 to requirements.in which was suggested in PR#1248, I've also tried grpcio-status<1.49.0 but that too didn't work out

I've also tried what was suggested here in this comment of PR#1240, but unfortunately none of them worked out and I'd to additionally pin the version of protobuf to 3.19.6 as an example version to meet version conditions set by TensorFlow 2.10 here in (https://github.com/tensorflow/tensorflow/blob/v2.10.0/tensorflow/tools/pip_package/setup.py#L96) which is
'protobuf >= 3.9.2, < 3.20'

@pingsutw Would pinning the protobuf version be a right fix for this issue?

There are two choices, either we'd have to pin the protobuf version if we want to have the latest TensorFlow 2.10 or pin the tensorflow version to 2.8.1 instead

[VPraharsha03]

@pingsutw Added tests, please review them, I'll be adding the documentation very soon

[samhita-alla]

@VPraharsha03 let me know after you work on our suggestions.

[samhita-alla]

Copied my feedback from the Slack thread:

• filename should be local_path in to_literal
• tf.io.read_file("tensor_data", name = None) should be tf.io.read_file(local_path)
• pathlib.Path(local_path).parent.mkdir(parents=True, exist_ok=True) isn’t required because tf.io.write_file will always creae a directory recursively.
• tf.io.parse_tensor(read_serial, out_type = python_val.dtype, name = None)
• You can send the tensor dtype using LiteralCollection:

  flytekit/tests/flytekit/unit/core/test_literals_resolver.py

  Lines 79 to 84 in 7c915bb

  	"my_list": Literal(
  	collection=LiteralCollection(
  	literals=[
  	Literal(scalar=Scalar(primitive=Primitive(integer=1))),
  	Literal(scalar=Scalar(primitive=Primitive(integer=2))),
  	Literal(scalar=Scalar(primitive=Primitive(integer=3))),

  I took a stab at it: https://gist.github.com/samhita-alla/013601cd8b3a86a4277cdda54fa51a00.

[VPraharsha03]

@samhita-alla Done with the changes, still have to fix linting issues

[VPraharsha03]

I'm sorry, I'd messed up badly while committing, I've opened a new PR here