Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove LiteralTypeForLiteral by creating IsInstance function #5909

Open
wants to merge 4 commits into
base: master
Choose a base branch
from

Conversation

Mecoli1219
Copy link
Contributor

@Mecoli1219 Mecoli1219 commented Oct 24, 2024

Tracking issue

#5908

Why are the changes needed?

The current Tuple IDL design is complicated by this function since if we want to guess the LiteralType from Literal, we have to store the name of the tuple and the name of each field inside the tuple in the Literal.

However, guessing the LiteralType is not necessary for Flyte since whenever we create a new literal, there must be a target type (maybe inputs or outputs, but they will always have a defined type). Also, almost all use cases of LiteralTypeForLiteral are followed by a function AreTypesCastable, we could simply combine these two functions into a function determining whether a Literal could be instantiated by a LiteralType. It is really similar to isinstance() function in Python.

What changes were proposed in this pull request?

  • I created an IsInstance() for the flytepropeller to check whether a Literal could be represented by a LiteralType.
  • Remove LiteralTypeForLiteral and update all necessary code in FlyteAdmin & FlytePropeller.
  • Update the unit tests
    • Change unit tests for LiteralTypeForLiteral to IsInstance
    • Update the error message for other cases.

How was this patch tested?

  • Unit testing
  • Built and run some Flytekit test

Setup process

Screenshots

Check all the applicable boxes

  • I updated the documentation accordingly.
  • All new and existing tests passed.
  • All commits are signed-off.

Related PRs

Docs link

Additional Concerns

Someone suggested that we shouldn't merge this PR before the release

Copy link

codecov bot commented Oct 24, 2024

Codecov Report

Attention: Patch coverage is 37.77778% with 112 lines in your changes missing coverage. Please review.

Project coverage is 36.92%. Comparing base (b0062e4) to head (1f58141).

Files with missing lines Patch % Lines
flytepropeller/pkg/compiler/validators/utils.go 30.00% 97 Missing and 8 partials ⚠️
...tepropeller/pkg/compiler/errors/compiler_errors.go 0.00% 6 Missing ⚠️
...ytepropeller/pkg/controller/nodes/array/handler.go 90.90% 0 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #5909      +/-   ##
==========================================
- Coverage   36.97%   36.92%   -0.06%     
==========================================
  Files        1318     1318              
  Lines      132506   132547      +41     
==========================================
- Hits        48994    48942      -52     
- Misses      79256    79345      +89     
- Partials     4256     4260       +4     
Flag Coverage Δ
unittests-datacatalog 51.58% <ø> (ø)
unittests-flyteadmin 54.01% <100.00%> (-0.04%) ⬇️
unittests-flytecopilot 30.99% <ø> (ø)
unittests-flytectl 62.29% <ø> (ø)
unittests-flyteidl 7.23% <ø> (ø)
unittests-flyteplugins 53.85% <ø> (ø)
unittests-flytepropeller 42.40% <34.88%> (-0.21%) ⬇️
unittests-flytestdlib 55.17% <ø> (+0.05%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@Future-Outlier Future-Outlier changed the title [DO NOT MERGE] Remove LiteralTypeForLiteral by creating IsInstance function [WIP][DO NOT MERGE] Remove LiteralTypeForLiteral by creating IsInstance function Oct 24, 2024
Copy link
Member

@Future-Outlier Future-Outlier left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you test every test case be affected in flytekit remote?
and provide screenshot?
I think this function is not well tested by unit test and integration test.

if _, ok := lit.GetValue().(*core.Literal_Collection); !ok {
return false
}
for _, x := range lit.GetCollection().Literals {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we add the tuple literal type but still represent the literal as a collection, how do we check if this collection is a tuple or not

Copy link
Contributor Author

@Mecoli1219 Mecoli1219 Nov 7, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The type of Literal for tuple will be Literal_Collection, but the LiteralType won't. We'll have to create another new InstanceChecker (perhaps tupleInstanceChecker) to deal with that.

func IsInstance(lit *core.Literal, t *core.LiteralType) bool {
instanceChecker := getInstanceChecker(t)

if lit.GetOffloadedMetadata() != nil {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i'd add a condition here to make sure the field is actually present.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sorry incomplete review... will get back to this later.

Signed-off-by: Mecoli1219 <[email protected]>
Signed-off-by: Mecoli1219 <[email protected]>
@Mecoli1219
Copy link
Contributor Author

@Future-Outlier I tested the IsInstance with the flytekit's unit tests and integration tests. I couldn't run eager in my environment, and it is not related to this update. Besides that, all test works successfully.
image
image

Signed-off-by: Mecoli1219 <[email protected]>
@Mecoli1219 Mecoli1219 changed the title [WIP][DO NOT MERGE] Remove LiteralTypeForLiteral by creating IsInstance function Remove LiteralTypeForLiteral by creating IsInstance function Dec 20, 2024
@Mecoli1219
Copy link
Contributor Author

Updated flytekit Integration test:
image
unit test:
image
image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: In progress
Development

Successfully merging this pull request may close these issues.

4 participants